Sigma 18-35mm f/1.8 DC HSM misdetected after upgrading to firmware 2.03

[lbschenkel]

Describe the bug
Sigma 18-35mm f/1.8 DC HSM on a Canon 70D body, after upgrading to firmware 2.03 via the Sigma USB mount, is misdetected. Apparently the new firmware changed LensType from 150 to 368.

Firmware changelog: https://www.sigma-global.com/en/download/lenses/firmware/popups/a_18_35_18/
Analogous report to exiftool: https://exiftool.org/forum/index.php?topic=3833.msg63023#msg63023

To Reproduce
Steps to reproduce the behaviour:

1. Upgrade lens to latest firmware (2.03)
2. exifv2 --version: 0.27.3
3. Lens has a new type and is misdetected (two RAW files, same lens and body, old and new firmware):

$ exiv2 -p v 20200921_1828.cr2 20201017_2113.cr2
20200921_1828.cr2  0x0016 CanonCs      LensType                    Short       1  150
20200921_1828.cr2  0x0017 CanonCs      Lens                        Short       3  35 18 1
20200921_1828.cr2  0x0095 Canon        LensModel                   Ascii      74  18-35mm
20200921_1828.cr2  0xa432 Photo        LensSpecification           Rational    4  18/1 35/1 0/1 0/1
20200921_1828.cr2  0xa434 Photo        LensModel                   Ascii       8  18-35mm
20200921_1828.cr2  0xa435 Photo        LensSerialNumber            Ascii      11  0000000000
20201017_2113.cr2  0x0016 CanonCs      LensType                    Short       1  368
20201017_2113.cr2  0x0017 CanonCs      Lens                        Short       3  35 18 1
20201017_2113.cr2  0x0095 Canon        LensModel                   Ascii      74  18-35mm F1.8 DC HSM | Art 013
20201017_2113.cr2  0xa432 Photo        LensSpecification           Rational    4  18/1 35/1 0/1 0/1
20201017_2113.cr2  0xa434 Photo        LensModel                   Ascii      30  18-35mm F1.8 DC HSM | Art 013
20201017_2113.cr2  0xa435 Photo        LensSerialNumber            Ascii      11  0000000000

$ exiv2 -p a 20200921_1828.cr2 20201017_2113.cr2
20200921_1828.cr2  Exif.CanonCs.LensType                        Short       1  Sigma 18-35mm f/1.8 DC HSM | A
20200921_1828.cr2  Exif.CanonCs.Lens                            Short       3  18.0 - 35.0 mm
20200921_1828.cr2  Exif.Canon.LensModel                         Ascii      74  18-35mm
20200921_1828.cr2  Exif.Photo.LensSpecification                 Rational    4  18/1 35/1 0/1 0/1
20200921_1828.cr2  Exif.Photo.LensModel                         Ascii       8  18-35mm
20200921_1828.cr2  Exif.Photo.LensSerialNumber                  Ascii      11  0000000000
20201017_2113.cr2  Exif.CanonCs.LensType                        Short       1  Sigma 14-24mm f/2.8 DG HSM | A
20201017_2113.cr2  Exif.CanonCs.Lens                            Short       3  18.0 - 35.0 mm
20201017_2113.cr2  Exif.Canon.LensModel                         Ascii      74  18-35mm F1.8 DC HSM | Art 013
20201017_2113.cr2  Exif.Photo.LensSpecification                 Rational    4  18/1 35/1 0/1 0/1
20201017_2113.cr2  Exif.Photo.LensModel                         Ascii      30  18-35mm F1.8 DC HSM | Art 013
20201017_2113.cr2  Exif.Photo.LensSerialNumber                  Ascii      11  0000000000

Expected behavior
Lens should be detected as Sigma 18-35mm f/1.8 DC HSM | A and not Sigma 14-24mm f/2.8 DG HSM | A.

Desktop (please complete the following information):

• OS: Arch Linux

Additional context
I hope this information is enough so I'm not attaching the RAW files here, but if needed I can attach them.

[clanmills]

You can probably fix this for yourself in the "configuration" file ~/.exiv2:

[canon]
150=Sigma 18-35mm f/1.8 DC HSM | A
368=Sigma 14-24mm f/2.8 DG HSM | A

You'll find this discussed on the exiv2 man page: https://exiv2.org/manpage.html

If this fixes it, can you close the issue. If this doesn't fix it, please get the images to me (with Dropbox or Google Drive) and I'll investigate.

[lbschenkel]

I believe that you meant:

[canon]
150=Sigma 18-35mm f/1.8 DC HSM | A
368=Sigma 18-35mm f/1.8 DC HSM | A

Which works but I'm reporting this with the intent of trying to make this work out of the box for all users [1] instead of just myself.

[1] if possible: I haven't looked at how exiv2 does it, but exiftool uses heuristics based on the combination of different tags

[lbschenkel]

The funny thing is that I can see code 368 assigned to this lens here:

exiv2/src/canonmn_int.cpp

Line 1057 in 5547d10

{ 368, "Sigma 18-35mm f/1.8 DC HSM | A" },

And it looks the change that introduced it matches my situation exactly: 4f1a907 (#458)

Not quite sure why my exiv2 still misdetects it, though. This is how Arch Linux builds it: https://github.com/archlinux/svntogit-packages/blob/packages/exiv2/trunk/PKGBUILD

Directions on how to proceed? Should I upload a RAW file?

[clanmills]

Lens recognition has been time sink on the engineering resources of Team Exiv2. So, I introduced the ~/.exiv2 "Configuration File" in 0.26 to save lots of work and give users an instant way to recognised their lens. You don't need to wait on the release cycles of exiv2 and your distribution. You get it fixed instantly.

I'm currently writing a book about Exiv2 as I would like to retire on my 70th birthday on 2021-01-18. There will be an open source graphics conference in May 2021 in Rennes and which I'd like to talk about my book and run a work shop on Image Metadata and Exiv2 Architecture. Assuming somebody takes over the maintenance of Exiv2, I'm going to make a proposal to "solve" the lens problem. The proposal is discussed here: http://clanmills.com/exiv2/book/#future

The lens problem is difficult. The lens isn't stored in the metadata. Different manufacturers use different ways to deal with this and it's very common that a number such as "368" is used to represent several lenses. Then we have to examine other metadata to make a guess about which lens is being used. My proposal deals with those possibilities and makes lens recognition a project that can be used by Exiv2, ExifTool and others. So, the maintenance burden of "the heuristic" is removed from Exiv2 and ExifTool and put into M2LScript.

For the moment, that's the way things are and I don't want to get involved in updating the existing lens recognition code. I'm working on the book and retiring. Will M2LScript come to life? We'll see.

[lbschenkel]

Sure, I understand. As I said, this report had the intention of getting this fixed upstream so other users could benefit from a fix -- I am aware that I can fix the problem for myself via the configuration file. And I'm also aware that lens detection is a tricky matter (especially for third-party lenses). But thank you for taking the time to explain anyway.

I didn't know that you were retired (or close to retiring). I'm sad for the project but I want to thank you for maintaining exiv2 for all this time.

Now that I found out that exiv2 is supposed to be detecting this lens, but it isn't, I'll try to build it debug it locally. I might be able to find out what's wrong and hopefully submit a patch (or at least describe what's going on).

[clanmills]

It's probably really easy to fix this. However, I like to have an image in the test suite for every lens to be sure that a future change won't undo your change. It's all rather tedious. Usually takes me about an hour to deal with a new lens, provided it doesn't collide with an existing lens.

The M2Lscript concept removes lens recognition from Exiv2 and ExifTool. M2Lscripts can be updated (over the internet) as new lenses appear.

[lbschenkel]

OK, here is what I found out. The lens is correctly detected in master, but not the 0.27 maintenance branch. I was able to bisect this branch and the commit that broke it was this: b7bd098

The original commit that introduced support for this lens, before it was undone by the commit above, was this: cf4f14c

[lbschenkel]

This is the diff against v0.27.3 that solved it for me:

diff --git a/src/canonmn_int.cpp b/src/canonmn_int.cpp
index b3443819..75107ec5 100644
--- a/src/canonmn_int.cpp
+++ b/src/canonmn_int.cpp
@@ -918,7 +918,7 @@ namespace Exiv2 {
         {  137, "Tamron SP 60mm f/2 Macro Di II"                            }, // 12
         {  137, "Sigma 10-20mm f/3.5 EX DC HSM"                             }, // 13
         {  137, "Tamron SP 24-70mm f/2.8 Di VC USD"                         }, // 14
-        {  137, "Sigma 18-35mm f/1.8 DC HSM"                                }, // 15
+        {  137, "Sigma 18-35mm f/1.8 DC HSM | A"                            }, // 15
         {  137, "Sigma 12-24mm f/4.5-5.6 DG HSM II"                         }, // 16
         {  137, "Sigma 70-300mm f/4-5.6 DG OS"                              }, // 17
         {  138, "Canon EF 28-80mm f/2.8-4L"                                 },
@@ -1147,6 +1147,7 @@ namespace Exiv2 {
         {  368, "Sigma 105mm f/1.4 DG HSM"                                  }, // 8
         {  368, "Sigma 14-24mm f/2.8 DG HSM"                                }, // 9
         {  368, "Sigma 70mm f/2.8 DG Macro"                                 }, // 10
+        {  368, "Sigma 18-35mm f/1.8 DC HSM | A"                            }, // 11
         {  488, "Canon EF-S 15-85mm f/3.5-5.6 IS USM"                       },
         {  489, "Canon EF 70-300mm f/4-5.6L IS USM"                         },
         {  490, "Canon EF 8-15mm f/4L Fisheye USM"                          },
@@ -1281,6 +1282,7 @@ namespace Exiv2 {
         { 250, printCsLensByFocalLength }, // not tested
         { 254, printCsLensByFocalLength },
         { 255, printCsLensByFocalLength }, // not tested
+        { 368, printCsLensByFocalLength },
         { 491, printCsLensByFocalLength },
         { 493, printCsLensByFocalLength }, // not tested
         { 624, printCsLensByFocalLengthTC },

(The first change was just to make the lens name consistent, in order to match lens type 150.)

I believe that the lack of { 368, printCsLensByFocalLength } was a bug introduced in b7bd098 because that commit introduced multiple lenses of type 368 without any means to disambiguate between them.

I hope this helps. I haven't run the test suite yet, so it's possible there's more to be done.

[clanmills]

I guess that I'm not permitted to say "No" to this. Please extract the .exv file from your image and I'll update the test suite. You extract the file with the command: $ exiv2 -ea --force --verbose my.image. The command reports the location of the .exv which you should attach here. It's just a little file of the metadata in your image. It's typically about 10k.

I'll add this, provided it doesn't break anything in test suite on either 0.27-maintenance or master.

[lbschenkel]

I have uploaded both .exv files to https://drive.google.com/file/d/1XaCRuWl0l4mkAPw-7TzZq71e--DwJAmq/view?usp=sharing
Please let me know when you have fetched it, so I can remove the file (because of the embedded serial numbers in the EXIF, I prefer not to have them permanently attached here).

I guess that I'm not permitted to say "No" to this.

I genuinely couldn't figure out if you were being ironic or not (English is not my mother tongue), so apologies if I'm appearing to be too entitled or too pushy. My intent is to try my best to do as much of the work as possible to get a fix upstream, so others with the same lens can also benefit (especially when using software such as Darktable). I don't want to create more work for you, so I can try to figure out how the test suite works and submit a (hopefully) ready-to-be-merged PR.

I didn't add anything new to the test suite, but with the diff above applied it still passed on my machine.