Encodings reported by chardetng-py don't always match up to python's decoding

[john-parton]

LookupError: unknown encoding: windows-874

In a previous version, I just passed the entire buffer to encoding_rs and had it handle decoding entirely in rust, but that was a little confusing.

Need more robust aliases

[john-parton]

Actually it might be even more complicated than that. https://bugs.python.org/issue25416

I think it's possible that encoding_rs and the python codec will have different output for legacy single byte encodings

[Mr0grog]

For what it’s worth, I went through and:

1. Compared the names/aliases. windows-874 is the only one chardetng can produce that does not map correctly in Python, so that’s already fully addressed.

2. Wrote a quick script to compare the single byte WHATWG encodings definitions (found at https://encoding.spec.whatwg.org/#legacy-single-byte-encodings or more conveniently in git at https://github.com/whatwg/encoding) with the Unicode consortium definitions (found at https://unicode.org/Public/MAPPINGS/). They are all basically identical except for a few exceptions in KOI8-U and windows-1255:

   MATCH: IBM866
   MATCH: ISO-8859-2
   MATCH: ISO-8859-4
   MATCH: ISO-8859-5
   MATCH: ISO-8859-6
   MATCH: ISO-8859-7
   MATCH: ISO-8859-8
   MATCH: ISO-8859-13
   Definitions for KOI8-U do not match!
     Mismatch at byte 174 (0xae)! WHATWG = point 1118 (0x045e) | Unicode = point 9565 (0x255d)
     Mismatch at byte 190 (0xbe)! WHATWG = point 1038 (0x040e) | Unicode = point 9580 (0x256c)
   MATCH: windows-874
   MATCH: windows-1250
   MATCH: windows-1251
   MATCH: windows-1252
   MATCH: windows-1253
   MATCH: windows-1254
   Definitions for windows-1255 do not match!
     Mismatch at byte 202 (0xca)! WHATWG = point 1466 (0x05ba) | Unicode = point <UNDEFINED>
   MATCH: windows-1256
   MATCH: windows-1257
   MATCH: windows-1258
   

   (The format here is byte <decimal> (<hex>) is the actual byte being decoded and point <decimal> (<hex>) is the unicode code point it should decode as.)

   Caveat: I allowed points that are control characters to be treated as undefined for this comparison, which I think is probably reasonable. There are a bunch more encodings that don’t match up otherwise (i.e. a byte is defined as some control character in WHATWG’s definition and is entirely undefined/unmapped in the Unicode Consortium’s definition, or vice versa).

These mismatches are definitely not ideal, but I think are also small enough not to be a problem (at least for the use case here: detecting the encoding of some bytes; not actually decoding/encoding them). None of these bytes show up in chardetng’s frequency data in src/data.rs (assuming I’m reading it correctly), which I think means they aren’t really being considered anyway, so their mapping shouldn’t have a notable impact.

On the other hand, the windows-1255 case does feature a byte that could cause errors when [strictly] decoding in Python, and based on chardetng’s docs, I think it wouldn’t otherwise give you an answer that does that, since it says it discards options with decoding failures/unmapped bytes. That said, I think it’s good advice to avoid ever decoding a sniffed encoding strictly in the first place; IIRC chardet and cChardet both occasionally give answers that are wrong enough that they don’t successfully decode the bytes and code I had using them seems to always use errors="ignore" or errors="replace" when decoding with their guesses. (This might all be worth mentioning in some addendum to the the docs, though… idunno.)

[Mr0grog]

Ah, some historical details on the ones that differ between WHATWG and Unicode, if you’re curious:

• KOI8-U as defined by WHATWG is actually KOI8-RU (the -ru encoding was never fully standardized, and is not available in Python). Apparently Internet Explorer historically interpreted KOI8-U this way, and so all other browsers had to follow suit. Discussion: Confusion between KOI8-U and KOI8-RU encodings whatwg/encoding#74 and https://www.w3.org/Bugs/Public/show_bug.cgi?id=17053

• In windows-1255, the actual implementation in Windows has historically mapped 0xCA to 0x05BA, even though all the specs and code sheets listed it as unmapped. Since WHATWG’s goals are to keep old content working (don’t break the web!), they decided to implement it following how Windows did it in practice, rather than how it was spec'd. Discussion: windows-1255 encoding: add mapping for 0xCA whatwg/encoding#73

[john-parton]

Thanks for looking into that. So it looks like if you use python to decode, it might not be correct. I think probably it's best to just document this and move on.

[john-parton]

I'm going to keep this issue open because it's lacking documentation, but it looks like the functional problems are well-addressed.

[Mr0grog]

Yes, good point! I should not have marked that PR as solving this.

I started looking into the multibyte encodings (I think I covered the single-byte ones well), and it seems like the situation is a bit more messy. I’m leaving some notes here so I or someone can document based on them.

Multibyte codecs that Chardetng detects:

• GBK
• Big5
• Shift_JIS
• ISO-2022-JP
• EUC-JP
• EUC-KR

GBK

See #149 for details. Basically, gb18030 is a superset of gbk, which is a superset of gb2312. Chardetng expects decoding with encoding_rs, which treats all of them the same (as gb18030). They are all different decoders in Python, though, so we should (and now do) transform GBK → gb18030, which will work as a decoder in Python for all [correctly detected] cases.

Shift_JIS

Oh dear, this is messy and complicated. Shift_JIS turns out to be a really old and limited spec that has been extended in a multitude of ways that are incompatible with each other.

⚠️ TL;DR: If Chardetng returns Shift_JIS, the right way to decode in Python is probably:

1. Try decoding as ms932.
2. If that fails, try decoding as shift_jis_2004 (which is a superset of all the others in this family).
3. If that fails, do whatever your fallback action is for bad detection.

(Caveat: if you want/need to behave like a web browser, skip step 2).

I think it would be lovely if that behavior was bundled up into the detect() function here (but in a way where you could call it separately if you are using EncodingDetector directly), and maybe also use it to inform the confidence of compat.detect(), but I think you could also make the case that it’s out of scope. It also adds in a fair amount of overhead if you aren’t ultimately planning to decode the bytes you’re dealing with (a good reason to put it in detect() and not EncodingDetector).

⚠️ End TL;DR

The most popular versions of Shift_JIS are:

• Shift_JIS is the original (although in many languages/frameworks, the thing named "shift_jis" is not actually this).

• Windows-31J, a.k.a. Windows-932 (in Python: ms932 or cp932) is the most widely used version according to WHATWG and Wikipedia, but I couldn’t find evidence for that cited anywhere (probably just because this was the official/first-party Japanese encoding on Windows). It is a superset of Shift_JIS, with a four exceptions (see below).

  The exceptions to being a superset: two notable exceptions (0x5C and 0x7E are treated differently) and two minor ones (0x815F and 0x8160 get visually similar, but not exact same characters). However, in Python, there is a STRICT_BUILD flag that, if false or not set (and it appears to be false on my machine) causes shift_jis to treat 0x5E and 0x7E like Windows-932, and makes Windows-932 handles errors in a slightly different way. I have no idea what environments might set this to true and get “accurate” behavior.

• Shift_JISx0213 is a standards-body superset of Shift_JIS that adds a lot of characters. (Note: because of the above strict build stuff, this may or may not be an exact superset of Shift_JIS on any given Python install. 😬)

• Shift_JIS_2004 is a standards-body update of of Shift_JISx0213. It adds 10 characters.

  On Python, it is not quite an exact superset because one character is different in a weird a subtle way that I’m not sure if is correct or a bug: The bytes 0xFC5A equate to the same glyph, but a different code point! In Shift_JISx0213, it’s U+9B1D and in Shift_JIS_2004, it’s U+9B1C. I think this is mainly a non-issue because the same bytes work with both, and decode to a perfectly equivalent glyph. The only problem is if you do:

  some_bytes.decode('shift_jisx0213').encode('shift_jis_2004')

  Or vice-versa. That issue is way outside the scope of this package, though.

WHATWG (and Chardetng) treats Windows-932 as Shift_JIS. So technically if Chardetng detects Shift_JIS, it really means windows-932 (or ms932 in Python terms). BUT the detection is loose enough and the overlap in these encodings is big enough that will typically detect Shift_JISx0213 or Shift_JIS_2004 or Shift_JIS content as Windows-932 and give Shift_JIS as a result.

So if Chardetng returns Shift_JIS, the right way to decode in Python is probably:

1. Try decoding as ms932.
2. If that fails, try decoding as shift_jis_2004 (which is a superset of all the others in this family).
3. If that fails, do whatever your fallback action is for bad detection.

(Caveat: if you want/need to behave like a web browser, skip step 2).

In an ideal world, I think that behavior would be bundled up into the detect() function here (but in a way where you could call it separately if you are using EncodingDetector directly), and maybe also use it to inform the confidence of compat.detect(), but I think you could also make the case that it’s out of scope. It also adds in a fair amount of overhead if you aren’t ultimately planning to decode the bytes you’re dealing with (a good reason to put it in detect() and not EncodingDetector).

---

I didn’t have a chance to dive into the other multibyte ones yet. Quick notes on EUC-KR and Big5, but these need more investigation:

EUC-KR

WHATWG’s and encoding_rs’s definition of EUC-KR is actually UHC/Windows-949, which is an extension on the original EUC-KR. I’m not sure whether that’s a strict superset (in which case this is straightforward and we should swap the name like we did for GBK/gb18030) or not, or any idiosyncracies in how Python treats the two (they are at least separate codecs in Python).

Big5

WHATWG’s and encoding_rs’s definition of Big5 is actually Big5-HKSCS. I’m not sure how much a strict superset this is, or anything else here. Needs more research.

ISO-2022-JP

Haven’t looked into at all yet.

EUC-JP

Haven’t looked into at all yet.

[john-parton]

Hm, I think writing up a condensed version of this--something like "these encodings tend to be difficult in general: "--and putting it in the docs might be sufficient.

It's obviously pretty tricky and I appreciate you looking into it.

If someone really wants to do what a browser might do, they should probably take the binary stream and pass it directly to encoding_rs. It's not too bad to add bindings. We could create and encoding_rs_py (lol) binding.

[Mr0grog]

That’s fair. Given that, I think we should probably also (in addition to the docs) rename Shift_JIS to ms932 (like was done with gbk → gb18030) since that is really the closest thing in Python’s repertoire to what chardetng means.

I’m still hoping to write some docs for this, but want to do the research on the other remaining multibyte codecs here first.

[Mr0grog]

More research:

ISO-2022-JP

ℹ️ TL;DR: None of Python’s built-in encodings is quite equivalent to the WHATWG version of this, but iso2022_jp_ext is the closest (and fairly narrow) superset. For safe usage, it probably makes sense to map ISO-2022-JP to iso2022_jp_ext. ℹ️

ISO-2022-JP has a similarly complicated backstory to GBK with incompatible families of extensions. ISO-2022 is a structure for making a codec that is stateful and switches between different sub-encodings when it encounters certain escape sequences (e.g. when it encounters the bytes ESC ( J it switches to interpreting everything after as if it were the roman component of JIS X 0201-1976, and when it encounters ESC $ B it switches to interpreting everything after as if it were JIS X 0208-1983).

Note: JIS X NNNN is an encoding standard, and JIS X NNNN-YYYY is the version of that standard published in year YYYY.

• Baseline ISO-2022-JP can switch between:

  1. ASCII
  2. JIS X 0201-1976 (roman characters)
  3. JIS X 0208-1978, and JIS X 0208-1983

  Both Python and WHATWG treat both the JIS X 0208 components as JIS X 0208-1983 instead of treating them differently. The -1983 version is a strict superset, so this really just means they are slightly less strict than the specification.

• WHATWG ISO-2022-JP is the same as baseline, but it treats the JIS X 0208-1978 version as if it were JIS X 0208-1983 (a superset) and it adds support for:

  1. JIS X 0201-1976 (kana characters) — this and ISO-2022-JP-EXT are the only versions that support the kana characters from JIS X 0201-1976.

• ISO-2022-JP-EXT is the same as baseline + support for:

  1. JIS X 0201-1976 (kana characters) (like the WHATWG version above)
  2. JIS X 0212-1990

  In Python, this also treats JIS X 0208-1978 as JIS X 0208-1983, like WHATWG does. It’s basically WHATWG + JIS X 0212-1990.

• ISO-2022-JP-1 is the same as baseline + adds support for:

  1. JIS X 0212-1990

• ISO-2022-JP-2 is the same as ISO-2022-JP-1 + adds support for:

  1. gb2312 (Chinese)
  2. KS x 1001-1992 (Korean)
  3. ISO-8859-1 (the “high” or extended latin part — bytes above 0xA0)
  4. ISO-8859-7 (the “high” or basic greek part — byes above 0xA0)

• ISO-2022-JP-3 does not follow from -JP-2. It is the same as baseline + adds support for:

  1. JIS X 0201-1976 (kana characters) (like the -ext and WHATWG versions above) BUT NOT IN PYTHON. Or at least, Wikipedia suggests this is supposed to be supported, but digging through the Python source confirms it is not in Python’s implementation. Trying this in actual Python (v3.12.0) also confirms it is not supported.
  2. JIS X 0213-2000

  The basic idea here was to replace all the complicated switches in ISO-2022-JP-2 with JIS X 0213, which pretty much supports it all in a more cohesive way.

• ISO-2022-JP-2004 is the same as ISO-2022-JP-3 but replaces the JIS X 0213-2000 bits with JIS X 0213-2004 (effectively a superset).

• There are more variations, but these are the most popular and the only ones relevant to Python or WHATWG/chardetng/encoding_rs.

So in Python, nothing is quite equivalent to the WHATWG version (which is what Chardetng is guessing when it says ISO-2022-JP). Straight-up iso2022_jp will fail to decode some things Chardetng thinks are OK, but iso2022_jp_ext is a strict superset of what Chardetng is looking for (the only superset!), so should always succeed. A bit of fiddling around with various characters that are only supported in some of the sub-encodings listed here seems to confirm this in practice (using C Python 3.12.0).

If we are going ahead with remapping names for compatibility, it probably makes sense to map ISO-2022-JP → iso2022_jp_ext.

EUC-JP

ℹ️ TL;DR: As far as decoding goes WHATWG/chardet/encoding_rs’s concept of EUC-JP is the same as Python’s. We shouldn’t need to treat this one specially. ℹ️

EUC encodings are like ISO-2022 and Shift-JIS in that they contain several sub-encodings. Like ISO-2022, they support more/are a bit more flexible than Shift-JIS, but uses a switching mechanism based on what bytes are allowed where, like Shift-JIS, rather than statefully switching modes like ISO-2022. (If you’re familiar with UTF-8, it’s similar.)

For Japanese, there are three popular versions:

• EUC-JP combines ASCII, JIS X 0201 kana characters (like ISO-2022-JP-EXT), JIS X 0208, and JIS X 0212. In some versions, the ASCII portion is JIS X 0201 roman characters instead, but both Python and WHATWG/encoding_rs/chardetng use ASCII.

  In WHATWG and encoding_rs, the EUC-JP codec does not support encoding JIS X 0212 characters. But they are supported for decoding so we don’t really care for our purposes here.

• EUC-JISx0213 combines ASCII and JIS X 0213-2000 (0213 can handle all the stuff from 0201, 0208, and 0212).

  Python changes behavior slightly here based on the STRICT_BUILD build-time flag — the same as Shift-JIS. The impact of this is pretty minor and probably not worth worrying about here.

• EUC-JIS-2004 is the same as EUC-JISx0213 but upgrades from JIS X 0213-2000 to JIS X 0213-2004 (just a newer version of the standard with a few additional characters).

Neither the WHATWG standard nor encoding_rs/chardetng use anything from JIS X 0213, and pretty much exactly match behavior. We don’t need to do anything special for this family of encodings.

---

Still remaining:

• EUC-KR
• Big5

[john-parton]

That all makes sense to me. I wonder if documenting it is sufficient, or if we should emit a warning of some kind.

By putting the mapping logic directly into the rust code, we have created a minor problem if the user wants to know what chardetng actually outputs. For instance, if they want to pass the output to a binding of encoding_rs.

Additionally, if we want to emit a warning using the python warnings module, I'm not sure we can really do that because the mapping is already done in rust.

I think perhaps deviating very slightly from the rust struct and adding a compat flag to the guess method might make sense