RFC: allow flexible or better binarization

[bertsky]

Tesseract has always included its own, internal binarization – which is not based on Leptonica and is of rather bad quality (custom global Otsu implementation without normalization). Leptonica does have lots of nice adaptive local normalization and thresholding implementations, but they are not utilized.

Since Tesseract 4.0, recognition does not (under normal circumstances) use that binarized image, but uses the greyscale-converted raw image. If the input image was already bitonal, then it still works. However, what the LSTM model expects depends on what data it was trained on. (Original tessdata models were all trained on artificial/clean greyscale IIRC.)

But the binarized image is still needed for all segmentation and layout analysis (OSD, separator detection, picture detection). And to make matters worse, segmentation also makes use of the greyscale image and threshold values from the binarizer – in order to get a better approximation/interpolation of blob outlines (ComputeEdgeOffsets). If the input image was already bitonal, then a fallback method is used which is not as accurate (ComputeBinaryOffsets).

Now the user (both CLI and API) is in a dilemma:

1. Present the originial/raw image:
   • Segmentation on some images may be suboptimal, because good binarization is hard.
   • Recognition might expect a (colour/contrast-) normalized or even binarized (e.g. from tesstrain) image and thus be suboptimal.
2. Present an externally binarized image:
   • Segmentation on other images may be suboptimal, because the blob outlines are inaccurate.
   • Recognition might expect a greyscale image and thus be suboptimal.

So what do we do? Allow delegating to Leptonica's pixContrastNorm, pixSauvolaBinarizeTiled etc. methods via parameter variable? Or extend the API to allow passing the threshold values of an external binarization? How do we inform/document this and encapsulate an LSTM model's expectations?

There are many more aspects, but I just wanted to open the discussion.

If requested, I can provide example images (of bad segmentation due to internal Otsu or bad segmentation due to inaccurate interpolation from bitonal input; of recognition perplexed by background masked to white), as well as pointers to code.

[wrznr]

@bertsky Many thanks for bringing up this important issue. It might indeed be helpful to illustrate the matter with sample images (in order to get the wave of comments rolling).

[amitdo]

Thanks for opening this RFC. Very nice analysis.

I will add my comments later.

[amitdo]

Regarding ComputeEdgeOffsets, I assume the more accurate outlines only matter for the legacy engine and not for the nn engine. Correct? The nn enginr does not act on segmented glyphs like the legacy engine.

[amitdo]

Segmentation on some images may be suboptimal, because good binarization is hard.

Image binarization based on neural net might solve it.
With regular 'dumb' methods, you can use a brute force method. Any other idea?

[amitdo]

Recognition might expect a (colour/contrast-) normalized or even binarized (e.g. from tesstrain) image and thus be suboptimal.

In the short term we should document somewhere on what kind of images our models were trained. Currently, it's grayscale (with 0 and 255 as values).
In the long term, make it easy to query the model for this information via the API.

[MerlijnWajer]

(Not a core dev, but attempting to become a core user and help out some with dev)

So what do we do? Allow delegating to Leptonica's pixContrastNorm, pixSauvolaBinarizeTiled etc. methods via parameter variable? Or extend the API to allow passing the threshold values of an external binarization? How do we inform/document this and encapsulate an LSTM model's expectations?

Perhaps the API could be passed a callback function that takes the to-be-thresholded image and returns the transform of the image into thresholded values, rather than passing values in via the API (but perhaps that's what you meant). This could then be utilised by the command-line frontend to potentially even invoke external scripts/executables from within that function (not great for performance, but flexible).

Allowing various (predefined) thresholding methods could also be done through some plugin structure, perhaps?

[amitdo]

https://groups.google.com/g/tesseract-ocr/c/bNk9lYa-xmw

[Robyer]

What about adding new method to provide both "original" and "binarized" images? That way client can use own binarization method while different parts of Tesseract can still use specific image variant based on their needs.

[amitdo]

@bertsky, @stweil

Suggested plan:

• Step 1: Get rid of Tesseract's Otsu. Replace it with Leptonica's Otsu. Hopefully, this will give us comparable/better ocr output. If this works well, move to the next step.
• Step 2: Add more binarization methods from Leptonica.
• Step 3: Add a method that get pix_binary, pix thresholds from an outside source (produced by a separate library/program).