Page numbers not detected in various cases

[MerlijnWajer]

Environment

• Tesseract Version: Various, 4.1.1, 5.0.0 v20201231
• Platform: Linux, 64 bit

Current Behavior:

In some cases, Tesseract fully automatic page segmentation does not pick up page numbers that are quite visible. Here is an example (as hocr result viewer):

https://archive.org/services/hocr-view/view?identifier=sim_architectural-record_1931-12_70_6&pageno=27

I have taken the liberty of hosting some images with this problem here (and can attempt to surface more if that is helpful):

https://archive.org/~merlijn/tesseract-pagenumbers/

I don't believe that the problem is related to binarisation. I've tried to run the Java viewer (https://tesseract-ocr.github.io/tessdoc/ViewerDebugging.html) to look at the results, but didn't become much wiser, as it simply shows the page numbers not being picked up.

Expected Behavior:

Tesseract picks up the page number as well.

[stweil]

The standard models were trained with long lines of text, so maybe they simply did not teach Tesseract how very short text lines (like page numbers) look like. This can be tested with new models like frak2021 which were at least trained with some line numbers and other short lines.

[stweil]

I tried frak2021 with some of the test images and had the same issue: none of the page numbers was recognized.

[wollmers]

Binarisation is not the cause, even if it looks not optimal when I output tessinput.tif with command line option -c tessedit_write_images=true:

With default --psm and -l eng for sim_architectural-record_1931-12_70_6_0027.jpg:

A CONGREGATIONAL CHURCH IN

Fuermann and Sons

Outward thrust of roof trusses is counteracted by steel struts in the walls with steel rods going
under the floor slab between struts, thus effecting a saving in masonry.

 

With default --psm 6 the headline is missing and one speckle added as comma:

[52 lines of noise from the images]
Fuermann and Sons
, Outward thrust of roof trusses is counteracted by steel struts in the walls with steel rods going
under the floor slab between struts, thus effecting a saving in masonry.
416 DECEMBER, 1931

The default psm detects "something" in sim_architectural-record_1931-12_70_6_0027.psmno.hocr but it looks like a bug and generates empty lines:

   <div class='ocr_carea' id='block_1_4' title="bbox 0 0 3356 4750">
    <p class='ocr_par' id='par_1_4' lang='eng' title="bbox 0 0 3356 4750">
     <span class='ocr_line' id='line_1_5' title="bbox 0 0 3356 4750; baseline 0 0; x_size 2375; x_descenders -1187.5; x_ascenders 1187.5">
      <span class='ocrx_word' id='word_1_37' title='bbox 0 0 3356 4750; x_wconf 95'> </span>
     </span>
    </p>
   </div>

It's not the model. eng and deu from tessdata_best are very fine for modern texts after ~1900.

It clearly is caused by the weak page segmentation. Seems to be confused by the footer area not in line with the vertical borders of the body. IMHO page segmentation needs a complete redesign and should be trainable from ground truth layout patterns. It also should recognise and mark different zones like text, tables, captions, header, footer, halftone, drawings, decorative separators etc.

There are two work-arounds:

• Use another page segmentation, cut the page into line images and OCR them in line mode.
• OCR the pages with default psm and a second time with psm 6. Then merge the results.

[stweil]

This is a small single line image which I made from the examples and which also shows the issue:

Tesseract has no problems with this image if the black border on top or bottom is removed.

[wollmers]

This is a small single line image which I made from the examples and which also shows the issue:

Interesting. With this image tessinput.tif is negated. But that's not the bug. With a negated input image and the then white bars cropped away the result is perfect. Cropping one bar away gives only the page number as result.

Seems the region classification (text area detection) has a serious problem. I guess it is full with opinionated default options and thresholds.

[wollmers]

Tried to boil it down a little bit with the image https://archive.org/~merlijn/tesseract-pagenumbers/sim_biblical-theology-bulletin_spring-1990_20_1_0004.jpg. It has good quality and influence of bad image quality can be excluded.

First tried with different modes of --psm. Only --psm 6 detects the page number, but has some noise coming from the dark scan border.

Good recognition but no page number: psm 1, 3, 4, 11, 12.
Just noise: 5.
Recognises multiline initial M (of "Many") and keeps any: 11, 12.

Then tried a cut-out of the page number alone:

Recognises page number: psm 6, 7, 10.
Empty result: 4, 11, 12.
Result o: 5.
Result ee ee ee (Diplopia?): 8, 13.

--psm 12 of pagenumber only gives some messages:

$ tesseract biblical.page.jpg biblical.page.psm12 --psm 12 --tessdata-dir /usr/local/share/tessdata txt hocr
Tesseract Open Source OCR Engine v5.0.0-alpha-773-gd33ed with Leptonica
Too few characters. Skipping this page
OSD: Weak margin (0.00) for 1 blob text block, but using orientation anyway: 0
Too few characters. Skipping this page
OSD: Weak margin (0.00) for 1 blob text block, but using orientation anyway: 0

This means different treatment by psm, which is intended, but ignores wrongly ignores small text.

Maybe it's related to PSM_COL_FIND_ENABLED:

$ grep -R PSM_COL_FIND_ENABLED src
src/textord/colfind.cpp:    if (!PSM_COL_FIND_ENABLED(pageseg_mode)) {
src/ccmain/pagesegmain.cpp:  if (!PSM_COL_FIND_ENABLED(pageseg_mode) &&
src/ccmain/pagesegmain.cpp: * If !PSM_COL_FIND_ENABLED(pageseg_mode), then no attempt is made to divide
src/ccmain/pagesegmain.cpp:  if (!PSM_COL_FIND_ENABLED(pageseg_mode)) v_lines.clear();

$ grep -R PSM_COL_FIND_ENABLED include
include/tesseract/publictypes.h:inline bool PSM_COL_FIND_ENABLED(int pageseg_mode) {

PSM_COL_FIND_ENABLED means psm = 1 .. 3.
PSM_SPARSE means psm = 11 .. 12
PSM_BLOCK_FIND_ENABLED means psm = 1 .. 4.

This

tesseract/src/ccmain/pagesegmain.cpp

Line 137 in 0fb170b

if (PSM_OSD_ENABLED(pageseg_mode) || PSM_BLOCK_FIND_ENABLED(pageseg_mode) ||

excludes --psm 6:

  if (PSM_OSD_ENABLED(pageseg_mode) || PSM_BLOCK_FIND_ENABLED(pageseg_mode) ||
      PSM_SPARSE(pageseg_mode)) {
    auto_page_seg_ret_val =
        AutoPageSeg(pageseg_mode, blocks, &to_blocks,
                    enable_noise_removal ? &diacritic_blobs : nullptr, osd_tess, osr);
    if (pageseg_mode == PSM_OSD_ONLY) {
      return auto_page_seg_ret_val;
    }

But disabling enable_noise_removal has no effect.

Must be elsewhere.

[MerlijnWajer]

Here is another case that might help: https://archive.org/~merlijn/tesseract-pagenumbers/effectivenessoft00rick_0033.jpg

the entire line Insert Table 6 about here is missing. Presumably because it is below a (detected) line separator, perhaps?

[artunit]

This happens a lot with scanned major papers and thesis materials on my campus, for example the page number here. We have a simple script that uses OpenCV to pick up missed regions but it would be great if there was a flag or command line parameter to get Tesseract to target this kind of thing directly.

[stweil]

Thanks for this nice script. Do you think that the same processing could be added to Tesseract, but based on Leptonica instead of OpenCV and using C++ of course?

[artunit]

This might be possible, I have done a little C++ work with Leptonica on newspaper layouts, and I have been flummoxed by page numbers there too, but I am sure it's more a matter of fine-tuning. It's a bit beyond my usual tinkering, but I will take a look.

[artunit]

As near as I can tell, Leptonica is used for identifying photos in images in Tesseract and some data functions, but the segmentation is based on a custom connected components approach. It's possible to get a bit more by reducing kMinMediumSizeRatio in blobbox.cpp, for example, changing it from 0.25 to 0.1 , will pick up @MerlijnWajer's "Insert Table 6 about here" text but it's some intricate plumbing to modify beyond that. Post-processing might actually be easier and opens the door to leveraging the confidence value for deciding what's cruft and what's valuable.

[wollmers]

@MerlijnWajer

Here is another case that might help: https://archive.org/~merlijn/tesseract-pagenumbers/effectivenessoft00rick_0033.jpg

the entire line Insert Table 6 about here is missing. Presumably because it is below a (detected) line separator, perhaps?

In this case it works with --psm 6:

$ tesseract effectivenessoft00rick_0033.jpg - --psm 6 --tessdata-dir /usr/local/share/tessdata
Warning: Invalid resolution 0 dpi. Using 70 instead.
Ricker 13.
their camp time. This was found to be significant at the .001
level of confidence.
Insert Table 6 about here
Comparisons were also made between the ex-camper and
[...]

But I would expect that Tesseract outputs the line separators, because they are characters.

It should not be necessary to play around with --psm or other command line options.

[wollmers]

@stweil

Thanks for this nice script. Do you think that the same processing could be added to Tesseract, but based on Leptonica instead of OpenCV and using C++ of course?

IMHO it's not necessary to code this logic because it's still there, but the thresholds are wrong or not adaptive to different layouts.

[MerlijnWajer]

As near as I can tell, Leptonica is used for identifying photos in images in Tesseract and some data functions, but the segmentation is based on a custom connected components approach. It's possible to get a bit more by reducing kMinMediumSizeRatio in blobbox.cpp, for example, changing it from 0.25 to 0.1 , will pick up @MerlijnWajer's "Insert Table 6 about here" text but it's some intricate plumbing to modify beyond that. Post-processing might actually be easier and opens the door to leveraging the confidence value for deciding what's cruft and what's valuable.

I will toy around a bit with this number. I wonder if it ought to be based on DPI some how (maybe it is, with the line_size).

For example, if I take the sim_architectural-record_1931-12_70_6_0027 image and resize it to 1000x1385px (from 3432x4752px), tesseract 4.1.1 (and also 5.0.0-rc1) picks up the page number without problems. This also seems to suggest that @wollmers was on to something when he suggested to downscale images to ~300 DPI for better accuracy.