Assertion when running lstmtraining

[stweil]

Running lstmtraining for frk language with 50000 iterations terminated with an assertion.

$ lstmtraining -U /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.unicharset --script_dir ~/src/github/tesseract-ocr/langdata --net_spec '[1,36,0,1 Ct5,5,16 Mp3,3 Lfys64 Lfx128 Lrx128 Lfx256 O1c105]' --model_output /home/stweil/src/github/tesseract-ocr/tesseract/frk/output/base --train_listfile /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.training_files.txt --eval_listfile /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.training_files.txt --max_iterations 50000
...
At iteration 15778/49900/49900, Mean rms=0.37%, delta=0.112%, char train=0.381%, word train=1.506%, skip ratio=0%,  wrote checkpoint.

At iteration 15788/50000/50000, Mean rms=0.363%, delta=0.104%, char train=0.346%, word train=1.387%, skip ratio=0%,  wrote checkpoint.

Finished! Error rate = 0.26
num_docs > 0:Error:Assert failed:in file ../../../../ccstruct/imagedata.cpp, line 648

I used latest Tesseract sources, a slightly modified font list and a longer training text for frk training.
A previous run with 10000 iterations and nearly the same conditions did not raise the assertion:

...
2 Percent improvement time=807, best error was 3.911 @ 8211
At iteration 9018/10000/10000, Mean rms=0.835%, delta=0.465%, char train=1.729%, word train=6.095%, skip ratio=0%,  New best char error = 1.729Deserialize failed wrote best model:/home/stweil/src/github/tesseract-ocr/tesseract/tutorial/frkoutput/base1.729_9018.lstm wrote checkpoint.

Finished! Error rate = 1.729

[Shreeshrii]

I have found that the error goes away when NOT using --eval_listfile, please try without the following

…

--eval_listfile /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.training_files.txt Though this means that there is no regular eval during training. ShreeDevi

____________________________________________________________ भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

On Mon, Mar 13, 2017 at 12:42 PM, Stefan Weil ***@***.***> wrote: Running lstmtraining for frk language with 50000 iterations terminated with an assertion. $ lstmtraining -U /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.unicharset --script_dir ~/src/github/tesseract-ocr/langdata --net_spec '[1,36,0,1 Ct5,5,16 Mp3,3 Lfys64 Lfx128 Lrx128 Lfx256 O1c105]' --model_output /home/stweil/src/github/tesseract-ocr/tesseract/frk/output/base --train_listfile /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.training_files.txt --eval_listfile /home/stweil/src/github/tesseract-ocr/tesseract/frk/train/frk.training_files.txt --max_iterations 50000 ... At iteration 15778/49900/49900, Mean rms=0.37%, delta=0.112%, char train=0.381%, word train=1.506%, skip ratio=0%, wrote checkpoint. At iteration 15788/50000/50000, Mean rms=0.363%, delta=0.104%, char train=0.346%, word train=1.387%, skip ratio=0%, wrote checkpoint. Finished! Error rate = 0.26 num_docs > 0:Error:Assert failed:in file ../../../../ccstruct/imagedata.cpp, line 648 I used latest Tesseract sources, a slightly modified font list and a longer training text for frk training. A previous run with 10000 iterations and nearly the same conditions did not raise the assertion: ... 2 Percent improvement time=807, best error was 3.911 @ 8211 At iteration 9018/10000/10000, Mean rms=0.835%, delta=0.465%, char train=1.729%, word train=6.095%, skip ratio=0%, New best char error = 1.729Deserialize failed wrote best model:/home/stweil/src/github/tesseract-ocr/tesseract/tutorial/frkoutput/base1.729_9018.lstm wrote checkpoint. Finished! Error rate = 1.729 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#757>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE2_o-S-16Nb7AFsGsCO3w_L80p8t9Fuks5rlOxhgaJpZM4Ma6ed> .

[amitdo]

A link to the relevant line in the source code:

tesseract/ccstruct/imagedata.cpp

Line 648 in 134a253

ASSERT_HOST(num_docs > 0);

[amitdo]

Maybe it's a memory problem. How much RAM do you have in that machine?

[amitdo]

About fonts, I believe that for LSTM training Ray used much more fonts for each language than with the old engine.

[stweil]

The training machine has 23 GiB RAM plus 23 GiB swap, and about 35 GiB of that memory are available for the training. Maybe some Debian GNU Linux defaults set a smaller limit for single processes, but I don't think we have a memory problem.

[stweil]

According to font_properties, Ray used about 6000 fonts. I used 12 fonts. The training result was pretty good for the old engine and unusable for LSTM.

[amitdo]

So little RAM? A University server, I guess... :-)

[stweil]

A server which wants to process 700000 pages of a journal printed in fraktur.

[stweil]

I have found that the error goes away when NOT using --eval_listfile, please try without the following ...

Yes, the assertion does not occur when I omit --eval_listfile.

[Shreeshrii]

On Mon, Mar 13, 2017 at 3:42 PM, Amit D. ***@***.***> wrote: I used 12 fonts. The training result was pretty good for the old engine and unusable for LSTM. 12 fonts is not enough for LSTM. Use as much fonts as you can find.

​@stweil You can increase the number of box/tiff pairs by adding --exposures "-1 0 1" or even --exposures "-2 -1 0 1 2" ​with the same fonts to get images which are lighter and darker than the original font. ``` training/tesstrain.sh --fonts_dir /usr/share/fonts --lang frk \ --linedata_only --noextract_font_properties --exposures "-1 0 1" \ --langdata_dir ../langdata --tessdata_dir ./tessdata \ --output_dir ~/tesstutorial/frk ``` Also, please check whether the fonts you are using have support for the paragraph marker etc, otherwise they might get dropped as unrenderable. @theraysmith I think it will be useful if training using non-synthetic box/tiff pairs is also supported for LSTM. Thanks.

[Shreeshrii]

@stweil I had generated box files for different Fraktur font alphabet images using makebox. However these need to be reviewed for correctness and tabs need to be added at end of lines.

I do not recognize the letters so can't update them. jtessboxeditor could be used for adding tabs.

https://github.com/paalberti/tesseract-dan-fraktur/files/721936/fraktur-png-box-to-be-corrected.zip

[stweil]

@Shreeshrii, that's a nice collection of Fraktur fonts, but several of the image not even include all normal ASCII characters. All images are missing the long s character (ſ) which is very important for all Fraktur texts. Also missing are all forms of ligatures (combinations of certain characters, like for example ffi, which need a special rendering).

[Shreeshrii]

I know. I do not have those fonts,but found these images on the net on some font sites. If you or Ray have the resources to get these fonts, you can use them to create appropriate trainingdata. ShreeDevi

…

____________________________________________________________ भजन - कीर्तन - आरती @ http://bhajans.ramparivar.com

On Tue, Mar 14, 2017 at 11:16 AM, Stefan Weil ***@***.***> wrote: @Shreeshrii <https://github.com/Shreeshrii>, that's a nice collection of Fraktur fonts, but several of the image not even include all normal ASCII characters. All images are missing the long s character (ſ) which is very important for all Fraktur texts. Also missing are all forms of ligatures (combinations of certain characters, like for example ffi, which need a special rendering). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#757 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE2_o_my9wQq3XxnJtuzD2QWSpwu0Pt4ks5rlimqgaJpZM4Ma6ed> .

[Shreeshrii]

These are the freely available Fraktur fonts that I found:

FRAKTUR_FONTS=(
  "CaslonishFraxx Medium" \
  "Cloister Black, Semi-Light" \
  "Proclamate Light, Semi-Light" \
  "UnifrakturCook" \
  "UnifrakturMaguntia" \
  "UnifrakturMaguntia16" \
  "UnifrakturMaguntia17" \
  "UnifrakturMaguntia18" \
  "UnifrakturMaguntia19" \
  "UnifrakturMaguntia20" \
  "UnifrakturMaguntia21" \
    "Walbaum-Fraktur" \
)

sample files via text2image that I had posted in the issue

https://github.com/paalberti/tesseract-dan-fraktur/files/721956/frk.box-tif-pairs.zip

[stweil]

Did you see the new page about fonts which I added to the wiki? Maybe you want to add information there.