[12.0][REF] Generate intermediary clean CSV files #2
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Example : EFD ICMS IPI pdf Outubro 2019 p20-21 Fixed by deleting the break breaking the for loop parsing the raw CSV files when a register's row seems empty The commit add other small cleaning.
And other refactoring : - Clean the register's header strings - Extract the in_out register's requirements from extract_fields_spec()
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
2 times, most recently
from
f0e3540 to
0172915
Compare
Refactor to build 3 types of csv in 4 steps :
1. extract **raw CSV** from pdf with camelot
2. build **"accurate" CSV** from raw CSV
- 1 CSV for each module
- 1 header for each CSV
- 1 line for each field
- No field's cells modification (neither loss of information),
only mapping which cell is under which CSV's column
- For each field, adding the field's register name and the
page number to check back in the pdf
- option to apply "camelot_row_patches" if necessary
3. build a **"usable" python dictionary** from the "authentic CSV"
- with "interpreted" values like "required", "type" and "int_size"
4. build **"usable" CSV** or JSON or whatever from the dict
with **no fields modification**, just built from the dict values
And from these "usabale" dictionnaries :
5. build **"odoo-usable" CSV** _from the dict_ with additional columns
to create Odoo objects from the CSV lines.
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
0172915 to
34df33a
Compare
In order to help hard-coding the modules headers based on 'manual' observation of all the modules headers displayed by this method 'get_all_headers'.
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
988ee8e to
66e0ae4
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
2 times, most recently
from
e9979ca to
0bb9343
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
2 times, most recently
from
3a69655 to
0eb1bdb
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
0eb1bdb to
3121507
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
fa23706 to
0af1165
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
2 times, most recently
from
d37707d to
aedae11
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
aedae11 to
a7a2e5d
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
d102b58 to
5ba5798
Compare
clementmbr
force-pushed
the
12.0-ref-extract-register
branch
from
8c644f5 to
ff53ac1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Separate
extract_registers_spec()andextract_fields_spec()from generate.py in a new file to be launched separately and return intermediary CSV files.