Skip to content

Fix llama tokenizer #22402

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Apr 3, 2023
Merged

Fix llama tokenizer #22402

Changes from 1 commit
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
d3b2050
draft
ArthurZucker Mar 24, 2023
0da5717
update tokenization limma and conversion script
ArthurZucker Mar 27, 2023
a2f2b19
more udpates
ArthurZucker Mar 27, 2023
7cf369c
initial commit
ArthurZucker Mar 27, 2023
ff70e2c
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Mar 27, 2023
0cf62ec
style
ArthurZucker Mar 27, 2023
1f1a4bf
default pad to None
ArthurZucker Mar 27, 2023
7e3ea59
draft tokenization tests
ArthurZucker Mar 29, 2023
33dfe79
Merge branch 'main' of https://github.com/huggingface/transformers in…
ArthurZucker Mar 30, 2023
ea0c834
update test
ArthurZucker Mar 30, 2023
3bd23e6
update tokenization tests
ArthurZucker Mar 30, 2023
8edf9b1
nits
ArthurZucker Mar 30, 2023
a334056
update
ArthurZucker Mar 30, 2023
5b9be49
versioning test
ArthurZucker Mar 30, 2023
1b6532a
major fix
ArthurZucker Mar 30, 2023
d988805
fix more testst
ArthurZucker Mar 30, 2023
6bdf1e0
finish fixing special masks
ArthurZucker Mar 30, 2023
1ab5d7c
last nit
ArthurZucker Mar 30, 2023
512f77f
more nits
ArthurZucker Mar 30, 2023
4c3c2e2
add encode decode tests
ArthurZucker Mar 30, 2023
3175977
add more
ArthurZucker Mar 30, 2023
1871030
fix token type ids
ArthurZucker Mar 30, 2023
fd60fb6
style
ArthurZucker Mar 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion tests/test_tokenization_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3932,7 +3932,7 @@ def test_clean_up_tokenization_spaces(self):
tokenizer_fast.save_pretrained(tmp_dir_2)
tokenizer = BertTokenizer.from_pretrained(tmp_dir_2)

assert tokenizer_fast.clean_up_tokenization_spaces is False
assert tokenizer.clean_up_tokenization_spaces is False
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is such a small nit that I included it 😅

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After rebasing, this test fails for me :( just reproduced on main:

>       assert decoded == "[CLS] this shouldn ' t be! he ' ll go. [SEP]"
E       assert "[CLS] this s...'ll go. [SEP]" == "[CLS] this s... ll go. [SEP]"
E         - [CLS] this shouldn ' t be! he ' ll go. [SEP]
E         ?                   - -        - -
E         + [CLS] this shouldn't be! he'll go. [SEP]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is not pointing to the correct part of the test. If the cleanup_tokenization_spaces is indeed False, the fail can happen for cache reasons or anything else (also failed for me at some point).
Will check again

decoded = tokenizer.decode(tokens)
assert decoded == "[CLS] this shouldn ' t be ! he ' ll go . [SEP]"

Expand Down