Saving and loading tokenizers with torch.save fails

[mittalsuraj18]

🐛 Bug

Information

Model I am using (Bert, XLNet ...): Albert

Language I am using the model on (English, Chinese ...): English

The problem arises when using:

• the official example scripts: (give details below)
• my own modified scripts: (give details below)

The tasks I am working on is:

• an official GLUE/SQUaD task: (give the name)
• my own task or dataset: (give details below)

To reproduce

Steps to reproduce the behavior:

1. Load albert base tokenizer using AutoTokenizer.from_pretrained
2. Save it to a file using torch.save
3. Delete ~/.cache/torch/transformers directory
4. Now try to load from the file using torch.load
5. Loading fails as the cached file does not exist

import transformers
import torch
token = transformers.AutoTokenizer.from_pretrained("albert-base-v2")
torch.save({"token":token}, "./token.pt")

Delete ~/.cache/torch/ directory
Then Run

import torch
torch.load("./token.pt")

Expected behavior

Tokenizer should load successfully.

Environment info

• transformers version: 2.11.0
• Platform: Linux-4.19.104-microsoft-standard-x86_64-with-debian-bullseye-sid
• Python version: 3.7.6
• PyTorch version (GPU?): 1.3.1+cpu (False)
• Tensorflow version (GPU?): not installed (NA)
• Using GPU in script?: no
• Using distributed or parallel set-up in script?: no

[bugface]

I met save issue before. Based on my experience, it is more related to PyTorch.
See https://pytorch.org/tutorials/beginner/saving_loading_models.html, you should use state_dict to save and load model.

[mittalsuraj18]

But @FacingBugs, this is an issue with the tokenizers and not the models

[bugface]

For tokenizer, using Transformer package provided API:

tokenizer.save_pretrained(your_output_model_dir)

[mittalsuraj18]

@FacingBugs actually I have raised this bug because it was causing an issue in another library which uses this package
flairNLP/flair#1712
And since torch.save is mostly used to persist the models and dependencies for pytorch based learning, I believe the fix should be implemented in the transformers library itself rather than other dependent libraries which may add on top of transformers to provide their custom pytorch models in which case torch.save would mostly be used to save the models.

[bugface]

I think according to the PyTorch documentation, torch.save() is not recommended.

I cite this from the documentation:
"However in this case, the serialized data is bound to the specific classes and the exact directory structure used, so it can break in various ways when used in other projects, or after some serious refactors."

For my personal experience with the Transformer package, the xx.save_pretrained() works for most of the cases (models, tokenizers, configs). For the tokenizer, I think the package actually saved several other files besides the vocab file. I think using the save_pretrained method should be the best practice.

Hope this can help you.

[mittalsuraj18]

Wouldn’t this make persisting other models on top of transformers difficult because now we have to save and track multiple files instead of a single file?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.