Skip to content

Add a numeric check to the PaliGemma2 conversion script #2012

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

james77777778
Copy link
Collaborator

@james77777778 james77777778 commented Dec 6, 2024

Fix #2005
This PR adds a numeric check to the PaliGemma2 conversion script.

I noticed that the outputs are a bit weird when running on the CPU (Ryzen 7700). However, they work fine on the GPU (RTX 4070).

Here are the results (prompt: "describe en\n"):

Variant big_vision KerasHub
"pali_gemma2_3b_pt_448" cpu "A brown dairy" "A brown dairy cow on the beach" 🔶
"pali_gemma2_3b_pt_448" gpu "A cow on the beach" "A cow on the beach"
"pali_gemma2_3b_ft_docci_448" cpu "A brown cow is standing on the shore of the shore of the shore of the shore of the shore of the shore. The sun is casting a shade over the" "A brown cow is standing on the shore of the shore of the shore of the shore of the shore of the shore of the shore. The" 🔶
"pali_gemma2_3b_ft_docci_448" gpu "A cow on a beach. The cow is brown with a white spot on its head and a white tag on its ear. The cow is" "A cow on a beach. The cow is brown with a white spot on its head and a white tag on its ear. The cow is"
  • ✅: matched
  • 🔶: partially matched

cc @divyashreepathihalli

@github-actions github-actions bot added the Gemma Gemma model specific issues label Dec 6, 2024
Copy link
Member

@mattdangerw mattdangerw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks; look good! Kinda awkward the big_vision is so difficult to install, but we can go with this for now.

We could consider a safetensors converter and validator for this at some point. Not that transformers is a better source of comparison than big_vision. But more easily installable and would fit with other builtin converters we have.

@mattdangerw mattdangerw merged commit bc2e8d7 into keras-team:master Dec 7, 2024
10 checks passed
@james77777778 james77777778 deleted the add-numeric-check-paligemma2 branch December 7, 2024 02:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Gemma Gemma model specific issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add a numeric check to the PaliGemma2 conversion script
2 participants