[Bugfix] Fix Gemma3 multimodal placeholder replacement

[tristanleclercq]

google/gemma-3-4b-it fails when run via the openai server with multimodal input with the following error:

RuntimeError: Expected there to be 1 prompt updates corresponding to 1 image items, but instead found 0 prompt updates! Either the prompt text has missing/incorrect tokens for multi-modal inputs, or there is a problem with your implementation of merged multi-modal processor for this model (usually arising from an inconsistency between `_call_hf_processor` and `_get_prompt_updates`).

I observed the issue with engine V0, I didn't try with engine V1.

The issue can be observed for messages of this form:

{'role': 'system', 'content': [{'text': 'Describe the image in a short sentence.', 'type': 'text'}]}, {'role': 'user', 'content': [{'text': 'random text', 'type': 'text'}, {'text': 'image: ', 'type': 'text'}, {'image_url': , my_image_url}]}]

The issue comes from the fact that if the input message containing the image also has text parts (the user message in the example above), then the chat_utils._get_full_multimodal_text_prompt method will put the image placeholder at the front like so '<start_of_image>\nrandom text. Then the Gemma3MultiModalProcessor._get_prompt_updates => get_replacement_gemma3 => get_image_repl will replace the placeholder with Gemma3Processor.full_image_sequence = f"\n\n{tokenizer.boi_token}{image_tokens_expanded}{tokenizer.eoi_token}\n\n", which will result in 3 consecutive \n characters after the eoi_token. The \n\n\n which will be tokenized as the 109 token, and when passed to _find_mm_placeholders, the 109 token will be replaced by [107, 108], instead of [108, 107], which won't allow the matching condition in _iter_placeholders to ever evaluate to True.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[MichealZeng0420]

This error noy only happens on Gemma3. WhenI tried tp inference other VLMs( which is supported) , I suffered the same error and it puzzled me a lot. I attempted to clearly follow the prompt format which provided in examples file (https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py) to slove this problem.

[DarkLight1337]

Thanks for fixing, can you add a test to prevent future regressions?

[DarkLight1337]

This error noy only happens on Gemma3. WhenI tried tp inference other VLMs( which is supported) , I suffered the same error and it puzzled me a lot. I attempted to clearly follow the prompt format which provided in examples file (https://github.com/vllm-project/vllm/blob/main/examples/offline_inference/vision_language_multi_image.py) to slove this problem.

Could you elaborate? I suggest opening a new issue and explaining it there