Preparing Biomedical Multimodal (Text+Images/Tables/Formulas) Dataset for Gemma Fine-tuning

[virologist]

Hi,@gemma team

I’m working on a biomedical/medical domain fine-tuning dataset converted from PDFs to Markdown. The content contains:
• Images (medical diagrams, charts)
• Complex tables (clinical data)
• Mathematical formulas (drug dosage equations, statistical models)
Could the Gemma team advise:
1. Best practices for handling multimodal elements in Markdown format during dataset preparation?
2. Recommended preprocessing steps for preserving semantic relationships between text & non-text elements?
3. Any existing tutorials/documentation for similar technical/academic domain fine-tuning?

Any help would be greatly appreciated!
Yang

[Gopi-Uppari]

Hi @virologist,

Multimodal Handling:

Images: Use alt-text or captions; for image-supported models (like Gemma 3), include image fields in JSON.

Tables: Extract with tools like Camelot or Tabula, then convert to Markdown or structured text.

Formulas: Preserve using LaTeX format to retain structure and accuracy.

Preprocessing Tips:

Maintain structure (headings, lists, references) using Markdown.

Place images/tables near related text for context.

Clean text but keep domain-specific symbols and notations.

Fine-Tuning Gemma:

Use format: {"input_text": ..., "output_text": ...} for instruction tuning.

For multimodal fine-tuning, use Hugging Face tools with image support.

Tokenization and domain adaptation are key—ensure terms and formulas are well-preserved.

Could you please refer to Gemma fine-tuning guide

Thank you.

[Gopi-Uppari]

Could you please confirm if this issue is resolved for you with the above comment ? Please feel free to close the issue if it is resolved ?

Thank you.