Skip to content

Add Gemini test infrastructure. #1343

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

umaannamalai
Copy link
Contributor

This PR adds test infrastructure to support embeddings and text completions in google-genai Gemini LLMs including:

  • An updated tox file to run tests for google-genai
  • A mocked server to return prerecorded Gemini responses to test against so the real Gemini backend with a API key is not needed
  • A new conftest.py file that configures and yields test clients, the main server, and includes logic to intercept real Gemini requests and responses and log them out to a log file for later use to update the existing mocked server with new prompts

@umaannamalai umaannamalai requested a review from a team as a code owner April 1, 2025 03:41
Copy link

github-actions bot commented Apr 1, 2025

🦙 MegaLinter status: ✅ SUCCESS

Descriptor Linter Files Fixed Errors Warnings Elapsed time
✅ ACTION actionlint 4 0 0 0.52s
✅ MARKDOWN markdownlint 7 0 0 0 0.99s
✅ MARKDOWN markdown-link-check 7 0 0 15.92s
✅ PYTHON ruff 895 0 0 0 0.58s
✅ PYTHON ruff-format 895 0 0 0 0.32s
✅ YAML prettier 10 0 0 0 1.03s
✅ YAML v8r 10 0 0 4.82s
✅ YAML yamllint 10 0 0 0.52s

See detailed report in MegaLinter reports

MegaLinter is graciously provided by OX Security

@mergify mergify bot removed the merge-conflicts label Apr 1, 2025
@mergify mergify bot added the tests-failing label Apr 1, 2025
@codecov-commenter
Copy link

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 81.29%. Comparing base (9223f7a) to head (25705b1).

Additional details and impacted files
@@                        Coverage Diff                         @@
##           feature-gemini-instrumentation    #1343      +/-   ##
==================================================================
- Coverage                           81.48%   81.29%   -0.19%     
==================================================================
  Files                                 204      204              
  Lines                               22821    22821              
  Branches                             3617     3617              
==================================================================
- Hits                                18595    18553      -42     
- Misses                               3013     3050      +37     
- Partials                             1213     1218       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@umaannamalai umaannamalai merged commit 2cadc57 into feature-gemini-instrumentation Apr 1, 2025
46 of 52 checks passed
@umaannamalai umaannamalai deleted the gemini-test-infrastructure branch April 1, 2025 19:58
@mergify mergify bot removed the tests-failing label Apr 1, 2025
umaannamalai added a commit that referenced this pull request Apr 7, 2025
* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
umaannamalai added a commit that referenced this pull request Apr 30, 2025
* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
umaannamalai added a commit that referenced this pull request Apr 30, 2025
* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
umaannamalai added a commit that referenced this pull request Apr 30, 2025
* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
@umaannamalai umaannamalai added this to the v10.11.0 milestone May 1, 2025
TimPansino pushed a commit that referenced this pull request May 1, 2025
* Add Gemini test infrastructure. (#1343)

* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add Gemini embeddings.

* [MegaLinter] Apply linters fixes

* Address review comments.

* Key into kwargs dict.

* Update embedding error logic.

* Fix embeddings error tests for py39.

* [MegaLinter] Apply linters fixes

* Add Gemini test infrastructure. (#1343)

* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add Gemini embeddings.

* [MegaLinter] Apply linters fixes

* Address review comments.

* Key into kwargs dict.

* Update embedding error logic.

* Fix embeddings error tests for py39.

* [MegaLinter] Apply linters fixes

* Add Gemini text generations.

* Bump tests.

* [MegaLinter] Apply linters fixes

* Fix py39 errors test failure.

* [MegaLinter] Apply linters fixes

* Add comments.

* Update request message logic.

* [MegaLinter] Apply linters fixes

* Add string-based tests.

* Add Gemini test infrastructure. (#1343)

* Add Gemini test infra.

* Pull tox.ini changes.

* [MegaLinter] Apply linters fixes

* Remove mock package dependency.

* [MegaLinter] Apply linters fixes

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>

* Add Gemini embeddings.

* [MegaLinter] Apply linters fixes

* Address review comments.

* Key into kwargs dict.

* Update embedding error logic.

* Fix embeddings error tests for py39.

* [MegaLinter] Apply linters fixes

* Add Gemini text generations.

* Bump tests.

* Fix py39 errors test failure.

* [MegaLinter] Apply linters fixes

* Add comments.

* [MegaLinter] Apply linters fixes

* Add Gemini embeddings.

* Address review comments.

* Key into kwargs dict.

* Fix embeddings error tests for py39.

* [MegaLinter] Apply linters fixes

* Update request message logic.

* Add string-based tests.

* [MegaLinter] Apply linters fixes

* Address review.

* Temporarily pin redis tests.

* Temporarily pin redis tests to below v5.3.0.

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Hannah Stepanek <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants