fix thinking for gemini models #1113
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Description# Summary By MatterAI
🔄 What ChangedThis pull request refactors the handling of 'thinking' (chain-of-thought) messages for Gemini models across both Google Vertex AI and Google Generative AI providers. Key changes include:
🔍 Impact of the ChangeThis fix ensures that 'thinking' messages from Gemini models are correctly processed and displayed, separating them from the main content. It improves the accuracy of token usage reporting for reasoning steps and enhances the flexibility of the API response by providing structured content blocks when strict OpenAI compliance is not required. This directly addresses the issue of incorrect thinking output handling. 📁 Total Files Changed4 files were changed in this pull request. 🧪 Test AddedManual testing was performed to verify the changes:
🔒Security VulnerabilitiesNo security vulnerabilities were detected in the changes. MotivationCloses #1112, which addresses a bug related to the incorrect handling of thinking output for Gemini models. Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related IssuesTip Quality Recommendations
Sequence DiagramsequenceDiagram
participant Client
participant GatewayAPI as Gateway API (/chat/completions)
participant GoogleVertexAIProvider as Google Vertex AI Provider
participant GoogleProvider as Google Provider
participant GoogleVertexAIAPI as Google Vertex AI API
participant GoogleGenerativeAIAPI as Google Generative AI API
Client->>GatewayAPI: POST /chat/completions (params: {..., thinking: {budget_tokens, type}, ...})
GatewayAPI->>GoogleVertexAIProvider: chatComplete(params)
GatewayAPI->>GoogleProvider: chatComplete(params)
GoogleVertexAIProvider->>GoogleVertexAIProvider: transformGenerationConfig(params)
GoogleProvider->>GoogleProvider: transformGenerationConfig(params)
alt For Google Vertex AI
GoogleVertexAIProvider->>GoogleVertexAIAPI: generateContent(generationConfig: {thinking_config: {include_thoughts, thinking_budget}})
GoogleVertexAIAPI-->>GoogleVertexAIProvider: response (usageMetadata: {thoughtsTokenCount}, candidates: [{content: {parts: [{text, thought}, {functionCall}]}}])
GoogleVertexAIProvider->>GoogleVertexAIProvider: GoogleChatCompleteResponseTransform(response)
GoogleVertexAIProvider->>GoogleVertexAIProvider: GoogleChatCompleteStreamChunkTransform(parsedChunk)
end
alt For Google Generative AI
GoogleProvider->>GoogleGenerativeAIAPI: generateContent(generationConfig: {thinking_config: {include_thoughts, thinking_budget}})
GoogleGenerativeAIAPI-->>GoogleGenerativeAIAPI: response (usageMetadata: {thoughtsTokenCount}, candidates: [{content: {parts: [{text, thought}, {functionCall}]}}])
GoogleProvider->>GoogleProvider: GoogleChatCompleteResponseTransform(response)
GoogleProvider->>GoogleProvider: GoogleChatCompleteStreamChunkTransform(parsedChunk)
end
GoogleChatCompleteResponseTransform->>GoogleChatCompleteResponseTransform: Process content parts
GoogleChatCompleteResponseTransform->>GoogleChatCompleteResponseTransform: Identify 'thought' and 'text' parts
GoogleChatCompleteResponseTransform->>GoogleChatCompleteResponseTransform: Construct 'contentBlocks' array ({type: 'thinking', thinking: part.text} or {type: 'text', text: part.text})
GoogleChatCompleteResponseTransform->>GoogleChatCompleteResponseTransform: Add 'content_blocks' to message if !strictOpenAiCompliance
GoogleChatCompleteResponseTransform->>GoogleChatCompleteResponseTransform: Map tool_calls
GoogleChatCompleteResponseTransform->>GoogleChatCompleteResponseTransform: Update usage with 'thoughtsTokenCount' in 'completion_tokens_details.reasoning_tokens'
GoogleChatCompleteStreamChunkTransform->>GoogleChatCompleteStreamChunkTransform: Process content parts in stream
GoogleChatCompleteStreamChunkTransform->>GoogleChatCompleteStreamChunkTransform: Identify 'thought' and 'text' parts
GoogleChatCompleteStreamChunkTransform->>GoogleChatCompleteStreamChunkTransform: Construct 'contentBlocks' array ({index, delta: {thinking}} or {index, delta: {text}})
GoogleChatCompleteStreamChunkTransform->>GoogleChatCompleteStreamChunkTransform: Add 'content_blocks' to message if !strictOpenAiCompliance
GoogleChatCompleteStreamChunkTransform->>GoogleChatCompleteStreamChunkTransform: Update usage with 'thoughtsTokenCount' in 'completion_tokens_details.reasoning_tokens'
GoogleVertexAIProvider-->>GatewayAPI: Transformed Response
GoogleProvider-->>GatewayAPI: Transformed Response
GatewayAPI-->>Client: Final API Response
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
csgulati09
previously approved these changes
May 27, 2025
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
b4s36t4
reviewed
Jun 10, 2025
VisargD
approved these changes
Jun 11, 2025
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
closes #1112
testing done:
example payload:
{ "model": "gemini-2.5-flash-preview-04-17", "max_tokens": 1000, "thinking": { "budget_tokens": 100, "type": "enabled" }, "stream": false, "messages": [ { "role": "user", "content": [ { "type": "text", "text": "if tomatina tomatino has fathered tralleliala trallela, and batatina batata is tomatinas sister, how is she related to trallelia?" } ] } ] }