Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: remove default temperature on all QueryTransformer implementations #2258

Open
wants to merge 2 commits into
base: main
Choose a base branch
from
Open

fix: remove default temperature on all QueryTransformer implementations #2258

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3609c91
fix: remove default temperature on all QueryTransformer implementations
apappascs Feb 17, 2025
dd40808
update documentation
apappascs Feb 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion ...springframework/ai/rag/preretrieval/query/transformation/CompressionQueryTransformer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ public Query transform(Query query) {
.user(user -> user.text(this.promptTemplate.getTemplate())
.param("history", formatConversationHistory(query.history()))
.param("query", query.text()))
.options(ChatOptions.builder().temperature(0.0).build())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ThomasVitale Is there any specific reason why we had the temperature set here?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's recommended to set the temperature to a low value when performing query transformations to ensure more accurate results that will improve the retrieval quality. After removing this, it will be up to the developer to set the temperature on the ChatClient.Builder when instantiating a QueryTransformer. We need to specify that in the documentation as a pre-requisite to use a query transformer, because the typical default temperature is too high and will affect the retrieval quality.

All these modular RAG APIs and implementations are still in an experimental state. This is good input to perhaps review some design choices and improve the APIs, ensuring both good out-of-the-box experience, but also flexibility.

Side note:
This is not the first place where we see problems with the OpenAI reasoning models behaving differently. It feels like something that should be handled at the ChatModel API level so to have a general handling of those differences in one place, rather than addressing incompatibilities wherever they occur.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for the feedback, @ThomasVitale. I have updated the documentation based on your comment.

.options(ChatOptions.builder().build())
.call()
.content();

Expand Down
2 changes: 1 addition & 1 deletion ...org/springframework/ai/rag/preretrieval/query/transformation/RewriteQueryTransformer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ public Query transform(Query query) {
.user(user -> user.text(this.promptTemplate.getTemplate())
.param("target", this.targetSearchSystem)
.param("query", query.text()))
.options(ChatOptions.builder().temperature(0.0).build())
.options(ChatOptions.builder().build())
.call()
.content();

Expand Down
2 changes: 1 addition & 1 deletion ...springframework/ai/rag/preretrieval/query/transformation/TranslationQueryTransformer.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@ public Query transform(Query query) {
.user(user -> user.text(this.promptTemplate.getTemplate())
.param("targetLanguage", this.targetLanguage)
.param("query", query.text()))
.options(ChatOptions.builder().temperature(0.0).build())
.options(ChatOptions.builder().build())
.call()
.content();

Expand Down
2 changes: 2 additions & 0 deletions ...docs/src/main/antora/modules/ROOT/pages/api/retrieval-augmented-generation.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -156,6 +156,8 @@ Pre-Retrieval modules are responsible for processing the user query to achieve t
A component for transforming the input query to make it more effective for retrieval tasks, addressing challenges
such as poorly formed queries, ambiguous terms, complex vocabulary, or unsupported languages.

IMPORTANT: When using a `QueryTransformer`, it's recommended to configure the `ChatClient.Builder` with a low temperature (e.g., 0.0) to ensure more deterministic and accurate results, improving retrieval quality. The default temperature for most chat models is typically too high for optimal query transformation, leading to reduced retrieval effectiveness.

===== CompressionQueryTransformer

A `CompressionQueryTransformer` uses a large language model to compress a conversation history and a follow-up query
Expand Down