fix: remove default temperature on all QueryTransformer implementations #2258
Conversation
|
@apappascs Thanks for the PR! |
| @@ -84,7 +84,7 @@ public Query transform(Query query) { | |||
| .user(user -> user.text(this.promptTemplate.getTemplate()) | |||
| .param("history", formatConversationHistory(query.history())) | |||
| .param("query", query.text())) | |||
| .options(ChatOptions.builder().temperature(0.0).build()) | |||
There was a problem hiding this comment.
@ThomasVitale Is there any specific reason why we had the temperature set here?
There was a problem hiding this comment.
It's recommended to set the temperature to a low value when performing query transformations to ensure more accurate results that will improve the retrieval quality. After removing this, it will be up to the developer to set the temperature on the ChatClient.Builder when instantiating a QueryTransformer. We need to specify that in the documentation as a pre-requisite to use a query transformer, because the typical default temperature is too high and will affect the retrieval quality.
All these modular RAG APIs and implementations are still in an experimental state. This is good input to perhaps review some design choices and improve the APIs, ensuring both good out-of-the-box experience, but also flexibility.
Side note:
This is not the first place where we see problems with the OpenAI reasoning models behaving differently. It feels like something that should be handled at the ChatModel API level so to have a general handling of those differences in one place, rather than addressing incompatibilities wherever they occur.
There was a problem hiding this comment.
Thank you for the feedback, @ThomasVitale. I have updated the documentation based on your comment.
Signed-off-by: Alexandros Pappas <[email protected]>
apappascs
force-pushed
the
fix/remove-default-temperature-querytransformer
branch
from
1ca2107 to
282d7cb
Compare
Signed-off-by: Alexandros Pappas <[email protected]>
apappascs
force-pushed
the
fix/remove-default-temperature-querytransformer
branch
from
282d7cb to
dd40808
Compare
resolving #2243