ReadTimeoutException with Spring AI for Moderate-Sized Prompts (<4K characters) #2195
Answered
by
LiveNathan
LiveNathan
asked this question in
Q&A
-
|
I've encountered an unexpected Working Scenario
Failing Scenario
Error DetailsQuestions
Environment
Has anyone else encountered similar timeout issues with relatively small prompts? Passing Test@Test
void givenShortUserMessage_whenProcessingWithChatClient_thenResponseIsSuccessful() {
ChatClient chatClient = ChatClient.create(chatModel);
String userMessage = """
## How to rate industry alignment
Using the following scale, rate how closely each keyword aligns with the industry. The specific values represent key reference points or milestones, but your rating should lie on the continuous scale between 0 and 100 in proximity to those milestones.
100 = Perfect alignment: The keyword is highly relevant to the food and beverage, cafe culture, coffee shops industry.
50 = Partial alignment: The keyword is somewhat related or adjacent to this industry.
0 = Misalignment: The keyword is not related to this industry at all. In addition, ensure that keywords related to are give a low rating.
## List of keywords to analyze
150 ml to ounces, 150ml to ounces, 20 oz in pounds
## Output format to return
Set of keywordsWithThemeAndConfidence objects. Each object must include:
- The keyword
- A confidence rating (0–100) indicating how well it aligns with the specified industry.""";
ResponseEntity<ChatResponse, AiResponseKeywordsAligned> response = chatClient.prompt()
.system("You are a keyword research assistant specialized in SEO and Google Keyword Planner, helping users discover high-quality, long-tail keywords for niche businesses.")
.user(userMessage)
.call()
.responseEntity(AiResponseKeywordsAligned.class);
assertThat(response)
.as("Check that the response is not null")
.isNotNull();
AiResponseKeywordsAligned entity = response.entity();
assertThat(entity)
.as("Check that the entity is not null")
.isNotNull();
assertThat(entity.keywordsWithThemeAndConfidence())
.as("Check if AI returned any keywords with confidence")
.isNotNull()
.isNotEmpty();
}Failing Test@Test
void givenLongUserMessage_whenProcessingWithChatClient_thenResponseIsSuccessful() {
ChatClient chatClient = ChatClient.create(chatModel);
String userMessage = """
## How to rate industry alignment
Using the following scale, rate how closely each keyword aligns with the industry. The specific values represent key reference points or milestones, but your rating should lie on the continuous scale between 0 and 100 in proximity to those milestones.
100 = Perfect alignment: The keyword is highly relevant to the food and beverage, cafe culture, coffee shops industry.
50 = Partial alignment: The keyword is somewhat related or adjacent to this industry.
0 = Misalignment: The keyword is not related to this industry at all. In addition, ensure that keywords related to are give a low rating.
## List of keywords to analyze
150 ml to ounces, 150ml to ounces, 20 oz in pounds, 5 ounces to g, 5 ounces to gr, 5 ounces to grams, 6'2 in cm, 80 ml to ounces, a gallon of water is how many water bottles, bahama mama recipe, benefits of central business district, best afternoon tea london, best cafe near me, best espresso martini recipe, best local coffee shops, blue in spanish, cafe in meiner nähe, cafes to study, caffeine in diet coke, caffeine in espresso shot, chai tea benefits, chocolate cafe, chocolate house, chocolate rain, chocolate rain song, chocolate world, cinnamon tea benefits, coffee loophole recipe, coffee shops with wifi near me, cranberry in spanish, daiquiri recipe, dark chocolate benefits, definition for wholesome, est to ist, exit to eden, factory tea bar, french martini recipe, fruits start with i, gentleman in moscow book, gin and tonic recipe, ginger tea benefits, green bean casserole with fresh green beans, health food store, hershey chocolate world, hershey's chocolate world, hibiscus tea benefits, homemade hot chocolate, horchata recipe, hot chocolate near me, hot chocolate recipe, hot chocolate run, how many bottle of waters is a gallon, how many bottle waters are in a gallon, how many bottled waters are in a gallon, how many grams of sugar in a tablespoon of sugar, how many grams of sugar in a teaspoon, how many grams of sugar in a teaspoon of sugar, how many ounces of water to drink a day, how many water bottle are in a gallon, how many water bottles are in a gallon, how many water bottles are in a gallon of water, how many water bottles is a gallon of water, how many water bottles is in a gallon, how much caffeine in matcha, how much sugar can of coke, how much sugar i n can of coke, how much sugar in a can of coca cola, how much sugar in a can of coke, how much sugar in can of coke, how much sugar in can of soda, how much sugar is in a can of coke, how much sugar is in a can of pop, how much sugar is in a soda can, how much sugar is in coke can, how much sugars in a can of coke, how to descale a keurig, how to make hot chocolate, how to make hummingbird nectar, hugo spritz recipe, ingredients in coke, is carbonated water bad for you, is coconut water good for you, is coffee good for you, is coke zero bad for you, is dark chocolate good for you, is diet coke bad for you, is fluoride bad for you, is matcha good for you, is oat milk good for you, is sparkling water good for you, is vitamin water good for you, juice bar, kakawa chocolate house, lemon drop recipe, loco moco recipe, long island iced tea recipe, lychee martini recipe, lyrics to escape the pina colada song, mai tai recipe, mango smoothie recipe, martini recipe, matcha benefits, mg of caffeine in coffee, movita juice bar, nekter juice bar, oolong tea benefits, oz in grams, peppermint tea benefits, pina colada recipe, pomegranate benefits, raspberry leaf tea benefits, raw milk benefits, tea bar, time for tokyo, time in shan, toothsome chocolate emporium menu, virgin pina colada recipe, water in spanish, what's in spanish, whiskey drink recipes, whole foods coffee bar
## Output format to return
Set of keywordsWithThemeAndConfidence objects. Each object must include:
- The keyword
- A confidence rating (0–100) indicating how well it aligns with the specified industry.""";
ResponseEntity<ChatResponse, AiResponseKeywordsAligned> response = chatClient.prompt()
.system("You are a keyword research assistant specialized in SEO and Google Keyword Planner, helping users discover high-quality, long-tail keywords for niche businesses.")
.user(userMessage)
.call()
.responseEntity(AiResponseKeywordsAligned.class);
assertThat(response)
.as("Check that the response is not null")
.isNotNull();
AiResponseKeywordsAligned entity = response.entity();
assertThat(entity)
.as("Check that the entity is not null")
.isNotNull();
assertThat(entity.keywordsWithThemeAndConfidence())
.as("Check if AI returned any keywords with confidence")
.isNotNull()
.isNotEmpty();
}Test Configuration@SpringBootConfiguration
public class OpenAiTestConfiguration {
@Bean
public OpenAiApi openAiApi() {
return new OpenAiApi(getApiKey());
}
@Bean
public OpenAiChatModel openAiChatModel(OpenAiApi api) {
return new OpenAiChatModel(api, OpenAiChatOptions.builder()
.model(ChatModel.GPT_4_O_MINI)
.temperature(0.2)
.build());
}
private String getApiKey() {
String apiKey = System.getenv("SPRING_AI_OPENAI_API_KEY");
if (!StringUtils.hasText(apiKey)) {
throw new IllegalArgumentException("You must provide an API key via OPENAI_API_KEY.");
}
return apiKey;
}
}Response Objectpublic record AiResponseKeywordsAligned(
Set<KeywordWithConfidence> keywordsWithThemeAndConfidence) {
}public record KeywordWithConfidence(
@NotBlank
String keyword,
@Min(0)
@Max(100)
int industryAlignmentConfidence
) {
}Full Stacktrace |
Beta Was this translation helpful? Give feedback.
Answered by
LiveNathan
Feb 8, 2025
Replies: 1 comment 6 replies
-
|
Absolutely the same situation in my case. Please share the solution if you find it. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
Thanks! I'm not using the streaming response, but I guess I could switch to it. |
Beta Was this translation helpful? Give feedback.
-
|
Ok! If you decide to investigate the issue, have a look at how the As I'm using WebFlux, choosing the non-blocking variant was a no-brainer. And the appearance in Spring AI of something like: Mono<AiResponseKeywordsAligned> response = chatClient.prompt()
.user(userMessage)
.call()
.toMono(AiResponseKeywordsAligned.class);is just a matter of time, IMHO. Good luck! |
Beta Was this translation helpful? Give feedback.
-
|
Oh, interesting! Ok, I'll take a look. |
Beta Was this translation helpful? Give feedback.
-
|
It worked! Here's my final passing test: @Test
void givenLongResponseTime_whenProcessingWithStreamingChatClient_thenResponseIsSuccessful() {
ChatClient chatClient = ChatClient.create(chatModel);
var converter = new BeanOutputConverter<>(new ParameterizedTypeReference<AiResponseKeywordsAligned>() {
});
String systemPrompt = "You are a keyword research assistant specialized in SEO and Google Keyword Planner, helping users discover high-quality, long-tail keywords for niche businesses. {format}";
String userMessage = """
## How to rate industry alignment
Using the following scale, rate how closely each keyword aligns with the industry. The specific values represent key reference points or milestones, but your rating should lie on the continuous scale between 0 and 100 in proximity to those milestones.
100 = Perfect alignment: The keyword is highly relevant to the food and beverage, cafe culture, coffee shops industry.
50 = Partial alignment: The keyword is somewhat related or adjacent to this industry.
0 = Misalignment: The keyword is not related to this industry at all. In addition, ensure that keywords related to are give a low rating.
## List of keywords to analyze
150 ml to ounces, 150ml to ounces, 20 oz in pounds, 5 ounces to g, 5 ounces to gr, 5 ounces to grams, 6'2 in cm, 80 ml to ounces, a gallon of water is how many water bottles, bahama mama recipe, benefits of central business district, best afternoon tea london, best cafe near me, best espresso martini recipe, best local coffee shops, blue in spanish, cafe in meiner nähe, cafes to study, caffeine in diet coke, caffeine in espresso shot, chai tea benefits, chocolate cafe, chocolate house, chocolate rain, chocolate rain song, chocolate world, cinnamon tea benefits, coffee loophole recipe, coffee shops with wifi near me, cranberry in spanish, daiquiri recipe, dark chocolate benefits, definition for wholesome, est to ist, exit to eden, factory tea bar, french martini recipe, fruits start with i, gentleman in moscow book, gin and tonic recipe, ginger tea benefits, green bean casserole with fresh green beans, health food store, hershey chocolate world, hershey's chocolate world, hibiscus tea benefits, homemade hot chocolate, horchata recipe, hot chocolate near me, hot chocolate recipe, hot chocolate run, how many bottle of waters is a gallon, how many bottle waters are in a gallon, how many bottled waters are in a gallon, how many grams of sugar in a tablespoon of sugar, how many grams of sugar in a teaspoon, how many grams of sugar in a teaspoon of sugar, how many ounces of water to drink a day, how many water bottle are in a gallon, how many water bottles are in a gallon, how many water bottles are in a gallon of water, how many water bottles is a gallon of water, how many water bottles is in a gallon, how much caffeine in matcha, how much sugar can of coke, how much sugar i n can of coke, how much sugar in a can of coca cola, how much sugar in a can of coke, how much sugar in can of coke, how much sugar in can of soda, how much sugar is in a can of coke, how much sugar is in a can of pop, how much sugar is in a soda can, how much sugar is in coke can, how much sugars in a can of coke, how to descale a keurig, how to make hot chocolate, how to make hummingbird nectar, hugo spritz recipe, ingredients in coke, is carbonated water bad for you, is coconut water good for you, is coffee good for you, is coke zero bad for you, is dark chocolate good for you, is diet coke bad for you, is fluoride bad for you, is matcha good for you, is oat milk good for you, is sparkling water good for you, is vitamin water good for you, juice bar, kakawa chocolate house, lemon drop recipe, loco moco recipe, long island iced tea recipe, lychee martini recipe, lyrics to escape the pina colada song, mai tai recipe, mango smoothie recipe, martini recipe, matcha benefits, mg of caffeine in coffee, movita juice bar, nekter juice bar, oolong tea benefits, oz in grams, peppermint tea benefits, pina colada recipe, pomegranate benefits, raspberry leaf tea benefits, raw milk benefits, tea bar, time for tokyo, time in shan, toothsome chocolate emporium menu, virgin pina colada recipe, water in spanish, what's in spanish, whiskey drink recipes, whole foods coffee bar
## Output format to return
Set of keywordsWithThemeAndConfidence objects. Each object must include:
- The keyword
- A confidence rating (0–100) indicating how well it aligns with the specified industry.""";
log.info("User message: {}", userMessage);
Flux<String> response = chatClient.prompt()
.system(sp -> sp.text(systemPrompt).param("format", converter.getFormat()))
.user(userMessage)
.stream()
.content();
assertThat(response)
.as("Check that the response is not null")
.isNotNull();
List<String> chunks = response.collectList().block();
assertThat(chunks)
.as("Check that we received multiple chunks")
.hasSizeGreaterThan(1);
String content = String.join("", chunks);
AiResponseKeywordsAligned result = assertDoesNotThrow(() ->
converter.convert(content),
"Response should be convertible to AiResponseKeywordsAligned");
log.info("Result: {}", result);
assertThat(result)
.as("Check that the entity is not null")
.isNotNull();
assertThat(result.keywordsWithThemeAndConfidence())
.as("Check if AI returned any keywords with confidence")
.isNotNull()
.isNotEmpty();
}I did also try to get a test to pass that would return a Flux of ChatResponse, but I could never get it to work. This should probably be a new post, but I'll put it here in case anyone wants to see it. It just keeps failing because it says the content is null. @Test
void givenLongResponseTime_whenProcessingWithStreamingChatClient_thenReturnChatResponse() {
ChatClient chatClient = ChatClient.create(chatModel);
var converter = new BeanOutputConverter<>(new ParameterizedTypeReference<AiResponseKeywordsAligned>() {
});
log.info("Converter: {}", converter.getFormat());
String systemPrompt = "You are a keyword research assistant specialized in SEO and Google Keyword Planner, helping users discover high-quality, long-tail keywords for niche businesses. {format}";
String userMessage = """
## How to rate industry alignment
Using the following scale, rate how closely each keyword aligns with the industry. The specific values represent key reference points or milestones, but your rating should lie on the continuous scale between 0 and 100 in proximity to those milestones.
100 = Perfect alignment: The keyword is highly relevant to the food and beverage, cafe culture, coffee shops industry.
50 = Partial alignment: The keyword is somewhat related or adjacent to this industry.
0 = Misalignment: The keyword is not related to this industry at all. In addition, ensure that keywords related to are give a low rating.
## List of keywords to analyze
150 ml to ounces, 150ml to ounces, 20 oz in pounds, 5 ounces to g, 5 ounces to gr, 5 ounces to grams, 6'2 in cm, 80 ml to ounces, a gallon of water is how many water bottles, bahama mama recipe, benefits of central business district, best afternoon tea london, best cafe near me, best espresso martini recipe, best local coffee shops, blue in spanish, cafe in meiner nähe, cafes to study, caffeine in diet coke, caffeine in espresso shot, chai tea benefits, chocolate cafe, chocolate house, chocolate rain, chocolate rain song, chocolate world, cinnamon tea benefits, coffee loophole recipe, coffee shops with wifi near me, cranberry in spanish, daiquiri recipe, dark chocolate benefits, definition for wholesome, est to ist, exit to eden, factory tea bar, french martini recipe, fruits start with i, gentleman in moscow book, gin and tonic recipe, ginger tea benefits, green bean casserole with fresh green beans, health food store, hershey chocolate world, hershey's chocolate world, hibiscus tea benefits, homemade hot chocolate, horchata recipe, hot chocolate near me, hot chocolate recipe, hot chocolate run, how many bottle of waters is a gallon, how many bottle waters are in a gallon, how many bottled waters are in a gallon, how many grams of sugar in a tablespoon of sugar, how many grams of sugar in a teaspoon, how many grams of sugar in a teaspoon of sugar, how many ounces of water to drink a day, how many water bottle are in a gallon, how many water bottles are in a gallon, how many water bottles are in a gallon of water, how many water bottles is a gallon of water, how many water bottles is in a gallon, how much caffeine in matcha, how much sugar can of coke, how much sugar i n can of coke, how much sugar in a can of coca cola, how much sugar in a can of coke, how much sugar in can of coke, how much sugar in can of soda, how much sugar is in a can of coke, how much sugar is in a can of pop, how much sugar is in a soda can, how much sugar is in coke can, how much sugars in a can of coke, how to descale a keurig, how to make hot chocolate, how to make hummingbird nectar, hugo spritz recipe, ingredients in coke, is carbonated water bad for you, is coconut water good for you, is coffee good for you, is coke zero bad for you, is dark chocolate good for you, is diet coke bad for you, is fluoride bad for you, is matcha good for you, is oat milk good for you, is sparkling water good for you, is vitamin water good for you, juice bar, kakawa chocolate house, lemon drop recipe, loco moco recipe, long island iced tea recipe, lychee martini recipe, lyrics to escape the pina colada song, mai tai recipe, mango smoothie recipe, martini recipe, matcha benefits, mg of caffeine in coffee, movita juice bar, nekter juice bar, oolong tea benefits, oz in grams, peppermint tea benefits, pina colada recipe, pomegranate benefits, raspberry leaf tea benefits, raw milk benefits, tea bar, time for tokyo, time in shan, toothsome chocolate emporium menu, virgin pina colada recipe, water in spanish, what's in spanish, whiskey drink recipes, whole foods coffee bar
## Output format to return
Set of keywordsWithThemeAndConfidence objects. Each object must include:
- The keyword
- A confidence rating (0–100) indicating how well it aligns with the specified industry.""";
Flux<ChatResponse> response = chatClient.prompt()
.system(sp -> sp.text(systemPrompt).param("format", converter.getFormat()))
.user(userMessage)
.stream()
.chatResponse();
List<ProcessedResponse> processedResponses = response
.<ProcessedResponse>handle((chatResponse, sink) -> {
ChatResponseMetadata metadata = chatResponse.getMetadata();
String content = chatResponse.getResult().getOutput().getContent();
// Validate content before deserialization
if (content == null || content.isBlank()) {
log.error("Received empty or null content from ChatResponse: {}", chatResponse);
sink.error(new RuntimeException("Empty content received from ChatResponse"));
return;
}
AiResponseKeywordsAligned entity;
try {
// Use the converter to map the response to the entity
entity = converter.convert(content);
} catch (Exception e) {
log.error("Failed to convert response content to AiResponseKeywordsAligned", e);
sink.error(new RuntimeException("Error deserializing ChatResponse content", e));
return;
}
sink.next(new ProcessedResponse(metadata, entity));
})
.collectList()
.block();
assertThat(processedResponses)
.isNotEmpty()
.allSatisfy(processedResponse -> {
assertThat(processedResponse.entity())
.isNotNull()
.extracting(AiResponseKeywordsAligned::keywordsWithThemeAndConfidence)
.isNotNull();
assertThat(processedResponse.metadata())
.isNotNull();
});
log.info("Processed Responses: {}", processedResponses);
} |
Beta Was this translation helpful? Give feedback.
Answer selected by
LiveNathan
-
|
Great! |
Beta Was this translation helpful? Give feedback.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
It worked!
Here's my final passing test: