Skip to content

TokenTextSplitter fails with "Failed to deserialize the error entity" #2767

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
khauser opened this issue Apr 16, 2025 · 0 comments
Open

TokenTextSplitter fails with "Failed to deserialize the error entity" #2767

khauser opened this issue Apr 16, 2025 · 0 comments
Labels
azure ETL

Comments

@khauser
Copy link

khauser commented Apr 16, 2025

Bug description
The chunking process of some "test execution"-files fails always at the same point. The file it's seems to be is about 8MB big testrunner.log.

Is this a problem of the Azure Search API or of spring-ai?

I tried several variations of TokenTextSplitter parameters.

2025-04-16T16:56:37.824+02:00�[0;39m �[32m INFO�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mc.i.i.a.c.AnalyzeTestExecutionController�[0;39m �[2m:�[0;39m {
  "id" : "d05a595b-1d1f-4a6e-ad5a-b193891e127c",
  "text" : "The testrunner is waiting for the AS: http://iste-run-tp5601-ts2450244-icm-as:7743/status/ReadinessProbe\n/intershop/bin/testrunner.sh: 15: waitforurl: not found\nAS was started: http://iste-run-tp5601-ts2450244-icm-as:7743/status/ReadinessProbe\n\nCommand Line:\njava -showversion -XX:+UseG1GC -Xms768m -Xmx1536m -Detest.home=/intershop/cartridges/etest/release -Dserver.name=testrunner -Djava.system.class.loader=com.intershop.beehive.runtime.IntershopClassLoader -Dintershop.ServerConfig=/intershop/serverdir.properties -javaagent:/intershop/cartridges/runtime/release/lib/ish-runtime.jar -Djdk.tls.client.protocols=TLSv1.2 -cp /intershop/cartridges/runtime/release/lib/ish-runtime.jar:/intershop/system-conf/cluster:/intershop/testrunner/lib/* -Dis.cartridgesClasspathLayout=release -Dis.cartridgeRepositories=/intershop/cartridges:/intershop/customizations -Dis.libraryRepositories=/intershop/lib:/intershop/customizations -Dis.cartridges=\"app_bo_test\" com.intershop.testrunner.IshTestrunner -s=tests.remote.com.intershop.smc.suite.SMC00TestSuite -o=/data/testplans/ff3c226c-31a4-46d3-b867-d2d2701397d7/testsuites/2450244/workspace/helm/icm-2.13.9/result\n\n[0.025s][warning][cds] Archived non-system classes are disabled because the java.system.class.loader property is specified (value = \"com.intershop.beehive.runtime.IntershopClassLoader\"). To use archived non-system classes, this property must not be set\nCan't find location '/intershop/customizations' for 'CARTRIDGES'\nCan't find location '/intershop/customizations' for 'LIBRARIES'\n243 library versions from library list files\nINFO: Agent Main of com.intershop.beehive.runtime.internal.agent.Agent first time executed\nopenjdk version \"21.0.6\" 2025-01-21 LTS\nOpenJDK Runtime Environment Temurin-21.0.6+7 (build 21.0.6+7-LTS)\nOpenJDK 64-Bit Server VM Temurin-21.0.6+7 (build 21.0.6+7-LTS, mixed mode, sharing)\n[2025-04-14 22:31:06.483 UTC] [main] INFO  c.intershop.testrunner.IshTestrunner - Testrunner output direcotry :/data/testplans/ff3c226c-31a4-46d3-b867-d2d2701397d7/testsuites/2450244/workspace/helm/icm-2.13.9/result was created :false\n[2025-04-14 22:31:06.574 UTC] [main] INFO  c.intershop.testrunner.SuiteHandler - Following tests were found in Suite: [tests.remote.com.intershop.smc.operation.servergroup.general.ServerGroupTest, tests.remote.com.intershop.smc.operation.site.general.SiteManagementTest, tests.remote.com.intershop.smc.smc.application.general.ApplicationTest, tests.remote.com.intershop.smc.smc.file.browsing.FileBrowserTest, tests.remote.com.intershop.smc.smc.file.download.FileDownloadTest, tests.remote.com.intershop.smc.smc.file.listing.FileListTest, tests.remote.com.intershop.smc.smc.installation.cluster.ClusterSnapshotTest, tests.remote.com.intershop.smc.smc.installation.files.InstallationFileTest, tests.remote.com.intershop.smc.smc.installation.heapdump.HeapdumpTest, tests.remote.com.intershop.smc.smc.installation.threaddump.ThreaddumpTest, tests.remote.com.intershop.smc.smc.orm.caching.ORMCacheTest, tests.remote.com.intershop.smc.smc.process.general.ProcessOverviewTest, tests.remote.com.intershop.smc.smc.process.locking.LockingTest, tests.remote.com.intershop.smc.smc.process.search.ProcessSearchTest, tests.remote.com.intershop.smc.smc.server.configuration.ApplicationServerConfigurationTest, tests.remote.com.intershop.smc.smc.server.logging.LoggingTest, tests.remote.com.intershop.smc.operation.servergroup.general.ServerGroupTest, tests.remote.com.intershop.smc.smc.serviceconfiguration.general.ServiceConfigurationTest, tests.remote.com.intershop.smc.smc.serviceconfiguration.listing.ServiceConfigurationListTest, tests.remote.com.intershop.smc.operation.site.general.SiteManagementTest, tests.remote.com.intershop.smc.smc.thread.listing.",
  "media" : null,
  "metadata" : {
    "fileName" : "istestrunner.log",
    "testPathId" : null,
    "filePath" : "workspace/helm/icm-2.13.9/result/istestrunner.log",
    "testSuiteId" : 2450244
  },
  "score" : null
}
�[2m2025-04-16T17:05:13.306+02:00�[0;39m �[33m WARN�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mc.a.c.i.s.HttpResponseBodyDecoder       �[0;39m �[2m:�[0;39m Failed to deserialize the error entity.
Unsupported text Content-Type Type: class com.azure.search.documents.implementation.models.ErrorResponse
�[2m2025-04-16T17:05:13.307+02:00�[0;39m �[31mERROR�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mc.azure.search.documents.SearchClient   �[0;39m �[2m:�[0;39m Status code 413, "The page was not displayed because the request entity is too large."
�[2m2025-04-16T17:05:13.321+02:00�[0;39m �[32mDEBUG�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mo.s.web.servlet.DispatcherServlet       �[0;39m �[2m:�[0;39m Failed to complete request: com.azure.core.exception.HttpResponseException: Status code 413, "The page was not displayed because the request entity is too large."
�[2m2025-04-16T17:05:13.327+02:00�[0;39m �[31mERROR�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mo.a.c.c.C.[.[.[.[dispatcherServlet]     �[0;39m �[2m:�[0;39m Servlet.service() for servlet [dispatcherServlet] in context with path [/vaadin-app] threw exception [Request processing failed: com.azure.core.exception.HttpResponseException: Status code 413, "The page was not displayed because the request entity is too large."] with root cause

java.lang.IllegalStateException: Unsupported text Content-Type Type: class com.azure.search.documents.implementation.models.ErrorResponse
	at com.azure.core.util.serializer.JacksonAdapter.deserializeText(JacksonAdapter.java:452) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.util.serializer.JacksonAdapter.lambda$deserialize$9(JacksonAdapter.java:353) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.util.serializer.JacksonAdapter.useAccessHelper(JacksonAdapter.java:488) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.util.serializer.JacksonAdapter.deserialize(JacksonAdapter.java:346) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.deserialize(HttpResponseBodyDecoder.java:176) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.deserializeBody(HttpResponseBodyDecoder.java:150) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseBodyDecoder.decodeByteArray(HttpResponseBodyDecoder.java:67) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.serializer.HttpResponseDecoder$HttpDecodedResponse.getDecodedBody(HttpResponseDecoder.java:93) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.SyncRestProxy.ensureExpectedStatus(SyncRestProxy.java:131) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.SyncRestProxy.handleRestReturnType(SyncRestProxy.java:211) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.SyncRestProxy.invoke(SyncRestProxy.java:86) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.implementation.http.rest.RestProxyBase.invoke(RestProxyBase.java:124) ~[azure-core-1.55.3.jar:1.55.3]
	at com.azure.core.http.rest.RestProxy.invoke(RestProxy.java:95) ~[azure-core-1.55.3.jar:1.55.3]
	at jdk.proxy2/jdk.proxy2.$Proxy146.indexSync(Unknown Source) ~[na:na]
	at com.azure.search.documents.implementation.DocumentsImpl.indexWithResponse(DocumentsImpl.java:778) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.implementation.util.Utility.lambda$indexDocumentsWithResponse$1(Utility.java:160) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.implementation.util.Utility.executeRestCallWithExceptionHandling(Utility.java:187) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.implementation.util.Utility.indexDocumentsWithResponse(Utility.java:158) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.SearchClient.indexDocumentsWithResponse(SearchClient.java:769) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.SearchClient.uploadDocumentsWithResponse(SearchClient.java:422) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at com.azure.search.documents.SearchClient.uploadDocuments(SearchClient.java:382) ~[azure-search-documents-11.7.6.jar:11.7.6]
	at org.springframework.ai.vectorstore.azure.AzureVectorStore.doAdd(AzureVectorStore.java:182) ~[spring-ai-azure-store-1.0.0-M7.jar:1.0.0-M7]
	at org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore.lambda$add$1(AbstractObservationVectorStore.java:85) ~[spring-ai-vector-store-1.0.0-M7.jar:1.0.0-M7]
	at io.micrometer.observation.Observation.observe(Observation.java:498) ~[micrometer-observation-1.14.5.jar:1.14.5]
	at org.springframework.ai.vectorstore.observation.AbstractObservationVectorStore.add(AbstractObservationVectorStore.java:85) ~[spring-ai-vector-store-1.0.0-M7.jar:1.0.0-M7]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:125) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.processFileEntries(AnalyzeTestExecutionController.java:94) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.loadVectorData(AnalyzeTestExecutionController.java:68) ~[main/:na]
	at com.intershop.iste.ai.controller.AnalyzeTestExecutionController.loadTestPlan(AnalyzeTestExecutionController.java:78) ~[main/:na]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:258) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:191) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:986) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:891) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564) ~[tomcat-embed-core-10.1.39.jar:6.0]
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885) ~[spring-webmvc-6.2.5.jar:6.2.5]
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658) ~[tomcat-embed-core-10.1.39.jar:6.0]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:195) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) ~[tomcat-embed-websocket-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.CorsFilter.doFilterInternal(CorsFilter.java:91) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.ServerHttpObservationFilter.doFilterInternal(ServerHttpObservationFilter.java:114) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-6.2.5.jar:6.2.5]
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116) ~[spring-web-6.2.5.jar:6.2.5]
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:164) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:140) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:483) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:397) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:905) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) ~[tomcat-embed-core-10.1.39.jar:10.1.39]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]

�[2m2025-04-16T17:05:13.346+02:00�[0;39m �[32mDEBUG�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mo.s.web.servlet.DispatcherServlet       �[0;39m �[2m:�[0;39m "ERROR" dispatch for GET "/vaadin-app/error?testPlanId=5601", parameters={masked}
�[2m2025-04-16T17:05:13.371+02:00�[0;39m �[32mDEBUG�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mo.s.w.s.v.ContentNegotiatingViewResolver�[0;39m �[2m:�[0;39m Selected 'text/html' given [text/html, text/html;q=0.8]
�[2m2025-04-16T17:05:13.381+02:00�[0;39m �[32mDEBUG�[0;39m �[35m37196�[0;39m �[2m--- [ai] [nio-6060-exec-1] �[0;39m�[36mo.s.web.servlet.DispatcherServlet       �[0;39m �[2m:�[0;39m Exiting from "ERROR" dispatch, status 500

Environment
Spring AI version: 1.0.0-M7
Java: Java(TM) SE Runtime Environment Oracle GraalVM 21.0.2+13.1 (build 21.0.2+13-LTS-jvmci-23.1-b30)
Please provide as many details as possible: Spring AI version, Java version, which vector store you use if any, etc

Steps to reproduce
build.gradle

    implementation group: 'com.azure', name: 'azure-search-documents', version: "11.7.6"
    implementation(group: 'org.springframework.ai', name: 'spring-ai-starter-model-azure-openai') {
        exclude group: 'com.azure', module: 'azure-search-documents'
    }
    implementation group: 'org.springframework.ai', name: 'spring-ai-autoconfigure-model-azure-openai'

    implementation group: 'org.springframework.ai', name: 'spring-ai-starter-vector-store-azure'
    implementation group: 'org.springframework.ai', name: 'spring-ai-autoconfigure-vector-store-azure'

    implementation group: 'org.springframework.ai', name: 'spring-ai-rag'
    implementation group: 'org.springframework.ai', name: 'spring-ai-azure-store'

    implementation group: 'com.azure', name: 'azure-core-http-okhttp', version: "1.12.10"

Controller.java

                try (InputStream inputStream = isteBIService.getLogFile(testSuiteId, entry.getPath());
                                ByteArrayOutputStream outputStream = new ByteArrayOutputStream())
                {

                    inputStream.transferTo(outputStream);

                    String content = outputStream.toString(StandardCharsets.UTF_8);

                    Document doc = Document.builder()
                                           .id("suite-" + testSuiteId + "/" + entry.getName())
                                           .text(content)
                                           .build();

                    //var tokenTextSplitter = TokenTextSplitter.builder().withChunkSize(800).build();
                    var tokenTextSplitter = new TokenTextSplitter(1024, 350, 5, 20000, true);
                    //var tokenTextSplitter = new TokenTextSplitter(512, 250, 5, 10000, true);
                    List<Document> splitDocuments = tokenTextSplitter.apply(List.of(doc));
                    for (Document splitDocument : splitDocuments)
                    {
                        splitDocument.getMetadata().put("fileName", entry.getName());
                        splitDocument.getMetadata().put("filePath", entry.getPath());
                        splitDocument.getMetadata().put("testSuiteId", testSuiteId);
                        splitDocument.getMetadata().put("testPathId", null);
                    }
                    log.info(new ObjectMapper().writerWithDefaultPrettyPrinter()
                                               .writeValueAsString(splitDocuments.get(0)));
                    store.add(splitDocuments);
                }
                catch(IOException e)
                {
                    throw new UncheckedIOException("Fehler beim Lesen des Log-Inhalts: " + entry.getPath(), e);
                }

Expected behavior
A clear and concise description of what you expected to happen.

Minimal Complete Reproducible example
Please provide a failing test or a minimal complete verifiable example that reproduces the issue.
Bug reports that are reproducible will take priority in resolution over reports that are not reproducible.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
azure ETL
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@markpollack @khauser