Enhance terms lookup query to take a query clause #18195

srikanthpadakanti · 2025-05-05T01:42:30Z

Description

Enhances the Terms lookup query to accept a query clause instead of only supporting a single docId. This expands functionality by enabling users to define document matches through flexible query logic rather than relying solely on fixed document IDs.

This change includes:

Support for Terms lookup query to use a query clause.
Validation logic to prevent using both id and query simultaneously.
Associated unit test updates to cover query-based lookup behavior.

Related Issues

Resolves #17599

Check List

[ X] Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-05-05T01:49:53Z

❌ Gradle check result for 085edf2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

server/src/main/java/org/opensearch/indices/TermsLookup.java

server/src/main/java/org/opensearch/index/query/TermsQueryBuilder.java

server/src/main/java/org/opensearch/indices/TermsLookup.java

github-actions · 2025-05-09T20:11:07Z

❌ Gradle check result for edfa605: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-14T17:27:39Z

❌ Gradle check result for f7146ff: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-14T22:27:29Z

❌ Gradle check result for 90f90c7: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-14T23:53:33Z

❌ Gradle check result for 997ddbe: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-15T01:03:56Z

❌ Gradle check result for 9c85868: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-05-15T18:23:35Z

❌ Gradle check result for 6f52602: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…use rather than docId Signed-off-by: Srikanth Padakanti <[email protected]>

Signed-off-by: Srikanth Padakanti <[email protected]>

…checks and updated unit tests Signed-off-by: Srikanth Padakanti <[email protected]>

Signed-off-by: Srikanth Padakanti <[email protected]>

github-actions · 2025-05-15T19:29:41Z

❌ Gradle check result for 6ab5410: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Srikanth Padakanti <[email protected]>

github-actions · 2025-05-15T20:04:46Z

❌ Gradle check result for 101c8ad: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti · 2025-05-19T19:54:36Z

@kotwanikunal @Bukhtawar @andrross @sandeshkr419
Can someone please review my PR.

Signed-off-by: Srikanth Padakanti <[email protected]>

github-actions · 2025-05-19T21:17:49Z

✅ Gradle check result for 29c4d55: SUCCESS

github-actions · 2025-05-20T15:43:10Z

❌ Gradle check result for ab7abdb: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

kotwanikunal · 2025-05-20T19:51:51Z

server/src/main/java/org/opensearch/indices/TermsLookup.java

-
-    public String id() {
-        return id;
+        out.writeOptionalWriteable(query);


You need to add in a version check here as well

Suggested change

out.writeOptionalWriteable(query);

if (out.getVersion().onOrAfter(Version.V_3_0_0)) {

out.writeOptionalWriteable(query);

}

kotwanikunal · 2025-05-20T19:53:46Z

server/src/main/java/org/opensearch/indices/TermsLookup.java

@@ -90,6 +114,11 @@ public TermsLookup(StreamInput in) throws IOException {
        if (in.getVersion().onOrAfter(Version.V_2_17_0)) {
            store = in.readBoolean();
        }
+        if (in.getVersion().onOrAfter(Version.V_3_0_0)) {
+            if (in.readBoolean()) {


I don't think you should be doing the in.readBoolean() explicitly.
ReadOptionalWriteable should take care of it for you.

Reference:

public <T extends Writeable> T readOptionalWriteable(Writeable.Reader<T> reader) throws IOException { if (readBoolean()) { T t = reader.read(this); if (t == null) { throw new IOException( "Writeable.Reader [" + reader + "] returned null which is not allowed and probably means it screwed up the stream." ); } return t; } else { return null; } }

kotwanikunal · 2025-05-20T20:01:31Z

server/src/main/java/org/opensearch/index/query/TermsQueryBuilder.java

+            SearchResponse response;
+            try {
+                response = shardContext.getClient()
+                    .search(


I am wondering if there is a better location for fetching rather than rewrites. If I understand correctly, rewrite can be called multiple times through the query flow. We would have wasted search calls.

The actionGet() is also blocking the search thread on this query for a network call - which is not ideal.

Can we initialize this value at the start of the query creation and keep using the same value as a class var?
Also, can you please extract the query specific mechanism into a separate method for easy readability?

kotwanikunal · 2025-05-20T20:02:08Z

server/src/main/java/org/opensearch/index/query/TermsQueryBuilder.java

+        if (termsLookup != null && termsLookup.query() != null) {
+            QueryBuilder rewrittenQuery = termsLookup.query().rewrite(context);
+
+            SearchResponse response = context.getClient()


Can we store this value and reuse it for rewrite phases?
Also can we avoid blocking calls on the search thread?

Yeah, this will need to be asynchronous, probably similar to how the current terms lookup query is implemented:

OpenSearch/server/src/main/java/org/opensearch/index/query/TermsQueryBuilder.java

Lines 609 to 612 in 52b8b05

queryRewriteContext.registerAsyncAction((client, listener) -> fetch(termsLookup, client, ActionListener.map(listener, list -> {

supplier.set(list);

return null;

})));

.github/workflows/wrapper.yml

andrross · 2025-05-20T20:33:37Z

.idea/runConfigurations/Debug_OpenSearch.xml

@@ -6,6 +6,10 @@
    <option name="HOST" value="localhost" />
    <option name="PORT" value="5005" />
    <option name="AUTO_RESTART" value="true" />
+    <RunnerSettings RunnerId="Debug">


Please revert this change unless this is intentional.

andrross · 2025-05-20T20:38:27Z

server/src/main/java/org/opensearch/index/query/TermsQueryBuilder.java

+        if (termsLookup != null && termsLookup.query() != null) {
+            QueryBuilder rewrittenQuery = termsLookup.query().rewrite(context);
+
+            SearchResponse response = context.getClient()


Yeah, this will need to be asynchronous, probably similar to how the current terms lookup query is implemented:

OpenSearch/server/src/main/java/org/opensearch/index/query/TermsQueryBuilder.java

Lines 609 to 612 in 52b8b05

queryRewriteContext.registerAsyncAction((client, listener) -> fetch(termsLookup, client, ActionListener.map(listener, list -> {

supplier.set(list);

return null;

})));

opensearch-trigger-bot · 2025-06-20T15:22:45Z

This PR is stalled because it has been open for 30 days with no activity.

vamshin · 2025-06-26T21:32:23Z

@srikanthpadakanti are you planning to continue working on this? We are targeting this feature for 3.2 release

kotwanikunal reviewed May 8, 2025

View reviewed changes

github-actions bot added enhancement Enhancement or improvement to existing feature or request Search:Query Capabilities labels May 15, 2025

srikanthpadakanti force-pushed the FeatureRequest17599 branch from da0ba03 to 964f452 Compare May 15, 2025 18:53

srikanthpadakanti added 14 commits May 15, 2025 13:56

FeatureRequest17599: Enhancing Terms lookup query to take a query cla…

c3533a7

…use rather than docId Signed-off-by: Srikanth Padakanti <[email protected]>

Checkpoint 2 fixed the index issue when query is null

5d07141

Signed-off-by: Srikanth Padakanti <[email protected]>

Checkpoint 3 fixed the optional argument id, query handling

25a2bf8

Signed-off-by: Srikanth Padakanti <[email protected]>

Debugging the parsing issue

da93337

Signed-off-by: Srikanth Padakanti <[email protected]>

Address the PR comments

cbafd35

Signed-off-by: Srikanth Padakanti <[email protected]>

Both id and query is working checkpoint

6cc6ab0

Signed-off-by: Srikanth Padakanti <[email protected]>

Both id and query is working checkpoint

1cd1dd5

Signed-off-by: Srikanth Padakanti <[email protected]>

Both id and query is working and basic unittests

a95a3b5

Signed-off-by: Srikanth Padakanti <[email protected]>

Fix: Ensure TermsLookup supports both id and query; added validation …

ba6785a

…checks and updated unit tests Signed-off-by: Srikanth Padakanti <[email protected]>

Fix: Ensure TermsLookup supports both id and query; added validation …

f560563

…checks and updated unit tests Signed-off-by: Srikanth Padakanti <[email protected]>

Fix: Ensure TermsLookup supports both id and query; added validation …

741290d

…checks and updated unit tests Signed-off-by: Srikanth Padakanti <[email protected]>

Fix: Ensure TermsLookup supports both id and query; added validation …

f4af35e

…checks and updated unit tests Signed-off-by: Srikanth Padakanti <[email protected]>

Add CHANGELOG entry for feature request opensearch-project#17599

8e5330f

Signed-off-by: Srikanth Padakanti <[email protected]>

Add CHANGELOG entry for feature request opensearch-project#17599

6ab5410

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti force-pushed the FeatureRequest17599 branch from 964f452 to 6ab5410 Compare May 15, 2025 18:56

Add retry logic to gradle wrapper validation

101c8ad

Signed-off-by: Srikanth Padakanti <[email protected]>

opensearch-ci-bot mentioned this pull request May 16, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ClientYamlTestSuiteIT #14319

Open

srikanthpadakanti requested review from ashking94, Bukhtawar, CEHENKLE, cwperks, dbwiddis, gbbafna, mch2, msfroh, owaiskazi19, reta, Rishikesh1159, sachinpkale, saratvemulapalli, shwetathareja, sohami and VachaShah as code owners May 19, 2025 19:16

Merge branch 'main' into FeatureRequest17599

2ddb9c3

Signed-off-by: Srikanth Padakanti <[email protected]>

srikanthpadakanti added 3 commits May 19, 2025 14:55

Merge branch 'opensearch-project:main' into FeatureRequest17599

f84fcb6

Add the test to cover the missing part as per codecoverage report

33e47bf

Signed-off-by: Srikanth Padakanti <[email protected]>

spotlessApply

29c4d55

Signed-off-by: Srikanth Padakanti <[email protected]>

Merge branch 'opensearch-project:main' into FeatureRequest17599

ab7abdb

kotwanikunal reviewed May 20, 2025

View reviewed changes

andrross reviewed May 20, 2025

View reviewed changes

andrross changed the title ~~Feature request17599~~ Enhance terms lookup query to take a query clause May 20, 2025

opensearch-trigger-bot bot added the stalled Issues that have stalled label Jun 20, 2025

	queryRewriteContext.registerAsyncAction((client, listener) -> fetch(termsLookup, client, ActionListener.map(listener, list -> {
	supplier.set(list);
	return null;
	})));

Enhance terms lookup query to take a query clause #18195

Are you sure you want to change the base?

Enhance terms lookup query to take a query clause #18195

Conversation

srikanthpadakanti commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented May 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented May 9, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 14, 2025

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

github-actions bot commented May 15, 2025

Uh oh!

srikanthpadakanti commented May 19, 2025

Uh oh!

github-actions bot commented May 19, 2025

Uh oh!

github-actions bot commented May 20, 2025

Uh oh!

kotwanikunal May 20, 2025

Choose a reason for hiding this comment

Uh oh!

kotwanikunal May 20, 2025

Choose a reason for hiding this comment

Uh oh!

kotwanikunal May 20, 2025

Choose a reason for hiding this comment

Uh oh!

kotwanikunal May 20, 2025

Choose a reason for hiding this comment

Uh oh!

andrross May 20, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

andrross May 20, 2025

Choose a reason for hiding this comment

Uh oh!

andrross May 20, 2025

Choose a reason for hiding this comment

Uh oh!

opensearch-trigger-bot bot commented Jun 20, 2025

Uh oh!

vamshin commented Jun 26, 2025

Uh oh!

Uh oh!

srikanthpadakanti commented May 5, 2025 •

edited

Loading