[GRPC] SearchService and Search GRPC endpoint v1 #17830

karenyrx · 2025-04-08T04:12:21Z

Description

Scaffold initial search service and search GRPC endpoint. Supports MatchAll, TermQuery, and MatchNone. TODOs are left in the code for supporting aggregations, suggest, ext, other more involved query types, etc.
Bumps the opensearch-protobuf jar verson to 0.2.0 to add in more proto fixes for the Search endpoint.

Test Plan

Unit tests
Manual party testing with HTTP (ongoing). e.g.
a) Send a MatchAll query with "size" parameter in both the URL parameters as well as request body.


grpcurl -import-path ~/OpenSearch -d @ -proto ~/OpenSearch/search_service.proto -plaintext localhost:9400 org.opensearch.protobufs.services.SearchService/Search <<EOM
{
  "size": "3",
  "source": {
    "bool_value": true
  },
  "request_body": {
    "query": {
      "match_all": {}
    },
    "size": 40
  }
}
EOM

{
  "responseBody": {
    "took": "6",
    "timedOut": false,
    "shards": {
      "successful": 3,
      "total": 3
    },
    "hits": {
      "total": {
        "totalHits": {
          "relation": "TOTAL_HITS_RELATION_EQ",
          "value": "1"
        }
      },
      "hits": [
        {
          "index": "my_index",
          "id": "2",
          "score": {
            "floatValue": 1
          },
          "source": "eyJkb2MiOnsieWVhciI6MjAxMywidGl0bGUiOiJSdXNoIn0sInRpbWVzdGFtcCI6IjIwMjUtMDQtMDhUMTI6MjQ6MzAuOTEyMjY5ODI5WiJ9",
          "metaFields": {
            "fields": {
              "_routing": {
                "listValue": {
                  "value": [
                    {
                      "string": "0"
                    }
                  ]
                }
              }
            }
          }
        }
      ],
      "maxScore": {
        "floatValue": 1
      }
    }
  }
}

curl -XGET "http://localhost:9200/_search?size=1&_source=true" -H 'Content-Type: application/json' -d '               
{
 "size": "40",
 "query": {
   "match_all": {}
 }
} ' | jq

{
  "took": 5,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my_index",
        "_id": "2",
        "_score": 1,
        "_routing": "0",
        "_source": {
          "doc": {
            "year": 2013,
            "title": "Rush"
          },
          "timestamp": "2025-04-08T12:24:30.912269829Z"
        }
      }
    ]
  }
}

SearchRequest (same for both GRPC and HTTP)

SearchRequest{searchType=QUERY_THEN_FETCH, indices=[], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"size":1,"query":{"match_all":{"boost":1.0}},"_source":{"includes":[],"excludes":[]}}, cancelAfterTimeInterval=null, pipeline=null, phaseTook=null}

b) TermQuery

grpcurl -import-path ~/OpenSearch -d @ -proto ~/OpenSearch/search_service.proto -plaintext localhost:9400 org.opensearch.protobufs.services.SearchService/Search <<EOM
{
 "request_body": [
    {
      "query": {
        "term": {
          "title": {
            "value": {
              "string_value": "Rush"
            },
	     "case_insensitive": true
          }
        }
      }
    }
  ],
  "index": "my_index"
}
EOM


{
  "responseBody": {
    "took": "64",
    "timedOut": false,
    "shards": {
      "successful": 1,
      "total": 1
    },
    "hits": {
      "total": {
        "totalHits": {
          "relation": "TOTAL_HITS_RELATION_EQ",
          "value": "1"
        }
      },
      "hits": [
        {
          "index": "my_index",
          "id": "3",
          "score": {
            "floatValue": 1
          },
          "source": "eyAidGl0bGUiOiAiUnVzaCIsICJ5ZWFyIjogMjAxM30=",
          "metaFields": {}
        }
      ],
      "maxScore": {
        "floatValue": 1
      }
    }
  }
}

curl -XGET "http://localhost:9200/my_index/_search" -H 'Content-Type: application/json' -d'
{
  "query": {
    "term": {
      "title": {
        "value": "Rush",
         "case_insensitive": true
      }
    }
  }
}' | jq

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 3,
    "successful": 3,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "my_index",
        "_id": "3",
        "_score": 1,
        "_source": {
          "title": "Rush",
          "year": 2013
        }
      }
    ]
  }
}

SearchRequest (same for both GRPC and HTTP)

SearchRequest{searchType=QUERY_THEN_FETCH, indices=[my_index], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"query":{"term":{"title":{"value":"Rush","case_insensitive":true,"boost":1.0}}}}, cancelAfterTimeInterval=null, pipeline=null, phaseTook=null}

c) MatchNone with source_includes field (compare POJOs not the response)


grpcurl -import-path ~/OpenSearch -d @ -proto ~/OpenSearch/search_service.proto -plaintext localhost:9400 org.opensearch.protobufs.services.SearchService/Search <<EOM
{
  "source": {
    "bool_value": true
  },
  "source_includes": [
    "title"
  ],
  "request_body": {
    "query": {
      "match_none": {}
    }
  }
}
EOM

{
  "responseBody": {
    "took": "5",
    "timedOut": false,
    "shards": {
      "successful": 1,
      "total": 1
    },
    "hits": {
      "total": {
        "totalHits": {
          "relation": "TOTAL_HITS_RELATION_EQ"
        }
      },
      "maxScore": {
        "nullValue": "NULL_VALUE_NULL"
      }
    }
  }
}

curl -XGET "http://localhost:9200/_search?_source=true&_source_includes=title" -H 'Content-Type: application/json' -d '
{
 "query": {
   "match_none": {}
 }
} ' | jq

{
  "took": 8,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 0,
      "relation": "eq"
    },
    "max_score": null,
    "hits": []
  }
}

SearchRequest (same for both GRPC and HTTP)

SearchRequest{searchType=QUERY_THEN_FETCH, indices=[], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, expand_wildcards_hidden=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=null, allowPartialSearchResults=null, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"query":{"match_none":{"boost":1.0}},"_source":{"includes":["title"],"excludes":[]}}, cancelAfterTimeInterval=null, pipeline=null, phaseTook=null}

The documents ingested 3 of the above tests were:

curl -vv -XPOST "http://localhost:9200/_bulk" -H 'Content-Type: application/json' -d'
{"index": {"_index": "my_index", "_id": "1"}}
{"title":"OpenSearch Basics", "price": "15"}


{"index": {"_index": "my_index", "_id": "2"}}
{"title":"OpenSearch Advanced", "author": "Cordelia M", "price": "28"}


{"index": {"_index": "my_index", "_id": "3"}}
{ "title": "Rush", "year": 2013}

'

Related Issues

#16783

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-04-08T04:27:14Z

❌ Gradle check result for 11b9615: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T11:48:03Z

❌ Gradle check result for ddee102: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T12:29:28Z

❌ Gradle check result for d950579: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T16:57:20Z

❌ Gradle check result for 9d23b96: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T18:21:24Z

❕ Gradle check result for 5ee1000: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-04-08T18:22:39Z

Codecov Report

Attention: Patch coverage is 80.96970% with 157 lines in your changes missing coverage. Please review.

Project coverage is 72.36%. Comparing base (7b6108b) to head (1a98708).
Report is 10 commits behind head on main.

Files with missing lines	Patch %	Lines
.../proto/request/search/SearchRequestProtoUtils.java	66.22%	35 Missing and 16 partials ⚠️
.../request/search/SearchSourceBuilderProtoUtils.java	81.63%	10 Missing and 8 partials ⚠️
...rpc/proto/response/search/SearchHitProtoUtils.java	80.55%	5 Missing and 9 partials ⚠️
...ponse/search/SearchResponseSectionsProtoUtils.java	18.18%	4 Missing and 5 partials ⚠️
...oto/request/search/InnerHitsBuilderProtoUtils.java	86.27%	3 Missing and 4 partials ⚠️
...c/proto/request/search/ProtoActionsProtoUtils.java	40.00%	1 Missing and 5 partials ⚠️
...t/search/query/AbstractQueryBuilderProtoUtils.java	25.00%	5 Missing and 1 partial ⚠️
...oto/request/search/sort/SortBuilderProtoUtils.java	62.50%	5 Missing and 1 partial ⚠️
...proto/request/search/IndicesOptionsProtoUtils.java	87.17%	0 Missing and 5 partials ⚠️
...rpc/proto/request/search/SearchTypeProtoUtils.java	28.57%	3 Missing and 2 partials ⚠️
... and 12 more

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17830      +/-   ##
============================================
- Coverage     72.44%   72.36%   -0.09%     
- Complexity    66483    66727     +244     
============================================
  Files          5409     5450      +41     
  Lines        308282   309135     +853     
  Branches      44759    44998     +239     
============================================
+ Hits         223344   223703     +359     
- Misses        66608    67142     +534     
+ Partials      18330    18290      -40

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-04-08T20:43:41Z

❌ Gradle check result for c8015b9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-04-08T23:09:02Z

❌ Gradle check result for 67bed87: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

andrross · 2025-04-08T23:40:58Z

@karenyrx Can you rebase in the latest test fixes from main and fix the CHANGELOG conflict?

server/src/main/java/org/opensearch/action/support/IndicesOptions.java

...rc/main/java/org/opensearch/plugin/transport/grpc/listeners/SearchRequestActionListener.java

Signed-off-by: Karen Xu <[email protected]>

github-actions · 2025-04-09T00:23:52Z

❌ Gradle check result for ada3465: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Karen Xu <[email protected]>

github-actions · 2025-04-09T01:35:59Z

❕ Gradle check result for 5bb04e1: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

Signed-off-by: Andrew Ross <[email protected]>

finnegancarroll · 2025-04-08T16:44:28Z

.../org/opensearch/plugin/transport/grpc/proto/request/common/FetchSourceContextProtoUtils.java

+        }
+
+        if (fetchSource != null || sourceIncludes != null || sourceExcludes != null) {
+            return new FetchSourceContext(fetchSource == null ? true : fetchSource, sourceIncludes, sourceExcludes);


If fetchSource is explicitly set to false here fetchSource != null is true. Should we default fetchSource to false and change the condition to if (fetchSource || ...?

This is the equivalent REST-side implementation: https://github.com/opensearch-project/OpenSearch/blob/main/server/src/main/java/org/opensearch/search/fetch/subphase/FetchSourceContext.java#L135 which also does the same check. Following this, I think this GRPC code is accurate?

...main/java/org/opensearch/plugin/transport/grpc/proto/request/common/ObjectMapProtoUtils.java

...rc/main/java/org/opensearch/plugin/transport/grpc/proto/request/common/ScriptProtoUtils.java

...ava/org/opensearch/plugin/transport/grpc/proto/request/search/CollapseBuilderProtoUtils.java

.../exceptions/shardoperationfailedexception/ReplicationResponseShardInfoFailureProtoUtils.java

.../proto/response/exceptions/shardoperationfailedexception/SnapshotShardFailureProtoUtils.java

...ain/java/org/opensearch/plugin/transport/grpc/proto/response/search/SearchHitProtoUtils.java

...st/java/org/opensearch/plugin/transport/grpc/listeners/SearchRequestActionListenerTests.java

github-actions · 2025-04-09T18:57:40Z

❌ Gradle check result for 148f712: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

…ity, add more javadocs Signed-off-by: Karen Xu <[email protected]>

github-actions · 2025-04-09T22:08:21Z

✅ Gradle check result for 1a98708: SUCCESS

…17830) Signed-off-by: Karen Xu <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Signed-off-by: Sriram Ganesh <[email protected]>

…17830) Signed-off-by: Karen Xu <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Andrew Ross <[email protected]> Signed-off-by: Harsh Kothari <[email protected]>

karenyrx force-pushed the search-4 branch from 11b9615 to ddee102 Compare April 8, 2025 11:40

karenyrx force-pushed the search-4 branch 2 times, most recently from 4110529 to d950579 Compare April 8, 2025 12:03

karenyrx force-pushed the search-4 branch 2 times, most recently from c9a59dd to 9d23b96 Compare April 8, 2025 15:56

peterzhuamazon mentioned this pull request Apr 8, 2025

[RELEASE] Release version 3.0.0 opensearch-project/opensearch-build#3747

Closed

66 tasks

karenyrx force-pushed the search-4 branch 2 times, most recently from 7a4bf87 to 5ee1000 Compare April 8, 2025 17:16

karenyrx force-pushed the search-4 branch from 6546519 to 67bed87 Compare April 8, 2025 22:11

karenyrx force-pushed the search-4 branch from ed49fd2 to 81f6004 Compare April 8, 2025 23:14

andrross reviewed Apr 8, 2025

View reviewed changes

server/src/main/java/org/opensearch/action/support/IndicesOptions.java Outdated Show resolved Hide resolved

andrross reviewed Apr 8, 2025

View reviewed changes

...rc/main/java/org/opensearch/plugin/transport/grpc/listeners/SearchRequestActionListener.java Outdated Show resolved Hide resolved

karenyrx added 4 commits April 8, 2025 23:59

[GRPC] SearchService and Search GRPC endpoint v1

b4fcede

Signed-off-by: Karen Xu <[email protected]>

more tests

3354776

Signed-off-by: Karen Xu <[email protected]>

empty commit to retrigger flaky test CI

d685fdf

Signed-off-by: Karen Xu <[email protected]>

address comments

52285b2

Signed-off-by: Karen Xu <[email protected]>

karenyrx force-pushed the search-4 branch from 81f6004 to 52285b2 Compare April 9, 2025 00:00

make BulkRequestActionListener observer final too

ada3465

Signed-off-by: Karen Xu <[email protected]>

karenyrx marked this pull request as ready for review April 9, 2025 00:02

karenyrx requested review from anasalkouz and ashking94 as code owners April 9, 2025 00:02

karenyrx requested review from shwetathareja, sohami, VachaShah, bugmakerrrrrr, jainankitk and linuxpi as code owners April 9, 2025 00:02

empty commit for CI

5bb04e1

Signed-off-by: Karen Xu <[email protected]>

Merge branch 'main' into search-4

148f712

Signed-off-by: Andrew Ross <[email protected]>

finnegancarroll approved these changes Apr 9, 2025

View reviewed changes

karenyrx force-pushed the search-4 branch 2 times, most recently from 164c00b to 6be7f0c Compare April 9, 2025 21:03

fix floating javadocs, add null checks, restrict public method visibl…

1a98708

…ity, add more javadocs Signed-off-by: Karen Xu <[email protected]>

karenyrx force-pushed the search-4 branch from 6be7f0c to 1a98708 Compare April 9, 2025 21:06

andrross approved these changes Apr 9, 2025

View reviewed changes

andrross merged commit b7dca4a into opensearch-project:main Apr 9, 2025
31 of 32 checks passed

BrewTestBot mentioned this pull request May 6, 2025

opensearch 3.0.0 Homebrew/homebrew-core#222665

Merged

[GRPC] SearchService and Search GRPC endpoint v1 #17830

[GRPC] SearchService and Search GRPC endpoint v1 #17830

Uh oh!

Conversation

karenyrx commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Test Plan

Related Issues

Check List

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

codecov bot commented Apr 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

github-actions bot commented Apr 8, 2025

Uh oh!

andrross commented Apr 8, 2025

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

finnegancarroll Apr 8, 2025

Choose a reason for hiding this comment

Uh oh!

karenyrx Apr 9, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

github-actions bot commented Apr 9, 2025

Uh oh!

Uh oh!

Uh oh!

karenyrx commented Apr 8, 2025 •

edited

Loading

codecov bot commented Apr 8, 2025 •

edited

Loading