BooleanQuery rewrite for must_not RangeQuery clauses

[peteralfonsi]

Description

This PR automatically rewrites boolean queries which have a must_not RangeQuery clause to instead use a should clause of the complement of that range. This can be 2-30x faster depending on the query. See #17586 where this is described in more detail.

Example original query (on nyc_taxis):

"bool" : { 
  "must_not": [ 
    {"range" : {"dropoff_datetime" : {"gte": "01/07/2015", "lte": "01/09/2015", "format": "dd/MM/yyyy"}}}
  ]
}

Rewritten query:

"bool": { 
  "must":{
    "bool":{
      "should": [
        {"range" : {"dropoff_datetime" : {"lt": "01/07/2015", "format": "dd/MM/yyyy"}}},
        {"range" : {"dropoff_datetime" : {"gt": "01/09/2015", "format": "dd/MM/yyyy"}}}
      ]
    }
  }
}

Some benchmark numbers from http_logs and nyc_taxis (excluded ranges are on @timestamp and dropoff_datetime fields respectively). "Originally written as" means whether the query was sent to OpenSearch with a must_not clause, or if it was sent already rewritten with should clauses. Ideally, after the changes are applied, these p50s should be the same.

Excluded range	Originally written as	Dataset	p50 before changes (ms)	p50 after changes (ms)
6/10 - 6/13	must_not	http_logs	259	38.2
6/10 - 6/13	should	http_logs	34.2	39.5
6/9 - 6/10	must_not	http_logs	269	30.8
6/9 - 6/10	should	http_logs	26.3	30.8
7/1 - 9/1	must_not	nyc_taxis	873	408
7/1 - 9/1	should	nyc_taxis	427	405
1/1 - 9/1	must_not	nyc_taxis	1214	111
1/1 - 9/1	should	nyc_taxis	116	111
1/1 12:00 - 1/1 12:01	must_not	nyc_taxis	599	19.5
1/1 12:00 - 1/1 12:01	should	nyc_taxis	19.3	20.2

I believe the small differences between runs (for example, 7/1-9/1 should going from 427 -> 405 ms, when we'd expect no change) is just due to variation between different runs/instances. This is expected from what I've seen in tiered caching benchmarks. I've done a few runs and the direction/magnitude of the changes vary.

Related Issues

Part of #17586

Check List

• Functionality includes testing.
• [N/A] API changes companion pull request created, if applicable.
• [N/A] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for d9eee10: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 25367bb: SUCCESS

[peteralfonsi]

Sorry for the very long delay, was caught up with other things. Looks like the method described by @froh is actually doable at QueryBuilder rewrite time, since we have access to LeafReaderContexts via IndexSearcher via QueryShardContext. I've added this check. I also reran the benchmarks to make sure getting PointValues wasn't slowing things down, but there was no change in latencies.

[github-actions]

❕ Gradle check result for fc49ddd: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[msfroh]

Sorry for the delay in getting back to this one. I accidentally lost track of it.

The additional changes look safer, thanks! I just had a small clean-up suggestion on getting the LeafReaderContexts.

[github-actions]

❌ Gradle check result for feed2e8: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for c273159: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[peteralfonsi]

Hey @msfroh , any further comments on this one?

[github-actions]

❌ Gradle check result for c4d29f2: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[peteralfonsi]

Flaky test: #18302

[github-actions]

✅ Gradle check result for 58d9a45: SUCCESS

[peteralfonsi]

Hey @msfroh , just bumping on this