Introduce execution_hint for Cardinality aggregation
#17312
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
Signed-off-by: Siddharth Rayabharam <[email protected]>
asimmahmood1
requested review from
anasalkouz,
andrross,
ashking94,
Bukhtawar,
CEHENKLE,
cwperks,
dblock,
dbwiddis,
gbbafna,
jed326,
kotwanikunal,
mch2,
msfroh,
nknize,
owaiskazi19,
reta,
Rishikesh1159,
sachinpkale,
saratvemulapalli and
shwetathareja
as code owners
Signed-off-by: Asim Mahmood <[email protected]>
rishabhmaurya
approved these changes
Feb 15, 2025
sandeshkr419
approved these changes
Feb 17, 2025
cwperks
approved these changes
Feb 18, 2025
msfroh
approved these changes
Feb 20, 2025
This was referenced Feb 20, 2025
|
The backport to To backport manually, run these commands in your terminal: # Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch/backport-2.x
# Create a new branch
git switch --create backport/backport-17312-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 e3a6ccadc942c64e83bd224031bc4d1c6ab14623
# Push it to GitHub
git push --set-upstream origin backport/backport-17312-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch/backport-2.xThen, create a pull request where the |
3 tasks
asimmahmood1
added a commit
to asimmahmood1/OpenSearch
that referenced
this pull request
Feb 21, 2025
…oject#17312) - backport to 2.19 --------- Signed-off-by: Siddharth Rayabharam <[email protected]> Signed-off-by: Asim Mahmood <[email protected]> Signed-off-by: Asim M <[email protected]> Co-authored-by: Siddharth Rayabharam <[email protected]> Co-authored-by: Craig Perkins <[email protected]> (cherry picked from commit e3a6cca)
3 tasks
andrross
pushed a commit
that referenced
this pull request
Feb 21, 2025
…kport to 2.19 (#17420) * Introduce `execution_hint` for Cardinality aggregation (#17312) - backport to 2.19 --------- Signed-off-by: Siddharth Rayabharam <[email protected]> Signed-off-by: Asim Mahmood <[email protected]> Signed-off-by: Asim M <[email protected]> Co-authored-by: Siddharth Rayabharam <[email protected]> Co-authored-by: Craig Perkins <[email protected]> (cherry picked from commit e3a6cca) * Update changelog Signed-off-by: Asim Mahmood <[email protected]> --------- Signed-off-by: Asim Mahmood <[email protected]>
cwperks
pushed a commit
that referenced
this pull request
Feb 27, 2025
…kport to 2.x (#17419) * Introduce `execution_hint` for Cardinality aggregation (#17312) --------- Signed-off-by: Siddharth Rayabharam <[email protected]> Signed-off-by: Asim Mahmood <[email protected]> Signed-off-by: Asim M <[email protected]> Co-authored-by: Siddharth Rayabharam <[email protected]> Co-authored-by: Craig Perkins <[email protected]> (cherry picked from commit e3a6cca) * Update changlog Signed-off-by: Asim Mahmood <[email protected]> * Update min version to 2_19_1 * since 2.19 version has been merged: #17420 Signed-off-by: Asim Mahmood <[email protected]> --------- Signed-off-by: Asim Mahmood <[email protected]>
3 tasks
asimmahmood1
added a commit
to asimmahmood1/opensearch-benchmark-workloads
that referenced
this pull request
Mar 13, 2025
…execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <[email protected]>
6 tasks
gkamat
pushed a commit
to opensearch-project/opensearch-benchmark-workloads
that referenced
this pull request
Apr 3, 2025
…execution hint ordinals (#579) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <[email protected]> * Update check to include version less than 6.0.0 to handle old ES versions Signed-off-by: Asim Mahmood <[email protected]> * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name Signed-off-by: Asim Mahmood <[email protected]> --------- Signed-off-by: Asim Mahmood <[email protected]>
opensearch-trigger-bot bot
pushed a commit
to opensearch-project/opensearch-benchmark-workloads
that referenced
this pull request
Apr 3, 2025
…execution hint ordinals (#579) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <[email protected]> * Update check to include version less than 6.0.0 to handle old ES versions Signed-off-by: Asim Mahmood <[email protected]> * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name Signed-off-by: Asim Mahmood <[email protected]> --------- Signed-off-by: Asim Mahmood <[email protected]> (cherry picked from commit 2bbfefc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gkamat
pushed a commit
to opensearch-project/opensearch-benchmark-workloads
that referenced
this pull request
Apr 3, 2025
…execution hint ordinals (#579) (#601) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality * Update check to include version less than 6.0.0 to handle old ES versions * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name --------- (cherry picked from commit 2bbfefc) Signed-off-by: Asim Mahmood <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
opensearch-trigger-bot bot
pushed a commit
to opensearch-project/opensearch-benchmark-workloads
that referenced
this pull request
Apr 16, 2025
…execution hint ordinals (#579) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality Signed-off-by: Asim Mahmood <[email protected]> * Update check to include version less than 6.0.0 to handle old ES versions Signed-off-by: Asim Mahmood <[email protected]> * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name Signed-off-by: Asim Mahmood <[email protected]> --------- Signed-off-by: Asim Mahmood <[email protected]> (cherry picked from commit 2bbfefc) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
gkamat
pushed a commit
to opensearch-project/opensearch-benchmark-workloads
that referenced
this pull request
Apr 16, 2025
…execution hint ordinals (#579) (#614) * Modify big5 cardinality-agg-high from agent.name to event.id and use execution hint ordinals * as part of follow up to opensearch-project/OpenSearch#17312 * event.id has almost 7x count as agent.name, so it serves as better benchmark for high cardinality * Update check to include version less than 6.0.0 to handle old ES versions * Allow cardinality-agg-high field to be overridden * useful for testing previous field: agent.name --------- (cherry picked from commit 2bbfefc) Signed-off-by: Asim Mahmood <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Introduce
execution_hintfor Cardinality aggregation - Revive PR #15764This is intended as short term solution as a workaround to users that see large latency regression when upgrading to OS 2.x. The main risk here is ordinals take up more memory, i.e. depends on size of cardinality. So for very high cardinality, there is a risk of OOM at shard node. This is marked as power user.
I plan to follow up with #15269 which is more useful setting. This will allow the user select a higher threshold for using ordinals, e.g. a shard with large heap can use ordinals more often, but still fall back to the slower direct as needed.
Related Issues
Resolves #16837.
Check List
Testing
Tested using big5 dataset, see #16837 (comment)
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.