Skip to content

Increase default maxMergeAtOnce from 10 to 30 #17774

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Apr 16, 2025

Conversation

linuxpi
Copy link
Contributor

@linuxpi linuxpi commented Apr 3, 2025

Description

  • Increase default maxMergeAtOnce from 10 to 30. This is inline with recent guidance from Lucene
  • We also introduce new dynamic cluster setting to tune maxMergeAtOnce at cluster level. This helps users playaround with this setting at cluster level incase the new value is not suitable for their workload

Related Issues

Resolves #17051

Check List

  • Functionality includes testing.
  • [ ] API changes companion pull request created, if applicable.
  • Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

github-actions bot commented Apr 3, 2025

❌ Gradle check result for 97aef8a: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@linuxpi linuxpi force-pushed the max-merge-at-once branch from 97aef8a to 4709b11 Compare April 3, 2025 09:17
Copy link
Contributor

github-actions bot commented Apr 3, 2025

❌ Gradle check result for 4709b11: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@linuxpi linuxpi force-pushed the max-merge-at-once branch 3 times, most recently from 9fe9c14 to f37545f Compare April 3, 2025 19:17
@github-actions github-actions bot added enhancement Enhancement or improvement to existing feature or request Indexing:Performance labels Apr 3, 2025
Copy link
Contributor

github-actions bot commented Apr 3, 2025

❌ Gradle check result for f37545f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@linuxpi linuxpi force-pushed the max-merge-at-once branch from f37545f to 70a31d9 Compare April 4, 2025 04:13
Copy link
Contributor

github-actions bot commented Apr 4, 2025

❌ Gradle check result for 70a31d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@linuxpi
Copy link
Contributor Author

linuxpi commented Apr 4, 2025

❌ Gradle check result for 70a31d9: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Test Result (2 failures / -1)

[org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod](https://build.ci.opensearch.org/job/gradle-check/55575/testReport/junit/org.opensearch.repositories.s3/S3BlobContainerRetriesTests/classMethod/)
[org.opensearch.repositories.s3.S3BlobContainerRetriesTests.classMethod](https://build.ci.opensearch.org/job/gradle-check/55575/testReport/junit/org.opensearch.repositories.s3/S3BlobContainerRetriesTests/classMethod_2/)

Copy link
Contributor

github-actions bot commented Apr 9, 2025

❌ Gradle check result for 1b75c1f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Varun Bansal <[email protected]>
@linuxpi
Copy link
Contributor Author

linuxpi commented Apr 10, 2025

{"run-benchmark-test": "id_5"}

Copy link
Contributor

The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/2819/ . Final results will be published once the job is completed.

@opensearch-ci-bot
Copy link
Collaborator

Benchmark Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/2819/

Metric Task Value Unit
Cumulative indexing time of primary shards 68.1115 min
Min cumulative indexing time across primary shards 68.1115 min
Median cumulative indexing time across primary shards 68.1115 min
Max cumulative indexing time across primary shards 68.1115 min
Cumulative indexing throttle time of primary shards 0 min
Min cumulative indexing throttle time across primary shards 0 min
Median cumulative indexing throttle time across primary shards 0 min
Max cumulative indexing throttle time across primary shards 0 min
Cumulative merge time of primary shards 46.9466 min
Cumulative merge count of primary shards 47
Min cumulative merge time across primary shards 46.9466 min
Median cumulative merge time across primary shards 46.9466 min
Max cumulative merge time across primary shards 46.9466 min
Cumulative merge throttle time of primary shards 13.4438 min
Min cumulative merge throttle time across primary shards 13.4438 min
Median cumulative merge throttle time across primary shards 13.4438 min
Max cumulative merge throttle time across primary shards 13.4438 min
Cumulative refresh time of primary shards 1.66777 min
Cumulative refresh count of primary shards 68
Min cumulative refresh time across primary shards 1.66777 min
Median cumulative refresh time across primary shards 1.66777 min
Max cumulative refresh time across primary shards 1.66777 min
Cumulative flush time of primary shards 5.66712 min
Cumulative flush count of primary shards 53
Min cumulative flush time across primary shards 5.66712 min
Median cumulative flush time across primary shards 5.66712 min
Max cumulative flush time across primary shards 5.66712 min
Total Young Gen GC time 1.069 s
Total Young Gen GC count 45
Total Old Gen GC time 0 s
Total Old Gen GC count 0
Store size 19.3562 GB
Translog size 5.12227e-08 GB
Heap used for segments 0 MB
Heap used for doc values 0 MB
Heap used for terms 0 MB
Heap used for norms 0 MB
Heap used for points 0 MB
Heap used for stored fields 0 MB
Segment count 21
Min Throughput index-append 540.73 docs/s
Mean Throughput index-append 561.54 docs/s
Median Throughput index-append 562.84 docs/s
Max Throughput index-append 572.43 docs/s
50th percentile latency index-append 6461.27 ms
90th percentile latency index-append 10829.9 ms
99th percentile latency index-append 13925 ms
100th percentile latency index-append 16689.3 ms
50th percentile service time index-append 6468.43 ms
90th percentile service time index-append 10829.9 ms
99th percentile service time index-append 13925 ms
100th percentile service time index-append 16689.3 ms
error rate index-append 0 %
Min Throughput wait-until-merges-finish 0 ops/s
Mean Throughput wait-until-merges-finish 0 ops/s
Median Throughput wait-until-merges-finish 0 ops/s
Max Throughput wait-until-merges-finish 0 ops/s
100th percentile latency wait-until-merges-finish 288327 ms
100th percentile service time wait-until-merges-finish 288327 ms
error rate wait-until-merges-finish 0 %
Min Throughput default 19.94 ops/s
Mean Throughput default 19.95 ops/s
Median Throughput default 19.95 ops/s
Max Throughput default 19.95 ops/s
50th percentile latency default 6.42141 ms
90th percentile latency default 6.79896 ms
99th percentile latency default 7.73649 ms
100th percentile latency default 9.27276 ms
50th percentile service time default 5.45831 ms
90th percentile service time default 5.72186 ms
99th percentile service time default 6.72184 ms
100th percentile service time default 8.83811 ms
error rate default 0 %
Min Throughput term 19.96 ops/s
Mean Throughput term 19.97 ops/s
Median Throughput term 19.97 ops/s
Max Throughput term 19.97 ops/s
50th percentile latency term 6.5946 ms
90th percentile latency term 7.03288 ms
99th percentile latency term 8.79745 ms
100th percentile latency term 8.86341 ms
50th percentile service time term 5.73491 ms
90th percentile service time term 5.98297 ms
99th percentile service time term 7.79356 ms
100th percentile service time term 8.13715 ms
error rate term 0 %
Min Throughput phrase 19.94 ops/s
Mean Throughput phrase 19.95 ops/s
Median Throughput phrase 19.95 ops/s
Max Throughput phrase 19.96 ops/s
50th percentile latency phrase 7.34276 ms
90th percentile latency phrase 7.82581 ms
99th percentile latency phrase 9.12514 ms
100th percentile latency phrase 9.55732 ms
50th percentile service time phrase 6.52448 ms
90th percentile service time phrase 6.79349 ms
99th percentile service time phrase 8.4708 ms
100th percentile service time phrase 8.63116 ms
error rate phrase 0 %
Min Throughput articles_monthly_agg_uncached 19.89 ops/s
Mean Throughput articles_monthly_agg_uncached 19.91 ops/s
Median Throughput articles_monthly_agg_uncached 19.91 ops/s
Max Throughput articles_monthly_agg_uncached 19.92 ops/s
50th percentile latency articles_monthly_agg_uncached 8.72557 ms
90th percentile latency articles_monthly_agg_uncached 9.17101 ms
99th percentile latency articles_monthly_agg_uncached 10.2116 ms
100th percentile latency articles_monthly_agg_uncached 12.2384 ms
50th percentile service time articles_monthly_agg_uncached 7.88876 ms
90th percentile service time articles_monthly_agg_uncached 8.13233 ms
99th percentile service time articles_monthly_agg_uncached 9.37703 ms
100th percentile service time articles_monthly_agg_uncached 11.936 ms
error rate articles_monthly_agg_uncached 0 %
Min Throughput articles_monthly_agg_cached 20.02 ops/s
Mean Throughput articles_monthly_agg_cached 20.02 ops/s
Median Throughput articles_monthly_agg_cached 20.02 ops/s
Max Throughput articles_monthly_agg_cached 20.02 ops/s
50th percentile latency articles_monthly_agg_cached 3.95554 ms
90th percentile latency articles_monthly_agg_cached 4.30356 ms
99th percentile latency articles_monthly_agg_cached 4.78483 ms
100th percentile latency articles_monthly_agg_cached 5.04632 ms
50th percentile service time articles_monthly_agg_cached 3.04695 ms
90th percentile service time articles_monthly_agg_cached 3.1966 ms
99th percentile service time articles_monthly_agg_cached 3.63803 ms
100th percentile service time articles_monthly_agg_cached 3.74083 ms
error rate articles_monthly_agg_cached 0 %
Min Throughput scroll 12.54 pages/s
Mean Throughput scroll 12.57 pages/s
Median Throughput scroll 12.56 pages/s
Max Throughput scroll 12.63 pages/s
50th percentile latency scroll 786.456 ms
90th percentile latency scroll 846.254 ms
99th percentile latency scroll 1026.95 ms
100th percentile latency scroll 1030.92 ms
50th percentile service time scroll 783.844 ms
90th percentile service time scroll 844.138 ms
99th percentile service time scroll 1024.33 ms
100th percentile service time scroll 1028.87 ms
error rate scroll 0 %

@opensearch-ci-bot
Copy link
Collaborator

Benchmark Baseline Comparison Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/59/

Metric Task Baseline Contender Diff Unit
Cumulative indexing time of primary shards 65.68 68.1115 2.4314 min
Min cumulative indexing time across primary shard 65.68 68.1115 2.4314 min
Median cumulative indexing time across primary shard 65.68 68.1115 2.4314 min
Max cumulative indexing time across primary shard 65.68 68.1115 2.4314 min
Cumulative indexing throttle time of primary shards 0 0 0 min
Min cumulative indexing throttle time across primary shard 0 0 0 min
Median cumulative indexing throttle time across primary shard 0 0 0 min
Max cumulative indexing throttle time across primary shard 0 0 0 min
Cumulative merge time of primary shards 39.3801 46.9466 7.5665 min
Cumulative merge count of primary shards 46 47 1
Min cumulative merge time across primary shard 39.3801 46.9466 7.5665 min
Median cumulative merge time across primary shard 39.3801 46.9466 7.5665 min
Max cumulative merge time across primary shard 39.3801 46.9466 7.5665 min
Cumulative merge throttle time of primary shards 9.53325 13.4438 3.91053 min
Min cumulative merge throttle time across primary shard 9.53325 13.4438 3.91053 min
Median cumulative merge throttle time across primary shard 9.53325 13.4438 3.91053 min
Max cumulative merge throttle time across primary shard 9.53325 13.4438 3.91053 min
Cumulative refresh time of primary shards 1.75272 1.66777 -0.08495 min
Cumulative refresh count of primary shards 65 68 3
Min cumulative refresh time across primary shard 1.75272 1.66777 -0.08495 min
Median cumulative refresh time across primary shard 1.75272 1.66777 -0.08495 min
Max cumulative refresh time across primary shard 1.75272 1.66777 -0.08495 min
Cumulative flush time of primary shards 5.5599 5.66712 0.10722 min
Cumulative flush count of primary shards 52 53 1
Min cumulative flush time across primary shard 5.5599 5.66712 0.10722 min
Median cumulative flush time across primary shard 5.5599 5.66712 0.10722 min
Max cumulative flush time across primary shard 5.5599 5.66712 0.10722 min
Total Young Gen GC time 1.341 1.069 -0.272 s
Total Young Gen GC count 47 45 -2
Total Old Gen GC time 0 0 0 s
Total Old Gen GC count 0 0 0
Store size 19.3505 19.3562 0.00574 GB
Translog size 5.12227e-08 5.12227e-08 0 GB
Heap used for segments 0 0 0 MB
Heap used for doc values 0 0 0 MB
Heap used for terms 0 0 0 MB
Heap used for norms 0 0 0 MB
Heap used for points 0 0 0 MB
Heap used for stored fields 0 0 0 MB
Segment count 22 21 -1
Min Throughput index-append 566.277 540.733 -25.5443 docs/s
Mean Throughput index-append 576.205 561.544 -14.6609 docs/s
Median Throughput index-append 574.787 562.836 -11.9511 docs/s
Max Throughput index-append 592.484 572.43 -20.0541 docs/s
50th percentile latency index-append 6167.14 6461.27 294.132 ms
90th percentile latency index-append 10618.9 10829.9 211.045 ms
99th percentile latency index-append 13323 13925 602.019 ms
100th percentile latency index-append 14282 16689.3 2407.25 ms
50th percentile service time index-append 6166.83 6468.43 301.603 ms
90th percentile service time index-append 10618.9 10829.9 211.045 ms
99th percentile service time index-append 13323 13925 602.019 ms
100th percentile service time index-append 14282 16689.3 2407.25 ms
error rate index-append 0 0 0 %
Min Throughput wait-until-merges-finish 0.00770644 0.00346828 -0.00424 ops/s
Mean Throughput wait-until-merges-finish 0.00770644 0.00346828 -0.00424 ops/s
Median Throughput wait-until-merges-finish 0.00770644 0.00346828 -0.00424 ops/s
Max Throughput wait-until-merges-finish 0.00770644 0.00346828 -0.00424 ops/s
100th percentile latency wait-until-merges-finish 129761 288327 158566 ms
100th percentile service time wait-until-merges-finish 129761 288327 158566 ms
error rate wait-until-merges-finish 0 0 0 %
Min Throughput default 19.9365 19.9378 0.00126 ops/s
Mean Throughput default 19.9454 19.947 0.00159 ops/s
Median Throughput default 19.946 19.9476 0.00162 ops/s
Max Throughput default 19.9531 19.955 0.00194 ops/s
50th percentile latency default 6.11374 6.42141 0.30767 ms
90th percentile latency default 6.62015 6.79896 0.17881 ms
99th percentile latency default 8.46682 7.73649 -0.73033 ms
100th percentile latency default 8.71038 9.27276 0.56238 ms
50th percentile service time default 5.19068 5.45831 0.26763 ms
90th percentile service time default 5.74812 5.72186 -0.02626 ms
99th percentile service time default 7.63904 6.72184 -0.91719 ms
100th percentile service time default 7.83741 8.83811 1.0007 ms
error rate default 0 0 0 %
Min Throughput term 19.9925 19.9645 -0.02797 ops/s
Mean Throughput term 19.9935 19.9692 -0.02425 ops/s
Median Throughput term 19.9935 19.9695 -0.02399 ops/s
Max Throughput term 19.9944 19.9732 -0.02121 ops/s
50th percentile latency term 5.78687 6.5946 0.80773 ms
90th percentile latency term 6.21231 7.03288 0.82057 ms
99th percentile latency term 7.22824 8.79745 1.56921 ms
100th percentile latency term 7.6593 8.86341 1.20412 ms
50th percentile service time term 4.96673 5.73491 0.76818 ms
90th percentile service time term 5.24574 5.98297 0.73723 ms
99th percentile service time term 6.64294 7.79356 1.15062 ms
100th percentile service time term 7.06486 8.13715 1.0723 ms
error rate term 0 0 0 %
Min Throughput phrase 19.9859 19.9434 -0.04253 ops/s
Mean Throughput phrase 19.9877 19.9515 -0.03623 ops/s
Median Throughput phrase 19.9879 19.9518 -0.03611 ops/s
Max Throughput phrase 19.9895 19.9581 -0.03137 ops/s
50th percentile latency phrase 6.25985 7.34276 1.08291 ms
90th percentile latency phrase 6.693 7.82581 1.1328 ms
99th percentile latency phrase 7.6234 9.12514 1.50174 ms
100th percentile latency phrase 7.98679 9.55732 1.57052 ms
50th percentile service time phrase 5.40679 6.52448 1.11769 ms
90th percentile service time phrase 5.80616 6.79349 0.98732 ms
99th percentile service time phrase 6.90334 8.4708 1.56747 ms
100th percentile service time phrase 7.66794 8.63116 0.96322 ms
error rate phrase 0 0 0 %
Min Throughput articles_monthly_agg_uncached 19.9282 19.8919 -0.03628 ops/s
Mean Throughput articles_monthly_agg_uncached 19.9382 19.9068 -0.03141 ops/s
Median Throughput articles_monthly_agg_uncached 19.9386 19.9077 -0.03094 ops/s
Max Throughput articles_monthly_agg_uncached 19.9465 19.9193 -0.02727 ops/s
50th percentile latency articles_monthly_agg_uncached 8.07963 8.72557 0.64594 ms
90th percentile latency articles_monthly_agg_uncached 8.76648 9.17101 0.40452 ms
99th percentile latency articles_monthly_agg_uncached 13.9695 10.2116 -3.75788 ms
100th percentile latency articles_monthly_agg_uncached 15.4819 12.2384 -3.24348 ms
50th percentile service time articles_monthly_agg_uncached 7.25845 7.88876 0.63031 ms
90th percentile service time articles_monthly_agg_uncached 7.80462 8.13233 0.32772 ms
99th percentile service time articles_monthly_agg_uncached 13.2744 9.37703 -3.89733 ms
100th percentile service time articles_monthly_agg_uncached 14.7347 11.936 -2.79879 ms
error rate articles_monthly_agg_uncached 0 0 0 %
Min Throughput articles_monthly_agg_cached 19.9994 20.0168 0.01743 ops/s
Mean Throughput articles_monthly_agg_cached 19.9997 20.0196 0.0199 ops/s
Median Throughput articles_monthly_agg_cached 19.9996 20.0194 0.01977 ops/s
Max Throughput articles_monthly_agg_cached 19.9999 20.0232 0.0233 ops/s
50th percentile latency articles_monthly_agg_cached 3.06824 3.95554 0.8873 ms
90th percentile latency articles_monthly_agg_cached 3.5376 4.30356 0.76595 ms
99th percentile latency articles_monthly_agg_cached 3.91523 4.78483 0.86959 ms
100th percentile latency articles_monthly_agg_cached 4.22852 5.04632 0.8178 ms
50th percentile service time articles_monthly_agg_cached 2.27114 3.04695 0.77581 ms
90th percentile service time articles_monthly_agg_cached 2.46253 3.1966 0.73407 ms
99th percentile service time articles_monthly_agg_cached 2.89271 3.63803 0.74532 ms
100th percentile service time articles_monthly_agg_cached 2.9877 3.74083 0.75313 ms
error rate articles_monthly_agg_cached 0 0 0 %
Min Throughput scroll 12.5479 12.5429 -0.00505 pages/s
Mean Throughput scroll 12.5788 12.5704 -0.00834 pages/s
Median Throughput scroll 12.5717 12.5641 -0.00763 pages/s
Max Throughput scroll 12.642 12.6269 -0.01507 pages/s
50th percentile latency scroll 678.986 786.456 107.47 ms
90th percentile latency scroll 704.473 846.254 141.782 ms
99th percentile latency scroll 735.855 1026.95 291.092 ms
100th percentile latency scroll 738.554 1030.92 292.363 ms
50th percentile service time scroll 676.37 783.844 107.474 ms
90th percentile service time scroll 703.059 844.138 141.079 ms
99th percentile service time scroll 733.405 1024.33 290.927 ms
100th percentile service time scroll 735.709 1028.87 293.162 ms
error rate scroll 0 0 0 %

Copy link
Contributor

❌ Gradle check result for fb43f2f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@linuxpi
Copy link
Contributor Author

linuxpi commented Apr 11, 2025

{"run-benchmark-test": "id_5"}

Copy link
Contributor

The Jenkins job url is https://build.ci.opensearch.org/job/benchmark-pull-request/2832/ . Final results will be published once the job is completed.

@opensearch-ci-bot
Copy link
Collaborator

Benchmark Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-pull-request/2832/

Metric Task Value Unit
Cumulative indexing time of primary shards 70.8808 min
Min cumulative indexing time across primary shards 70.8808 min
Median cumulative indexing time across primary shards 70.8808 min
Max cumulative indexing time across primary shards 70.8808 min
Cumulative indexing throttle time of primary shards 0 min
Min cumulative indexing throttle time across primary shards 0 min
Median cumulative indexing throttle time across primary shards 0 min
Max cumulative indexing throttle time across primary shards 0 min
Cumulative merge time of primary shards 43.359 min
Cumulative merge count of primary shards 51
Min cumulative merge time across primary shards 43.359 min
Median cumulative merge time across primary shards 43.359 min
Max cumulative merge time across primary shards 43.359 min
Cumulative merge throttle time of primary shards 11.5249 min
Min cumulative merge throttle time across primary shards 11.5249 min
Median cumulative merge throttle time across primary shards 11.5249 min
Max cumulative merge throttle time across primary shards 11.5249 min
Cumulative refresh time of primary shards 1.68547 min
Cumulative refresh count of primary shards 72
Min cumulative refresh time across primary shards 1.68547 min
Median cumulative refresh time across primary shards 1.68547 min
Max cumulative refresh time across primary shards 1.68547 min
Cumulative flush time of primary shards 5.79782 min
Cumulative flush count of primary shards 58
Min cumulative flush time across primary shards 5.79782 min
Median cumulative flush time across primary shards 5.79782 min
Max cumulative flush time across primary shards 5.79782 min
Total Young Gen GC time 1.133 s
Total Young Gen GC count 45
Total Old Gen GC time 0 s
Total Old Gen GC count 0
Store size 19.3593 GB
Translog size 5.12227e-08 GB
Heap used for segments 0 MB
Heap used for doc values 0 MB
Heap used for terms 0 MB
Heap used for norms 0 MB
Heap used for points 0 MB
Heap used for stored fields 0 MB
Segment count 24
Min Throughput index-append 524.92 docs/s
Mean Throughput index-append 548.01 docs/s
Median Throughput index-append 547.17 docs/s
Max Throughput index-append 564.58 docs/s
50th percentile latency index-append 6767.74 ms
90th percentile latency index-append 11141.3 ms
99th percentile latency index-append 14070.5 ms
100th percentile latency index-append 16417.3 ms
50th percentile service time index-append 6766.65 ms
90th percentile service time index-append 11141.7 ms
99th percentile service time index-append 14070.5 ms
100th percentile service time index-append 16417.3 ms
error rate index-append 0 %
Min Throughput wait-until-merges-finish 0 ops/s
Mean Throughput wait-until-merges-finish 0 ops/s
Median Throughput wait-until-merges-finish 0 ops/s
Max Throughput wait-until-merges-finish 0 ops/s
100th percentile latency wait-until-merges-finish 311011 ms
100th percentile service time wait-until-merges-finish 311011 ms
error rate wait-until-merges-finish 0 %
Min Throughput default 19.94 ops/s
Mean Throughput default 19.95 ops/s
Median Throughput default 19.95 ops/s
Max Throughput default 19.95 ops/s
50th percentile latency default 6.11614 ms
90th percentile latency default 6.55495 ms
99th percentile latency default 7.33515 ms
100th percentile latency default 9.1971 ms
50th percentile service time default 5.32737 ms
90th percentile service time default 5.49961 ms
99th percentile service time default 6.54216 ms
100th percentile service time default 8.32832 ms
error rate default 0 %
Min Throughput term 19.97 ops/s
Mean Throughput term 19.98 ops/s
Median Throughput term 19.98 ops/s
Max Throughput term 19.98 ops/s
50th percentile latency term 6.65456 ms
90th percentile latency term 7.17538 ms
99th percentile latency term 7.92695 ms
100th percentile latency term 9.24842 ms
50th percentile service time term 5.91284 ms
90th percentile service time term 6.17016 ms
99th percentile service time term 6.9159 ms
100th percentile service time term 8.3457 ms
error rate term 0 %
Min Throughput phrase 19.97 ops/s
Mean Throughput phrase 19.98 ops/s
Median Throughput phrase 19.98 ops/s
Max Throughput phrase 19.98 ops/s
50th percentile latency phrase 7.49959 ms
90th percentile latency phrase 7.9724 ms
99th percentile latency phrase 10.0227 ms
100th percentile latency phrase 10.1286 ms
50th percentile service time phrase 6.70445 ms
90th percentile service time phrase 6.96798 ms
99th percentile service time phrase 9.01897 ms
100th percentile service time phrase 9.25478 ms
error rate phrase 0 %
Min Throughput articles_monthly_agg_uncached 19.88 ops/s
Mean Throughput articles_monthly_agg_uncached 19.9 ops/s
Median Throughput articles_monthly_agg_uncached 19.9 ops/s
Max Throughput articles_monthly_agg_uncached 19.91 ops/s
50th percentile latency articles_monthly_agg_uncached 8.94665 ms
90th percentile latency articles_monthly_agg_uncached 9.35324 ms
99th percentile latency articles_monthly_agg_uncached 10.9521 ms
100th percentile latency articles_monthly_agg_uncached 13.4554 ms
50th percentile service time articles_monthly_agg_uncached 8.12095 ms
90th percentile service time articles_monthly_agg_uncached 8.28273 ms
99th percentile service time articles_monthly_agg_uncached 10.2674 ms
100th percentile service time articles_monthly_agg_uncached 12.5234 ms
error rate articles_monthly_agg_uncached 0 %
Min Throughput articles_monthly_agg_cached 20.02 ops/s
Mean Throughput articles_monthly_agg_cached 20.02 ops/s
Median Throughput articles_monthly_agg_cached 20.02 ops/s
Max Throughput articles_monthly_agg_cached 20.02 ops/s
50th percentile latency articles_monthly_agg_cached 3.92349 ms
90th percentile latency articles_monthly_agg_cached 4.36253 ms
99th percentile latency articles_monthly_agg_cached 4.81747 ms
100th percentile latency articles_monthly_agg_cached 4.92981 ms
50th percentile service time articles_monthly_agg_cached 3.10599 ms
90th percentile service time articles_monthly_agg_cached 3.24139 ms
99th percentile service time articles_monthly_agg_cached 3.73509 ms
100th percentile service time articles_monthly_agg_cached 3.80212 ms
error rate articles_monthly_agg_cached 0 %
Min Throughput scroll 12.55 pages/s
Mean Throughput scroll 12.57 pages/s
Median Throughput scroll 12.57 pages/s
Max Throughput scroll 12.64 pages/s
50th percentile latency scroll 753.774 ms
90th percentile latency scroll 762.508 ms
99th percentile latency scroll 782.12 ms
100th percentile latency scroll 786.307 ms
50th percentile service time scroll 751.707 ms
90th percentile service time scroll 759.845 ms
99th percentile service time scroll 779.759 ms
100th percentile service time scroll 783.813 ms
error rate scroll 0 %

@opensearch-ci-bot
Copy link
Collaborator

Benchmark Baseline Comparison Results

Benchmark Results for Job: https://build.ci.opensearch.org/job/benchmark-compare/64/

Metric Task Baseline Contender Diff Unit
Cumulative indexing time of primary shards 68.0928 70.8808 2.78798 min
Min cumulative indexing time across primary shard 68.0928 70.8808 2.78798 min
Median cumulative indexing time across primary shard 68.0928 70.8808 2.78798 min
Max cumulative indexing time across primary shard 68.0928 70.8808 2.78798 min
Cumulative indexing throttle time of primary shards 0 0 0 min
Min cumulative indexing throttle time across primary shard 0 0 0 min
Median cumulative indexing throttle time across primary shard 0 0 0 min
Max cumulative indexing throttle time across primary shard 0 0 0 min
Cumulative merge time of primary shards 44.0849 43.359 -0.72588 min
Cumulative merge count of primary shards 53 51 -2
Min cumulative merge time across primary shard 44.0849 43.359 -0.72588 min
Median cumulative merge time across primary shard 44.0849 43.359 -0.72588 min
Max cumulative merge time across primary shard 44.0849 43.359 -0.72588 min
Cumulative merge throttle time of primary shards 10.5567 11.5249 0.96818 min
Min cumulative merge throttle time across primary shard 10.5567 11.5249 0.96818 min
Median cumulative merge throttle time across primary shard 10.5567 11.5249 0.96818 min
Max cumulative merge throttle time across primary shard 10.5567 11.5249 0.96818 min
Cumulative refresh time of primary shards 1.9055 1.68547 -0.22003 min
Cumulative refresh count of primary shards 74 72 -2
Min cumulative refresh time across primary shard 1.9055 1.68547 -0.22003 min
Median cumulative refresh time across primary shard 1.9055 1.68547 -0.22003 min
Max cumulative refresh time across primary shard 1.9055 1.68547 -0.22003 min
Cumulative flush time of primary shards 6.05538 5.79782 -0.25757 min
Cumulative flush count of primary shards 59 58 -1
Min cumulative flush time across primary shard 6.05538 5.79782 -0.25757 min
Median cumulative flush time across primary shard 6.05538 5.79782 -0.25757 min
Max cumulative flush time across primary shard 6.05538 5.79782 -0.25757 min
Total Young Gen GC time 1.116 1.133 0.017 s
Total Young Gen GC count 43 45 2
Total Old Gen GC time 0 0 0 s
Total Old Gen GC count 0 0 0
Store size 19.357 19.3593 0.00223 GB
Translog size 5.12227e-08 5.12227e-08 0 GB
Heap used for segments 0 0 0 MB
Heap used for doc values 0 0 0 MB
Heap used for terms 0 0 0 MB
Heap used for norms 0 0 0 MB
Heap used for points 0 0 0 MB
Heap used for stored fields 0 0 0 MB
Segment count 20 24 4
Min Throughput index-append 541.751 524.916 -16.8348 docs/s
Mean Throughput index-append 557.057 548.013 -9.04377 docs/s
Median Throughput index-append 556.373 547.166 -9.20697 docs/s
Max Throughput index-append 571.724 564.583 -7.14111 docs/s
50th percentile latency index-append 6281.69 6767.74 486.051 ms
90th percentile latency index-append 10332.7 11141.3 808.624 ms
99th percentile latency index-append 13996.7 14070.5 73.7556 ms
100th percentile latency index-append 16870.8 16417.3 -453.438 ms
50th percentile service time index-append 6281.43 6766.65 485.224 ms
90th percentile service time index-append 10329.2 11141.7 812.47 ms
99th percentile service time index-append 13996.7 14070.5 73.7556 ms
100th percentile service time index-append 16870.8 16417.3 -453.438 ms
error rate index-append 0 0 0 %
Min Throughput wait-until-merges-finish 0.00449621 0.00321532 -0.00128 ops/s
Mean Throughput wait-until-merges-finish 0.00449621 0.00321532 -0.00128 ops/s
Median Throughput wait-until-merges-finish 0.00449621 0.00321532 -0.00128 ops/s
Max Throughput wait-until-merges-finish 0.00449621 0.00321532 -0.00128 ops/s
100th percentile latency wait-until-merges-finish 222409 311011 88601.4 ms
100th percentile service time wait-until-merges-finish 222409 311011 88601.4 ms
error rate wait-until-merges-finish 0 0 0 %
Min Throughput default 19.9279 19.9363 0.00843 ops/s
Mean Throughput default 19.9382 19.9453 0.00708 ops/s
Median Throughput default 19.9389 19.9458 0.00694 ops/s
Max Throughput default 19.9468 19.9529 0.00612 ops/s
50th percentile latency default 5.7131 6.11614 0.40304 ms
90th percentile latency default 6.32237 6.55495 0.23258 ms
99th percentile latency default 7.73253 7.33515 -0.39738 ms
100th percentile latency default 10.6514 9.1971 -1.45426 ms
50th percentile service time default 4.96133 5.32737 0.36604 ms
90th percentile service time default 5.39531 5.49961 0.1043 ms
99th percentile service time default 7.02869 6.54216 -0.48653 ms
100th percentile service time default 9.59823 8.32832 -1.26991 ms
error rate default 0 0 0 %
Min Throughput term 19.9697 19.9723 0.00258 ops/s
Mean Throughput term 19.9741 19.9766 0.0025 ops/s
Median Throughput term 19.9744 19.9773 0.00287 ops/s
Max Throughput term 19.9775 19.9801 0.00257 ops/s
50th percentile latency term 5.45692 6.65456 1.19764 ms
90th percentile latency term 5.88127 7.17538 1.29411 ms
99th percentile latency term 7.3253 7.92695 0.60165 ms
100th percentile latency term 7.49817 9.24842 1.75025 ms
50th percentile service time term 4.68178 5.91284 1.23106 ms
90th percentile service time term 4.92894 6.17016 1.24122 ms
99th percentile service time term 6.39105 6.9159 0.52485 ms
100th percentile service time term 6.75437 8.3457 1.59133 ms
error rate term 0 0 0 %
Min Throughput phrase 19.9378 19.9748 0.03705 ops/s
Mean Throughput phrase 19.9474 19.9782 0.0308 ops/s
Median Throughput phrase 19.9479 19.9782 0.03026 ops/s
Max Throughput phrase 19.9555 19.9809 0.02541 ops/s
50th percentile latency phrase 6.89606 7.49959 0.60353 ms
90th percentile latency phrase 7.53209 7.9724 0.44031 ms
99th percentile latency phrase 7.85864 10.0227 2.16406 ms
100th percentile latency phrase 7.92706 10.1286 2.20154 ms
50th percentile service time phrase 6.36397 6.70445 0.34048 ms
90th percentile service time phrase 6.58656 6.96798 0.38142 ms
99th percentile service time phrase 6.97922 9.01897 2.03975 ms
100th percentile service time phrase 7.65689 9.25478 1.59789 ms
error rate phrase 0 0 0 %
Min Throughput articles_monthly_agg_uncached 19.914 19.8843 -0.02974 ops/s
Mean Throughput articles_monthly_agg_uncached 19.9258 19.9003 -0.02553 ops/s
Median Throughput articles_monthly_agg_uncached 19.9264 19.901 -0.02542 ops/s
Max Throughput articles_monthly_agg_uncached 19.936 19.9138 -0.02218 ops/s
50th percentile latency articles_monthly_agg_uncached 9.07999 8.94665 -0.13334 ms
90th percentile latency articles_monthly_agg_uncached 9.54059 9.35324 -0.18736 ms
99th percentile latency articles_monthly_agg_uncached 12.4247 10.9521 -1.47259 ms
100th percentile latency articles_monthly_agg_uncached 12.986 13.4554 0.46939 ms
50th percentile service time articles_monthly_agg_uncached 8.35999 8.12095 -0.23903 ms
90th percentile service time articles_monthly_agg_uncached 8.55371 8.28273 -0.27099 ms
99th percentile service time articles_monthly_agg_uncached 11.9086 10.2674 -1.64118 ms
100th percentile service time articles_monthly_agg_uncached 12.6714 12.5234 -0.14791 ms
error rate articles_monthly_agg_uncached 0 0 0 %
Min Throughput articles_monthly_agg_cached 20.0152 20.0165 0.00134 ops/s
Mean Throughput articles_monthly_agg_cached 20.0176 20.0192 0.00157 ops/s
Median Throughput articles_monthly_agg_cached 20.0174 20.0189 0.00155 ops/s
Max Throughput articles_monthly_agg_cached 20.0208 20.0227 0.00189 ops/s
50th percentile latency articles_monthly_agg_cached 3.3985 3.92349 0.52499 ms
90th percentile latency articles_monthly_agg_cached 3.95189 4.36253 0.41064 ms
99th percentile latency articles_monthly_agg_cached 4.21578 4.81747 0.60169 ms
100th percentile latency articles_monthly_agg_cached 4.36166 4.92981 0.56815 ms
50th percentile service time articles_monthly_agg_cached 2.71434 3.10599 0.39165 ms
90th percentile service time articles_monthly_agg_cached 3.04454 3.24139 0.19685 ms
99th percentile service time articles_monthly_agg_cached 3.21404 3.73509 0.52105 ms
100th percentile service time articles_monthly_agg_cached 3.39575 3.80212 0.40637 ms
error rate articles_monthly_agg_cached 0 0 0 %
Min Throughput scroll 12.5498 12.5456 -0.00418 pages/s
Mean Throughput scroll 12.5818 12.575 -0.00689 pages/s
Median Throughput scroll 12.5745 12.5682 -0.0063 pages/s
Max Throughput scroll 12.6476 12.6351 -0.01253 pages/s
50th percentile latency scroll 645.563 753.774 108.211 ms
90th percentile latency scroll 668.882 762.508 93.626 ms
99th percentile latency scroll 687.181 782.12 94.939 ms
100th percentile latency scroll 687.956 786.307 98.3508 ms
50th percentile service time scroll 642.965 751.707 108.742 ms
90th percentile service time scroll 666.54 759.845 93.3048 ms
99th percentile service time scroll 684.884 779.759 94.8755 ms
100th percentile service time scroll 685.32 783.813 98.4922 ms
error rate scroll 0 0 0 %

Copy link
Contributor

❌ Gradle check result for 9d5c6b5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

✅ Gradle check result for 9d5c6b5: SUCCESS

@sachinpkale sachinpkale merged commit 3593def into opensearch-project:main Apr 16, 2025
30 of 31 checks passed
@linuxpi linuxpi deleted the max-merge-at-once branch April 16, 2025 04:32
Lindsay-00 pushed a commit to Lindsay-00/OpenSearch that referenced this pull request Apr 22, 2025
* Increase default maxMergeAtOnce from 10 to 30

Signed-off-by: Varun Bansal <[email protected]>

* add changelog entry

Signed-off-by: Varun Bansal <[email protected]>

* Handle the cases when index setting are removed and added

Signed-off-by: Varun Bansal <[email protected]>

* Improve logging

Signed-off-by: Varun Bansal <[email protected]>

---------

Signed-off-by: Varun Bansal <[email protected]>
Signed-off-by: Lingxi Chen <[email protected]>
Lindsay-00 pushed a commit to Lindsay-00/OpenSearch that referenced this pull request Apr 22, 2025
* Increase default maxMergeAtOnce from 10 to 30

Signed-off-by: Varun Bansal <[email protected]>

* add changelog entry

Signed-off-by: Varun Bansal <[email protected]>

* Handle the cases when index setting are removed and added

Signed-off-by: Varun Bansal <[email protected]>

* Improve logging

Signed-off-by: Varun Bansal <[email protected]>

---------

Signed-off-by: Varun Bansal <[email protected]>
Signed-off-by: Lingxi Chen <[email protected]>
x-INFiN1TY-x pushed a commit to x-INFiN1TY-x/OpenSearch_Local that referenced this pull request Apr 24, 2025
* Increase default maxMergeAtOnce from 10 to 30

Signed-off-by: Varun Bansal <[email protected]>

* add changelog entry

Signed-off-by: Varun Bansal <[email protected]>

* Handle the cases when index setting are removed and added

Signed-off-by: Varun Bansal <[email protected]>

* Improve logging

Signed-off-by: Varun Bansal <[email protected]>

---------

Signed-off-by: Varun Bansal <[email protected]>
Signed-off-by: Tanishq Ranjan <[email protected]>
Harsh-87 pushed a commit to Harsh-87/OpenSearch that referenced this pull request May 7, 2025
* Increase default maxMergeAtOnce from 10 to 30

Signed-off-by: Varun Bansal <[email protected]>

* add changelog entry

Signed-off-by: Varun Bansal <[email protected]>

* Handle the cases when index setting are removed and added

Signed-off-by: Varun Bansal <[email protected]>

* Improve logging

Signed-off-by: Varun Bansal <[email protected]>

---------

Signed-off-by: Varun Bansal <[email protected]>
Signed-off-by: Harsh Kothari <[email protected]>
Harsh-87 pushed a commit to Harsh-87/OpenSearch that referenced this pull request May 7, 2025
* Increase default maxMergeAtOnce from 10 to 30

Signed-off-by: Varun Bansal <[email protected]>

* add changelog entry

Signed-off-by: Varun Bansal <[email protected]>

* Handle the cases when index setting are removed and added

Signed-off-by: Varun Bansal <[email protected]>

* Improve logging

Signed-off-by: Varun Bansal <[email protected]>

---------

Signed-off-by: Varun Bansal <[email protected]>
Signed-off-by: Harsh Kothari <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing:Performance
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Increase maxMergeAtOnce parameter in OpenSearch 3.0
4 participants