[Blog] Improve OpenSearch cluster performance by separating search and indexing workloads #3772
Conversation
Signed-off-by: Archer <[email protected]>
Signed-off-by: Archer <[email protected]>
Naarcha-AWS
requested review from
elfisher,
AMoo-Miki,
nknize,
krisfreedain,
peterzhuamazon,
CEHENKLE,
dtaivpp,
kolchfa-aws,
nateynateynate and
natebower
as code owners
There was a problem hiding this comment.
@Naarcha-AWS Editorial review complete. Please see my comment and changes and tag me for approval when addressed. Thanks!
|
|
||
| | | |Cluster 2(5P, 3R) |Cluster 3 (5P, 1R, 1S) |Difference | | ||
| |--- |--- |--- |--- |--- | | ||
| |Index Throughput |Median |58520.82413 |59818.80119 |2.16985 | |
There was a problem hiding this comment.
| |Index Throughput |Median |58520.82413 |59818.80119 |2.16985 | | |
| |Index throughput |Median |58520.82413 |59818.80119 |2.16985 | |
| | |Maximum |59169.29297 |60240.02344 |1.77744 | | ||
| | | | | | | | ||
| | | | | | | | ||
| |Query Latency |p50 |6842.04297 |5358.19263 |-27.69311 | |
There was a problem hiding this comment.
| |Query Latency |p50 |6842.04297 |5358.19263 |-27.69311 | | |
| |Query latency |p50 |6842.04297 |5358.19263 |-27.69311 | |
|
|
||
| ### Result summary | ||
|
|
||
| From these comparisons, we can see that for simultaneous indexing and search workloads, Cluster 3 is slightly better (~2% improvement) and query latency in Cluster 3 is much lower (~34% improvement). This is achieved with ~8% less cost per hour compared to Cluster 1. |
There was a problem hiding this comment.
| From these comparisons, we can see that for simultaneous indexing and search workloads, Cluster 3 is slightly better (~2% improvement) and query latency in Cluster 3 is much lower (~34% improvement). This is achieved with ~8% less cost per hour compared to Cluster 1. | |
| From these comparisons, we can see that Cluster 3 is slightly better (~2% improvement) for simultaneous indexing and search workloads and has much lower query latency (~34% improvement). This is achieved with ~8% lower cost per hour compared to Cluster 1. |
|
|
||
| ## Conclusion | ||
|
|
||
| Separating indexing and search workloads in OpenSearch gives you greater control to scale, tune, and optimize your cluster. Whether you're prioritizing throughput, reducing latency, or managing costs, this approach helps meet the needs of modern applications. |
There was a problem hiding this comment.
| Separating indexing and search workloads in OpenSearch gives you greater control to scale, tune, and optimize your cluster. Whether you're prioritizing throughput, reducing latency, or managing costs, this approach helps meet the needs of modern applications. | |
| Separating indexing and search workloads in OpenSearch gives you greater control over scaling, tuning, and optimizing your cluster. Whether you want to prioritize throughput, reduce latency, or manage costs, this approach helps meet the needs of modern applications. |
|
|
||
| ## Future work | ||
|
|
||
| Currently, coordinator nodes handle both search and indexing requests. For full separation, consider routing indexing and search traffic to distinct coordinator node groups. |
There was a problem hiding this comment.
| Currently, coordinator nodes handle both search and indexing requests. For full separation, consider routing indexing and search traffic to distinct coordinator node groups. | |
| Coordinator nodes currently handle both search and indexing requests. For full separation, consider routing indexing and search traffic to distinct coordinator node groups. |
Signed-off-by: Nathan Bower <[email protected]>
Signed-off-by: Nathan Bower <[email protected]>
There was a problem hiding this comment.
@Naarcha-AWS LGTM!
@pajuric Please update the meta and this will be ready to publish.
Signed-off-by: Naarcha-AWS <[email protected]>
|
@Naarcha-AWS - Is there an open blog issue for this? If not, please open one now so you can be added to the publishing schedule. |
|
@Naarcha-AWS - for following the blog process, you guys just jumped to the front of the line. I'll add the meta, and we'll plan to publish this next Tuesday, May 27. |
| authors: | ||
| - naarcha | ||
| - pudyodu | ||
| date: 2025-05-09 |
| categories: | ||
| - technical-posts | ||
| meta_keywords: OpenSearch, shard roles, indexing, search separation, performance tuning | ||
| meta_description: Learn how separating indexing and search workloads in OpenSearch can improve performance, scalability, and cost efficiency using dedicated roles and node types. |
There was a problem hiding this comment.
Update meta with the following:
meta_keywords: OpenSearch cluster performance, index shards, search workloads, node roles, indexing and search separation, OpenSearch benchmark, performance tuning, optimizing your cluster
meta_description: Optimize OpenSearch cluster performance by separating indexing and search operations. Discover write/search replicas and dedicated search nodes, with benchmark data showing 34% lower query latency through workload separation.
Signed-off-by: Naarcha-AWS <[email protected]>
|
@pajuric: Adjusted. |
|
|
||
| **Comparing Cluster 1 and Cluster 3** | ||
|
|
||
| | | |Cluster 1 (5P, 3R) |Cluster 3 (5P, 1R, 1S) |Difference | |
There was a problem hiding this comment.
Difference is in Percentage in this table and next table. Do we need to annotate it with %?
There was a problem hiding this comment.
Not necessarily, I think its implicit by the numbers. If we do make an update to the blog, I can add this to the list.
|
@natebower - This blog is published and can be closed out. https://opensearch.org/blog/improve-opensearch-cluster-performance-by-separating-search-and-indexing-workloads/ |
Description
Check List
By submitting this pull request, I confirm that my contribution is made under the terms of the BSD-3-Clause License.