Search Replica Allocation and Recovery #17457

vinaykpud · 2025-02-25T21:37:39Z

Description

Modified the search replica allocation based on node attribute
In this PR we restrict Search replica to assign to only nodes with search role
Also in this PR, we have made changes to treat search and regular replicas differently so unable to allocate one do not block the other: [RW Separation] Treat Regular and Search Replicas Separately to Prevent Allocation Blocking #17421
I also fixed the recovery of search replica when there is a node left scenario: [RW Separation] Search replica recovery flow breaks when search shard allocated to new node after node drop #17334

Related Issues

Resolves #17422
Resolves #17421
Resolves #17334
Related to #15306

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

github-actions · 2025-02-25T22:37:03Z

❌ Gradle check result for f9c54c5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-02-26T06:09:53Z

❕ Gradle check result for 2d5b977: UNSTABLE

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2025-02-26T06:10:18Z

Codecov Report

Attention: Patch coverage is 94.02985% with 4 lines in your changes missing coverage. Please review.

Project coverage is 72.34%. Comparing base (dcad6b8) to head (79a8da2).
Report is 20 commits behind head on main.

Files with missing lines	Patch %	Lines
...java/org/opensearch/index/shard/StoreRecovery.java	92.30%	3 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #17457      +/-   ##
============================================
- Coverage     72.46%   72.34%   -0.12%     
+ Complexity    65757    65717      -40     
============================================
  Files          5311     5311              
  Lines        305001   305052      +51     
  Branches      44230    44237       +7     
============================================
- Hits         221022   220696     -326     
- Misses        65932    66280     +348     
- Partials      18047    18076      +29

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

github-actions · 2025-03-19T03:12:40Z

❌ Gradle check result for 1ecdbc3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-19T04:11:04Z

❌ Gradle check result for 1ecdbc3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-19T04:50:50Z

❌ Gradle check result for 1ecdbc3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Vinay Krishna Pudyodu <[email protected]>

github-actions · 2025-03-19T07:01:16Z

❌ Gradle check result for d197bd0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-19T15:15:18Z

❌ Gradle check result for d197bd0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2025-03-19T16:12:11Z

❕ Gradle check result for d197bd0: UNSTABLE

TEST FAILURES:

      1 org.opensearch.snapshots.DedicatedClusterSnapshotRestoreIT.testSnapshotWithStuckNode

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

CHANGELOG.md

...n/java/org/opensearch/cluster/routing/allocation/decider/SearchReplicaAllocationDecider.java

...r/src/main/java/org/opensearch/cluster/routing/allocation/allocator/LocalShardsBalancer.java

Signed-off-by: Vinay Krishna Pudyodu <[email protected]>

github-actions · 2025-03-20T01:01:03Z

❕ Gradle check result for 79a8da2: UNSTABLE

TEST FAILURES:

      1 org.opensearch.action.admin.cluster.node.tasks.ResourceAwareTasksTests.testTaskResourceTrackingDuringTaskCancellation

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

* Restrict Search Replicas to Allocate only to Search dedicated node Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * fixed the javadoc Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * fixed tests Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Treat Regular and Search Replicas Separately to Prevent Allocation Blocking Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Updated tests and some refactor Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fixed SearchReplica recovery scenario for same node and new node Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Updated the logic for SearchReplica recovery scenario for new node Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fixed nits after self review Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Modified the search replica allocation based on node attribute Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * fixed PR comments Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Revert "Fixed SearchReplica recovery scenario for same node and new node" This reverts commit de1e719. Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Separated the recovery flow method for search replica Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Revert "fixed PR comments" This reverts commit 8fe8dcf. Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Added unit tests in IndexShardTests Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * updated method name and minor refactor Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Removed search replica recovery logic from internalRecoverFromStore method Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Added integ test to cover search node restart scenario Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Applied search node role in tests and removed searchonly attribute Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Fixed failing test Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Removed unwanted comment Signed-off-by: Vinay Krishna Pudyodu <[email protected]> * Address PR comments Signed-off-by: Vinay Krishna Pudyodu <[email protected]> --------- Signed-off-by: Vinay Krishna Pudyodu <[email protected]>

github-actions bot added bug Something isn't working Search:Performance labels Feb 25, 2025

opensearch-ci-bot mentioned this pull request Feb 26, 2025

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

vinaykpud marked this pull request as ready for review February 26, 2025 07:53

vinaykpud reopened this Mar 19, 2025

vinaykpud closed this Mar 19, 2025

vinaykpud reopened this Mar 19, 2025

vinaykpud closed this Mar 19, 2025

vinaykpud reopened this Mar 19, 2025

Fixed failing test

501ba77

Signed-off-by: Vinay Krishna Pudyodu <[email protected]>

vinaykpud marked this pull request as ready for review March 19, 2025 06:36

opensearch-ci-bot mentioned this pull request Mar 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for ReactorNetty4StreamingStressIT #15840

Closed

Removed unwanted comment

d197bd0

Signed-off-by: Vinay Krishna Pudyodu <[email protected]>

opensearch-ci-bot mentioned this pull request Mar 19, 2025

[AUTOCUT] Gradle Check Flaky Test Report for S3BlobContainerRetriesTests #17551

Open

vinaykpud closed this Mar 19, 2025

vinaykpud reopened this Mar 19, 2025

vinaykpud closed this Mar 19, 2025

vinaykpud reopened this Mar 19, 2025

mch2 reviewed Mar 19, 2025

View reviewed changes

Address PR comments

79a8da2

Signed-off-by: Vinay Krishna Pudyodu <[email protected]>

opensearch-ci-bot mentioned this pull request Mar 21, 2025

[AUTOCUT] Gradle Check Flaky Test Report for SearchReplicaRestoreIT #17648

Open

mch2 approved these changes Mar 21, 2025

View reviewed changes

mch2 added the skip-changelog label Mar 21, 2025

mch2 merged commit 1acba95 into opensearch-project:main Mar 21, 2025
34 of 35 checks passed

This was referenced Apr 28, 2025

[AUTOCUT] Gradle Check Flaky Test Report for IndexShardTests #14321

Open

[AUTOCUT] Gradle Check Flaky Test Report for SearchIT #18129

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Search Replica Allocation and Recovery #17457

Search Replica Allocation and Recovery #17457

Uh oh!

vinaykpud commented Feb 25, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

github-actions bot commented Feb 26, 2025

Uh oh!

codecov bot commented Feb 26, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 20, 2025

Uh oh!

Uh oh!

Uh oh!

Search Replica Allocation and Recovery #17457

Search Replica Allocation and Recovery #17457

Uh oh!

Conversation

vinaykpud commented Feb 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Feb 25, 2025

Uh oh!

github-actions bot commented Feb 26, 2025

Uh oh!

codecov bot commented Feb 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

github-actions bot commented Mar 19, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Mar 20, 2025

Uh oh!

Uh oh!

Uh oh!

vinaykpud commented Feb 25, 2025 •

edited

Loading

codecov bot commented Feb 26, 2025 •

edited

Loading