Fix checkpoint handling to prevent segment replication infinite loop

[ashking94]

Description

This fixes an issue in segrep flow which causes infinite segrep loop. The analysis is present in the issue - #18605.

Related Issues

Resolves #18605

Check List

• Functionality includes testing.
• [ ] API changes companion pull request created, if applicable.
• [ ] Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

✅ Gradle check result for ff5a898: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 33.33333% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.84%. Comparing base (b1d6f55) to head (ed9e60a).
Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
...s/replication/SegmentReplicationTargetService.java	33.33%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #18636      +/-   ##
============================================
+ Coverage     72.79%   72.84%   +0.05%     
- Complexity    68362    68466     +104     
============================================
  Files          5563     5563              
  Lines        314174   314176       +2     
  Branches      45554    45555       +1     
============================================
+ Hits         228703   228871     +168     
+ Misses        66887    66764     -123     
+ Partials      18584    18541      -43     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Gradle check result for 27e3218: SUCCESS

[ashking94]

I have not been able to repro the issue in ITs. I have tried reproducing it, will try to understand why doesn't it happen on local.

[ashking94]

I have not been able to repro the issue in ITs. I have tried reproducing it, will try to understand why doesn't it happen on local.

I was able to modify an unit test that was showing the behaviour where the cyclic loop of segrep is seen always.

REPRODUCE WITH: ./gradlew 'null' --tests 'org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testStartReplicationListenerSuccess' -Dtests.seed=B17175E6A29DDE01 -Dtests.locale=bg-Cyrl-BG -Dtests.timezone=Europe/Vatican -Druntime.java=21

org.mockito.exceptions.verification.TooManyActualInvocations: 
segmentReplicationTargetService.processLatestReceivedCheckpoint(
    <any>,
    <any>
);
Wanted 1 time:
-> at org.opensearch.indices.replication.SegmentReplicationTargetServiceTests.testStartReplicationListenerSuccess(SegmentReplicationTargetServiceTests.java:628)
But was 27 times:
-> at org.opensearch.indices.replication.SegmentReplicationTargetService.afterIndexShardStarted(SegmentReplicationTargetService.java:235)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)
-> at org.opensearch.indices.replication.SegmentReplicationTargetService$1.onReplicationDone(SegmentReplicationTargetService.java:351)

[github-actions]

✅ Gradle check result for ed9e60a: SUCCESS

[ashking94]

@mch2 I have tried making the above change which should fix the recursion loop. Do review if you can.