Multi-Writer Prevention : Conditional Upload Flow & Logic

[x-INFiN1TY-x]

Related Issues/RFCs:

• Implements the preferred solution ("Versioned & mutable metadata file with Conditional writes") from [RFC] Leverage Conditional APIs for Multi-Writer Detection in Remote Store Clusters ([RFC] Leverage Conditional APIs for Multi-Writer Detection in Remote Store Clusters #17763).
• Addresses data consistency concerns discussed in [Remote translog] Auto restore an index if it has been in red state for atleast x minutes/seconds #6736 and [Remote Translog] Phase 1 - Fail requests on isolated shard if no successful leader checker call in last x minutes to enable auto restore #6737.

---

Problem Statement

OpenSearch clusters using remote-backed storage are susceptible to data inconsistencies when multiple primary shards concurrently attempt to upload segment metadata—particularly during network partitions or primary failovers. A stale (previously active) primary might overwrite metadata written by the newly promoted one. This risk undermines cluster safety and complicates automation in recovery flows.

Current multi-writer detection mechanisms are not robust enough to handle this reliably.

---

Solution Overview

This PR introduces ETag-based conditional writes to the remote segment metadata upload process. ETags (version identifiers from cloud storage systems like S3, GCS, or Azure) allow OpenSearch to safely coordinate access to shared resources. This mechanism ensures only the correct primary shard can write metadata, while stale primaries self-detect and fence themselves.

Key enhancements:

1. ETag-Based Conditional Writes:
   Primary shards attach the known ETag to each metadata upload using the If-Match condition. If the ETag doesn't match the current version in the remote store, the write is rejected (HTTP 412 Precondition Failed).

2. Fixed Metadata Filename:
   To enable ETag-based coordination, segment metadata is now always written to a fixed filename (e.g., "segment_metadata") instead of legacy dynamic filenames.

3. Stale Primary Self-Fencing:

   • On promotion, the new primary performs a non-conditional (forced) metadata upload after clearing its local ETag knowledge. This updates the remote file and its ETag.
   • If the old primary tries to write using a stale ETag, the write fails. This triggers a controlled failShard() operation, fencing off the stale node.

4. ETag Lifecycle Managed at Shard Level:
   IndexShard now caches the ETag for its segment metadata file and updates it based on the success/failure of remote operations.

This design shifts writer validation from OpenSearch into the remote store’s atomic operations—improving correctness and simplifying state coordination.

---

Key Implementation Details

IndexShard

• Introduces a MetadataETagCache per shard to hold the latest known ETag.

• Provides methods:

  • getMetadataETag()
  • updateMetadataETag()
  • clearMetadataETag()

• On primary promotion, invokes initiateNonConditionalRemoteMetadataUpload():

  • Clears cached ETag to trigger an unconditional upload.
  • Performs an overwrite that establishes a new ETag and “claims” primary ownership.
  • Handles transient errors gracefully, relying on future refreshes to retry.

RemoteStoreRefreshListener

• During each metadata upload:

  • Retrieves the current ETag from the shard.
  • Invokes uploadMetadata(...) with the ETag and a structured ActionListener.

• On success: Updates shard’s cached ETag.

• On Precondition Failed: Treats this as a stale primary detection, clears ETag, and calls failShard() for fencing.

• Logs other failures without failing the shard.

RemoteSegmentStoreDirectory

• Accepts a versionIdentifier (ETag) and enhanced ActionListener.

• Constructs ConditionalWriteOptions based on the ETag:

  • If ETag is present → ifMatch
  • If ETag is null → unconditional upload

• Always uses "segment_metadata" as the remote filename.

RemoteDirectory & BlobStore

• copyFrom() method now takes ConditionalWriteOptions.
• Passes them through to the underlying blobContainer.writeBlobConditionally(...) for storage-provider-specific handling.

---

Testing

Unit tests in RemoteSegmentStoreDirectoryTests have been expanded to verify:

• ETag propagation and conditional write correctness.
• Proper fencing behavior on ETag mismatches.
• Correct switching between conditional and unconditional uploads.

Related Issues

• Implements part of RFC #17763

• Parent Meta Issue: [META] Implement Conditional APIs for Multi-Writer Detection

• Supersedes Introduce Interface changes to Support atomic conditional-writes #18065

Check List

• New functionality has been documented.
• Public documentation issue/PR created
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

---

Visualizing the Changes

1. Sequence Diagram: Stale Primary Self-Fencing Mechanism
   
   Demonstrates how an outdated ETag causes a 412 failure, triggering the stale primary’s self-initiated failShard().

2. Architecture: Core Components for Conditional Metadata Upload
   
   Shows interaction flow between IndexShard, RemoteStoreRefreshListener, RemoteSegmentStoreDirectory, RemoteDirectory, and BlobStore, including ETag usage.

3. Flowchart: New Primary’s Metadata Ownership Claim
   
   Shows how a new primary clears its ETag cache, performs a non-conditional upload, and updates its local ETag before assuming control.

[x-INFiN1TY-x]

Please note that this PR depends on the following downstream changes, which are currently under review. Until they are merged, the Gradle build will fail:

opensearch-project/OpenSearch #18064

opensearch-project/OpenSearch #18092

opensearch-project/OpenSearch #18093

[github-actions]

❌ Gradle check result for f1024f3: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?