Skip to content

Introduce Interfaces to Support atomic conditional-writes #18064

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

x-INFiN1TY-x
Copy link

@x-INFiN1TY-x x-INFiN1TY-x commented Apr 24, 2025

Description

This PR introduces interfaces for conditional-write support and defines new option and response types, updating the BlobContainer interface, and extending AsyncMultiStreamBlobContainer to support asynchronous conditional uploads.

  • New Types

    • ConditionalWriteOptions — an object that explicitly models each supported precondition.

    • ConditionalWriteResponse — a typed result that carries provider-generated version identifiers and success status.

  • Conditional APIs Interfaces for BlobContainer

    • writeBlobConditionally

    • writeBlobWithMetadataConditionally

  • Async Conditional API

    • Introduces asyncBlobUploadConditionally to AsyncMultiStreamBlobContainer.

    • Provides the corresponding override in AsyncMultiStreamEncryptedBlobContainer to ensure encrypted uploads honor the same contract.

Supported Preconditions

Precondition Azure AWS S3 GCS
If-None-Match ETag ≠ existing If-None-Match header
If-Match ETag == existing If-Match header generation match
If-Unmodified-Since ifMetagenerationNotModifiedSince

Atomic Write Workflow

  1. Caller constructs a ConditionalWriteOptions specifying the desired precondition.

  2. On success, the blob (and optional metadata) are written, returning a ConditionalWriteResponse that includes the new version identifier for if-match writes.

  3. On failure, the provider’s standard precondition-failed exception is propagated.

Note: The async implementation logic—including multipart handling via AsyncTransferManager—is being delivered in the companion PR: #18093. This PR is scoped purely to defining and aligning the interfaces and types needed to support that capability.

Related Issues

Check List

  • New functionality has been documented.
  • Public documentation issue/PR created
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Copy link
Contributor

✅ Gradle check result for 72cb9c6: SUCCESS

Copy link

codecov bot commented Apr 24, 2025

Codecov Report

Attention: Patch coverage is 0% with 2 lines in your changes missing coverage. Please review.

Project coverage is 72.49%. Comparing base (8948c1f) to head (99e6943).

Files with missing lines Patch % Lines
...org/opensearch/common/blobstore/BlobContainer.java 0.00% 2 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main   #18064      +/-   ##
============================================
- Coverage     72.50%   72.49%   -0.01%     
+ Complexity    67167    67113      -54     
============================================
  Files          5473     5473              
  Lines        310095   310097       +2     
  Branches      45062    45062              
============================================
- Hits         224841   224813      -28     
- Misses        66899    66915      +16     
- Partials      18355    18369      +14     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@skumawat2025
Copy link
Contributor

@x-INFiN1TY-x Please rebase your changes.

Copy link
Contributor

✅ Gradle check result for 99e6943: SUCCESS

@x-INFiN1TY-x x-INFiN1TY-x changed the title Introduce Interface changes to Support atomic conditional-writes Introduce Interfaces to Support atomic conditional-writes Apr 24, 2025
@x-INFiN1TY-x x-INFiN1TY-x marked this pull request as ready for review April 25, 2025 04:46
Signed-off-by: Tanishq Ranjan <[email protected]>
Copy link
Contributor

❌ Gradle check result for 6461fe0: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@x-INFiN1TY-x
Copy link
Author

x-INFiN1TY-x commented Apr 27, 2025

Flaky Test : : #17552

  • SegmentReplicationResizeRequestIT

Copy link
Contributor

@astute-decipher astute-decipher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Member

@ashking94 ashking94 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we add the required interfaces in AsyncMultiStreamBlobContainer?

@x-INFiN1TY-x x-INFiN1TY-x requested a review from a team as a code owner May 28, 2025 12:36
Copy link
Contributor

❌ Gradle check result for 766204d: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Copy link
Contributor

❌ Gradle check result for a928b49: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

@x-INFiN1TY-x
Copy link
Author

x-INFiN1TY-x commented May 28, 2025

Flaky Test : : #17937

  • RethrottleTests.

@x-INFiN1TY-x
Copy link
Author

can we add the required interfaces in AsyncMultiStreamBlobContainer?
cc : @ashking94 @Bukhtawar @astute-decipher

Thanks for raising this,
To enable asynchronous conditional uploads across providers, I’ve introduced the appropriate interface in AsyncMultiStreamBlobContainer, along with the corresponding override in AsyncMultiStreamEncryptedBlobContainer, ensuring consistent support for encrypted use cases.

This PR is specifically focused on defining and aligning the necessary interfaces to support this capability across the relevant components. The implementation details—including multipart handling login via AsyncTransferManager—are addressed in the related PR : #18093.

Please let me know if there’s anything further that needs to be adjusted, here's a diag detailing the intended downstream usage for the concerned interfaces. (Under review in #18093)
Mermaid Chart - Create complex, visual diagrams with text  A smarter way of creating diagrams -2025-05-28-172238

@x-INFiN1TY-x x-INFiN1TY-x requested a review from ashking94 June 2, 2025 05:26
Comment on lines +37 to +43
private ConditionalWriteOptions(Builder builder) {
this.ifNotExists = builder.ifNotExists;
this.ifMatch = builder.ifMatch;
this.ifUnmodifiedSince = builder.ifUnmodifiedSince;
this.versionIdentifier = builder.versionIdentifier;
this.unmodifiedSince = builder.unmodifiedSince;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can there be case where the conditionalWriteOptions has multiple conditions and are contradicting? If yes do we need to have validation for same?

Copy link
Author

@x-INFiN1TY-x x-INFiN1TY-x Jun 11, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants