[Remote Store] Upload segments to remote store post refresh #3460

sachinpkale · 2022-05-27T08:21:56Z

Description

This PR is combining 4 commits merged to a feature branch feature/durability-enhancements. Following are the PRs corresponding to each commit:

Issues Resolved

[Remote Segment Store] Define RemoteDirectory interface #2993
[Remote Store] Add index level setting to enable/disable remote store #3141
[Remote Segment Store] Add another instance of Store to InternalEngine that uses RemoteDirectory #2994
[Remote Segment Store] Upload new segment files to remote store as a part of refresh flow #2995

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

opensearch-ci-bot · 2022-05-27T08:50:40Z

❌ Gradle Check failure bfc06600d121d96d7d0d45b0ac7828da799036f8
Log 5626

Reports 5626

opensearch-ci-bot · 2022-05-27T09:27:58Z

✅ Gradle Check success 82d55c6
Log 5627

Reports 5627

nknize

First pass through this it isn't clear to me where the line is between internal implementation and external APIs. Allowing third-party plugins to override internals is dangerous (as we've seen w/ the Engine). Can we be more clear here (through access modifiers, annotations, and javadocs) what we're enabling extensions to override? Hopefuly that surface area is minimal so we can carefully control the extension points.

nknize · 2022-05-31T22:42:14Z

server/src/main/java/org/opensearch/index/store/RemoteDirectory.java

+ * the remote store. Implementation of remaining methods will be added as remote store is integrated with
+ * replication, peer recovery etc.
+ */
+public class RemoteDirectory extends Directory {


provides an abstraction layer

It's a concrete implementation that abstracts the storing of files in a remote store, no? Can we make this class final so plugins do not extend? And let's mark w/ javadoc annotation @opensearch.internal as well to document this as an internal class that will not be extended outside the core.

If there is an intent to allow third-party plugins to extend this then we need an abstract class with very thoughtful overrides. Enabling a full scale override of the directory is dangerously trappy.

Right, we don't want third-party plugins to extend RemoteDirectory. I will make it final and mark it @opensearch.internal.

nknize · 2022-05-31T22:47:06Z

server/src/main/java/org/opensearch/index/store/RemoteDirectoryFactory.java

+ *
+ * @opensearch.internal
+ */
+public class RemoteDirectoryFactory implements IndexStorePlugin.RemoteDirectoryFactory {


Why is this internal when the RemoteDirectory class is flagged @opensearch.api? Have we clearly documented what can be (if anything) overriden by third party plugins? If so we should update the javadocs here to include that documentation otherwise we risk plugins recklessly overriding core internals.

RemoteDirectory class should also be marked as @opensearch.internal, will make the change.

nknize · 2022-05-31T22:50:41Z

/cc @saratvemulapalli can you have a look at this as well through the lens of extensions and extension points? Let's be careful to lock down dangerous ports of entry.

mch2 · 2022-05-31T23:09:19Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -325,7 +327,8 @@ public IndexShard(
        final Runnable globalCheckpointSyncer,
        final RetentionLeaseSyncer retentionLeaseSyncer,
        final CircuitBreakerService circuitBreakerService,
-        @Nullable final SegmentReplicationCheckpointPublisher checkpointPublisher
+        @Nullable final SegmentReplicationCheckpointPublisher checkpointPublisher,


I think we'll need to ensure these refresh listeners run in sequence and perhaps combine them when both features are enabled. I'd imagine the segrep checkpoint publisher needs to wait until files have completed upload to a remote store before alerting replicas new segments are available to pull from that store. The alternative is implementing a pull strategy where replicas periodically ping the store for metadata on its latest checkpoint. With node to node replication we are using push through this segrep listener.

As I understand, listeners are called in sequence. If not, as you suggested, we need a mechanism to pass the info on when the upload is complete. I am not sure about combining these listeners though. It will just make them tightly coupled.

I'm not sure there is much risk in creating one listener here given how much these features overlap. Or extending RemoteStoreRefreshListener to take a SegmentReplicationCheckpointPublisher and publishing within the remote store listener only if the segrep index setting is enabled. We'd then disable the checkpointListener if remote store & segrep are both enabled. Hiding all of that logic in a single listener makes sense to me.

Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

opensearch-ci-bot · 2022-06-01T04:55:17Z

✅ Gradle Check success 2a4c5a0
Log 5699

Reports 5699

andrross · 2022-06-01T20:24:57Z

server/src/main/java/org/opensearch/index/IndexService.java

+            RemoteStoreRefreshListener remoteStoreRefreshListener = null;
+            if (this.indexSettings.isRemoteStoreEnabled()) {
+                try {
+                    Repository repository = repositoriesService.repository("dragon-stone");


Is there any reason not to put the repository name into the index settings now? I know this isn't finalized as it is still behind an experimental feature flag, but it seems like you could replace the boolean with a repository name to avoid hardcoding a name.

@andrross We may not go ahead with index specific repositories. A repository for remote store will be created for the cluster and data of each remote_store enabled index will be stored in the same repository (open to suggestions). We are tracking the required change here: #3286

Could the repository name be the same as clusterUUID?

In the current change, we hard-coded repository name as dragon-stone but we would be making it configurable (as part of #3286). IMO, it is better to get this input from user instead of using a pre-defined name. This is mostly to avoid conflict where the repository with the same name already exists.

@sachinpkale what is the reason to couple IndexService with RepositoriesService? I think the RemoteDirectoryFactory should make the decision what kind of the repository should be created (by name or by index/cluster settings).

If we add logic of fetching repository name in RemoteDirectoryFactory, it would tightly couple remote directory with the repository name. It should be caller's responsibility to provide repository to be used.

Not really. If the repository name is provided by caller (in-place of Repository instance), not only we decouple the IndexService from RepositoriesService but we also open the possibilities to create remote directory instances which are not backed by Repository.

This makes it extensible to create remote directory instance for another repository as well (For example, if we want to re-use remote directory for snapshots in the future).

That is done by name in any case, since in both cases it would reference same repository. IndexService should not deal with repositories / snapshotting APIs, nor depend on it.

If the repository name is provided by caller

Are you suggesting to pass RepositoriesService as well as RepositoryName to newDirectory method?

Just repository name, the factory could inject RepositoriesService (or any other means to reference the source)

Tracking it here: #3547

Why would the repository name need to differ from the cluster name or UUID? I don't imagine we expect a many repositories to one cluster relationship? I also think this is a nice way to link the decoupled remote storage to the cluster?

@nknize I changed the repository name to cluster UUID.

sachinpkale · 2022-06-06T05:52:18Z

@nknize I have resolved/replied on your comments. Can you please take a look?

reta · 2022-06-06T17:21:48Z

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

+    @Override
+    public void afterRefresh(boolean didRefresh) throws IOException {
+        if (didRefresh) {
+            Set<String> localFiles = Arrays.stream(storeDirectory.listAll()).collect(Collectors.toSet());


Please, Set<String> localFiles = Set.of(storeDirectory.listAll());

Changed as suggested.

opensearch-ci-bot · 2022-06-08T09:44:49Z

❌ Gradle Check failure a71332f21cae87e50f4ff7d3ee53a1a0cbeba30d
Log 5851

Reports 5851

opensearch-ci-bot · 2022-06-08T10:03:11Z

✅ Gradle Check success 2f5957fa42ae9138e57046cc741de77057b29ef1
Log 5852

Reports 5852

Bukhtawar

Lets add ITs for remote store upload

Bukhtawar · 2022-06-09T13:04:31Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

@@ -325,7 +327,8 @@ public IndexShard(
        final Runnable globalCheckpointSyncer,
        final RetentionLeaseSyncer retentionLeaseSyncer,
        final CircuitBreakerService circuitBreakerService,
-        @Nullable final SegmentReplicationCheckpointPublisher checkpointPublisher
+        @Nullable final SegmentReplicationCheckpointPublisher checkpointPublisher,
+        final RemoteStoreRefreshListener remoteStoreRefreshListener


Lets make it Nullable

Bukhtawar · 2022-06-09T13:16:35Z

server/src/main/java/org/opensearch/index/shard/IndexShard.java

+        if (remoteStoreRefreshListener != null && shardRouting.primary()) {
+            internalRefreshListener.add(remoteStoreRefreshListener);
+        }


Lets inject the required listener in the indexShard constructor

Tracking as part of: #3548

Bukhtawar · 2022-06-09T13:37:36Z

server/src/main/java/org/opensearch/index/shard/RemoteStoreRefreshListener.java

+    private final Set<String> filesUploadedToRemoteStore;
+    private static final Logger logger = LogManager.getLogger(RemoteStoreRefreshListener.class);
+
+    public RemoteStoreRefreshListener(Directory storeDirectory, Directory remoteDirectory) throws IOException {


Lets create a NOOPRemoteStoreRefreshListener for replica

Tracking here: #3548

opensearch-ci-bot · 2022-06-09T14:02:39Z

❌ Gradle Check failure 63276935344b5135be5b5fdb43efceeefbefbe0e
Log 5868

Reports 5868

…hListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]>

Signed-off-by: Sachin Kale <[email protected]>

opensearch-ci-bot · 2022-06-09T15:54:55Z

✅ Gradle Check success eb10c3c
Log 5869

Reports 5869

Changes have been addressed

* Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

…ch-project#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

* Bump reactor-netty-core from 1.0.16 to 1.0.19 in /plugins/repository-azure (#3360) * Bump reactor-netty-core in /plugins/repository-azure Bumps [reactor-netty-core](https://github.com/reactor/reactor-netty) from 1.0.16 to 1.0.19. - [Release notes](https://github.com/reactor/reactor-netty/releases) - [Commits](reactor/reactor-netty@v1.0.16...v1.0.19) --- updated-dependencies: - dependency-name: io.projectreactor.netty:reactor-netty-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [Type removal] _type removal from mocked responses of scroll hit tests (#3377) Signed-off-by: Suraj Singh <[email protected]> * [Type removal] Remove _type deprecation from script and conditional processor (#3239) * [Type removal] Remove _type deprecation from script and conditional processor Signed-off-by: Suraj Singh <[email protected]> * Spotless check apply Signed-off-by: Suraj Singh <[email protected]> * [Type removal] Remove _type from _bulk yaml test, scripts, unused constants (#3372) * [Type removal] Remove redundant _type deprecation checks in bulk request Signed-off-by: Suraj Singh <[email protected]> * [Type removal] bulk yaml tests validating deprecation on _type and removal from scripts Signed-off-by: Suraj Singh <[email protected]> * Fix Lucene-snapshots repo for jdk 17. (#3396) Signed-off-by: Marc Handalian <[email protected]> * Replace internal usages of 'master' term in 'server/src/internalClusterTest' directory (#2521) Signed-off-by: Tianli Feng <[email protected]> * [REMOVE] Cleanup deprecated thread pool types (FIXED_AUTO_QUEUE_SIZE) (#3369) Signed-off-by: Andriy Redko <[email protected]> * [Type removal] _type removal from tests of yaml tests (#3406) * [Type removal] _type removal from tests of yaml tests Signed-off-by: Suraj Singh <[email protected]> * Fix spotless failures Signed-off-by: Suraj Singh <[email protected]> * Fix assertion failures Signed-off-by: Suraj Singh <[email protected]> * Fix assertion failures in DoSectionTests Signed-off-by: Suraj Singh <[email protected]> * Add release notes for version 2.0.0 (#3410) Signed-off-by: Rabi Panda <[email protected]> * [Upgrade] Lucene-9.2.0-snapshot-ba8c3a8 (#3416) Upgrades to latest snapshot of lucene 9.2.0 in preparation for GA release. Signed-off-by: Nicholas Walter Knize <[email protected]> * Fix release notes for 2.0.0-rc1 version (#3418) This change removes some old commits from the 2.0.0-rc1 release notes. These commits were already released as part of 1.x releases. Add back some missing type removal commits to the 2.0.0 release notes Signed-off-by: Rabi Panda <[email protected]> * Bump version 2.1 to Lucene 9.2 after upgrade (#3424) Bumps Version.V_2_1_0 lucene version to 9.2 after backporting upgrage. Signed-off-by: Nicholas Walter Knize <[email protected]> * Bump com.gradle.enterprise from 3.10 to 3.10.1 (#3425) Bumps com.gradle.enterprise from 3.10 to 3.10.1. --- updated-dependencies: - dependency-name: com.gradle.enterprise dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump reactor-core from 3.4.17 to 3.4.18 in /plugins/repository-azure (#3427) Bumps [reactor-core](https://github.com/reactor/reactor-core) from 3.4.17 to 3.4.18. - [Release notes](https://github.com/reactor/reactor-core/releases) - [Commits](reactor/reactor-core@v3.4.17...v3.4.18) --- updated-dependencies: - dependency-name: io.projectreactor:reactor-core dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump gax-httpjson from 0.101.0 to 0.103.1 in /plugins/repository-gcs (#3426) Bumps [gax-httpjson](https://github.com/googleapis/gax-java) from 0.101.0 to 0.103.1. - [Release notes](https://github.com/googleapis/gax-java/releases) - [Changelog](https://github.com/googleapis/gax-java/blob/main/CHANGELOG.md) - [Commits](https://github.com/googleapis/gax-java/commits) --- updated-dependencies: - dependency-name: com.google.api:gax-httpjson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * [segment replication]Introducing common Replication interfaces for segment replication and recovery code paths (#3234) * RecoveryState inherits from ReplicationState + RecoveryTarget inherits from ReplicationTarget Signed-off-by: Poojita Raj <[email protected]> * Refactoring: mixedClusterVersion error fix + move Stage to ReplicationState Signed-off-by: Poojita Raj <[email protected]> * pull ReplicationListener into a top level class + add javadocs + address review comments Signed-off-by: Poojita Raj <[email protected]> * fix javadoc Signed-off-by: Poojita Raj <[email protected]> * review changes Signed-off-by: Poojita Raj <[email protected]> * Refactoring the hierarchy relationship between repl and recovery Signed-off-by: Poojita Raj <[email protected]> * style fix Signed-off-by: Poojita Raj <[email protected]> * move package common under replication Signed-off-by: Poojita Raj <[email protected]> * rename to replication Signed-off-by: Poojita Raj <[email protected]> * rename and doc changes Signed-off-by: Poojita Raj <[email protected]> * [Type removal] Remove type from BulkRequestParser (#3423) * [Type removal] Remove type handling in bulk request parser Signed-off-by: Suraj Singh <[email protected]> * [Type removal] Remove testTypesStillParsedForBulkMonitoring as it is no longer present in codebase Signed-off-by: Suraj Singh <[email protected]> * Adding CheckpointRefreshListener to trigger when Segment replication is turned on and Primary shard refreshes (#3108) * Intial PR adding classes and tests related to checkpoint publishing Signed-off-by: Rishikesh1159 <[email protected]> * Putting a Draft PR with all changes in classes. Testing is still not included in this commit. Signed-off-by: Rishikesh1159 <[email protected]> * Wiring up index shard to new engine, spotless apply and removing unnecessary tests and logs Signed-off-by: Rishikesh1159 <[email protected]> * Adding Unit test for checkpointRefreshListener Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotless check Signed-off-by: Rishikesh1159 <[email protected]> * Fixing import statements * Signed-off-by: Rishikesh1159 <[email protected]> * removing unused constructor in index shard Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments from last commit Signed-off-by: Rishikesh1159 <[email protected]> * Adding package-info.java files for two new packages Signed-off-by: Rishikesh1159 <[email protected]> * Adding test for null checkpoint publisher and addreesing PR comments Signed-off-by: Rishikesh1159 <[email protected]> * Add docs for indexshardtests and remove shard.refresh Signed-off-by: Rishikesh1159 <[email protected]> * Add a new Engine implementation for replicas with segment replication enabled. (#3240) * Change fastForwardProcessedSeqNo method in LocalCheckpointTracker to persisted checkpoint. This change inverts fastForwardProcessedSeqNo to fastForwardPersistedSeqNo for use in Segment Replication. This is so that a Segrep Engine can match the logic of InternalEngine where the seqNo is incremented with each operation, but only persisted in the tracker on a flush. With Segment Replication we bump the processed number with each operation received index/delete/noOp, and invoke this method when we receive a new set of segments to bump the persisted seqNo. Signed-off-by: Marc Handalian <[email protected]> * Extract Translog specific engine methods into an abstract class. This change extracts translog specific methods to an abstract engine class so that other engine implementations can reuse translog logic. Signed-off-by: Marc Handalian <[email protected]> * Add a separate Engine implementation for replicas with segment replication enabled. This change adds a new engine intended to be used on replicas with segment replication enabled. This engine does not wire up an IndexWriter, but still writes all operations to a translog. The engine uses a new ReaderManager that refreshes from an externally provided SegmentInfos. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless checks. Signed-off-by: Marc Handalian <[email protected]> * Fix :server:compileInternalClusterTestJava compilation. Signed-off-by: Marc Handalian <[email protected]> * Fix failing test naming convention check. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. - Removed isReadOnlyReplica from overloaded constructor and added feature flag checks. - Updated log msg in NRTReplicationReaderManager - cleaned up store ref counting in NRTReplicationEngine. Signed-off-by: Marc Handalian <[email protected]> * Fix spotless check. Signed-off-by: Marc Handalian <[email protected]> * Remove TranslogAwareEngine and build translog in NRTReplicationEngine. Signed-off-by: Marc Handalian <[email protected]> * Fix formatting Signed-off-by: Marc Handalian <[email protected]> * Add missing translog methods to NRTEngine. Signed-off-by: Marc Handalian <[email protected]> * Remove persistent seqNo check from fastForwardProcessedSeqNo. Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> * Add test specific to translog trimming. Signed-off-by: Marc Handalian <[email protected]> * Javadoc check. Signed-off-by: Marc Handalian <[email protected]> * Add failEngine calls to translog methods in NRTReplicationEngine. Roll xlog generation on replica when a new commit point is received. Signed-off-by: Marc Handalian <[email protected]> * Rename master to cluster_manager in the XContent Parser of ClusterHealthResponse (#3432) Signed-off-by: Tianli Feng <[email protected]> * Bump hadoop-minicluster in /test/fixtures/hdfs-fixture (#3359) Bumps hadoop-minicluster from 3.3.2 to 3.3.3. --- updated-dependencies: - dependency-name: org.apache.hadoop:hadoop-minicluster dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs (#3358) * Bump avro from 1.10.2 to 1.11.0 in /plugins/repository-hdfs Bumps avro from 1.10.2 to 1.11.0. --- updated-dependencies: - dependency-name: org.apache.avro:avro dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Fix testSetAdditionalRolesCanAddDeprecatedMasterRole() by removing the initial assertion (#3441) Signed-off-by: Tianli Feng <[email protected]> * Replace internal usages of 'master' term in 'server/src/test' directory (#2520) * Replace the non-inclusive terminology "master" with "cluster manager" in code comments, internal variable/method/class names, in `server/src/test` directory. * Backwards compatibility is not impacted. * Add a new unit test `testDeprecatedMasterNodeFilter()` to validate using `master:true` or `master:false` can filter the node in [Cluster Stats](https://opensearch.org/docs/latest/opensearch/rest-api/cluster-stats/) API, after the `master` role is deprecated in PR #2424 Signed-off-by: Tianli Feng <[email protected]> * Removing unused method from TransportSearchAction (#3437) * Removing unused method from TransportSearchAction Signed-off-by: Ankit Jain <[email protected]> * Set term vector flags to false for ._index_prefix field (#1901). (#3119) * Set term vector flags to false for ._index_prefix field (#1901). Signed-off-by: Vesa Pehkonen <[email protected]> * Replaced the FieldType copy ctor with ctor for the prefix field and replaced setting the field type parameters with setIndexOptions(). (#1901) Signed-off-by: Vesa Pehkonen <[email protected]> * Added tests for term vectors. (#1901) Signed-off-by: Vesa Pehkonen <[email protected]> * Fixed code formatting error. Signed-off-by: Vesa Pehkonen <[email protected]> Co-authored-by: sdp <[email protected]> * [BUG] Fixing org.opensearch.monitor.os.OsProbeTests > testLogWarnCpuMessageOnlyOnes when cgroups are available but cgroup stats is not (#3448) Signed-off-by: Andriy Redko <[email protected]> * [Segment Replication] Add SegmentReplicationTargetService to orchestrate replication events. (#3439) * Add SegmentReplicationTargetService to orchestrate replication events. This change introduces boilerplate classes for Segment Replication and a target service to orchestrate replication events. It also includes two refactors of peer recovery components for reuse. 1. Rename RecoveryFileChunkRequest to FileChunkRequest and extract code to handle throttling into ReplicationTarget. 2. Extracts a component to execute retryable requests over the transport layer. Signed-off-by: Marc Handalian <[email protected]> * Code cleanup. Signed-off-by: Marc Handalian <[email protected]> * Make SegmentReplicationTargetService component final so that it can not be extended by plugins. Signed-off-by: Marc Handalian <[email protected]> * Bump azure-core-http-netty from 1.11.9 to 1.12.0 in /plugins/repository-azure (#3474) Bumps [azure-core-http-netty](https://github.com/Azure/azure-sdk-for-java) from 1.11.9 to 1.12.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-core-http-netty_1.11.9...azure-core_1.12.0) --- updated-dependencies: - dependency-name: com.azure:azure-core-http-netty dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update to Apache Lucene 9.2 (#3477) Signed-off-by: Andriy Redko <[email protected]> * Bump protobuf-java from 3.20.1 to 3.21.1 in /plugins/repository-hdfs (#3472) Signed-off-by: dependabot[bot] <[email protected]> * [Upgrade] Lucene-9.3.0-snapshot-823df23 (#3478) Upgrades to latest snapshot of lucene 9.3.0. Signed-off-by: Nicholas Walter Knize <[email protected]> * Filter out invalid URI and HTTP method in the error message of no handler found for a REST request (#3459) Filter out invalid URI and HTTP method of a error message, which shown when there is no handler found for a REST request sent by user, so that HTML special characters <>&"' will not shown in the error message. The error message is return as mine-type `application/json`, which can't contain active (script) content, so it's not a vulnerability. Besides, no browsers are going to render as html when the mine-type is that. While the common security scanners will raise a false-positive alarm for having HTML tags in the response without escaping the HTML special characters, so the solution only aims to satisfy the code security scanners. Signed-off-by: Tianli Feng <[email protected]> * Support use of IRSA for repository-s3 plugin credentials (#3475) * Support use of IRSA for repository-s3 plugin credentials Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> * Address code review comments Signed-off-by: Andriy Redko <[email protected]> * Bump google-auth-library-oauth2-http from 0.20.0 to 1.7.0 in /plugins/repository-gcs (#3473) * Bump google-auth-library-oauth2-http in /plugins/repository-gcs Bumps google-auth-library-oauth2-http from 0.20.0 to 1.7.0. --- updated-dependencies: - dependency-name: com.google.auth:google-auth-library-oauth2-http dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Use variable to define the version of dependency google-auth-library-java Signed-off-by: Tianli Feng <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Tianli Feng <[email protected]> * [Segment Replication] Added source-side classes for orchestrating replication events (#3470) This change expands on the existing SegmentReplicationSource interface and its corresponding Factory class by introducing an implementation where the replication source is a primary shard (PrimaryShardReplicationSource). These code paths execute on the target. The primary shard implementation creates the requests to be send to the source/primary shard. Correspondingly, this change also defines two request classes for the GET_CHECKPOINT_INFO and GET_SEGMENT_FILES requests as well as an abstract superclass. A CopyState class has been introduced that captures point-in-time, file-level details from an IndexShard. This implementation mirrors Lucene's NRT CopyState implementation. Finally, a service class has been introduce for segment replication that runs on the source side (SegmentReplicationSourceService) which handles these two types of incoming requests. This includes private handler classes that house the logic to respond to these requests, with some functionality stubbed for now. The service class also uses a simple map to cache CopyState objects that would be needed by replication targets. Unit tests have been added/updated for all new functionality. Signed-off-by: Kartik Ganesh <[email protected]> * [Dependency upgrade] google-oauth-client to 1.33.3 (#3500) Signed-off-by: Suraj Singh <[email protected]> * move bash flag to set statement (#3494) Passing bash with flags to the first argument of /usr/bin/env requires its own flag to interpret it correctly. Rather than use `env -S` to split the argument, have the script `set -e` to enable the same behavior explicitly in preinst and postinst scripts. Also set `-o pipefail` for consistency. Closes: #3492 Signed-off-by: Cole White <[email protected]> * Support use of IRSA for repository-s3 plugin credentials: added YAML Rest test case (#3499) Signed-off-by: Andriy Redko <[email protected]> * Bump azure-storage-common from 12.15.0 to 12.16.0 in /plugins/repository-azure (#3517) * Bump azure-storage-common in /plugins/repository-azure Bumps [azure-storage-common](https://github.com/Azure/azure-sdk-for-java) from 12.15.0 to 12.16.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.15.0...azure-storage-blob_12.16.0) --- updated-dependencies: - dependency-name: com.azure:azure-storage-common dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce (#3516) * Bump google-oauth-client from 1.33.3 to 1.34.0 in /plugins/discovery-gce Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.33.3 to 1.34.0. - [Release notes](https://github.com/googleapis/google-oauth-java-client/releases) - [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-oauth-java-client@v1.33.3...v1.34.0) --- updated-dependencies: - dependency-name: com.google.oauth-client:google-oauth-client dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Fix the support of RestClient Node Sniffer for version 2.x and update tests (#3487) Fix the support of RestClient Node Sniffer for OpenSearch 2.x, and update unit tests for OpenSearch. The current code contains the logic to be compatible with Elasticsearch 2.x version, which is conflict with OpenSearch 2.x, so removed that part of legacy code. * Update the script create_test_nodes_info.bash to dump the response of Nodes Info API GET _nodes/http for OpenSearch 1.0 and 2.0 version, which used for unit test. * Remove the support of Elasticsearch version 2.x for the Sniffer * Update unit test to validate the Sniffer compatible with OpenSearch 1.x and 2.x * Update the API response parser to meet the array notation (in ES 6.1 and above) for the node attributes setting. It will result the value of `node.attr` setting will not be parsed as array in the Sniffer, when using the Sniffer on cluster in Elasticsearch 6.0 and above. * Replace "master" node role with "cluster_manager" in unit test Signed-off-by: Tianli Feng <[email protected]> * Bump com.diffplug.spotless from 6.6.1 to 6.7.0 (#3513) Bumps com.diffplug.spotless from 6.6.1 to 6.7.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment (#3357) * Bump guava from 18.0 to 23.0 in /plugins/ingest-attachment Bumps [guava](https://github.com/google/guava) from 18.0 to 23.0. - [Release notes](https://github.com/google/guava/releases) - [Commits](google/guava@v18.0...v23.0) --- updated-dependencies: - dependency-name: com.google.guava:guava dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Add more ingorance of using internal java API sun.misc.Unsafe Signed-off-by: Tianli Feng <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Tianli Feng <[email protected]> * Added bwc version 2.0.1 (#3452) Signed-off-by: Kunal Kotwani <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> * Add release notes for 1.3.3 (#3549) Signed-off-by: Xue Zhou <[email protected]> * [Upgrade] Lucene-9.3.0-snapshot-b7231bb (#3537) Upgrades to latest snapshot of lucene 9.3; including reducing maxFullFlushMergeWaitMillis in LuceneTest.testWrapLiveDocsNotExposeAbortedDocuments to 0 ms to ensure aborted docs are not merged away in the test with the new mergeOnRefresh default policy. Signed-off-by: Nicholas Walter Knize <[email protected]> * [Remote Store] Upload segments to remote store post refresh (#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Fixing VerifyVersionConstantsIT test failure (#3574) Signed-off-by: Andriy Redko <[email protected]> * Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic (#3571) * Bump jettison from 1.4.1 to 1.5.0 in /plugins/discovery-azure-classic Bumps [jettison](https://github.com/jettison-json/jettison) from 1.4.1 to 1.5.0. - [Release notes](https://github.com/jettison-json/jettison/releases) - [Commits](jettison-json/jettison@jettison-1.4.1...jettison-1.5.0) --- updated-dependencies: - dependency-name: org.codehaus.jettison:jettison dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1 in /plugins/repository-gcs (#3573) * Bump google-api-services-storage in /plugins/repository-gcs Bumps google-api-services-storage from v1-rev20200814-1.30.10 to v1-rev20220608-1.32.1. --- updated-dependencies: - dependency-name: com.google.apis:google-api-services-storage dependency-type: direct:production ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Upgrade Google HTTP Client to 1.42.0 Signed-off-by: Xue Zhou <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Xue Zhou <[email protected]> * Add flat_skew setting to node overload decider (#3563) * Add flat_skew setting to node overload decider Signed-off-by: Rishab Nahata <[email protected]> * Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment (#3572) * Bump xmlbeans from 5.0.3 to 5.1.0 in /plugins/ingest-attachment Bumps xmlbeans from 5.0.3 to 5.1.0. --- updated-dependencies: - dependency-name: org.apache.xmlbeans:xmlbeans dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce (#3570) * Bump google-oauth-client from 1.34.0 to 1.34.1 in /plugins/discovery-gce Bumps [google-oauth-client](https://github.com/googleapis/google-oauth-java-client) from 1.34.0 to 1.34.1. - [Release notes](https://github.com/googleapis/google-oauth-java-client/releases) - [Changelog](https://github.com/googleapis/google-oauth-java-client/blob/main/CHANGELOG.md) - [Commits](googleapis/google-oauth-java-client@v1.34.0...v1.34.1) --- updated-dependencies: - dependency-name: com.google.oauth-client:google-oauth-client dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> * Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider (#3428) * Fix for bug showing incorrect awareness attributes count in AwarenessAllocationDecider Signed-off-by: Anshu Agarwal <[email protected]> * Added bwc version 1.3.4 (#3552) Signed-off-by: GitHub <[email protected]> Co-authored-by: opensearch-ci-bot <[email protected]> * Support dynamic node role (#3436) * Support unknown node role Currently OpenSearch only supports several built-in nodes like data node role. If specify unknown node role, OpenSearch node will fail to start. This limit how to extend OpenSearch to support some extension function. For example, user may prefer to run ML tasks on some dedicated node which doesn't serve as any built-in node roles. So the ML tasks won't impact OpenSearch core function. This PR removed the limitation and user can specify any node role and OpenSearch will start node correctly with that unknown role. This opens the door for plugin developer to run specific tasks on dedicated nodes. Issue: #2877 Signed-off-by: Yaliang Wu <[email protected]> * fix cat nodes rest API spec Signed-off-by: Yaliang Wu <[email protected]> * fix mixed cluster IT failure Signed-off-by: Yaliang Wu <[email protected]> * add DynamicRole Signed-off-by: Yaliang Wu <[email protected]> * change generator method name Signed-off-by: Yaliang Wu <[email protected]> * fix failed docker test Signed-off-by: Yaliang Wu <[email protected]> * transform role name to lower case to avoid confusion Signed-off-by: Yaliang Wu <[email protected]> * transform the node role abbreviation to lower case Signed-off-by: Yaliang Wu <[email protected]> * fix checkstyle Signed-off-by: Yaliang Wu <[email protected]> * add test for case-insensitive role name change Signed-off-by: Yaliang Wu <[email protected]> * Rename package 'o.o.action.support.master' to 'o.o.action.support.clustermanager' (#3556) * Rename package org.opensearch.action.support.master to org.opensearch.action.support.clustermanager Signed-off-by: Tianli Feng <[email protected]> * Rename classes with master term in the package org.opensearch.action.support.master Signed-off-by: Tianli Feng <[email protected]> * Deprecate classes in org.opensearch.action.support.master Signed-off-by: Tianli Feng <[email protected]> * Remove pakcage o.o.action.support.master Signed-off-by: Tianli Feng <[email protected]> * Move package-info back Signed-off-by: Tianli Feng <[email protected]> * Move package-info to new folder Signed-off-by: Tianli Feng <[email protected]> * Correct the package-info Signed-off-by: Tianli Feng <[email protected]> * Fixing flakiness of ShuffleForcedMergePolicyTests (#3591) Signed-off-by: Andriy Redko <[email protected]> * Deprecate classes in org.opensearch.action.support.master (#3593) Signed-off-by: Tianli Feng <[email protected]> * Add release notes for version 2.0.1 (#3595) Signed-off-by: Kunal Kotwani <[email protected]> * Fix NPE when minBound/maxBound is not set before being called. (#3605) Signed-off-by: George Apaaboah <[email protected]> * Added bwc version 2.0.2 (#3613) Co-authored-by: opensearch-ci-bot <[email protected]> * Fix false positive query timeouts due to using cached time (#3454) * Fix false positive query timeouts due to using cached time Signed-off-by: Ahmad AbuKhalil <[email protected]> * delegate nanoTime call to SearchContext Signed-off-by: Ahmad AbuKhalil <[email protected]> * add override to SearchContext getRelativeTimeInMillis to force non cached time Signed-off-by: Ahmad AbuKhalil <[email protected]> * Fix random gradle check failure issue 3584. (#3627) * [Segment Replication] Add components for segment replication to perform file copy. (#3525) * Add components for segment replication to perform file copy. This change adds the required components to SegmentReplicationSourceService to initiate copy and react to lifecycle events. Along with new components it refactors common file copy code from RecoverySourceHandler into reusable pieces. Signed-off-by: Marc Handalian <[email protected]> * Deprecate public methods and variables with master term in package 'org.opensearch.action.support.master' (#3617) Signed-off-by: Tianli Feng <[email protected]> * Add replication orchestration for a single shard (#3533) * implement segment replication target Signed-off-by: Poojita Raj <[email protected]> * test added Signed-off-by: Poojita Raj <[email protected]> * changes to tests + finalizeReplication Signed-off-by: Poojita Raj <[email protected]> * fix style check Signed-off-by: Poojita Raj <[email protected]> * addressing comments + fix gradle check Signed-off-by: Poojita Raj <[email protected]> * added test + addressed review comments Signed-off-by: Poojita Raj <[email protected]> * [BUG] opensearch crashes on closed client connection before search reply (#3626) * [BUG] opensearch crashes on closed client connection before search reply Signed-off-by: Andriy Redko <[email protected]> * Addressing code review comments Signed-off-by: Andriy Redko <[email protected]> * Add all deprecated method in the package with new name 'org.opensearch.action.support.clustermanager' (#3644) Signed-off-by: Tianli Feng <[email protected]> * Introduce TranslogManager implementations decoupled from the Engine (#3638) * Introduce decoupled translog manager interfaces Signed-off-by: Bukhtawar Khan <[email protected]> * Adding onNewCheckpoint to Start Replication on Replica Shard when Segment Replication is turned on (#3540) * Adding onNewCheckpoint and it's test to start replication. SCheck for latestcheckpoint and replaying logic is removed from this commit and will be added in a different PR Signed-off-by: Rishikesh1159 <[email protected]> * Changing binding/inject logic and addressing comments from PR Signed-off-by: Rishikesh1159 <[email protected]> * Applying spotless check Signed-off-by: Rishikesh1159 <[email protected]> * Moving shouldProcessCheckpoint() to IndexShard, and removing some trace logs Signed-off-by: Rishikesh1159 <[email protected]> * applying spotlessApply Signed-off-by: Rishikesh1159 <[email protected]> * Adding more info to log statement in targetservice class Signed-off-by: Rishikesh1159 <[email protected]> * applying spotlessApply Signed-off-by: Rishikesh1159 <[email protected]> * Addressing comments on PR Signed-off-by: Rishikesh1159 <[email protected]> * Adding teardown() in SegmentReplicationTargetServiceTests. Signed-off-by: Rishikesh1159 <[email protected]> * fixing testShouldProcessCheckpoint() in SegmentReplicationTargetServiceTests Signed-off-by: Rishikesh1159 <[email protected]> * Removing CheckpointPublisherProvider in IndicesModule Signed-off-by: Rishikesh1159 <[email protected]> * spotless check apply Signed-off-by: Rishikesh1159 <[email protected]> * Remove class org.opensearch.action.support.master.AcknowledgedResponse (#3662) * Remove class org.opensearch.action.support.master.AcknowledgedResponse Signed-off-by: Tianli Feng <[email protected]> * Remove class org.opensearch.action.support.master.AcknowledgedRequest RequestBuilder ShardsAcknowledgedResponse Signed-off-by: Tianli Feng <[email protected]> * Restore AcknowledgedResponse and AcknowledgedRequest to package org.opensearch.action.support.master (#3669) Signed-off-by: Tianli Feng <[email protected]> * [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm) (#3656) * [BUG] Custom POM configuration for ZIP publication produces duplicit tags (url, scm) Signed-off-by: Andriy Redko <[email protected]> * Added test case for pluginZip with POM Signed-off-by: Andriy Redko <[email protected]> * Support both Gradle 6.8.x and Gradle 7.4.x Signed-off-by: Andriy Redko <[email protected]> * Adding 2.2.0 Bwc version to main (#3673) * Upgraded to t-digest 3.3. (#3634) * Revert renaming method onMaster() and offMaster() in interface LocalNodeMasterListener (#3686) Signed-off-by: Tianli Feng <[email protected]> * Upgrading AWS SDK dependency for native plugins (#3694) * Merge branch 'feature/point_in_time' of https://github.com/opensearch-project/OpenSearch into fb Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> Co-authored-by: Suraj Singh <[email protected]> Co-authored-by: Marc Handalian <[email protected]> Co-authored-by: Tianli Feng <[email protected]> Co-authored-by: Andriy Redko <[email protected]> Co-authored-by: Rabi Panda <[email protected]> Co-authored-by: Nick Knize <[email protected]> Co-authored-by: Poojita Raj <[email protected]> Co-authored-by: Rishikesh Pasham <[email protected]> Co-authored-by: Ankit Jain <[email protected]> Co-authored-by: vpehkone <[email protected]> Co-authored-by: sdp <[email protected]> Co-authored-by: Kartik Ganesh <[email protected]> Co-authored-by: Cole White <[email protected]> Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com> Co-authored-by: opensearch-ci-bot <[email protected]> Co-authored-by: Xue Zhou <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Co-authored-by: Xue Zhou <[email protected]> Co-authored-by: Rishab Nahata <[email protected]> Co-authored-by: Anshu Agarwal <[email protected]> Co-authored-by: Yaliang Wu <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]> Co-authored-by: George Apaaboah <[email protected]> Co-authored-by: Ahmad AbuKhalil <[email protected]> Co-authored-by: Bukhtawar Khan <[email protected]> Co-authored-by: Sarat Vemulapalli <[email protected]> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

…ch-project#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]>

…4380) * [Remote Store] Upload segments to remote store post refresh (#3460) * Add RemoteDirectory interface to copy segment files to/from remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add index level setting for remote store Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * Add RemoteDirectoryFactory and use RemoteDirectory instance in RefreshListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * Upload segment to remote store post refresh Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener (#3703) * Inject remote store in IndexShard instead of RemoteStoreRefreshListener Signed-off-by: Sachin Kale <[email protected]> * Pass supplier of RepositoriesService to RemoteDirectoryFactory Signed-off-by: Sachin Kale <[email protected]> * Create isRemoteStoreEnabled function for IndexShard Signed-off-by: Sachin Kale <[email protected]> * Explicitly close remoteStore on indexShard close Signed-off-by: Sachin Kale <[email protected]> * Change RemoteDirectory.close to a no-op Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]> * [Remote Store] Add remote store restore API implementation (#3642) * Add remote restore API implementation Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add support to add nested settings for remote store (#4060) * Add support to add nested settings for remote store Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add rest endpoint for remote store restore (#3576) * Add rest endpoint for remote store restore Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add validator that forces segment replication type before enabling remote store (#4175) * Add validator that forces segment replication type before enabling remote store Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Change remote_store setting validation message to make it more clear (#4199) * Change remote_store setting validation message to make it more clear Signed-off-by: Sachin Kale <[email protected]> * [Remote Store] Add RemoteSegmentStoreDirectory to interact with remote segment store (#4020) * Add RemoteSegmentStoreDirectory to interact with remote segment store Signed-off-by: Sachin Kale <[email protected]> * Use RemoteSegmentStoreDirectory instead of RemoteDirectory (#4240) * Use RemoteSegmentStoreDirectory instead of RemoteDirectory Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: #1968 Upload segments to remote store post refresh: #3460 Add rest endpoint for remote store restore: #3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: #4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: #1968 Upload segments to remote store post refresh: #3460 Add rest endpoint for remote store restore: #3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: #4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

opensearch-project#8047) I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Rishab Nahata <[email protected]>

@sachinpkale

I have nominated and maintainers have agreed to invite Sachin Kale (@sachinpkale) to be a co-maintainer. Sachin has kindly accepted. Sachin has led the design and implementation of the remote backed storage feature in OpenSearch. This feature was introduced as experimental in OpenSearch 2.3 and is planned for general availability in 2.9. Some significant issues and PRs authored by Sachin for this effort are as follows: Feature proposal: opensearch-project#1968 Upload segments to remote store post refresh: opensearch-project#3460 Add rest endpoint for remote store restore: opensearch-project#3576 Add RemoteSegmentStoreDirectory to interact with remote segment store: opensearch-project#4020 In total, Sachin has authored 57 PRs going back to May 2022. He also frequently reviews contributions from others and has reviewed nearly 100 PRs in the same time frame. Signed-off-by: Andrew Ross <[email protected]> Signed-off-by: Shivansh Arora <[email protected]>

sachinpkale requested review from a team and reta as code owners May 27, 2022 08:21

sachinpkale force-pushed the main branch from bfc0660 to 82d55c6 Compare May 27, 2022 08:58

Bukhtawar approved these changes May 30, 2022

View reviewed changes

nknize previously requested changes May 31, 2022

View reviewed changes

mch2 reviewed May 31, 2022

View reviewed changes

mch2 mentioned this pull request Jun 1, 2022

[Design Proposal] Add Remote Storage Options for Improved Durability #2700

Open

sachinpkale and others added 2 commits June 1, 2022 09:48

Add RemoteDirectory interface to copy segment files to/from remote store

b3b8160

Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

Add index level setting for remote store

87e2cc7

Signed-off-by: Sachin Kale <[email protected]> Co-authored-by: Sachin Kale <[email protected]>

sachinpkale force-pushed the main branch from 82d55c6 to 2a4c5a0 Compare June 1, 2022 04:21

sachinpkale requested a review from nknize June 1, 2022 04:23

andrross reviewed Jun 1, 2022

View reviewed changes

Bukhtawar self-requested a review June 6, 2022 12:02

Bukhtawar approved these changes Jun 6, 2022

View reviewed changes

reta reviewed Jun 6, 2022

View reviewed changes

sachinpkale force-pushed the main branch 2 times, most recently from a71332f to 2f5957f Compare June 8, 2022 09:26

sachinpkale mentioned this pull request Jun 9, 2022

[Remote Store] Move repository fetch logic from IndexService to RemoteDirectoryFactory #3547

Closed

sachinpkale force-pushed the main branch from 2f5957f to 6327693 Compare June 9, 2022 13:41

Bukhtawar reviewed Jun 9, 2022

View reviewed changes

sachinpkale and others added 2 commits June 9, 2022 20:53

Add RemoteDirectoryFactory and use RemoteDirectory instance in Refres…

9a6a252

…hListener Co-authored-by: Sachin Kale <[email protected]> Signed-off-by: Sachin Kale <[email protected]>

Upload segment to remote store post refresh

eb10c3c

Signed-off-by: Sachin Kale <[email protected]>

sachinpkale force-pushed the main branch from 6327693 to eb10c3c Compare June 9, 2022 15:24

sachinpkale mentioned this pull request Jun 9, 2022

[Remote Store] Inject the RemoteStoreRefreshListener in the indexShard constructor #3548

Closed

Bukhtawar merged commit fb13759 into opensearch-project:main Jun 10, 2022

sachinpkale mentioned this pull request Jun 25, 2022

[Remote Store] Inject remote store in IndexShard instead of RemoteStoreRefreshListener #3703

Merged

5 tasks

sachinpkale mentioned this pull request Sep 1, 2022

[Backport 2.x] [Remote Store] Backport remote segment store changes #4380

Merged

6 tasks

andrross mentioned this pull request Jun 12, 2023

Add @sachinpkale as an OpenSearch maintainer #8019

Merged

6 tasks

andrross mentioned this pull request Jun 13, 2023

[Backport 2.x] Add @sachinpkale as an OpenSearch maintainer (#8019) #8047

Merged

6 tasks

[Remote Store] Upload segments to remote store post refresh #3460

[Remote Store] Upload segments to remote store post refresh #3460

Uh oh!

Conversation

sachinpkale commented May 27, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issues Resolved

Check List

Uh oh!

opensearch-ci-bot commented May 27, 2022

Uh oh!

opensearch-ci-bot commented May 27, 2022

Uh oh!

nknize left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nknize commented May 31, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opensearch-ci-bot commented Jun 1, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

reta Jun 8, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sachinpkale commented Jun 6, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

opensearch-ci-bot commented Jun 8, 2022

Uh oh!

opensearch-ci-bot commented Jun 8, 2022

Uh oh!

Bukhtawar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sachinpkale commented May 27, 2022 •

edited

Loading

reta Jun 8, 2022 •

edited

Loading