[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

msfroh · 2023-10-12T23:55:06Z

Description

Manual backport of #10036

There are latency-sensitive users for whom the default 20% allowable accumulation of deletes is too high, so they force merge with "only_expunge_deletes".

Unfortunately, this has ignored the max segment size, so despite the word "only" in there, they get the terrible side-effect of producing segments larger than the max segment size. These oversized segments are unlikely to participate in future merges, until they have so many deletes that they could drop back below the max segment size, likely making the whole "too many deletes" problem even worse.

This commit fixes that so that "only_expunge_deletes" ONLY EXPUNGES DELETES without creating oversized segments.

Related Issues

N/A

Check List

New functionality includes testing.
- All tests pass
New functionality has been documented.
- New functionality has javadoc added
Commits are signed per the DCO using --signoff
Commit changes are listed out in CHANGELOG.md file (See: Changelog)
Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…n /plugins/repository-azure (opensearch-project#9147) (opensearch-project#9169) * Bump io.projectreactor.netty:reactor-netty-http Bumps [io.projectreactor.netty:reactor-netty-http](https://github.com/reactor/reactor-netty) from 1.1.8 to 1.1.9. - [Release notes](https://github.com/reactor/reactor-netty/releases) - [Commits](reactor/reactor-netty@v1.1.8...v1.1.9) --- updated-dependencies: - dependency-name: io.projectreactor.netty:reactor-netty-http dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit e1eade1) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…vadocDeclaration (Intellij Project_Default.xml) (opensearch-project#9121) (opensearch-project#9152) (cherry picked from commit 2a5e92a) Signed-off-by: LSchultebraucks <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#9148) (opensearch-project#9174) * Bump org.apache.maven:maven-model from 3.9.3 to 3.9.4 in /buildSrc Bumps [org.apache.maven:maven-model](https://github.com/apache/maven) from 3.9.3 to 3.9.4. - [Release notes](https://github.com/apache/maven/releases) - [Commits](apache/maven@maven-3.9.3...maven-3.9.4) --- updated-dependencies: - dependency-name: org.apache.maven:maven-model dependency-type: direct:production update-type: version-update:semver-patch ... * Update changelog --------- (cherry picked from commit 75f651b) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

* Moving get snapshot requests to listener based async calls Signed-off-by: Indrajit Banerjee <[email protected]> * “Empty-Commit” Signed-off-by: Indrajit Banerjee <[email protected]> --------- Signed-off-by: Indrajit Banerjee <[email protected]> Co-authored-by: Indrajit Banerjee <[email protected]>

Signed-off-by: Gagan Juneja <[email protected]>

* Fixing SnapshotResiliencyTests.testTransportGetSnapshotsAction with single listener object Signed-off-by: Indrajit Banerjee <[email protected]>

…idations (opensearch-project#8444) (opensearch-project#9188) Signed-off-by: Shephali Mittal <[email protected]>

…alRelocationNoPreferenceShardMovementPrimaryFirstEnabled (opensearch-project#9181) (opensearch-project#9202) (cherry picked from commit ccf0b9a) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…y-gcs (opensearch-project#9145) (opensearch-project#9195) * Bump io.grpc:grpc-context in /plugins/repository-gcs Bumps [io.grpc:grpc-context](https://github.com/grpc/grpc-java) from 1.56.1 to 1.57.1. - [Release notes](https://github.com/grpc/grpc-java/releases) - [Commits](grpc/grpc-java@v1.56.1...v1.57.1) --- updated-dependencies: - dependency-name: io.grpc:grpc-context dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Andriy Redko <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 7560afb) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…CustomString (opensearch-project#9175) (opensearch-project#9205) (cherry picked from commit b2af655) Signed-off-by: Poojita Raj <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ures/hdfs-fixture (opensearch-project#9146) (opensearch-project#9179) * Bump org.apache.zookeeper:zookeeper in /test/fixtures/hdfs-fixture Bumps org.apache.zookeeper:zookeeper from 3.8.2 to 3.9.0. --- updated-dependencies: - dependency-name: org.apache.zookeeper:zookeeper dependency-type: direct:production update-type: version-update:semver-minor ... * Update changelog --------- (cherry picked from commit d4f102b) Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com>

…evel (opensearch-project#9085) (opensearch-project#9211) * Use BucketCountThresholds in InternalTerms and InternalAggregations and do not apply shard level thresholds at slice level for Concurrent Segment Search * Addressing comments * Re-introduce shardSize member to InternalMultiTerms and InternalMappedTerms * Introduce LocalBucketCountThresholds for local size and min_doc_count values --------- Signed-off-by: Jay Deng <[email protected]>

….JSON (opensearch-project#9156) (opensearch-project#9210) This commit rote refactors nearly all instances and usages of the XContentType.JSON singleton instance defined in the :libs:opensearch-x-content library to the MediaTypeRegistry.JSON singleton instance defined in :libs:opensearch-core. This decouples the server and test implementations from the x-content library such that foundation classes can be further refactored from the :server module to the :opensearch-core library without requiring a strict dependency on the x-content library. This keeps opensearch-core library loosely coupled with the remaining :libs:opensearch-* libraries such that serverless and cloud-native implementations can selective import only the libraries needed for implementation. Signed-off-by: Nicholas Walter Knize <[email protected]> (cherry picked from commit 7278f43) --------- Signed-off-by: Nicholas Walter Knize <[email protected]>

Signed-off-by: Kunal Kotwani <[email protected]>

…search-project#9170) Signed-off-by: Sarthak Aggarwal <[email protected]> (cherry picked from commit 3713bcb)

… for ingestion (opensearch-project#9190) (opensearch-project#9251)

…/repository-azure (opensearch-project#9231) (opensearch-project#9260) * Bump com.azure:azure-storage-blob in /plugins/repository-azure Bumps [com.azure:azure-storage-blob](https://github.com/Azure/azure-sdk-for-java) from 12.22.3 to 12.23.0. - [Release notes](https://github.com/Azure/azure-sdk-for-java/releases) - [Commits](Azure/azure-sdk-for-java@azure-storage-blob_12.22.3...azure-storage-blob_12.23.0) --- updated-dependencies: - dependency-name: com.azure:azure-storage-blob dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 1162865) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ect#9271) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 (opensearch-project#9227) * Bump com.diffplug.spotless from 6.19.0 to 6.20.0 Bumps com.diffplug.spotless from 6.19.0 to 6.20.0. --- updated-dependencies: - dependency-name: com.diffplug.spotless dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 48ec206) * Force upgrade the vulnerable dependencies of hadoop-minicluster (opensearch-project#9252) * Force upgrade the vulnerable dependencies of hadoop-minicluster Signed-off-by: Thomas Farr <[email protected]> * Add changelog entry Signed-off-by: Thomas Farr <[email protected]> --------- Signed-off-by: Thomas Farr <[email protected]> (cherry picked from commit a1fc31c) * Bump com.google.code.gson:gson from 2.9.0 to 2.10.1 in /plugins/repository-gcs (opensearch-project#9230) * Bump com.google.code.gson:gson in /plugins/repository-gcs Bumps [com.google.code.gson:gson](https://github.com/google/gson) from 2.9.0 to 2.10.1. - [Release notes](https://github.com/google/gson/releases) - [Changelog](https://github.com/google/gson/blob/main/CHANGELOG.md) - [Commits](google/gson@gson-parent-2.9.0...gson-parent-2.10.1) --- updated-dependencies: - dependency-name: com.google.code.gson:gson dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Updating SHAs Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 5c283e0) * Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 (opensearch-project#9228) * Bump lycheeverse/lychee-action from 1.2.0 to 1.8.0 Bumps [lycheeverse/lychee-action](https://github.com/lycheeverse/lychee-action) from 1.2.0 to 1.8.0. - [Release notes](https://github.com/lycheeverse/lychee-action/releases) - [Commits](lycheeverse/lychee-action@v1.2.0...v1.8.0) --- updated-dependencies: - dependency-name: lycheeverse/lychee-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 4b21849) --------- Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thomas Farr <[email protected]>

…fix PIT/Scroll with remote store. (opensearch-project#9272) * [Segment Replication] Refactor file cleanup logic and fix PIT/Scroll with remote store. (opensearch-project#9111) * Remove divergent commit logic with segment replication. This change removes divergent commit paths for segrep node-node and remote store. All replicas with segrep enabled will perform local commits and ignore any incoming segments_n file. This changes the recovery sync with remote store to also exclude the segments_n so that only the fetched infos bytes are committed before an engine is opened. This change also updates deletion logic with segment replication to automatically delete when a file is decref'd to 0. Signed-off-by: Marc Handalian <[email protected]> * Add more NRTReplicationEngineTests. Signed-off-by: Marc Handalian <[email protected]> * Ensure old commit files are wiped on remote store sync before we commit a new segmentInfos. Signed-off-by: Marc Handalian <[email protected]> * Add more shard level tests. Signed-off-by: Marc Handalian <[email protected]> * Add test ensuring commits are cleaned up on replicas. Signed-off-by: Marc Handalian <[email protected]> * Self review. Signed-off-by: Marc Handalian <[email protected]> * Use refresh level sync before recovery Signed-off-by: Marc Handalian <[email protected]> * PR feedback. Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit c301544) * Fix test SegmentReplicationIndexShardTests.testPrimaryRestart. This test is specific to remote store and should not be run for node-node replication. Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit a33f67e) Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]>

…project#9274) (cherry picked from commit 3ddff4c) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…-project#9244) (opensearch-project#9273) (cherry picked from commit ab08d73) Signed-off-by: Marc Handalian <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…est (opensearch-project#9079) (opensearch-project#9278) * Unmute testIndexingWithSegRep rolling upgrade test * Assert on row count before processing the result * Filter rows with 0 doc count * Retry assertHitCount * Handle exception * Add comment --------- Signed-off-by: Suraj Singh <[email protected]>

…) (opensearch-project#9282) (cherry picked from commit db768d1) Signed-off-by: Jay Deng <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…andle restore … (opensearch-project#9261) --------- Signed-off-by: bansvaru <[email protected]>

…arch-project#8758) (opensearch-project#9268) Signed-off-by: Ashish Singh <[email protected]>

…search-project#9122) (opensearch-project#9275) * Make SearchTemplateRequest implement IndicesRequest.Replaceable * Add to CHANGELOG --------- (cherry picked from commit c73f727) Signed-off-by: Craig Perkins <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…oject#9292) * Adds attributes to startSpan * Update Changelog * Adds attributes to startSpan * Refactor code * Add java doc * Refactor code * Refactor code * Refactor code * Removes dependency --------- Signed-off-by: Gagan Juneja <[email protected]> Co-authored-by: Gagan Juneja <[email protected]>

opensearch-project#9293) (opensearch-project#9295) --------- (cherry picked from commit 5fdebee) Signed-off-by: bansvaru <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…annotations, their meaning and relations between them (opensearch-project#9223) (opensearch-project#9296) * [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them (opensearch-project#9223) * [BWC and API enforcement] Define the initial set of annotations, their meaning and relations between them Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: Peter Nied <[email protected]> Co-authored-by: Peter Nied <[email protected]> (cherry picked from commit 0b2c07a) * [BWC and API enforcement] Extend the element types the initial set of annotations could be applied to Signed-off-by: Andriy Redko <[email protected]> --------- Signed-off-by: Andriy Redko <[email protected]>

…rojects) (opensearch-project#9309) (opensearch-project#9319) Signed-off-by: Owais Kazi <[email protected]> (cherry picked from commit 301c766) Co-authored-by: Owais Kazi <[email protected]>

…arch-project#10315) * Fix: register mulitple extensions. * Updated CHANGELOG. * Added tests. --------- (cherry picked from commit 1d66af3) Signed-off-by: dblock <[email protected]>

…t#10204) (opensearch-project#10333) Signed-off-by: Rishav Sagar <[email protected]> Co-authored-by: Rishav Sagar <[email protected]>

…10289) (opensearch-project#10332) (cherry picked from commit 2098975) Signed-off-by: Jongwoo Han <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

… (opensearch-project#10319) (cherry picked from commit beb25b1) Signed-off-by: Andriy Redko <[email protected]>

…10301) (opensearch-project#10337) * Bump peter-evans/create-pull-request from 3 to 5 Bumps [peter-evans/create-pull-request](https://github.com/peter-evans/create-pull-request) from 3 to 5. - [Release notes](https://github.com/peter-evans/create-pull-request/releases) - [Commits](peter-evans/create-pull-request@v3...v5) --- updated-dependencies: - dependency-name: peter-evans/create-pull-request dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit fe80682) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…opensearch-project#10335) (cherry picked from commit e7909fc) Signed-off-by: Bhumika Saini <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…rt calls (opensearch-project#10192) (opensearch-project#10278) (cherry picked from commit 61495bf) Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…roject#9992) (opensearch-project#10312) * Configurable merge policy for index * additional setting to configure merge policy for timestamp based index * introduction of logbytesize merge policy as an option Signed-off-by: Rishabh Maurya <[email protected]> * remove the trace log not required anymore Signed-off-by: Rishabh Maurya <[email protected]> * Refactor the merge policy extraction logic Signed-off-by: Rishabh Maurya <[email protected]> * Rename constant DEFAULT to DEFAULT_POLICY Signed-off-by: Rishabh Maurya <[email protected]> * Simplify merge policy extraction and selection logic Signed-off-by: Rishabh Maurya <[email protected]> * missing javadoc error Signed-off-by: Rishabh Maurya <[email protected]> * Renaming log byte size policy setting with mb Signed-off-by: Rishabh Maurya <[email protected]> * Move validation exception to enum from setting defn Signed-off-by: Rishabh Maurya <[email protected]> * rename time_index to time_series_index Signed-off-by: Rishabh Maurya <[email protected]> --------- Signed-off-by: Rishabh Maurya <[email protected]> (cherry picked from commit fa66beb)

…oject#10200) (opensearch-project#10329) (cherry picked from commit 3a790c1) Signed-off-by: Jay Deng <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…earch-project#10317) There was a weird circular class-loading dependency between Settings and Settings.Builder that could result in ClassLoader deadlock if multiple threads trigger class-loading of Settings at the same time. (cherry picked from commit d3bf230) Signed-off-by: Michael Froh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Signed-off-by: Daniel (dB.) Doubrovkine <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Daniel (dB.) Doubrovkine <[email protected]>

…ch-project#10325) (opensearch-project#10344) (cherry picked from commit bddf0d3) Signed-off-by: Andriy Redko <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…pensearch-project#10346) * Add support for encrypted async blob read Signed-off-by: Kunal Kotwani <[email protected]> * Add async blob read support for encrypted containers Signed-off-by: Kunal Kotwani <[email protected]> --------- Signed-off-by: Kunal Kotwani <[email protected]> (cherry picked from commit c4c4ad8)

… (opensearch-project#10353) --------- (cherry picked from commit 7159e2e) Signed-off-by: Gaurav Bafna <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…ect#10356) (opensearch-project#10366) (cherry picked from commit e5024a8) Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…nsearch-project#10363) (opensearch-project#10368) (cherry picked from commit a0cb344) Signed-off-by: Ashish Singh <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…/fixtures/hdfs-fixture (opensearch-project#10299) (opensearch-project#10369) * Bump org.xerial.snappy:snappy-java in /test/fixtures/hdfs-fixture Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.4 to 1.1.10.5. - [Release notes](https://github.com/xerial/snappy-java/releases) - [Commits](xerial/snappy-java@v1.1.10.4...v1.1.10.5) --- updated-dependencies: - dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * Update changelog Signed-off-by: dependabot[bot] <[email protected]> --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: dependabot[bot] <dependabot[bot]@users.noreply.github.com> (cherry picked from commit 3ed7cc4) Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ion if system repository and mutation of immutable settings of system repository (opensearch-project#9839) (opensearch-project#10364) --------- (cherry picked from commit 699d235) Signed-off-by: Dharmesh 💤 <[email protected]>

…ct#10341) (opensearch-project#10361) (cherry picked from commit 677ff75) Signed-off-by: Sachin Kale <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…#10349) (opensearch-project#10373) * Refactor read context streams to async streams Signed-off-by: Kunal Kotwani <[email protected]> * Refactor multipart download to a more async model The previous approach of kicking off the stream requests for all parts of a file did not work well for very large files. For example, a 20GiB file uploaded in 16MiB parts will consist of 1200+ parts. When we attempted to initiate streaming for all parts concurrently, some parts would hit a client timeout after 2 minutes without being able to get a connection due to the other parts not having been completed in that time frame. This refactoring adds yet another layer of indirection in order to allow the code that is actually writing the destination file to control the rate at which streams are started. This should allow for downloading files consisting of arbitrarily many parts at any connection speed. This commit also wires in the download rate limiter so that the `indices.recovery.max_bytes_per_sec` is properly honored. Signed-off-by: Andrew Ross <[email protected]> --------- Signed-off-by: Kunal Kotwani <[email protected]> Signed-off-by: Andrew Ross <[email protected]> Co-authored-by: Kunal Kotwani <[email protected]> (cherry picked from commit 28f185b)

…ch-project#10347) (opensearch-project#10389) * Update multipart download path to write to temp files. This change updates ReadContextListener to first write parts to a temp location until all parts have been received. Signed-off-by: Marc Handalian <[email protected]> * Suppress forbidden IOUtils.fsync Signed-off-by: Marc Handalian <[email protected]> * Remove unnecessary logging format Signed-off-by: Marc Handalian <[email protected]> * sync directory after file rename Signed-off-by: Marc Handalian <[email protected]> * Remove flaky threadpool terminate test Signed-off-by: Marc Handalian <[email protected]> --------- Signed-off-by: Marc Handalian <[email protected]> (cherry picked from commit ca0dae6) Co-authored-by: Marc Handalian <[email protected]>

…10393) Signed-off-by: Gagan Juneja <[email protected]>

…ta inplace (opensearch-project#9569) (opensearch-project#10390) * [Remote Store] Add support to reload repository metadata inplace (opensearch-project#9569) --------- Signed-off-by: Bhumika Saini <[email protected]> * Empty commit to retrigger build Signed-off-by: Bhumika Saini <[email protected]> * Empty commit to retrigger build Signed-off-by: Bhumika Saini <[email protected]> --------- Signed-off-by: Bhumika Saini <[email protected]> Signed-off-by: Gaurav Bafna <[email protected]> Co-authored-by: Gaurav Bafna <[email protected]>

…search-project#10143) (opensearch-project#10381) * Add instrumentation in InboundHandler * Add CHANGELOG * Address review comment * Address review comment * Address review comment * Empty-Commit * Address review comment * Address review comment * Address review comment --------- (cherry picked from commit aac0f09) Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: Gagan Juneja <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Gagan Juneja <[email protected]>

…h-project#10016) (opensearch-project#10403) --------- (cherry picked from commit 1eec0f1) Signed-off-by: bansvaru <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…oject#10326) (opensearch-project#10384) (cherry picked from commit 71b6948) Signed-off-by: Sagar Upadhyaya <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

…o 1.24.0 in /buildSrc (opensearch-project#10386) Manual backport of opensearch-project#10205 Signed-off-by: Michael Froh <[email protected]>

…t#10036) (cherry picked from commit 1dde018) Signed-off-by: Michael Froh <[email protected]>

github-actions · 2023-10-13T00:22:38Z

Gradle Check (Jenkins) Run Completed with:

RESULT: FAILURE ❌
URL: https://build.ci.opensearch.org/job/gradle-check/27783/
CommitID: abb99b5
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
Is the failure a flaky test unrelated to your change?

github-actions · 2023-10-13T00:30:49Z

Compatibility status:

Checks if related components are compatible with change abb99b5

Incompatible components

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/neural-search.git]

reta and others added 30 commits August 8, 2023 10:15

Adds support for tracing runnable scenarios (opensearch-project#9187)

c3a0784

Signed-off-by: Gagan Juneja <[email protected]>

Backport/backport 8377 to 2.x (opensearch-project#9192)

545bf9d

* Fixing SnapshotResiliencyTests.testTransportGetSnapshotsAction with single listener object Signed-off-by: Indrajit Banerjee <[email protected]>

[Backport 2.x] Added Integration tests support for Telemetry span Val…

560e54e

…idations (opensearch-project#8444) (opensearch-project#9188) Signed-off-by: Shephali Mittal <[email protected]>

[2.x] Added bwc version 1.3.13. (opensearch-project#9240)

e752bc3

Signed-off-by: Kunal Kotwani <[email protected]>

introducing codecs as their algo name (opensearch-project#9123) (open…

32e8187

…search-project#9170) Signed-off-by: Sarthak Aggarwal <[email protected]> (cherry picked from commit 3713bcb)

[Remote Store] Fix couple of Remote Store flaky test and use bulk api…

eb5d8e1

… for ingestion (opensearch-project#9190) (opensearch-project#9251)

[Backport 2.x][Remote Store] Introduce RemoteStoreRestoreService to h…

d30199f

…andle restore … (opensearch-project#9261) --------- Signed-off-by: bansvaru <[email protected]>

Fix local segments stats update in RemoteStoreRefreshListener (opense…

77df735

…arch-project#8758) (opensearch-project#9268) Signed-off-by: Ashish Singh <[email protected]>

The spotless should take care of unified imports ordering (libs/ subp…

4110ab0

…rojects) (opensearch-project#9309) (opensearch-project#9319) Signed-off-by: Owais Kazi <[email protected]> (cherry picked from commit 301c766) Co-authored-by: Owais Kazi <[email protected]>

dblock and others added 27 commits October 3, 2023 10:41

Fix: register mulitple extensions. (opensearch-project#10256) (opense…

5f5f2f8

…arch-project#10315) * Fix: register mulitple extensions. * Updated CHANGELOG. * Added tests. --------- (cherry picked from commit 1d66af3) Signed-off-by: dblock <[email protected]>

Add unreferenced file cleanup count to merge stats (opensearch-projec…

f366dc3

…t#10204) (opensearch-project#10333) Signed-off-by: Rishav Sagar <[email protected]> Co-authored-by: Rishav Sagar <[email protected]>

Bump netty from 4.1.97.Final to 4.1.99.Final (opensearch-project#10306)…

537e4fc

… (opensearch-project#10319) (cherry picked from commit beb25b1) Signed-off-by: Andriy Redko <[email protected]>

[Backport 2.x] - Add Telemetry metrics framework (opensearch-project#…

92a261b

…10393) Signed-off-by: Gagan Juneja <[email protected]>

[Backport 2.x] Bump org.apache.commons:commons-compress from 1.23.0 t…

fea095d

…o 1.24.0 in /buildSrc (opensearch-project#10386) Manual backport of opensearch-project#10205 Signed-off-by: Michael Froh <[email protected]>

Honor max segment size during only_expunge_deletes (opensearch-projec…

abb99b5

…t#10036) (cherry picked from commit 1dde018) Signed-off-by: Michael Froh <[email protected]>

msfroh closed this Oct 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

Uh oh!

msfroh commented Oct 12, 2023

Uh oh!

github-actions bot commented Oct 13, 2023

Uh oh!

github-actions bot commented Oct 13, 2023

Uh oh!

Uh oh!

[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

[Backport 2.x] Honor max segment size during only_expunge_deletes #10602

Uh oh!

Conversation

msfroh commented Oct 12, 2023

Description

Related Issues

Check List

Uh oh!

github-actions bot commented Oct 13, 2023

Gradle Check (Jenkins) Run Completed with:

Uh oh!

github-actions bot commented Oct 13, 2023

Compatibility status:

Incompatible components

Skipped components

Compatible components

Uh oh!

Uh oh!