Skip to content

CNDB-11762: Remove StorageAttachedIndexQueryPlan#postProcessor #1422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Dec 9, 2024

Conversation

adelapena
Copy link

StorageAttachedIndexQueryPlan#postProcessor seems redundant because coordinator-side sorting+triming of ANN query results is already done in SelectStatement. If that's the case, we should remove it.

Checklist before you submit for review

  • Make sure there is a PR in the CNDB project updating the Converged Cassandra version
  • Use NoSpamLogger for log lines that may appear frequently in the logs
  • Verify test results on Butler
  • Test coverage for new/modified code is > 80%
  • Proper code formatting
  • Proper title for each commit staring with the project-issue number, like CNDB-1234
  • Each commit has a meaningful description
  • Each commit is not very long and contains related changes
  • Renames, moves and reformatting are in distinct commits

@adelapena adelapena self-assigned this Nov 15, 2024
@adelapena adelapena marked this pull request as draft November 15, 2024 17:53
@adelapena adelapena marked this pull request as ready for review November 15, 2024 20:30
@adelapena adelapena marked this pull request as draft November 15, 2024 20:31
@adelapena adelapena marked this pull request as ready for review November 18, 2024 13:21
@jbellis
Copy link

jbellis commented Nov 18, 2024

it looks like the whole postReconciliationProcessing -> postProcessor chain can be removed since the only remaining postProcessor is a no-op?

@adelapena
Copy link
Author

it looks like the whole postReconciliationProcessing -> postProcessor chain can be removed since the only remaining postProcessor is a no-op?

Probably yes, since I think we don't have 3rd party 2i implementations using it for sorting.

I suspect that in the long term sorting will be better placed at the same level as postReconciliationProcessing, in the org.apache.cassandra.db package, rather than embedded in CQL. But for now, we can remove it and keep using the CQL version of sorting, which seems more evolved (and faster!).

Copy link

Quality Gate Failed Quality Gate failed

Failed conditions
64.7% Coverage on New Code (required ≥ 80%)

See analysis details on SonarQube Cloud

@cassci-bot
Copy link

❌ Build ds-cassandra-pr-gate/PR-1422 rejected by Butler


4 new test failure(s) in 7 builds
See build details here


Found 4 new test failures

Test Explanation Branch history Upstream history
...Test.shadowedPrimaryKeysRequireDeeperSearch[dc] flaky 🔵🔴🔵🔵 🔵🔵🔵🔵🔵🔵🔵
...t,varint>,wide=false,scenario=POST_BUILD_QUERY] regression 🔴🔵🔵🔵 🔵🔵🔵🔵🔵🔵🔵
...set>,wide=true,scenario=MEMTABLE_QUERY] regression 🔴🔵🔵🔵 🔵🔵🔵🔵🔵🔵🔵
o.a.c.u.b.BinLogTest.testTruncationReleasesLogS... flaky 🔵🔴🔵🔴🔴🔴🔵 🔵🔵🔵🔵🔵🔵🔵

Found 38 known test failures

@adelapena
Copy link
Author

@michaeljmarshall will you have time to review this? It should be a nice performance improvement for coordinators.

Copy link
Member

@michaeljmarshall michaeljmarshall left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Sorry, I missed reviewing this sooner. I'm glad we'll be able to skip an extra sorting step in the coordinator.

@michaeljmarshall
Copy link
Member

The two test failures labeled as regressions are for detected leaks. I commented on https://github.com/riptano/cndb/issues/11910 with a reference to this PR. They are not related to this PR, so I am going to merge this now.

@michaeljmarshall michaeljmarshall merged commit c06c94c into main Dec 9, 2024
458 of 475 checks passed
@michaeljmarshall michaeljmarshall deleted the CNDB-11762-main branch December 9, 2024 19:24
@michaeljmarshall michaeljmarshall mentioned this pull request Dec 9, 2024
9 tasks
@adelapena
Copy link
Author

adelapena commented Dec 10, 2024

Thanks for reviewing and merging :). There wasn't a PR for adding this to CNDB yet, so I have created this one: https://github.com/riptano/cndb/pull/12076

That PR also includes CNDB-11950, for which I haven't found the matching PR bumping the CNDB version.

adelapena added a commit that referenced this pull request Dec 16, 2024
eolivelli pushed a commit that referenced this pull request Dec 16, 2024
This reverts commit c06c94c.

It seems the removal of `Index#postProcessor` by CNDB-11762 broke some
tests in CNDB's `MultiNodeBillingTest`. Unfortunately that patch [was
merged before creating the PR bumping the CC version used by
CNDB](#1422 (comment)).
[The CNDB PR](riptano/cndb#12076) was created
after that merging but it was superseded by other CC version bumps.

So I'm adding this reversal so we can investigate how the removal of
`Index#postProcessor` affects those tests.
djatnieks pushed a commit that referenced this pull request Dec 18, 2024
`StorageAttachedIndexQueryPlan#postProcessor` seems redundant because
coordinator-side sorting+triming of ANN query results is already done in
`SelectStatement`. If that's the case, we should remove it.

- [ ] Make sure there is a PR in the CNDB project updating the Converged
Cassandra version
- [ ] Use `NoSpamLogger` for log lines that may appear frequently in the
logs
- [ ] Verify test results on Butler
- [ ] Test coverage for new/modified code is > 80%
- [ ] Proper code formatting
- [ ] Proper title for each commit staring with the project-issue
number, like CNDB-1234
- [ ] Each commit has a meaningful description
- [ ] Each commit is not very long and contains related changes
- [ ] Renames, moves and reformatting are in distinct commits
djatnieks pushed a commit that referenced this pull request Dec 18, 2024
This reverts commit c06c94c.

It seems the removal of `Index#postProcessor` by CNDB-11762 broke some
tests in CNDB's `MultiNodeBillingTest`. Unfortunately that patch [was
merged before creating the PR bumping the CC version used by
CNDB](#1422 (comment)).
[The CNDB PR](riptano/cndb#12076) was created
after that merging but it was superseded by other CC version bumps.

So I'm adding this reversal so we can investigate how the removal of
`Index#postProcessor` affects those tests.
djatnieks pushed a commit that referenced this pull request Jan 17, 2025
`StorageAttachedIndexQueryPlan#postProcessor` seems redundant because
coordinator-side sorting+triming of ANN query results is already done in
`SelectStatement`. If that's the case, we should remove it.

- [ ] Make sure there is a PR in the CNDB project updating the Converged
Cassandra version
- [ ] Use `NoSpamLogger` for log lines that may appear frequently in the
logs
- [ ] Verify test results on Butler
- [ ] Test coverage for new/modified code is > 80%
- [ ] Proper code formatting
- [ ] Proper title for each commit staring with the project-issue
number, like CNDB-1234
- [ ] Each commit has a meaningful description
- [ ] Each commit is not very long and contains related changes
- [ ] Renames, moves and reformatting are in distinct commits
djatnieks pushed a commit that referenced this pull request Jan 17, 2025
This reverts commit c06c94c.

It seems the removal of `Index#postProcessor` by CNDB-11762 broke some
tests in CNDB's `MultiNodeBillingTest`. Unfortunately that patch [was
merged before creating the PR bumping the CC version used by
CNDB](#1422 (comment)).
[The CNDB PR](riptano/cndb#12076) was created
after that merging but it was superseded by other CC version bumps.

So I'm adding this reversal so we can investigate how the removal of
`Index#postProcessor` affects those tests.
djatnieks pushed a commit that referenced this pull request Jan 17, 2025
`StorageAttachedIndexQueryPlan#postProcessor` seems redundant because
coordinator-side sorting+triming of ANN query results is already done in
`SelectStatement`. If that's the case, we should remove it.

- [ ] Make sure there is a PR in the CNDB project updating the Converged
Cassandra version
- [ ] Use `NoSpamLogger` for log lines that may appear frequently in the
logs
- [ ] Verify test results on Butler
- [ ] Test coverage for new/modified code is > 80%
- [ ] Proper code formatting
- [ ] Proper title for each commit staring with the project-issue
number, like CNDB-1234
- [ ] Each commit has a meaningful description
- [ ] Each commit is not very long and contains related changes
- [ ] Renames, moves and reformatting are in distinct commits
djatnieks pushed a commit that referenced this pull request Jan 17, 2025
This reverts commit c06c94c.

It seems the removal of `Index#postProcessor` by CNDB-11762 broke some
tests in CNDB's `MultiNodeBillingTest`. Unfortunately that patch [was
merged before creating the PR bumping the CC version used by
CNDB](#1422 (comment)).
[The CNDB PR](riptano/cndb#12076) was created
after that merging but it was superseded by other CC version bumps.

So I'm adding this reversal so we can investigate how the removal of
`Index#postProcessor` affects those tests.
djatnieks pushed a commit that referenced this pull request Jan 30, 2025
`StorageAttachedIndexQueryPlan#postProcessor` seems redundant because
coordinator-side sorting+triming of ANN query results is already done in
`SelectStatement`. If that's the case, we should remove it.

- [ ] Make sure there is a PR in the CNDB project updating the Converged
Cassandra version
- [ ] Use `NoSpamLogger` for log lines that may appear frequently in the
logs
- [ ] Verify test results on Butler
- [ ] Test coverage for new/modified code is > 80%
- [ ] Proper code formatting
- [ ] Proper title for each commit staring with the project-issue
number, like CNDB-1234
- [ ] Each commit has a meaningful description
- [ ] Each commit is not very long and contains related changes
- [ ] Renames, moves and reformatting are in distinct commits
djatnieks pushed a commit that referenced this pull request Jan 30, 2025
This reverts commit c06c94c.

It seems the removal of `Index#postProcessor` by CNDB-11762 broke some
tests in CNDB's `MultiNodeBillingTest`. Unfortunately that patch [was
merged before creating the PR bumping the CC version used by
CNDB](#1422 (comment)).
[The CNDB PR](riptano/cndb#12076) was created
after that merging but it was superseded by other CC version bumps.

So I'm adding this reversal so we can investigate how the removal of
`Index#postProcessor` affects those tests.
djatnieks pushed a commit that referenced this pull request May 18, 2025
`StorageAttachedIndexQueryPlan#postProcessor` seems redundant because
coordinator-side sorting+triming of ANN query results is already done in
`SelectStatement`. If that's the case, we should remove it.

- [ ] Make sure there is a PR in the CNDB project updating the Converged
Cassandra version
- [ ] Use `NoSpamLogger` for log lines that may appear frequently in the
logs
- [ ] Verify test results on Butler
- [ ] Test coverage for new/modified code is > 80%
- [ ] Proper code formatting
- [ ] Proper title for each commit staring with the project-issue
number, like CNDB-1234
- [ ] Each commit has a meaningful description
- [ ] Each commit is not very long and contains related changes
- [ ] Renames, moves and reformatting are in distinct commits
djatnieks pushed a commit that referenced this pull request May 18, 2025
This reverts commit c06c94c.

It seems the removal of `Index#postProcessor` by CNDB-11762 broke some
tests in CNDB's `MultiNodeBillingTest`. Unfortunately that patch [was
merged before creating the PR bumping the CC version used by
CNDB](#1422 (comment)).
[The CNDB PR](riptano/cndb#12076) was created
after that merging but it was superseded by other CC version bumps.

So I'm adding this reversal so we can investigate how the removal of
`Index#postProcessor` affects those tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants