Skip to content

[Bug] Speedup revive scan in Pop Consumption and support server side reset offset #9112

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
3 tasks done
lizhimins opened this issue Jan 8, 2025 · 0 comments · Fixed by #9113
Closed
3 tasks done

Comments

@lizhimins
Copy link
Member

lizhimins commented Jan 8, 2025

Before Creating the Bug Report

  • I found a bug, not just asking a question, which should be created in GitHub Discussions.
  • I have searched the GitHub Issues and GitHub Discussions of this repository and believe that this is not a duplicate.
  • I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

Runtime platform environment

Linux 5.10

RocketMQ version

only develop branch about version 5.3.1

JDK Version

JDK11

Describe the Bug

The current development branch uses the rocksdb seek first API, which has certain performance issues. The related issues are as follows:

  1. [storage][pruner] Add index to avoid seek_to_first on pruner startup aptos-labs/aptos-core#3032
  2. iterator Seek slowdown after delete many keys  facebook/rocksdb#10300
  3. iterator.seek() so slow facebook/rocksdb#10510

Solution:

Actually, our application receives a delete request every time it is invoked by the user, and this is not something we can control. In some scenarios, the user will delete all keys under one prefix one by one, and the order of deletion is not consecutive, so we cannot convert the point deletes into range deletes.

Rocksdb official description:
https://github.com/facebook/rocksdb/wiki/Prefix-Seek

When the server-side offset feature is enabled, the Pop KV implementation does not effectively reset the offset.

Steps to Reproduce

I have written a test case, We can observe that using the seek first API results in higher latency.

image

What Did You Expect to See?

  1. The seek API has stable latency and fast
  2. We can reset offset when server offset feature is enable

What Did You See Instead?

None

Additional Context

No response

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant