rocketmq-all-5.3.2

What's Changed

This release introduces a new implementation of Pop consumption based on RocksDB (in the alpha phase, see RIP-73 for details). It improves and provides a migration plan for the capability of using RocksDB to store consumption queue indexes, adds support for the ability to cancel scheduled messages, enhances graceful online and offline capabilities, and includes numerous regular detail fixes related to HA, ACL, Metrics and Pop consumption.

• [ISSUE #8784] PrintMessageSubCommand support lmq by @RongtongJin in #8785
• [ISSUE #8790] Add Utils for put header to Metadata to avoid duplicate data by @dingshuangxi888 in #8792
• [ISSUE #8802] Update controller design.md by @luozongle01 in #8803
• [ISSUE #8798] Fix typo by @luozongle01 in #8799
• [ISSUE #8804] Clean offset when remove group offset by @leizhiyuan in #8805
• [ISSUE #8806] Fix autoBatch bug when connecting multiple RocketMQ clusters. by @luozongle01 in #8807
• [ISSUE #8810] Fix independent execution of e2e and benchmark deployments by @chi3316 in #8812
• [ISSUE #8816] Fix log typo. by @luozongle01 in #8817
• [ISSUE #8764] Implement consume lag estimation in cq rocksdb store by @LetLetMe in #8800
• [ISSUE #8824] Fix IllegalStateException caused by logical errors by @yx9o in #8825
• [ISSUE #8780] Implement asynchronous storage of ack/ck messages in pop consume to enhance performance by @RongtongJin in #8727
• [ISSUE #8835] When ck is in the buffer, incomplete ack will lead to message duplication. by @GenerousMan in #8836
• [ISSUE #8820] Fix variables match annotation (@RocketMQResource) by @mawen12 in #8821
• [ISSUE #8830] Fix atomic flush incorrect use and clean up code by @lizhanhui in #8830
• [ISSUE #8829] Fix make ConsumeQueueStore bottom most compression type configurable by @lizhanhui in #8841
• [ISSUE #8839] Remove unnecessary synchronized to improve concurrency by @lizhanhui in #8840
• [ISSUE #8833] Update netty version to 4.1.114 to fix CVE-2023-34462 by @ShannonDing in #8832
• [ISSUE #8846] Fix registerProducer should not be affected by concurrent scanNotAct… by @lizhanhui in #8847
• [ISSUE #8829] Support LMQ dispatch in case if Consume Queue Store is RocksDB-based by @lizhanhui in #8842
• [ISSUE #8850] Use correct log method by @mawen12 in #8851
• [ISSUE #8772] Remove lock mq step in broadcasting mode rebalancing by @redlsz in #8774
• [ISSUE #8442] [RIP-70] Extract adaptive lock mechanism by @3424672656 in #8663
• [ISSUE #8829] Feat: provide ConfigManagerV2 to make best uses of RocksDB by @lizhanhui in #8856
• [ISSUE #8822] Double write cq, reduce unnecessary switches by @LetLetMe in #8823
• [ISSUE #8875] Fix HAConnection leak by @crazywen in #8876
• [ISSUE #8725] Clean DefaultMQPushConsumer after start fail by @yuz10 in #8726
• [ISSUE #8829] Keep data version while reload and XXXConfigManagerV2 turns off sync by @lizhanhui in #8891
• [ISSUE #8885] Resolve the issue of inaccurate CK number statistics by @RongtongJin in #8886
• [ISSUE #8808] Resolve unsupported 'UseBiasedLocking' VM Option for JDK21 by @taomaree in #8809
• [ISSUE #8882] Change the compare method for acl signature to improve the security. by @dingshuangxi888 in #8883
• [ISSUE #8889] handle namespace outside the loop by @mawen12 in #8890
• [ISSUE #8906] Handle string toUpperCase outside the loop by @mawen12 in #8907
• [ISSUE #8599] Throw exception when receiving GO_AWAY twice to close channel. by @qianye1001 in #8862
• [ISSUE #8917] Topic route return none permission message queues for gRPC client by @dingshuangxi888 in #8919
• [ISSUE #8921] Add isWakeCommitWhenPutMessage for AIO by @zk-drizzle in #8922
• [ISSUE #8925] Fix the bug of update user for ACL 2.0 by @dingshuangxi888 in #8926
• [ISSUE #8909] Move nullcheck ahead by @jjastan in #8910
• [ISSUE #8942] Add incGroupAckNums and incGroupCkNums to LmqBrokerStatsManager by @RongtongJin in #8943
• [ISSUE #8829] Support for Persisting LMQ Consumer Offsets in Config V1 Using RocksDB by @RongtongJin in #8939
• [ISSUE #8945] Remove unnecessary operations from the critical section by @3424672656 in #8946
• [ISSUE #8940] Fix incorrect path for exportMetadataInRocksDBCommand by @RongtongJin in #8941
• [ISSUE #8935] Fix behind metrics unit error in timer message store by @GenerousMan in #8936
• [ISSUE #8765] Fix low performance of delay message when enable rocksdb consume queue by @yuz10 in #8766
• [ISSUE #8953] Improve IO for asynchronous delivery processes by @3424672656 in #8954
• [ISSUE #8829] Fix avoid memory overhead when there is large number of LMQ CQ by @lizhanhui in #8956
• [ISSUE #8947] Notify pop request before calculate consumer lag by @lizhimins in #8949
• [ISSUE #8933] feat: DefaultPullConsumer add balance switch. by @humkum in #8934
• [ISSUE #8929] Proxy adds message body empty check when send in grpc protocol by @qianye1001 in #8930
• [ISSUE #8877] Refactor lock in ReceiptHandleGroup to make the lock can be properly released when future can not be completed. by @qianye1001 in #8916
• [ISSUE #8955] Fix message buffer not release and dispatch thread exit in tiered storage by @lizhimins in #8965
• [ISSUE #8972] Adding the EnableLmqStats option allows monitoring of LMQ statistics at runtime by @RongtongJin in #8973
• [ISSUE #8961] Automatic recognition of address scheme in Topic Route by host by @dingshuangxi888 in #8962
• [ISSUE #8460] Set default broker name when revive found ack without broker name field by @lizhimins in #8981
• [ISSUE #8982] Dynamically install latest Go version for e2e pipeline by @chi3316 in #8985
• [ISSUE #8976] Modify file segment construct method by @lizhimins in #8977
• [ISSUE #8963] Fix code36 request sent to ns by @yx9o in #8964
• [ISSUE #8968] Introduce the clearRetryTopicWhenDeleteTopic option to enable precise external deletion of topics by @RongtongJin in #8969
• [ISSUE #8991] PrepareHeartbeatData should not be set by default subscriptionDataSet data by @weihubeats in #8992
• [ISSUE #7199] GrpcClientChannel header add null judgement by @weihubeats in #7238
• [ISSUE #9009] Optimize message accumulation display and Metrics in DefaultPullConsumer by @qianye1001 in #9010
• [ISSUE #8984] Fix the broker switch enableMixedMessageType doesn't work by @redlsz in #8986
• [ISSUE #8950] Remove Redundant nullcheck of configPath by @jjastan in #8951
• [ISSUE #8966] Feat: add remote address information to acl perm error by @humkum in #8967
• [ISSUE #9007] Fix client connection local ip is null in RemotingClient by @weihubeats in #9008
• [ISSUE #9014] Fix clusterAclConfigVersion command execution failed by @yx9o in #9017
• [ISSUE #7480] Fix the offset in the timerCheckPoint will not be corrected when the commitlog and consumeQueue are truncated by @RongtongJin in #7488
• [ISSUE #9015] Sync SysFlag and message body inflation status; allow omit of message body by @lizhanhui in #9016
• [ISSUE #8979] Add configurable switch for timer message retry logic by @chi3316 in #8980
• [ISSUE #9034] Refactor(LmqBrokerStatsManager): Refactor Redundant Code in LmqBrokerStatsManager by @asapple in #9034
• [ISSUE #8974] Support recalling of delay message by @imzs in #8975
• [ISSUE #9042] Update createTimerMessageStore call with new parameter by @chi3316 in #9041
• [ISSUE #9021] Correct the error message of acl command by @yx9o in #9022
• [ISSUE #8970] Remove redundant heartbeats by @weihubeats in #8971
• [ISSUE #8988] Support dispatchBehindMilliseconds by @guyinyou in #8989
• [ISSUE #9028] Adjust some error code for SYSTEM_ERROR by @absolute8511 in #9027
• [ISSUE #9054] Optimize log print when client consume message in pop mode by @mxsm in #9055
• [ISSUE #9002] When bytebuffer is not enough, do not throw exception by @leizhiyuan in #9003
• [ISSUE #9072] Fix the permission check for retry topic to get topic route. by @dingshuangxi888 in #9073
• [ISSUE #9025] [RIP-73] Pop Consumption Improvement Based on RocksDB by @lizhimins in #9048
• [ISSUE #8957] Remove excess traffic and fix cache inconsistencies by @3424672656 in #8958
• [ISSUE #9069] Fix the IndexFile ConcurrentModificationException in tiered storage by @wangshaojie4039 in #9071
• [ISSUE #9075] Avoid message type validate in message sync scenario. by @dingshuangxi888 in #9076
• [ISSUE #9080] Fix tranfer logic when get large messages from cache in tiered storage by @yuz10 in #9079
• [ISSUE #8974] Add feature switch of recalling, disable by default. by @imzs in #9067
• [ISSUE #9080] Not hold final message store config in fetcher by @lizhimins in #9086
• [ISSUE #9025] [RIP-73] Fix reset offset not effective in Pop Consumption by @lizhimins in #9087
• [ISSUE #9025] [RIP-73] Modify Pop Consumption rocksdb init config by @lizhimins in #9100
• [ISSUE #9106] Fix revive backoff retry not effective in Pop Consumption based on rocksdb by @redlsz in #9107
• [ISSUE #8998] No retry is required when the remaining time reaches zero by @3424672656 in #8999
• [ISSUE #9105] Fix the issue of duplicate consumption in LMQ by @RongtongJin in #9101
• [ISSUE #9108] Refactor ColdDataCgCtrService#getColdDataFlowCtrInfo by @yx9o in #9109
• [ISSUE #9064] Optimize transaction message callback check logic by @LilMosey in #9062
• [ISSUE #9112] Speedup revive scan in Pop Consumption and support server side reset offset by @lizhimins in #9113
• [ISSUE #9121] Fix CRC32 Check Failing When Value is 0 by @guyinyou in #9122
• [ISSEU #6426] Fix slave broker SubscriptionGroupConfig and MessageRequestMode updating atomically by @AuroraTwinkle in #8983
• [ISSUE #9111] Support export broker RocksDB Config to json file by @qianye1001 in #9114
• [ISSUE #8895] Fix NPE when broker shutdown and optimize the log by @qianye1001 in #9094
• [ISSUE #9128] Fix NPE when gRPC client ack message immediately after changing proxy by @qianye1001 in #9129
• [ISSUE #4570] Fix: Docker usage may occur error in volume mapping params, simple fix by @OneCodeMonkey in #9096
• [ISSUE #9119] Invoke async should handle raw exception instead of CompletionException by @gaoyf in #9120
• [ISSUE #9160] Ensure the requestCode increments sequentially by @KiteSoar in #9159
• [ISSUE #9152] Broker getConsumeStats supports inputting multiple topics by @qianye1001 in #9153
• [ISSUE #9149] Assign offset in offsetTable even if the subscription key not exist. by @dingshuangxi888 in #9150
• [ISSUE #9156] Use fastjson2 in AclUtils#getAclRPCHook by @yx9o in #9157
• [ISSUE #9174] Add a collection of predefined Groups and common che… by @ltamber in #9175
• [ISSUE #9172] Clean pull offset and reset offset when delete subscription group by @lizhimins in #9173
• [ISSUE #9181] Update fastjson version by @ltamber in #9180
• [ISSUE #9182] Fix NameServer will be not ready forever when set needWaitForService to true by @gaoyf in #9183
• [ISSUE #9187] The request should be rejected if the queueOffset equals maxOffset when changing the invisible time by @RongtongJin in #9186
• [ISSUE #9201] Cleanup dead code patterns and improve readability and maintainability. by @Kris20030907 in #9202
• [ISSUE #9191] Provide the ability to replace the remoting layer implementation for Proxy and Broker by @f1amingo in #9192
• [ISSUE #9203] Replace numbers with static variables defined in RequestCode by @xinzhuxiansheng in #9204
• [ISSUE #9206] Fix slave sync topic sub in rocksdb ha by @fujian-zfj in #9207
• [ISSUE #8340] RuntimeInfo and ClusterListSubCommand show ackThreadPoolQueueSize and ackThreadPoolQueueHeadWaitTimeMills by @RongtongJin in #8339
• [ISSUE #8589] Optimize RocksDB CQ shutdown when using DoubleWriteCQ by @qianye1001 in #9212
• [ISSUE #9213] Fix get the earliest time error when data is clean up in tiered storage by @lizhimins in #9214
• [ISSUE #9196] Broker return pop stats when receive notification by @qianye1001 in #9197
• [ISSUE #9217] Fix broker's inflight and available message counts incorrect when the pop consumer service is enabled by @lizhimins in #9218
• [ISSUE #9221] Extract some common code in BrokerPathConfigHelper by @yx9o in #9222
• [ISSUE #8127] Optimize the metric calculation logic of the time wheel by @3424672656 in #8128
• [ISSUE #9184] Optimize QueueLockManager#tryLock method in Pop by @mxsm in #9185

New Contributors

• @mawen12 made their first contribution in #8821
• @taomaree made their first contribution in #8809
• @jjastan made their first contribution in #8910
• @asapple made their first contribution in #9034
• @draw233 made their first contribution in #9037
• @LilMosey made their first contribution in #9062
• @AuroraTwinkle made their first contribution in #8983
• @OneCodeMonkey made their first contribution in #9096
• @Kris20030907 made their first contribution in #9124
• @KiteSoar made their first contribution in #9159
• @Lynxhide made their first contribution in #9158
• @xinzhuxiansheng made their first contribution in #9204

Full Changelog: rocketmq-all-5.3.1...rocketmq-all-5.3.2