rocketMQ4.9.2 不同broker之间的consumer offset没有同步，enableDLegerCommitLog=true

[zhongxiaoyi738]

[rocketmq@broker-01-68d4859b4d-b7gq7 bin]$ more /opt/conf/broker-01.conf
brokerClusterName = DefaultCluster
brokerName=broker-01
listenPort=31863
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911

must be unique

dLegerSelfId=n1
sendMessageThreadPoolNums=16

[rocketmq@broker-02-84cc5b59fb-d2n2r bin]$ more /opt/conf/broker-02.conf
brokerClusterName = DefaultCluster
brokerName=broker-02
listenPort=31866
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911

must be unique

dLegerSelfId=n2
sendMessageThreadPoolNums=16

[rocketmq@broker-03-584bb8879d-5b9tq bin]$ more /opt/conf/broker-03.conf
brokerClusterName = DefaultCluster
brokerName=broker-03
listenPort=31869
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911

must be unique

dLegerSelfId=n3
sendMessageThreadPoolNums=16

broker-02
#Topic #Group #Broker Name #QID #Broker Offset #Consumer Offset #Diff #LastTime
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 0 133 133 0 2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 1 121 121 0 2025-01-20 15:13:56
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 2 135 134 1 2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-02 3 121 121 0 2025-01-19 14:43:35

broker-03
#Topic #Group #Broker Name #QID #Broker Offset #Consumer Offset #Diff #LastTime
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 0 133 133 0 2025-01-19 14:43:05
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 1 121 121 0 2025-01-20 15:13:56
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 2 135 135 0 2025-02-27 16:39:54
ANALYSIS_TEMPLATE_TO_CARD_UPD ANALYSIS_TEMPLATE_TO_CARD_UPD broker-03 3 121 121 0 2025-01-19 14:43:35

[francisoliverlee]

any error logs?
DLeger Model not suggest, try Controller Mode

[zhongxiaoyi738]

brokerClusterName = DefaultCluster
brokerName=broker-01
listenPort=31863
brokerIP1=10.7.30.237
autoCreateTopicEnable=true
autoCreateSubscriptionGroup=true
maxMessageSize=65535
flushDiskType=ASYNC_FLUSH
deleteWhen=04
fileReservedTime=48
mapedFileSizeCommitLog=1073741824
mapedFileSizeConsumeQueue=30000
diskMaxUsedSpaceRatio=88
storePathRootDir=/home/rocketmq/store/node00
storePathCommitLog=/home/rocketmq/store/node00/commitlog
enableDLegerCommitLog=true
dLegerGroup=RaftNode00
dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
dLegerSelfId=n1
sendMessageThreadPoolNums=16

1、brokerName、dLegerGroup不一致
2、dLegerPeers 与 dLegerSelfId 不一致
请问这2个不一致会有影响么？
感觉BrokerController.handleSlaveSynchronize这个定时任务没有启动哦，jstack打印出来BrokerController只有一个线程

[francisoliverlee]

1、brokerName、dLegerGroup不一致
不影响

2、dLegerPeers 与 dLegerSelfId 不一致
dLegerSelfId是dLegerPeers的一部分。 具体看这里

check peer state change log in codes

[zhongxiaoyi738]

hi，我改了一下rocketMQ的源码（加了很多调试日志），新增加的日志如下：（masterAddrBak=null）

2025-03-11 15:45:30 INFO DLegerRoleChangeHandler_1 - xiaoyi----handleSlaveSynchronize role=SLAVE
2025-03-11 15:45:30 INFO DLegerRoleChangeHandler_1 - xiaoyi----handleSlaveSynchronize ready slaveSyncFuture
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----handleSlaveSynchronize run syncAll
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncConsumerOffset masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncDelayOffset masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncSubscriptionGroupConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879

[zhongxiaoyi738]

dLegerPeers=n1-broker-01.rocketmq-flow.svc.cluster.local:40911;n2-broker-02.rocketmq-flow.svc.cluster.local:40911;n3-broker-03.rocketmq-flow.svc.cluster.local:40911
dLegerSelfId=n1

我们这样配置，对么？
如果用域名，应该如何配置

[francisoliverlee]

满足这个逻辑即可

[zhongxiaoyi738]

补充，从节点的日志摘录（刚刚启动时有报错，但是后续恢复正常了）

[rocketmq@broker-02-7dcffbb679-jmxlb rocketmqlogs]$ more broker_default.log
2025-03-11 16:10:46 INFO main - instrumented a special java.util.Set into: sun.nio.ch.EPollSelectorImpl@458c1321

2025-03-11 16:11:46 INFO voteInvokeExecutor_3 - [n2][GetVoteResponse] {"code":200,"term":-1,"voteResult":"UNKNOWN"}
2025-03-11 16:11:47 INFO QuorumAckChecker-n2 - [n2][CANDIDATE] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
2025-03-11 16:11:49 INFO StateMaintainer - [n2] [PARSE_VOTE_RESULT] cost=2600 term=24 memberNum=3 allNum=2 acceptedNum=0 notReadyTermNum=0 biggerLedgerNum=0 alreadyHasLeader=false maxTerm=24 result=WAIT_TO_REVOTE
2025-03-11 16:11:49 INFO NettyServerPublicExecutor_3 - [n2][ChangeRoleToFollower] from term: 24 leaderId: n1 and currTerm: 24
2025-03-11 16:11:49 ERROR voteInvokeExecutor_4 - Send vote request failed, info[group=RaftNode00,term=24,code=200,local=null,remote=n3,leader=n2]
org.apache.rocketmq.remoting.exception.RemotingConnectException: connect to broker-03.rocketmq-flow.svc.cluster.local:40911 failed
at org.apache.rocketmq.remoting.netty.NettyRemotingClient.invokeAsync(NettyRemotingClient.java:535) ~[rocketmq-remoting-4.9.2.jar:4.9.2]
at io.openmessaging.storage.dledger.DLedgerRpcNettyService.lambda$vote$3(DLedgerRpcNettyService.java:164) ~[dledger-0.2.2.jar:na]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_322]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[na:1.8.0_322]
at java.lang.Thread.run(Thread.java:750) ~[na:1.8.0_322]
2025-03-11 16:11:49 INFO voteInvokeExecutor_4 - [n2][GetVoteResponse] {"code":200,"term":-1,"voteResult":"UNKNOWN"}
2025-03-11 16:11:50 INFO QuorumAckChecker-n2 - [n2][FOLLOWER] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
2025-03-11 16:11:53 INFO QuorumAckChecker-n2 - [n2][FOLLOWER] term=24 ledgerBegin=1374454 ledgerEnd=1746357 committed=1746357 watermarks={}
后面一直打印的日志，跟最后一行一致

[zhongxiaoyi738]

补充问题
rocketMQ4.9.2，开启enableDLegerCommitLog=true
topic、consumerOffset等元数据，是如何同步的？

[francisoliverlee]

DLeger模式只涉及commitlog是raft的， 元数据应该还是一样同步。不过貌似有个bug，dleger模式不能为null，不过我没有验证， 你可以打日志check下

rocketmq/broker/src/main/java/org/apache/rocketmq/broker/BrokerController.java

Line 1139 in c840b20

this.slaveSynchronize.setMasterAddr(null);

[zhongxiaoyi738]

请问你提到的这个bug有修复么？

我加了调试日志
2025-03-11 15:45:33 INFO BrokerControllerScheduledThread1 - xiaoyi----syncTopicConfig masterAddrBak=null, brokerController.getBrokerAddr()=10.2.30.171:31879