Skip to content

spring-kafka @KafkaListener lead to spring project deadlock Stuck and stopped problems #3835

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
CZMHHH opened this issue Apr 9, 2025 · 2 comments
Open

spring-kafka @KafkaListener lead to spring project deadlock Stuck and stopped problems #3835

CZMHHH opened this issue Apr 9, 2025 · 2 comments
Labels
status: waiting-for-triage type: bug

Comments

@CZMHHH
Copy link

CZMHHH commented Apr 9, 2025

this is my spring-kafka dependency

org.springframework.kafka
spring-kafka
3.3.1


org.apache.kafka
kafka-clients


    <dependency>
        <groupId>org.apache.kafka</groupId>
        <artifactId>kafka-clients</artifactId>
        <version>3.6.0</version>  <!-- 确保使用 2.0+ 或 3.0+ 版本 -->
    </dependency>

this is my spring config
spring.kafka.consumer.group-id=Consumer
spring.kafka.consumer.type=skuTagAutoUpdate
spring.kafka.consumer.enable-auto-commit=false
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.listener.ack-mode=manual
spring.kafka.properties.security.protocol=SASL_PLAINTEXT
spring.kafka.properties.sasl.mechanism=PLAIN

When publishing services in batches There are many Kafka rebalancing logs, like the following

Image
@CZMHHH
Copy link
Author

CZMHHH commented Apr 9, 2025

The machine will report these logs every once in a while
[2025-04-09T17:01:46.232][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:1072] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Request joining group due to: group is already rebalancing
[2025-04-09T17:01:46.233][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:343] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Revoke previously assigned partitions
[2025-04-09T17:01:46.233][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:242] userId[] partitions revoked:[]
[2025-04-09T17:01:46.233][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:576] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] (Re-)joining group
[2025-04-09T17:01:46.535][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:637] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully joined group with generation Generation{generationId=371, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:01:46.555][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:812] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully synced group in generation Generation{generationId=371, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:01:46.556][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:312] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Notifying assignor about the new Assignment(partitions=[])
[2025-04-09T17:01:46.556][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:324] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Adding newly assigned partitions:
[2025-04-09T17:01:46.556][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:247] userId[] partitions assigned:[]
[2025-04-09T17:02:34.561][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:1072] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Request joining group due to: group is already rebalancing
[2025-04-09T17:02:34.562][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:343] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Revoke previously assigned partitions
[2025-04-09T17:02:34.562][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:242] userId[] partitions revoked:[]
[2025-04-09T17:02:34.562][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:576] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] (Re-)joining group
[2025-04-09T17:02:34.687][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:637] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully joined group with generation Generation{generationId=372, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:02:34.707][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:812] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully synced group in generation Generation{generationId=372, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:02:34.707][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:312] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Notifying assignor about the new Assignment(partitions=[])
[2025-04-09T17:02:34.707][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:324] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Adding newly assigned partitions:
[2025-04-09T17:02:34.707][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:247] userId[] partitions assigned:[]
[2025-04-09T17:04:10.720][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:1072] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Request joining group due to: group is already rebalancing
[2025-04-09T17:04:10.720][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:343] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Revoke previously assigned partitions
[2025-04-09T17:04:10.720][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:242] userId[] partitions revoked:[]
[2025-04-09T17:04:10.720][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:576] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] (Re-)joining group
[2025-04-09T17:04:11.062][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:637] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully joined group with generation Generation{generationId=373, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:04:11.083][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:812] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully synced group in generation Generation{generationId=373, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:04:11.084][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:312] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Notifying assignor about the new Assignment(partitions=[])
[2025-04-09T17:04:11.084][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:324] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Adding newly assigned partitions:
[2025-04-09T17:04:11.084][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:247] userId[] partitions assigned:[]
[2025-04-09T17:05:20.097][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:1072] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Request joining group due to: group is already rebalancing
[2025-04-09T17:05:20.098][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:343] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Revoke previously assigned partitions
[2025-04-09T17:05:20.098][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:242] userId[] partitions revoked:[]
[2025-04-09T17:05:20.098][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:576] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] (Re-)joining group
[2025-04-09T17:05:20.199][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:637] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully joined group with generation Generation{generationId=374, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:05:20.219][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:812] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully synced group in generation Generation{generationId=374, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:05:20.220][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:312] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Notifying assignor about the new Assignment(partitions=[])
[2025-04-09T17:05:20.220][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:324] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Adding newly assigned partitions:
[2025-04-09T17:05:20.220][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:247] userId[] partitions assigned:[]
[2025-04-09T17:09:05.250][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:1072] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Request joining group due to: group is already rebalancing
[2025-04-09T17:09:05.251][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:343] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Revoke previously assigned partitions
[2025-04-09T17:09:05.251][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:242] userId[] partitions revoked:[]
[2025-04-09T17:09:05.251][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:576] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] (Re-)joining group
[2025-04-09T17:09:05.360][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:637] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully joined group with generation Generation{generationId=375, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:09:05.381][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:812] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully synced group in generation Generation{generationId=375, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:09:05.381][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:312] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Notifying assignor about the new Assignment(partitions=[])
[2025-04-09T17:09:05.381][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:324] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Adding newly assigned partitions:
[2025-04-09T17:09:05.381][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:247] userId[] partitions assigned:[]
[2025-04-09T17:09:41.386][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:1072] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Request joining group due to: group is already rebalancing
[2025-04-09T17:09:41.387][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:343] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Revoke previously assigned partitions
[2025-04-09T17:09:41.387][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:242] userId[] partitions revoked:[]
[2025-04-09T17:09:41.387][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:576] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] (Re-)joining group
[2025-04-09T17:09:41.520][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:637] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully joined group with generation Generation{generationId=376, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:09:41.540][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:812] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Successfully synced group in generation Generation{generationId=376, memberId='consumer-duer_gtm_out-sync-order-1-f17eb65d-9586-4fbf-bbbb-8cc93ce841f2', protocol='range'}
[2025-04-09T17:09:41.540][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:312] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Notifying assignor about the new Assignment(partitions=[])
[2025-04-09T17:09:41.540][INFO][skuTagAutoUpdate-0-C-1][][org.apache.kafka.clients.consumer.internals.ConsumerCoordinator Line:324] userId[] [Consumer clientId=consumer-duer_gtm_out-sync-order-1, groupId=duer_gtm_out-sync-order] Adding newly assigned partitions:
[2025-04-09T17:09:41.540][INFO][skuTagAutoUpdate-0-C-1][][org.springframework.kafka.listener.KafkaMessageListenerContainer Line:247] userId[] partitions assigned:[]

@artembilan
Copy link
Member

When publishing services in batches There are many Kafka rebalancing logs

The rebalancing of the consumer group has nothing to do with the producer.

It is not clear from your logs why rebalancing happens, but in most cases that is because a consumer does not return control back to the poll() method to keep this consumer from being indicated as a blocked and kicked off of the group.

Might be even the case that there is nothing to do with Spring, so you would raise the issue exactly here, but not in Apache Kafka itself.

I even would suggest to go to StackOverflow with more info about your project. That way it might be easier to identify the cause.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
status: waiting-for-triage type: bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@artembilan @CZMHHH