GH-2381 : Support kafka parallel-consumer

[chickenchickenlove]

Introduction

Hi, Spring Team.
This is chickenchickenlove, huge fan of you guys.
I saw this issue (Support kafka parallel-consumer), and this comment as well!

No plans currently; but contributions are welcome!

So, i'd like to implement that integrate Parallel Consumer to spring-kafka
This is skeleton code.

Please take a look when you have some free time. 🙇‍♂️
Thank you for your time!

Background

• Sometimes, Confluent Parallel consumer will be more helpful than kafka consumer, because parallel consumer have good parallel processing (If you want to know more, See this).
• Thus, i think it will be good option to spring-kafka as well.

How To use

@EnableKafka
@EnableParallelConsumer
@SpringBootApplication
public class SpringkafkasampleApplication {

	public static void main(String[] args) {
		SpringApplication.run(SpringkafkasampleApplication.class, args);
	}
}

• Annotated @EnableParallelConsumer to your Spring application. then, some spring bean related with parallel consumer, will be registered to ApplicationContext

@Component
public class MyTestClass implements ParallelConsumerCallback<String, String> {

    private final List<String> topics;
    public MyTestClass(List<String> topics) {
        this.topics = topics;
    }

    @Override
    public void accept(PollContext<String, String> context) {
        context.getConsumerRecordsFlattened().forEach(kvConsumerRecord -> System.out.println(kvConsumerRecord));
    }

    @Override
    public List<String> getTopics() {
        return this.topics;
    }
}

• and then, you should implement interface ParallelConsumerCallback and register their concrete class as spring bean by annotating @Component or @Configuration.
• then, ParallelConsumerFactory will start ParallelConsumer and ParallelConsumer process records with ParallelConsumerCallback concrete class.

ETC

• How about follow the way like @KafkaListener?
  • IMHO, it is unnecessary. because AFAIK, ParallelConsumer can allow only one callback. thus, it is not suitable for ParallelConsumer.
• Can we remove ParallelConsumerImportSelector?
  • Yes. if org.springframework.boot.autoconfigure.kafka.KafkaAutoConfiguration.java class support to register spring bean, ParallelConsumerImportSelector can be removed.

Update from Apr 27.

• I added some interfaces for parallelConsumer.

Thank you for your time, Again.

[pivotal-cla]

@chickenchickenlove Please sign the Contributor License Agreement!

Click here to manually synchronize the status of this Pull Request.

See the FAQ for frequently asked questions.

[pivotal-cla]

@chickenchickenlove Thank you for signing the Contributor License Agreement!

[sobychacko]

@chickenchickenlove Thank you for the PR and the kind words. Next month, we will release the first RC version of 3.2.0. Since this is an entirely new feature and involves dependencies from io.confluent (as opposed to org.apache for the regular consumer), this needs a detailed review to come up with a plan for supporting parallel consumers. I think you are in the right direction, but we will have an internal discussion on this matter. Depending on how things go, we may have to defer this feature to the 3.3.x release, but we will let you know.

[chickenchickenlove]

Hello, @sobychacko !
Thanks for your time 🙇‍♂️

I see! however, that's sounds great. 👍
If after internal discussions, spring-kafka team decides to introduce ParallelConsumer to spring-kafka, and there is anything more to do, Please let me know. I'm more than willing to help.

Also, Please let me know in case of a decision not to introduce ParallelConsumer to spring-kafka!"

Thanks for your time, again 🙇‍♂️

[tagorenathv]

Any update on this please ?

[sobychacko]

@tagorenathv We will look at this after the GA for 3.2.0 (this month) as part of the next minor version.

[BartlomiejSzczotka]

hello @sobychacko any plans to implement it in near future? Thank you in advance for info!

[sobychacko]

@BartlomiejSzczotka Yes, we will still look at this for the 3.3.0 version. It may not be possible for it to be ready in time for the upcoming M1 release, but it will likely be in the M2 timeframe. Thanks!

[chickenchickenlove]

Hi, @sobychacko !
FYI, this PR is currently a draft because we haven't reached a consensus yet.
This means that it reflects only my perspective.
I believe there are better ways to support ParallelConsumer, and we can find them through discussion.

Also, i made new commits for improving code quality at April 27, 28!

So, Take your times and please feel free to talk anything, any thoughs!
Once we reach a consensus, i will write test codes for this feature as well!

Thanks a lot to look at this PR! 😃

[chickenchickenlove]

Hi, @sobychacko .Gently ping. 🙇‍♂️

It's been a while.
Is the integration of the parallel consumer still under review?
Please let me know 🙏

[sobychacko]

@chickenchickenlove For various reasons on our end, we decided to postpone this feature to either the next minor (3.4.0) or the next major release (4.0.0). Either way, we will start working on it in a few months.

[chickenchickenlove]

Thanks for checking 👍
I will be waiting for cue 😃

[chickenchickenlove]

Hi, @sobychacko !
I've heard that 4.0.x is coming in a while.
So, I'm curious to know what you think about bringing ParallelConsumer to Spring-Kafka.
Please let me know, when you have time. 🙇‍♂️

[artembilan]

Spring for Apache Kafka 4.0 is going to be based on Apache Kafka 4.0.
I wonder how that https://github.com/confluentinc/parallel-consumer is going to be compatible.
Perhaps it would be better to wait until it has released respective version based on Apache Kafka 4.0 as well...

[chickenchickenlove]

@artembilan Thanks for your comment 🙇‍♂️
The latest parallel-consumer uses kafka 3.7.1. (link)
Ah, Yes, I agree with you!

I have a question.
If spring-kafka version becomes 4.x.x and, parallel-consumer supports apache kafka 4.0.0, in this situation,
could I try to implement parallel-consumer, should I postpone it?

Looking back, It might be better for spring-kafka not to support parallel-consumer,
because parallel-consumer is not as active as spring-kafka.
So, the slow updates of parallel-consumer could actually become an obstacle for spring-kafka in adopting newer versions of Apache Kafka.

WDYT?

[artembilan]

That's true, too.
We also don't want to have a module based on the out-dated and not supported library.
So, perhaps we just should not do this and continue to advertise our concurrency and reactive streams support in listener container.

[chickenchickenlove]

I see.
If you decide to close issue #2381,
Please let me know! Then, I will close this PR, too.
Or keep the issue, I will keep this PR as well. (I was also interested in implementing this feature, so if there is a possibility of it being implemented in the future, I would like to try implementing it myself. 😅 )

[sobychacko]

@chickenchickenlove We have an action item on our end to re-evaluate parallel consumer as part of the 4.0.0 work. I suggest, leave this open and we can make a decision on it once we actively start working on the 4.0.x changes.