Skip to content

Add support for Kafka 4.0.0 #11176

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Feb 28, 2025
Merged

Add support for Kafka 4.0.0 #11176

merged 6 commits into from
Feb 28, 2025

Conversation

scholzj
Copy link
Member

@scholzj scholzj commented Feb 22, 2025

Type of change

  • Enhancement / new feature

Description

This PR adds support for Kafka 4.0.0 to test the 4.0.0-RC0. The agreement is that this should be merged to allow further development and avoid rebasing.


The basic docs update with regards to the new Kafka version is done in this PR. But the larger docs update with regards to Log4j2 needs to be done separately.


The Kafka Agent has an issue because of backward incompatibilities between Jetty 9 used in Kafka 3 and Jetty 12 used in Kafka 4. The way this PR solves it is by creating two modules:

  • kafka-agent-3 for Kafka 3 with Jetty 9
  • kafka-agent for Kafka 4 with Jetty 12

During the container image build, the right version is copied. Once we drop support for Kafka 3.9, we will simply delete the kafka-agent-3 module.


The log4j2 support relies on the Log4j2 reloading features instead of on Kafka APIs. The logging changes are also currently not fully documented and unit tested.

Checklist

  • Write tests
  • Make sure all tests pass
  • Update documentation
  • Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
  • Update CHANGELOG.md

@scholzj scholzj added this to the 0.46.0 milestone Feb 22, 2025
@scholzj
Copy link
Member Author

scholzj commented Feb 23, 2025

/azp run upgrade

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 23, 2025

/azp run upgrade

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 24, 2025

/azp run regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 24, 2025

/azp run regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj scholzj force-pushed the add-support-for-kafka-4.0.0 branch from 7dc62c4 to 84f1eb2 Compare February 24, 2025 21:18
@scholzj
Copy link
Member Author

scholzj commented Feb 25, 2025

/azp run regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Jakub Scholz <[email protected]>
Signed-off-by: Jakub Scholz <[email protected]>
Signed-off-by: Jakub Scholz <[email protected]>
@scholzj scholzj force-pushed the add-support-for-kafka-4.0.0 branch from 6d906a9 to f3532e2 Compare February 25, 2025 20:01
Signed-off-by: Jakub Scholz <[email protected]>
@scholzj
Copy link
Member Author

scholzj commented Feb 25, 2025

/azp run upgrade

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 25, 2025

/azp run regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Jakub Scholz <[email protected]>
@scholzj
Copy link
Member Author

scholzj commented Feb 26, 2025

/azp run regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 26, 2025

/azp run feature-gates-regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 26, 2025

/azp run upgrade

1 similar comment
@scholzj
Copy link
Member Author

scholzj commented Feb 26, 2025

/azp run upgrade

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@see-quick see-quick left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one nit otherwise it look good to me.

Copy link
Member

@im-konge im-konge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, just one thing (but nothing important).
Thanks for going through all the test-cases and fixing the issues :)

@scholzj scholzj marked this pull request as ready for review February 27, 2025 16:34
@scholzj
Copy link
Member Author

scholzj commented Feb 27, 2025

/azp run regression

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

@scholzj
Copy link
Member Author

scholzj commented Feb 27, 2025

/azp run upgrade

Copy link

Azure Pipelines successfully started running 1 pipeline(s).

Copy link
Member

@ppatierno ppatierno left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Thanks Jakub!

@scholzj scholzj merged commit 0eb2ba7 into main Feb 28, 2025
25 checks passed
@scholzj scholzj deleted the add-support-for-kafka-4.0.0 branch February 28, 2025 10:39
@scholzj scholzj added this to Roadmap Feb 28, 2025
@scholzj scholzj moved this to 0.46.0 (Work In Progress) in Roadmap Feb 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: 0.46.0 (Work In Progress)
Development

Successfully merging this pull request may close these issues.

4 participants