[CORE-123098] - Upgrade to Kafka 3.9.0 #17
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR resolves linkedin#2178 Upgrading simplekdc version to "2.1.0" which supports a change that can correctly use security classes based on what version of IBM Semeru JDK(if applicable) is being used. There is no regression observed using Semeru, OpenJDK and Temurin JDKs. This newer version(released on 14 August 2024) also caters vulnerability in deps mentioned linkedin#2179 as **org.jboss.xnio:xnio-api** is updated to **3.8.16**[^1] [^1]:https://github.com/apache/directory-kerby/releases/tag/kerby-all-2.1.0#:~:text=Bump%20org.jboss.xnio%3Axnio%2Dapi%20from%203.8.15.Final%20to%203.8.16.Final).
…inkedin#2181) `log4j.properties` files are ignored in the test resources, after renamed, finally I was able to change the loglevels while unit/integration testing. I'm not sure if it was the issue on issue linkedin#2152, but this would be the fix for tests. Prod should work with the log4j.properties file as that is passed with -Dlog4j.configurationFile java opt
Fix 'the the' in the comments
## Summary Why: Improve PR quality and review-ability. What: modifies current PR template to be structured and require more details when submitting PRs. ## Expected Behavior PR must come with sufficient details to address or explain the issue. ## Actual Behavior PR template only requires link to the issue: ``` This PR resolves #<Replace-Me-With-The-Issue-Number-Addressed-By-This-PR>. ``` ## Steps to reproduce 1. either create a new PR or 2. see [the current template](https://github.com/linkedin/cruise-control/blob/c5545ef04618b5b42290edda2ee63eb6bfa2e1a6/docs/pull_request_template.md) ## Known Workarounds People voluntarily provide additional details ## Additional Evidence - n/a ## Categorization - [x] refactor
## Summary ### Why 1. GIthub Actions workflow are native GH workflows 2. Github Actions do not require additional non-github accounts unlike CircleCI 3. plenty of compute resources[^0] available for OSS projects 4. unlike CircleCI resource limits (don't have details) [^0]:https://docs.github.com/en/actions/administering-github-actions/usage-limits-billing-and-administration#availability ### What 1. creates CI workflow `ci.yaml` 2. creates Artifactory workflow: `artifactory.yaml` Workflow structure is documented in the spec[^1] [^1]:https://docs.github.com/en/actions/writing-workflows/workflow-syntax-for-github-actions ## Expected Behavior CI is expected to 1. execute unit tests 1. execute integration tests 1. execute hw platform unit tests 1. publish artifacts to the artifactory when a tag is published 1. provide ability to re-run tests on failures 1. report results to corresponding PR/branch which is to be used as quality gates for PR merging. ## Actual Behavior 1. current Circle CI integration provides [1] [2] [3] [4] from the expected behavior 4. but re-run-ing checks requires additional efforts like logging in into the Circle CI 5. which slows PR feedback loop as users may not have CircleCI credentials and knowledge of the system [1]:https://github.com/linkedin/cruise-control/blob/a298df86095532264f13ca7490cfabb8ff68839f/.circleci/config.yml#L51-L53 [2]:https://github.com/linkedin/cruise-control/blob/a298df86095532264f13ca7490cfabb8ff68839f/.circleci/config.yml#L51-L53 [3]:https://github.com/linkedin/cruise-control/blob/a298df86095532264f13ca7490cfabb8ff68839f/.circleci/config.yml#L5-L34 [4]:https://github.com/linkedin/cruise-control/blob/a298df86095532264f13ca7490cfabb8ff68839f/.circleci/config.yml#L94-L103 ## Steps to reproduce 1. see failed PR checks, ie linkedin#2133 ## Known Workarounds 1. asking PR authors to trigger build ## Migration Plan 1. add GH Actions integration along with CircleCI 2. confirm GH Actions provide equivalent or better functionality 3. remove CircleCI integration 4. ensure publishing via GH actions works ## Categorization - [x] refactor
## Summary 1. Why: when on VPN, I can't run Cruise Control tests as ZK is binding to local real ip address and local network is restricted. 2. What: changing to bind to 127.0.0.1 fixes it (got the idea from Kafka embedded ZK setup. I think it won't make any difference how automation or human would run the tests, pls correct me if I'm wrong.
## Summary 1. Why: to categorize documentation PRs 2. What: adds "documentation" category to the PR template ## Expected Behavior - when users make documentation changes - they should be able to specify documentation as a change category ## Actual Behavior - no documentation category to specify
## Summary 1. Why: Documentation for **min.num.brokers.violate.metric.limit.to.decrease.cluster.concurrency** is missing. 2. What: document the setting
…tion (linkedin#2202) * Add more logging to help debugging the time spent on goal based operation * Update cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/async/progress/OperationProgress.java Co-authored-by: Maryan Hratson <[email protected]> * Update cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/servlet/handler/async/runnable/GoalBasedOperationRunnable.java Co-authored-by: Maryan Hratson <[email protected]> * Update cruise-control/src/main/java/com/linkedin/kafka/cruisecontrol/servlet/handler/async/runnable/GoalBasedOperationRunnable.java Co-authored-by: Maryan Hratson <[email protected]> --------- Co-authored-by: Maryan Hratson <[email protected]>
…#2203) * Remove test-multi-arch job in Circle CI * Remove test-multi-arch definitions
## Summary 1. Why: The test failed sometimes with unexpected method calls. 2. What: The fix is preparing the test to accept invalidate method call too ## Expected Behavior Tests are running without failure ## Actual Behavior Tests are failing sometimes with unexpected method call. ## Steps to Reproduce 1. setup repeated run on e.g. `testCreateUserTask` in IDE 2. observe failure after multiple successful runs (for me it was failing after around 250 successful runs) ## Additional evidence ``` java.lang.AssertionError: On mock adobe#2 (zero indexed): Unexpected method calls: HttpSession.invalidate() at org.easymock.EasyMock.getAssertionError(EasyMock.java:2230) at org.easymock.EasyMock.verify(EasyMock.java:2058) at com.linkedin.kafka.cruisecontrol.servlet.UserTaskManagerTest.testCreateUserTask(UserTaskManagerTest.java:59) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) at org.junit.runners.ParentRunner.run(ParentRunner.java:413) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:112) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) at org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40) at org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60) at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) at com.sun.proxy.$Proxy5.processTestClass(Unknown Source) at org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176) at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:113) at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:65) at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69) at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74) ``` ## Categorization - [x] bugfix - [ ] new feature - [ ] refactor - [ ] CVE - [ ] other
This is a minor improvement to the README.md.
Update dependencies to fix CVEs: Zookeeper, Netty, Jetty, Nimbus JOSE+JWT
…ING_PARTITION_SIZE_THRESHOLD_MB values (linkedin#2212)
* Upgrading kafka to 3.8.0 - config properties rewriting and adding necessary dependencies # Conflicts: # gradle.properties * Upgrading kafka to 3.8.0 - using alternative for removed getAllTopicConfigs zk admin client method * Upgrading kafka to 3.8.0 - adding 3.8 zk client creation way * Upgrading kafka to 3.8.0 - adding 3.8 network client creation way * replication/quota/topic log constants moved in 3.8 again its value hasn't changed, only where it was stored, this way it's backward compatible * Update usages of Metadata to conform to kafka 3.7 interface --------- Co-authored-by: David Simon <[email protected]>
…ruiseControlMetricsReporterSampler (linkedin#2211)
* Upgrading kafka to 3.8.0 - config properties rewriting and adding necessary dependencies * Upgrading kafka to 3.8.0 - using alternative for removed getAllTopicConfigs zk admin client method * Upgrading kafka to 3.8.0 - adding 3.8 zk client creation way * Upgrading kafka to 3.8.0 - adding 3.8 network client creation way * replication/quota/topic log constants moved in 3.8 again its value hasn't changed, only where it was stored, this way it's backward compatible * Update usages of Metadata to conform to kafka 3.7 interface --------- Co-authored-by: David Simon <[email protected]>
azun
approved these changes
Apr 15, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR resolves [CORE-123098]. Syncing latest upstream changes.