Fix the KafkaRoller issue talking to controllers

[tinaselenge]

Type of change

Select the type of your PR

• Refactor

Description

• Use describeCluster to discover the active controller and get quorum information from it. This is because describeQuorum can be only sent directly to the active controller. Non active controllers return an error to the request.
• Remove version checks to simplify the changes as this PR will be targeting Strimzi that only supports Kafka version 3.9+.
• Refactor unit tests to remove ZK based testcases and add more coverage for KRaft based clusters.

Checklist

Please go through this checklist and make sure all applicable tasks have been done

• Write tests
• Make sure all tests pass
• Update documentation
• Check RBAC rights for Kubernetes / OpenShift roles
• Try your changes from Pod inside your Kubernetes and OpenShift cluster, not just locally
• Reference relevant issue(s) and close them after merging
• Update CHANGELOG.md
• Supply screenshots for visual changes, such as Grafana dashboards

[scholzj]

/azp run regression

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[showuon]

As discussed in the kafka call, we might need to workaround it by sending DESCRIBE_CLUSTER first to find the leader node, and then send the DESCRIBE_QUORUM to the active controller like the metadata-quorum script did.

[showuon]

On second thought, I think ideally, the retry should be done in the admin client side. Opened KAFKA-18230 for the improvement.

[tinaselenge]

@showuon that's great. Thank you!

[tinaselenge]

As discussed in the kafka call, we might need to workaround it by sending DESCRIBE_CLUSTER first to find the leader node, and then send the DESCRIBE_QUORUM to the active controller like the metadata-quorum script did.

Yes, I agree. I switched the PR to draft to implement describe cluster :)

[tinaselenge]

I have updated the PR so that KafkaRoller uses describeCluster when retrieving active controller id. When getting quorum information, it would create an admin client with just active controller's bootstrap address.

I also removed version check for Kafka version 3.9.0, since this change would be targeting a release that doesn't support older than 3.9.0. This simplifies the code much better.

There are a couple of TODOs I would like to discuss with others, particularly the one for the tcpProbe for individual node since this check does not make sense anymore where it was. @scholzj @ppatierno @katheris, maybe you have thoughts on those? Of course thoughts from anyone else would also be appreciated.

[tinaselenge]

TODO to discuss:
This could cause the Warning message with NOT_LEADER_OR_FOLLOWER but both would result in retrying the node if active controller is not found.

[scholzj]

What exactly are the situations when we expect to run into this? When there are no running brokers and only controllers?

[tinaselenge]

We are querying the active controller from the controllers, not brokers so no running brokers should not be a problem. I think they could return -1, if there is no leader elected yet/election is still in progress? Could be also an indication of quorum being in a bad state.

[ppatierno]

If we cannot get the active controller (for any reasons) how much is it dangerous returning true to roll the current controller we are checking vs returning false, not rolling and waiting a better time to get the active controller? Could it be "forever" because of a bad state in the quorum?

[tinaselenge]

In this case, if we cannot get the active controller, we cannot perform the quorum check. Currently, if we cannot to perform quorum check, we do not roll the pod, it throws UnforceableProblem. So the question, do we follow the same logic, or use the admin client bootstrapped with all controller nodes.

[tinaselenge]

Closing this PR, because instead of trying to fix this issue, we will wait for Kafka 4.0 and later versions that have the fix (KAFKA-18230) to be supported. #10941 (comment)