Skip to content

Add a new goal to distribute topic leaders fairly #2267

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: main
Choose a base branch
from

Conversation

hmit
Copy link

@hmit hmit commented Apr 17, 2025

Summary

  1. Why: In current CC goals, there is no goal to balance a topic leaders only which ensure a fair distribution of load for not log compacted topics. The rebalance looks at historic loads for replica placement and it's possible that load changes temporarily causing load-skew for brokers. A potential solution to this problem is to distribute topic leader partitions fairly across all the brokers.
  2. What: Add a new Topic Replica Leader distribution goal.

This PR adds a new goal to distribute Leaders Replica for every topic in a fair manner across all the brokers. If the partition count is not a multiple of brokers, some brokers do end up with extra partitions.

The thresholds can be configured using 3 new knobs added to AnalyzerConfig.

The new goal is modeled after existing TopicReplicaDistributionGoal with following differences:

  1. This is a hard goal
  2. It looks at leader replicas in a broker only
  3. For candidate brokers with equal number of leader replicas for a topic, it uses total leader replicas as a tie-break

Categorization

  • documentation
  • bugfix
  • new feature
  • refactor
  • security/CVE
  • other

hmitnflx added 2 commits April 9, 2025 11:46
…ers across brokers

* add config for TopicLeaderDistributionGoal
* create kaas map and build clustermodel from it
* use generous default, modify values in the test
* more changes, add tests which generate a html report for visual inspection
* plot per broker distribution instead of table, add all replica info (leader + follower), use RackAwareDistributionGoal
* try leadership movement as well, add more clusters
for candidate brokers, use total leader_cnt as 1st level tie-breaker and only then use broker id

introduce unit-tests for topic leaders replicas goal
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants