Skip to content

[improve][client]PIP-425:Support connecting with next available endpoint for multi-endpoint serviceUrls #24387

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: master
Choose a base branch
from

Conversation

AuroraTwinkle
Copy link
Contributor

@AuroraTwinkle AuroraTwinkle commented Jun 5, 2025

Fixes #22934 (comment)

Main Issue: #22934 (comment)

PIP: #24394

Motivation

As #22934 and #22933 mentioned, when most of the nodes in serviceurl are down (but there is at least one available node), creating consumers and producers through PulsarClient will most likely fail. I think this is not as expected. If the code is robust enough, as long as there is one available node, it should be accessible normally. Therefore, this PR is going to optimize the code logic, remove unavailable nodes through the feedback mechanism, and improve the success rate of PulsarClient requests.

By the way, #22935 removes faulty nodes through a regular health check mechanism, but this brings new problems (frequent creation of connections and increased system load), so this solution is abandoned. See #22934 (comment) for more details!

Modifications

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • Dependencies (add or upgrade a dependency)
  • The public API
  • The schema
  • The default values of configurations
  • The threading model
  • The binary protocol
  • The REST endpoints
  • The admin CLI options
  • The metrics
  • Anything that affects deployment

Documentation

  • doc
  • doc-required
  • doc-not-needed
  • doc-complete

Matching PR in forked repository

PR in forked repository:
AuroraTwinkle#8

@github-actions github-actions bot added the doc-not-needed Your PR changes do not impact docs label Jun 5, 2025
@AuroraTwinkle AuroraTwinkle force-pushed the fix/serviceNameResolver branch from 3910c18 to fbf6613 Compare June 6, 2025 06:05
@AuroraTwinkle AuroraTwinkle force-pushed the fix/serviceNameResolver branch from fbf6613 to e510a8c Compare June 6, 2025 06:36
@AuroraTwinkle AuroraTwinkle changed the title [fix][client]:fix problem that consumer or producer create failed when build PulsarClient with many unavailable broker nodes [fix][client]:fix problem that consumer or producer create failed when many nodes in PulsarClient serviceUrl become unavailable Jun 7, 2025
@AuroraTwinkle AuroraTwinkle changed the title [fix][client]:fix problem that consumer or producer create failed when many nodes in PulsarClient serviceUrl become unavailable [improve][client]:fix problem that consumer or producer create failed when many nodes in PulsarClient serviceUrl become unavailable Jun 7, 2025
@AuroraTwinkle AuroraTwinkle marked this pull request as ready for review June 7, 2025 10:02
@AuroraTwinkle AuroraTwinkle changed the title [improve][client]:fix problem that consumer or producer create failed when many nodes in PulsarClient serviceUrl become unavailable [improve][client]PIP-425:fix problem that consumer or producer create failed when many nodes in PulsarClient serviceUrl become unavailable Jun 7, 2025
@AuroraTwinkle AuroraTwinkle changed the title [improve][client]PIP-425:fix problem that consumer or producer create failed when many nodes in PulsarClient serviceUrl become unavailable [improve][client]PIP-425:Support connecting with next available endpoint for multi-endpoint serviceUrls Jun 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
doc-not-needed Your PR changes do not impact docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[BUG] consumer or producer will create failed frequently when build PulsarClient with many unavailable broker nodes
1 participant