CA-404062: Wrongly restart xapi when receiving HTTP errors #6201
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The xapi on a supporter host would restart when it received HTTP error from the xapi on the coordinator host. This breaks the pool.designate_new_master use case for a big pool, e.g. 64-host pool. In this case, some supporters may restart unexpectedly within the phase of committing new coordinator due to the logic above. Additionally, the purpose of this logic, explained by the error message, is not correct also. Not all HTTP errors are caused by "our master address is wrong". On the other hand, if a use case requires to restart the xapi, an more explicit logic should ensure that, instead of leveraging an implicit HTTP error code. Furhtermore, if a supporter indeed is connecting to a wrong coordinator, this should be a bug and can be recovered manually. Based on above arguments, the restarting xapi after receiving HTTP error is removed. This follows the TODO concluded in CA-36936 as well. Signed-off-by: Ming Lu <[email protected]>
Signed-off-by: Ming Lu <[email protected]>
psafont
approved these changes
Jan 2, 2025
|
The backoff code is quite messy, but fixing it is unrelated to the issue. |
gangj
reviewed
Jan 2, 2025
| Thread.delay !Db_globs.permanent_master_failure_retry_interval ; | ||
| !Db_globs.restart_fn () | ||
| error "Received HTTP error %s (%s) from the coordinator" http_code err_msg ; | ||
| reconnect () |
There was a problem hiding this comment.
Is it possible that the HTTP error is because "our master address is wrong" as the original comment?
There was a problem hiding this comment.
Oh, I see the detailed explanation in the commit message.
gangj
reviewed
Jan 2, 2025
ocaml/database/master_connection.ml
Outdated
| if !connection_timeout < 0. then ( | ||
| if not !surpress_no_timeout_logs then ( | ||
| debug | ||
| "Connection to master died. I will continue to retry \ |
There was a problem hiding this comment.
Might be good to update to "coordinator" for those logs.
There was a problem hiding this comment.
The most of the change is copy-and-paste. So I kept the code as it was.
gangj
approved these changes
Jan 2, 2025
minglumlu
added a commit
that referenced
this pull request
Jan 3, 2025
(This is a back porting from a [PR](#6201) against master branch) The xapi on a supporter host would restart when it received HTTP error from the xapi on the coordinator host. This breaks the pool.designate_new_master use case for a big pool, e.g. 64-host pool. In this case, some supporters may restart unexpectedly within the phase of committing new coordinator due to the logic above. Additionally, the purpose of this logic, explained by the error message, is not correct also. Not all HTTP errors are caused by "our master address is wrong". On the other hand, if a use case requires to restart the xapi, a more explicit logic should ensure that, instead of leveraging an implicit HTTP error code. Furhtermore, if a supporter indeed is connecting to a wrong coordinator, this should be a bug and can be recovered manually. Based on above arguments, the restarting xapi after receiving HTTP error is removed. This follows the TODO concluded in CA-36936 as well.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The xapi on a supporter host would restart when it received HTTP error
from the xapi on the coordinator host.
This breaks the pool.designate_new_master use case for a big pool, e.g.
64-host pool. In this case, some supporters may restart unexpectedly
within the phase of committing new coordinator due to the logic above.
Additionally, the purpose of this logic, explained by the error message,
is not correct also. Not all HTTP errors are caused by "our master
address is wrong".
On the other hand, if a use case requires to restart the xapi, an more
explicit logic should ensure that, instead of leveraging an implicit
HTTP error code. Furhtermore, if a supporter indeed is connecting to a
wrong coordinator, this should be a bug and can be recovered manually.
Based on above arguments, the restarting xapi after receiving HTTP error
is removed. This follows the TODO concluded in CA-36936 as well.