Conversation
* Fixed two `DetachedInstanceError` due to missing `RunModel.run_name`
and `JobModel.job_name`.
* Fixed `Could not fetch server_info for worker ...: ReadError('')`
logged as an error with a traceback, repeating until the worker is
ready. An SSH tunnel to a port nobody listens on yet fails with
`httpx.ReadError`, so this is the normal startup path. Both
`_probe_http_worker()` and `_get_router_workers()` now catch
`httpx.RequestError` and log it at the debug level.
Also:
* `_get_router_workers()` returns `None` instead of `[]` when the
response cannot be read. `[]` means "the router has no workers", so
an unexpected status, an unparsable body, or a missing `workers` key
made the sync re-register every worker.
* Bare `except Exception` clauses are replaced with the exceptions
actually expected, so a programming error is no longer reported as
an unready replica. `sync_router_workers_for_run_model()` logs such
errors itself instead of letting them escape to the pipeline worker,
which leaves the sync row locked until the lock expires.
* `_get_worker()` takes the replica address instead of two pre-built
URLs, and each probe builds the URL it registers the worker under.
`_get_http_worker()`, `_get_grpc_worker()`, and
`_discover_grpc_server_info()` are folded into loops over the
connection modes and the runtime types to probe.
Fixes: #4300
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DetachedInstanceErrordue to missingRunModel.run_nameandJobModel.job_name.Could not fetch server_info for worker ...: ReadError('')logged as an error with a traceback, repeating until the worker is ready. An SSH tunnel to a port nobody listens on yet fails withhttpx.ReadError, so this is the normal startup path. Both_probe_http_worker()and_get_router_workers()now catchhttpx.RequestErrorand log it at the debug level.Also:
_get_router_workers()returnsNoneinstead of[]when the response cannot be read.[]means "the router has no workers", so an unexpected status, an unparsable body, or a missingworkerskey made the sync re-register every worker.except Exceptionclauses are replaced with the exceptions actually expected, so a programming error is no longer reported as an unready replica.sync_router_workers_for_run_model()logs such errors itself instead of letting them escape to the pipeline worker, which leaves the sync row locked until the lock expires._get_worker()takes the replica address instead of two pre-built URLs, and each probe builds the URL it registers the worker under._get_http_worker(),_get_grpc_worker(), and_discover_grpc_server_info()are folded into loops over the connection modes and the runtime types to probe.Fixes: #4300