Repository navigation
Watchdog: fail early on unreachable sources, always write the manifest - #293
Merged
Merged
Conversation
Sources can now be network shares on other rigs, so list every missing or unreachable source up front instead of failing partway through the modality search.
The remote machine registers the session under its own data root, and the watchdog gets the launcher's session directory and the remote one through its network share (remote.session_directory_unc).
transfer() used to start the watchdog service first and raise before writing the manifest when that failed, so the session was not queued at all. The manifest is now written first: the service processes existing manifests in its flag_dir when it starts, so the transfer still happens. If the service cannot be started, transfer() still raises, and the error names the written manifest. Closes #290
bruno-f-cruz
changed the title
The manifest is already written and is processed when the service starts, so a failed start is now logged as an error naming the queued manifest instead of raised. Any error while checking or starting the service is caught, not only CalledProcessError.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Watchdog sources can now be network shares on other rigs (e.g.
\rig-b\data\<session>, passed alongside the local session folder)._create_manifest_from_sessionnow checks every source upfront and raises oneFileNotFoundErrorlisting all missing or unreachable ones, instead of failing partway through the modality search.The folder-based modality search itself is unchanged and already works on UNC paths. The check itself is independent of the XML-RPC PRs.
Note: this checks reachability from the launcher's account. The watchdog process reads the sources at
schedule_timeunder its own account, which needs access to the share too.Example
examples/client_behavior_launcher.pynow shows a transfer covering both machines:(Updated with
mainnow that #291 and #292 are merged; the example usesRemoteRepositoryfrom #292.)Always write the manifest (closes #290)
transfer()used to start the watchdog service first and raise before writing the manifest when that failed, so the session wasn't queued at all. The manifest is now written first, then the service is started if needed.This works because the service processes existing manifests on startup:
EventHandler._startup_manifest_check()in aind-watchdog-service 0.1.6 schedules every*manifest*.*file inflag_dir, and clabe writesmanifest_<session>.yaml. Starting the service never raises: if it can't be started (for any error, or if it still isn't running afterwards),transfer()logs an error naming the manifest, which will be transferred once the service runs.A source that can't be reached, or a manifest that fails to build, still raises without writing anything, as before.