Add EarthScope S3 workflow with credential and memory fixes - #33
Open
wangyinz wants to merge 1 commit into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add the contributed workflow under
notebooks/EarthScopeS3, as requested afterthe GeoLab follow-up investigation. This is deliberately in
mspass_tutorial,not a new EarthScope-specific subsystem in the MsPASS library.
The directory contains the five output-free notebooks, reader/worker modules,
an extracted testable driver completion, setup/validation instructions, offline
tests, and a small reproducible memory comparison. No CSS tables, credentials,
saved notebook output, or personal scratch prefix is included.
Fixes relative to the supplied notebook/module bundle
EarthScopeClient.user.get_boto3_session()directly.The old wrapper froze cached credentials, lost their actual expiration, and
invented
now + 55 minutes, allowing an already-expired token to remain in use.The SDK and S3 client remain alive until plugin teardown, including cleanup on
failed setup. Clients live in named worker plugins rather than spillable task data.
version fallback. Auth, service, transport, refresh, and upload failures keep
their original exceptions and stop the workflow instead of becoming repeated
misleading access messages or ignored
Falsereturns. S3 bodies and ownedserial clients close on failure as well as success.
reserve the native output vector. Source ensembles are reclaimed before
serialization. This does not change native pickle or add a serialization mode.
holding/arrival Cartesian product; include adjacent index days and use half-open
years; preserve unmatched station names and filter networks by value.
data rather than the empty output ensemble.
waveform retrieval to a three-day subset with window 1, count successful day
files, and stream metadata JSON export.
Compatibility and intentional limits
TimeSeriesEnsemblepickle per day, its member order,metadata/sample values, and the full scratch bucket user prefix. There is
no station-batch
.pklstreamformat or new MsPASS API.no unvalidated endpoint migration or worker-side save rewrite is introduced.
s3client; custom keys select matchingplugin names. Restart/re-register workers when replacing the old modules.
Fixing the input refresh provider does not grant/renew output authorization.
selected database and is not an idempotent resume operation; warnings are in
the README and notebooks.
Validation
S3/database boundaries, plus the real EarthScope SDK session builder and
botocore with an offline cache/clock.
initially near-expired credentials, refresh failures and cleanup. The credential
subset also passed with SDK 1.6.1 (main native run used SDK 1.7.2).
LocalCluster(processes=True)verifies plugin registration, worker-locallookup of an intentionally unpickleable client, and teardown. The subprocess
test has a timeout and process-group cleanup.
cleanup, error propagation, pagination, cross-year grouping, ordinary pickle
round trips, member order/values, full scratch prefix, and input weak references
before serialization.
from approximately 323 MiB to 259 MiB; both pickle streams were 64.16 MiB.
Reproduce with
benchmark_completion_memory.py legacyandfixedin separateprocesses. This is not a prediction for the reported 12–18 GiB GeoLab process.
git diff --checkpassed. Scoped GitHub Actions cover auth, notebook, and workerprocess tests on Python 3.10/3.13; native tests require a MsPASS environment.
Still requires collaborator/GeoLab validation
No live AWS/GeoLab authorization, 75–90 minute real credential renewal, CSS table
load, full-year run, or real scratch upload/peak-RSS result is claimed. The README
lists small-day, representative-day and heavy-day checks before scaling up.
This PR does not establish that every production memory problem is resolved.
Related context: mspass-team/mspass#1024, mspass-team/mspass#1029.