Skip to content

Add EarthScope S3 workflow with credential and memory fixes - #33

Open
wangyinz wants to merge 1 commit into
masterfrom
codex/earthscope-s3-workflow-auth
Open

Add EarthScope S3 workflow with credential and memory fixes#33
wangyinz wants to merge 1 commit into
masterfrom
codex/earthscope-s3-workflow-auth

Conversation

@wangyinz

@wangyinz wangyinz commented Sep 9, 2026

Copy link
Copy Markdown
Member

Summary

Add the contributed workflow under notebooks/EarthScopeS3, as requested after
the GeoLab follow-up investigation. This is deliberately in mspass_tutorial,
not a new EarthScope-specific subsystem in the MsPASS library.

The directory contains the five output-free notebooks, reader/worker modules,
an extracted testable driver completion, setup/validation instructions, offline
tests, and a small reproducible memory comparison. No CSS tables, credentials,
saved notebook output, or personal scratch prefix is included.

Fixes relative to the supplied notebook/module bundle

  • Credential expiry: reuse EarthScopeClient.user.get_boto3_session() directly.
    The old wrapper froze cached credentials, lost their actual expiration, and
    invented now + 55 minutes, allowing an already-expired token to remain in use.
    The SDK and S3 client remain alive until plugin teardown, including cleanup on
    failed setup. Clients live in named worker plugins rather than spillable task data.
  • Error propagation: only definite missing-object HEAD responses permit
    version fallback. Auth, service, transport, refresh, and upload failures keep
    their original exceptions and stop the workflow instead of becoming repeated
    misleading access messages or ignored False returns. S3 bodies and owned
    serial clients close on failure as well as success.
  • Completion memory: consume the disposable input list while merging and
    reserve the native output vector. Source ensembles are reclaimed before
    serialization. This does not change native pickle or add a serialization mode.
  • Data completeness: paginate listings; use all required days without a
    holding/arrival Cartesian product; include adjacent index days and use half-open
    years; preserve unmatched station names and filter networks by value.
  • Waveform handling: convert multi-rate streams once and detrend the decoded
    data rather than the empty output ensemble.
  • Notebook safety: remove credential-export cells and personal paths, default
    waveform retrieval to a three-day subset with window 1, count successful day
    files, and stream metadata JSON export.

Compatibility and intentional limits

  • Preserve one ordinary TimeSeriesEnsemble pickle per day, its member order,
    metadata/sample values, and the full scratch bucket user prefix. There is
    no station-batch .pklstream format or new MsPASS API.
  • Keep driver completion and the supplied legacy input access point/SDK role;
    no unvalidated endpoint migration or worker-side save rewrite is introduced.
  • The default plugin registration name is s3client; custom keys select matching
    plugin names. Restart/re-register workers when replacing the old modules.
  • Read-side SDK credentials and write-side s3fs credentials remain separate.
    Fixing the input refresh provider does not grant/renew output authorization.
  • A whole-day result can still be large. Initial metadata loading mutates the
    selected database and is not an idempotent resume operation; warnings are in
    the README and notebooks.

Validation

  • 37 tests passed locally, using real native MsPASS/ObsPy objects and fake
    S3/database boundaries, plus the real EarthScope SDK session builder and
    botocore with an offline cache/clock.
  • Credential tests cover cached tokens at minutes 41/55, rotation past 60/120,
    initially near-expired credentials, refresh failures and cleanup. The credential
    subset also passed with SDK 1.6.1 (main native run used SDK 1.7.2).
  • A real LocalCluster(processes=True) verifies plugin registration, worker-local
    lookup of an intentionally unpickleable client, and teardown. The subprocess
    test has a timeout and process-group cleanup.
  • Native tests cover real miniSEED decoding/windowing/detrending, response-body
    cleanup, error propagation, pagination, cross-year grouping, ordinary pickle
    round trips, member order/values, full scratch prefix, and input weak references
    before serialization.
  • Synthetic 64 MiB sample comparison: serialization RSS above baseline decreased
    from approximately 323 MiB to 259 MiB; both pickle streams were 64.16 MiB.
    Reproduce with benchmark_completion_memory.py legacy and fixed in separate
    processes. This is not a prediction for the reported 12–18 GiB GeoLab process.
  • Notebook code parsed, outputs/credential patterns checked, formatting and
    git diff --check passed. Scoped GitHub Actions cover auth, notebook, and worker
    process tests on Python 3.10/3.13; native tests require a MsPASS environment.

Still requires collaborator/GeoLab validation

No live AWS/GeoLab authorization, 75–90 minute real credential renewal, CSS table
load, full-year run, or real scratch upload/peak-RSS result is claimed. The README
lists small-day, representative-day and heavy-day checks before scaling up.
This PR does not establish that every production memory problem is resolved.

Related context: mspass-team/mspass#1024, mspass-team/mspass#1029.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant