Skip to content

Dev remote upgrade: stale lease rollback and same-version Broker Pack activation gaps #1468

Description

@luokerenx4

Observed while upgrading an SSH-managed container from an earlier 0.92.1 dev build to e4e6290.

  1. Host/container restart left the prior Alice lease with a fresh heartbeat. New Alice exited with OpenAlice cli-server is already running as pid ..., causing EARLYEXIT activation rollback although the old container was gone. A second installation followed by openalice server stop --home ... allowed a clean start. Inspect runtime lease identity across PID namespaces and shutdown ordering.
  2. Runtime was then on the new content identity, but Alpaca remained STK-only: installed Broker Pack and new Runtime both have product version 0.92.1. getBrokerPackLocalStatus checks only version inequality; rolling dev packs are not tied to dev content. The tested bundled Alpaca artifact was transferred into a separate immutable release, hash-verified, activated atomically, and UTA restarted. Crypto quote/orderbook and SPY indicative chain then succeeded.
  3. Runtime status continues to advertise pending activation despite launchRoot matching the current installed release. The provider status lacks contentIdentity while reconcileActivation expects it.

Current deployment is healthy and the requested capability is verified. Deferred code changes require installer/Guardian and broker-pack publication acceptance beyond the Alpaca adapter task. Fix the general upgrade contracts; avoid ad-hoc startup state cleanup or merely dropping locking.

Follow-up code audit:

  • packages/cli/src/lifecycle.mjs:resolveRuntimeProvider takes Bun identity only from an explicit provider or OPENALICE_RUNTIME_CONTENT_IDENTITY. The installer shim sets OPENALICE_CONTENT_IDENTITY, and the server command supplies no explicit provider. Unlike launch-context.ts, this route does not resolve the release.json identity. Runtime status on the actual new release still lacks provider.contentIdentity; activation-runtime.mjs therefore cannot confirm it.
  • src/services/broker-packs/installer.ts:getBrokerPackLocalStatus compares product version only. Catalog resolution also selects by product version/platform, while rolling dev CLI publication does not publish a corresponding commit-bound Broker Pack catalog. Fixing the comparison alone is insufficient.
  • packages/guardian-runtime/src/process-control.ts:readProcessStartedAt depends on ps and a 2-second timeout; failure returns null. isSameProcess treats null as same process. Container PID reuse plus stable OPENALICE_MACHINE_ID makes this fallback relevant. The production log proves stale-lease rejection after restart, but whether this exact incident was a ps timeout or other identity-read failure has not yet been established.
  • src/main.ts releases the writer lock only after sequential plugin stops; scripts/guardian/prod.mjs force-kills children after five seconds. Interrupted cleanup must therefore be handled by robust ownership identity, not assumed absent.
  • Native lifecycle test injects a correct explicit runtimeProvider.contentIdentity, bypassing the plain server run path that failed here.

Remote read-only probe compiled with release Bun 1.4.0 successfully read current Alice/Connector process start times and rejected an old expected start timestamp. Thus a permanent ps/Bun parsing failure is not reproduced; startup contention/identity availability during container replacement remains an unproven hypothesis. Do not label the incident as a confirmed parser defect.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions