feat: add runtime and host command observability

.github/workflows/ci.yml

@@ -75,6 +75,14 @@ jobs:
         run: docker network create --subnet 172.30.0.0/16 hack-dev
       - name: Run local and Docker E2E
         run: HACK_E2E_REQUIRE_DOCKER=1 HACK_E2E_REQUIRE_TMUX=1 bun run test:e2e:local:docker
+      - name: Smoke container resource metadata without optional fields
+        run: |
+          probe_id=$(docker create --network none --read-only alpine:3.20 true)
+          trap 'docker rm -f "$probe_id" >/dev/null' EXIT
+          bun scripts/inspect-container-resources.ts --container "$probe_id" > "$RUNNER_TEMP/resource-probe.json"
+          bun -e 'const r = await Bun.stdin.json(); if (r.probeStatus !== "not_running" || r.container.healthcheckIntervalNs !== null || r.container.writableLayerBytes !== null) throw new Error("Invalid optional resource metadata");' < "$RUNNER_TEMP/resource-probe.json"
+          bun scripts/inspect-container-resources.ts --container "$probe_id" --storage > "$RUNNER_TEMP/resource-probe.json"
+          bun -e 'const r = await Bun.stdin.json(); if (typeof r.container.writableLayerBytes !== "number") throw new Error("Missing writable-layer accounting");' < "$RUNNER_TEMP/resource-probe.json"
 
   test:
     runs-on: blacksmith-6vcpu-macos-15
@@ -99,3 +107,21 @@ jobs:
         run: bun run build
       - name: Build release smoke (no tests)
         run: bun run build:release --skip-tests --no-clean --out=dist/release-ci
+
+  linux-process-lifetime:
+    runs-on: blacksmith-4vcpu-ubuntu-2404
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Setup Bun
+        uses: oven-sh/setup-bun@v1
+        with:
+          bun-version: "1.3.9"
+      - name: Require native regression tools
+        run: |
+          command -v python3
+          command -v lsof
+      - name: Install dependencies
+        run: bun install --frozen-lockfile
+      - name: Run process and terminal regressions
+        run: bun test tests/daemon-command.test.ts tests/daemon-orphan.test.ts tests/host-exec-lifetime.test.ts tests/host-exec-tty.test.ts tests/shell-observation.test.ts

docs/env.md

@@ -247,7 +247,9 @@ the signal to the command. Commands run in an owned process group,
 so cancellation also stops their descendants, escalating to SIGKILL after two
 seconds if necessary. Hack returns 130 for SIGINT and 143 for SIGTERM. Interactive
 commands use a supervisor in their own foreground group on the same terminal,
-preserving stdin, separate output streams, and Ctrl-Z/foreground resume. The
+preserving stdin, separate output streams, and Ctrl-Z/foreground resume. Piped
+stdin also retains the controlling terminal, so commands can open `/dev/tty`
+for native authentication prompts. The
 supervisor holds group ownership until cancellation cleanup finishes. SIGKILL cannot be forwarded; supervisors must terminate the
 whole owned process tree when force-killing a wrapper. Commands have no implicit
 time limit, and normal completion preserves the command's exit status.
@@ -405,3 +407,49 @@ If you are writing new docs or new project setup flows, document the YAML overla
 - [Sessions](sessions.md)
 - [CLI reference](cli.md)
 - [Pulumi-style env config design](plans/2026-03-27-pulumi-style-env-config-design.md)
+
+
+### Host command lifetime and diagnostics
+
+`hack host exec` and `hack env exec` record payload-free execution metadata under
+`$HACK_HOME/host-commands` (normally `~/.hack/host-commands`). Records contain the
+executable basename, project, process identities, owned process group, intended
+lifetime, elapsed time, exit/cancellation/timeout outcome, and final child CPU/RSS
+accounting. Arguments, shell strings, environment values and command output are
+not recorded. Completed records expire after seven days during later executions;
+interrupted records remain available for review. A diagnostic storage failure
+warns on stderr and preserves command execution and exit status.
+
+```sh
+hack host exec --timeout 60 -- bun scripts/check.ts </dev/null
+hack env exec --timeout 30 -- bun scripts/probe.ts </dev/null
+hack host exec --lifetime persistent -- bun run dev
+hack host ps --json
+hack host ps --project my-project
+```
+
+`--timeout` is a positive number of seconds. It sends SIGTERM to the owned process
+group, escalates after two seconds, and returns 124. It currently requires
+non-TTY stdin; piping input or redirecting from `/dev/null` preserves a safe group
+boundary without changing interactive terminal job control. There is no default
+timeout. `--lifetime persistent` declares intent and does not detach or create a
+mux session; it cannot be combined with a timeout. Persistent commands still
+respond to explicit cancellation. Use lifecycle processes or `hack session` for
+managed persistent workspaces.
+
+`hack host ps` also inspects existing lifecycle sessions using their stored mux
+ownership evidence. It is read-only: missing wrappers and unverified groups are
+reported for review and never killed. A PID must match its recorded start time
+before live command metrics are attributed to it. Lifecycle definition freshness
+is not revalidated by this resource snapshot. Commands started before tracking was
+available cannot be retroactively assigned ownership.
+
+Live CPU time is the cumulative CPU of currently observed tree/group members;
+short-lived descendants already gone are not included. Final CPU comes from the
+OS-reaped child resource usage and is a separate accounting source. RSS sums may
+count shared memory more than once. Missing observations are null, not zero.
+SIGKILL cannot be intercepted: interrupted records may lack a completion result,
+and group members without a surviving identity are explicitly unverified.
+
+See [runtime performance diagnostics](performance.md) for project-listing timings,
+container details, watcher measurements and read-only cleanup previews.

docs/performance.md

@@ -0,0 +1,107 @@
+# Runtime performance diagnostics
+
+Use bounded, repeated measurements against the same inventory. Record CLI version,
+container IDs, daemon freshness and the workload window. A healthy endpoint is not
+evidence that every project is fast. Avoid treating stopped containers, large RSS,
+or bind mounts as a cause without measuring the associated work.
+
+## Project listing
+
+```sh
+hack projects --json --summary --timings
+hack projects --json --summary --timings --no-daemon
+hack projects --json --project my-project --timings
+hack daemon status --json
+```
+
+`--summary` returns project identity and status, service/container/branch/session
+counts, and a separate lifecycle host-process count. The default JSON response
+retains its full detail contract. Load one project's details with `--project`;
+`--summary` cannot be combined with `--details` or `--meta`. Older daemons without
+summary support fall back to direct discovery.
+
+`--timings` writes numeric profiling JSON to stderr and preserves JSON stdout.
+Phases include current-checkout registration, Docker listing, inspect/cache lookup,
+lifecycle discovery, registry reads/updates, project views, optional metadata,
+projection and JSON serialization. The measured handler duration excludes CLI
+startup and downstream stdout consumption. Daemon reads report request phases and
+cache age separately from the last refresh's phases. With older daemons these
+server-side timings may be unavailable.
+
+The daemon `/v1/metrics` endpoint includes `last_refresh_phases_ms`,
+`last_projects_serialization_ms`, and `last_projects_response_bytes`. These describe
+the most recent operations, not an aggregate benchmark. `/v1/projects` accepts
+`summary=true` and `profile=true`. A summary and `include_meta=true` are incompatible.
+Docker inspection requests only the runtime model's fields; environment values and
+container command arguments are not fetched for listing.
+
+## Host commands and container usage
+
+```sh
+hack host ps --json
+hack usage --project my-project --details --json
+hack usage --project my-project --details --watch
+```
+
+Host tracking, timeout and persistence semantics are described in [env.md](env.md).
+`usage --details` adds individual container CPU, memory, I/O, PID counts and mount
+types/locations to JSON, plus a per-container table in human output. The project
+filter includes its branch instances. Stats requests exclude stopped containers
+and synthetic lifecycle entries. Verified tracked host command trees also appear
+in host usage groups. The project filter also scopes these command groups and
+their totals, including branch instances. Shared host infrastructure remains
+visible; reporting does not transfer ownership of other processes.
+
+For a deeper, explicit read-only probe from this checkout:
+
+```sh
+bun scripts/inspect-container-resources.ts --container <id-or-name>
+bun scripts/inspect-container-resources.ts --container <id-or-name> --runtime node
+bun scripts/inspect-container-resources.ts --container <id-or-name> --storage
+```
+
+The probe reports selected Docker metadata, cgroup memory/current/peak/anonymous
+and file counters, OOM events, cumulative CPU, process RSS and inotify watch-entry
+counts. It requires Bun or Node inside a running container. Unsupported runtime,
+procfs/cgroup access or stopped containers are reported as unavailable; the script
+never starts containers. Process enumeration is capped at 128 processes and 4096
+file descriptors with a three-second scan budget. The observer's RSS is reported
+separately, but its allocation still affects cgroup totals. Watch entries are
+counts, not necessarily unique files or proof that watching is expensive. No argv,
+environment, application files or output logs are read by the probe.
+
+Compare at least two CPU/memory samples: cumulative counters alone cannot establish
+current load or a leak. Inspect anonymous memory, file cache and OOM events before
+changing limits. Frequent healthchecks against a full application route can trigger
+rendering/database work; use measured request costs to decide whether an application
+should provide a cheaper readiness route.
+
+## Watchers and caches
+
+Use mount metadata and actual watch counts before adding volumes or ignore rules.
+Measure the same tracked source-file reads on the host and in the container, with
+identical file counts and bytes. Repeated reads measure warmed behavior; do not
+call the first observed read cold without controlling caches. Do not flush system
+caches or write benchmark files into an active source tree as a default diagnostic.
+A shared source bind across many services establishes fan-out, not its CPU cost.
+
+## Stopped containers
+
+```sh
+hack projects prune --dry-run --json
+hack projects prune --project my-project --dry-run --json
+```
+
+The preview uses existing registry/runtime ownership and missing-path checks, and
+changes nothing. It reports candidates, not proof that their writable data is
+safe to discard. A missing working directory might require further review of
+worktree ownership or disconnected storage. Existing `projects prune --json`
+without `--dry-run` applies cleanup, so use the preview explicitly.
+
+The resource probe's `--storage` option adds writable-layer size and changed-path
+prefix counts. Sizes exclude named-volume data and are not exact reclaimed disk
+space. Large changes outside declared volume destinations may be application data;
+review or preserve them before removal. Successful dependency/setup containers can
+legitimately be stopped. Cleanup can recover storage and reduce inventory work;
+stopped containers do not execute CPU work. Never infer that broad pruning is a
+CPU remedy.

docs/reference/cli.md

@@ -1122,6 +1122,7 @@ hack usage [options]
 | Option | Description |
 | --- | --- |
 | `--project <name>` | Target a registered project by name (from ~/.hack/projects.json) |
+| `--details` | Show per-container usage and mount types |
 | `--include-global` | Include global infra projects under ~/.hack (e.g. logging stack) |
 | `--watch` | Refresh usage continuously |
 | `--interval <value>` | Refresh interval (ms) for --watch |
@@ -1154,6 +1155,9 @@ hack projects <subcommand> [options]
 | --- | --- |
 | `--project <name>` | Target a registered project by name (from ~/.hack/projects.json) |
 | `--details` | Show per-project service tables |
+| `--summary` | Return compact project counts with --json; load details with --project |
+| `--timings` | Write numeric listing phase timings to stderr (requires --json) |
+| `--no-daemon` | Read runtime directly instead of the daemon cache |
 | `--meta` | Include git/worktree/session/env metadata (implies --details) |
 | `--include-global` | Include global infra projects under ~/.hack (e.g. logging stack) |
 | `--all` | Include unregistered docker compose projects (best-effort) |
@@ -1178,6 +1182,7 @@ hack projects prune [options]
 | --- | --- |
 | `--project <name>` | Target a registered project by name (from ~/.hack/projects.json) |
 | `--include-global` | Include global infra projects under ~/.hack (e.g. logging stack) |
+| `--dry-run` | Report prune candidates without changing registry entries or containers |
 | `--json` | Output JSON (machine-readable) |
 | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) |
 | `--help, -h` | Show help |
@@ -2250,6 +2255,8 @@ Inject the selected Hack env overlay directly into a one-off host command withou
 | `--service <global|service>` | Target scope (global or a discovered service name) |
 | `--target <host|compose>` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) |
 | `--shell <command>` | Run a shell command string via /bin/sh -lc after env injection so `$VAR` expansion happens inside the child shell |
+| `--timeout <seconds>` | Bound a non-TTY host command; terminate its process group and return 124 on expiry |
+| `--lifetime <command|persistent>` | Declare intended lifetime for diagnostics (default: command; does not detach) |
 | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) |
 | `--help, -h` | Show help |
 | `--version, -v` | Show version |
@@ -2350,6 +2357,7 @@ Use hack host when a command should run on your host machine, not inside the com
 
 | Command | Summary |
 | --- | --- |
+| `hack host ps` | Inspect host command lifetimes, CPU and lifecycle ownership (read-only) |
 | `hack host exec [command...]` | Run a host command with project env injected |
 | `hack host shell` | Open a host shell with project env injected |
 
@@ -2361,6 +2369,26 @@ Use hack host when a command should run on your host machine, not inside the com
 | `--help, -h` | Show help |
 | `--version, -v` | Show version |
 
+## `hack host ps`
+
+Inspect host command lifetimes, CPU and lifecycle ownership (read-only)
+
+### Usage
+
+```bash
+hack host ps [options]
+```
+
+### Options
+
+| Option | Description |
+| --- | --- |
+| `--project <name>` | Target a registered project by name (from ~/.hack/projects.json) |
+| `--json` | Output JSON (machine-readable) |
+| `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) |
+| `--help, -h` | Show help |
+| `--version, -v` | Show version |
+
 ## `hack host exec [command...]`
 
 Run a host command with project env injected
@@ -2389,6 +2417,8 @@ Run a one-off command on the host with the selected Hack env overlay injected. U
 | `--scope <global|service>` | Resolve values for one env scope while still running the command on the host |
 | `--target <host|compose>` | Env view for host commands (default: host rewrites container-oriented addresses for local host execution) |
 | `--shell <command>` | Run a shell command string via /bin/sh -lc after env injection so `$VAR` expansion happens inside the child shell |
+| `--timeout <seconds>` | Bound a non-TTY host command; terminate its process group and return 124 on expiry |
+| `--lifetime <command|persistent>` | Declare intended lifetime for diagnostics (default: command; does not detach) |
 | `--no-interactive` | Never prompt: apply documented defaults or fail with E_INTERACTIVE_REQUIRED (also via HACK_NO_INTERACTIVE=1) |
 | `--help, -h` | Show help |
 | `--version, -v` | Show version |