diff --git a/docs/design/v022-daily-use.md b/docs/design/v022-daily-use.md new file mode 100644 index 00000000..0a52ad3b --- /dev/null +++ b/docs/design/v022-daily-use.md @@ -0,0 +1,138 @@ +# v0.22 design: daily-use reliability and workflow + +Planning and acceptance record for the v0.22 candidate ([#431](https://github.com/raiseCatError/notMyShell/issues/431)). Durable progress lives in the +[v0.22 ledger](../development/v022-ledger.md). Automated results are not physical-terminal validation. + +## Goal + +Make NMSh better for everyday terminal work without turning it into a shell, a terminal emulator or an IDE. The +release starts from reliability: no typed key may be lost, duplicated, misrouted, shown when it should be hidden, or +run unexpectedly. On that foundation it adds a small number of workflow improvements that reuse what NMSh already has +(`/view`, `/edit`, `/open`, the editor bridge, Ask, the queue). Everything else stays where the issue tracker puts it. + +## Baseline and the v0.21 release boundary + +- Baseline: `dev` at `a8b383c2` (v0.21 integrated: #411–#424, then #425, #427 and #428). +- `release/v0.21.0` is now `86bb3a6c`: PR #430 merged `dev` (`a8b383c2`) into it on 2026-10-18. Draft PR #429 is the + release itself; its physical QA has not run. +- **Rule:** v0.22 PRs target `dev` but stay **draft and unmerged until v0.21 has shipped**. A fix that v0.21's physical + QA needs lands on `dev` and reaches the release branch by another `dev` → `release/v0.21.0` merge, as #430 did, so + anything v0.22 merged into `dev` before the release would ride along. v0.22 never touches `release/v0.21.0`, its QA + evidence, `master` or the publishing workflow. +- After v0.21 ships (release PR merged into `master`, tag published), reconcile: bring `master` back into `dev` if the + runbook does, rebase the open v0.22 branches, and move v0.22 changelog entries into the new `[Unreleased]` section + (they are written into `dev`'s `[Unreleased]` today, beside v0.21's). + +## Scope + +### Required reliability fixes (adopted) + +| Slice | Issue | Summary | State | +| --- | --- | --- | --- | +| R1 | #426 | Keys typed before NMSh knows who reads them wait for the look instead of an 800 ms deadline; a runnable first sample is not "busy" | Implemented, draft PR | +| R2 | #356 | Fish key loss after a handed-over read: reproduce under load on the current harness; fix only what reproduces | Investigation | + +### High-value user-facing improvements (adopted, from accepted issues) + +| Slice | Issue | Summary | State | +| --- | --- | --- | --- | +| U1 | #366 | `/view` → open the file at the line on screen in the person's editor through the existing `/open` bridge; refusal messages offer the same | Draft PR #434 | + +### Optional, decided by the maintainer (2026-10-18) + +| Item | Issue | Decision and state | +| --- | --- | --- | +| D1 | #389 | A recalled history entry browses as one entry (Up/Down step entries until it is edited). Reproduced on `dev`; changes a behavior the maintainer triaged as acceptable and a test codifies. **Approved.** Draft PR #438; the `History n/N` indicator is a separate slice on an existing composer edge, draft PR #441 (based on #438) | +| D2 | #432 | Natural-language command suggestions from the composer ([design](v022-natural-language-commands.md)). **Approved as optional:** Off by default, local model first (draft PR #440); remote bring-your-own-key providers later, in a separate PR after a security review | +| D3 | #366 | Choose a terminal editor for "Open with" when `$VISUAL`/`$EDITOR` is unset (detected installed editors). **Approved.** Draft PR #439 | + +### Explicitly deferred (status unchanged) + +#330–#332 (workspace sidebar/panel program, paused), #337 and #338 (worktree and GitHub write workflows, paused), #339 +and #340 (managed-agent adapters, Remote Control research), #333–#335 (tmux Studio, Setup Cat profiles, keymap +remapping), #341 (executable mods), #398 (Learn, deferred by decision), #399 (host config writes), #304 (Theme Bridge +expansion), #296 (hosted demo), #15. The ongoing indices #14, #20 and #305 stay ongoing. Nothing here is reactivated by +v0.22 starting; none is a dependency of the adopted slices. + +## R1 — #426: input ownership under load + +**Behavior.** While a command runs and NMSh has no evidence yet that it reads input, the first key of a draft and an +Enter that would queue are held while the session looks (`input-look`). The hold now ends only when the look answers +or evidence arrives (bounded at 5 s), never at a fixed 800 ms. Until then nothing typed is drawn, sent or queued. A +look still running after 800 ms says so in the composer: `Checking whether is waiting for input · your typing +is kept`. Ctrl+C is never held. In the session, a first sample that sees a process on a CPU with nothing earlier to +compare is "runnable", not computing; when the shell holds its terminal, the follow-up sample decides. + +**Why not the issue's example (route text at 800 ms, hold only the Enter).** It fixes the queueing but still draws a +hidden reply in the composer whenever the look is slow, for a prompt-less `read -s` and for a prompted `sudo` or `ssh` +password alike. The unfixed baseline failed the live test's "the hidden answer is never drawn" assertion this way under +load. Holding until the owner is known keeps the privacy property independent of machine load. The cost is confined to +a loaded machine: type-ahead appears when the look answers instead of at 800 ms, with the note above. + +**Boundaries.** `TerminalApp` hold policy and `InputWatch` sampling only. No protocol change; older services are never +asked and behave as before. No change to queue semantics, Fish's reader handover or the direct-key path. + +**Tests.** Deterministic (mocked timers, a look the test settles): prompt-less line, hidden and prompted (`sudo`) +replies typed at once are never queued, drawn or run; keys after a held Enter stay behind it; the bound releases a +look that never answers; the command ending mid-hold runs the line once; Ctrl+C releases; quick looks add no delay. A +watch test for the runnable first sample (line and key reads) and for a busy shell staying not-waiting. Live: the +input-ownership, input-awareness, typing-after-read, query-order and queue PTY suites; the ownership file 4× in +parallel. + +**Performance.** No added work per key. A look is the existing `stty` + `ps` probe (one or two samples). Normal-load +latency is unchanged; the bound is 5 s. + +**Physical QA.** Ghostty and Terminal.app, zsh, Bash and Fish: `read -r reply; echo got=$reply` and `read -s` typed +at once and after a pause; `sudo -k true` typed at once; type-ahead during `sleep 5`; a heavy build in another window +(load) while repeating the reads: the note appears, nothing is queued or shown. + +## R2 — #356: Fish keys after a handed-over read + +v0.21 stabilization ran the reproducer 40/40 (macOS, Fish 4.9.3, 8× load) and 22/22 (Linux, Fish 3.6). v0.22 repeats +it on the v0.22 baseline at higher parallel load and with the #426 changes. The issue is not declared fixed without a +reproduction and a verified fix; a clean run is recorded on the issue and the status stays with the maintainer. + +## U1 — #366: from `/view` to the person's editor + +**Behavior.** In `/view` of a file, `o` opens that file at the first line on screen with the configured "Open with" +target: the surrounding editor (Zed, VS Code) is launched with argv; a terminal editor (`$VISUAL`/`$EDITOR`) gets a +reviewed command placed in the composer for the person to run, never executed silently. With no editor known, the +viewer says how to set one. `/view`'s refusals (too large, binary, not a regular file) name `/open ` as the way +on, where it applies. `e` (native `/edit`) is unchanged. + +**Boundaries.** Reuses `resolveHostActions`, `editorArguments` and the `/open` dispatch; no new launcher, no editor +registry, no new PTY ownership. Paths stay single argv elements. Viewing output (not a file) offers nothing new. + +**Tests.** Viewer key → action (`o` only for files, with the top visible line); dispatch for Zed/VS Code (spawn argv), +`$EDITOR` (composer text, nothing executed), none (message); narrow width and `NO_COLOR` footer. + +**Physical QA.** Ghostty and Terminal.app with `$EDITOR=nvim` and with Zed/VS Code integrated terminals. + +## D1 — #389 multiline history browsing (approved; #438, #441) + +Reproduction on `dev` (`a8b383c2`): recall a two-line entry with Up, press Up again: the caret moves to the entry's +first line; a third Up recalls the older entry (`tests/composerHistory.test.ts` "a recalled multiline command navigates +its own lines before older history" codifies it). An N-line entry costs N extra presses. Proposed minimal rule: while +the composer shows an unchanged recalled entry, Up/Down step entries; any edit, paste, caret movement other than Up/Down, +or selection ends browsing. Draft multiline editing is unchanged. Implementation would sit in `TerminalApp`'s Up/Down +routing and `ComposerHistory`; `CommandEditor` stays as is. The `History n/N` indicator is a separate step. + +## D2 — natural-language command suggestions (approved as optional; #440) + +See [the design](v022-natural-language-commands.md). Summary: one optional, off-by-default `CommandSuggester` behind +Ask, with a local provider (the person's own OpenAI-compatible local runtime) and an optional bring-your-own-key remote +provider; a request is explicit (`/btw` or a `?? ` prefix), the suggestion is shown for review and inserted into the +composer, and only the person's Enter runs it. Nothing about files, environment or transcripts is sent unless the +person attaches it. + +As decided, the first slice (#440) is the local model only, reached through `/btw`/Ask (no `?? ` prefix and no +composer mode yet), with Copy and Insert and no refinement turns. NMSh's own risk label comes first and the model's +words are quoted and attributed. + +## Release gates + +Per slice: regressions that fail on the old code, focused suites, `verify:fast`, and the full `npm run verify` before +the PR leaves draft. At integration: `verify:release` on the exact head, CI on the exact head (Ubuntu and macOS shards, +Node 22, Fedora subset, timing smoke), and physical QA per slice in Ghostty and Terminal.app. Local runs use +`NMSH_TEST_TMPDIR=/Volumes/T7Tmp/ai-dev` (Unix socket path limit). No release, tag or deployment without the +maintainer. diff --git a/docs/design/v022-natural-language-commands.md b/docs/design/v022-natural-language-commands.md new file mode 100644 index 00000000..d829dba6 --- /dev/null +++ b/docs/design/v022-natural-language-commands.md @@ -0,0 +1,138 @@ +# Natural-language command suggestions + +**Status: approved as optional (decision D2, 2026-10-18); first slice in draft PR #440.** Part of the +[v0.22 design](v022-daily-use.md); discussed in [#432](https://github.com/raiseCatError/notMyShell/issues/432). The first +slice is the local model only. Remote providers are a separate, later PR after a security and credential-handling +review. The decisions are at the end. + +## The need + +Some terminals let a person switch fluidly between typing commands and describing what they want ("find files over +100 MB changed this week"), then go straight back to commands. NMSh already has most of the pieces: Ask (`/btw`, +`/ask`, and `/btw ` straight from the composer), a deterministic resolver for NMSh's own capabilities, an +optional local model that may only choose among those capabilities, and command classifiers used by paste review and +`/track`. What it cannot do is propose an arbitrary shell command for an arbitrary terminal task. Ask's model contract +deliberately forbids that today ("a model cannot introduce a file, session, worktree or command"). + +## Principles + +1. **Nothing runs by itself.** A suggestion is text shown for review. The only way it runs is the person inserting it + into the composer and pressing Enter, exactly like a command they typed (visible, highlighted, history rules apply). +2. **A request is never shell syntax.** It is entered through an NMSh surface (`/btw`, or an explicit Ask mode in the + composer with its own marker), so it can never reach the shell by accident, even with the feature off. +3. **Send the minimum.** By default only the request text, the shell (zsh, Bash, Fish) and the OS family. No files, + environment, history, transcript, working directory or hostname. Anything more is attached per request, shown first. +4. **One Ask.** This is a new outcome inside the existing Ask, after its deterministic resolver, not a second assistant. +5. **Off by default; useful offline.** With no provider, Ask works as today and says how to enable suggestions. +6. **NMSh's own safety check wins.** The command is classified by NMSh's existing classifier, whatever the model says. + +## Flow + +``` +› /btw find files over 100 MB changed this week + (or: Ask mode in the composer, marked and hinted "Ask NMSh · Enter asks · Esc returns to commands") + +Ask NMSh + 1. Deterministic resolver (unchanged). An NMSh action or answer wins. + 2. Unsupported or unclear, and command suggestions are on → CommandSuggester. + + Suggested command · zsh · Claude Haiku 5.5 with your API key + find . -type f -size +100M -mtime -7 + Lists regular files under this directory over 100 MB, modified in the last 7 days. + Read-only (NMSh check) + Enter Insert into the composer · c Copy · r Refine · Esc Back + +› find . -type f -size +100M -mtime -7 ← the composer, back in command mode; Enter runs it +``` + +Refine sends one follow-up with the previous suggestion (no longer conversation). A suggestion NMSh classifies as +destructive, privileged or writing over the network is shown with that label and needs an explicit second key to +insert; it is never offered as a one-key action. + +## Architecture + +- `src/ask/commandSuggest.ts`: the `CommandSuggester` interface, the fixed prompt, the result schema and validation. + + ```ts + interface SuggestRequest { request: string; shell: 'zsh' | 'bash' | 'fish'; os: 'macos' | 'linux'; attachments?: Attachment[] } + interface CommandSuggestion { command: string; explanation: string } + type SuggestResult = {kind: 'suggestion'; suggestion: CommandSuggestion; usage?: {input: number; output: number}} + | {kind: 'none'; reason: string} | {kind: 'unavailable'; reason: string}; + interface CommandSuggester { id: string; label: string; suggest(request: SuggestRequest, signal: AbortSignal): Promise } + ``` + +- Providers, each optional and lazily loaded: + - **Local**: the existing user-global model service (`src/understanding/ModelService.ts`) gains a `command` task type + next to intent and folding, so every window shares one runtime (llama.cpp, or an already-running Ollama or LM + Studio). The recommended 0.6B model is fit for choosing NMSh actions, not for writing shell commands: suggestions + need a larger model the person chose. NMSh never downloads a model for this feature. + - **Anthropic** (bring your own key): the Messages API through the official `@anthropic-ai/sdk`, default model + `claude-haiku-5-5`, low effort, the result constrained by a JSON schema (`output_config.format`) and validated again + locally. + - **OpenAI-compatible endpoint** (optional third): a URL plus an optional key, which covers other hosted services + without per-vendor code. +- Ask gains a typed `suggestion` outcome; `AskPanel` renders it with the existing action controls. +- Settings → Ask → Command suggestions: provider (Off, Local, Anthropic, OpenAI-compatible), model, endpoint, and the + **name** of the environment variable that holds the key (NMSh stores the name, never the key). +- Validation before display: one line unless the person asked for a multi-line script, at most 1 KiB, no control + characters or escape sequences (rendered as visible marks if present, then refused), explanation at most 300 + characters. Anything else is a "could not suggest" result, never a partial command. +- Classification: the same classifier paste review and `/track` use (`src/tasks/repeatSafety.ts` and the shared command + classifier). The model returns no risk field: risk is NMSh's judgment only. + +## Security and privacy + +- **Credentials.** API keys from a provider's developer platform only, read from an environment variable the person + names, at request time, from NMSh's own process (not the managed shell). Never written to configuration, + transcripts, journals, logs or errors. NMSh does not read browser cookies, consumer app sessions or chat + subscriptions, and does not assume a consumer subscription includes API access. +- **What leaves the machine.** Request text, shell and OS family. Attachments (the last command and its exit status, or + one selected output block, bounded and passed through the existing report redaction) are opt-in per request and shown + before sending. The first use of a remote provider shows exactly what will be sent and to which host. +- **Prompt injection.** The request is the person's own text; an attached output block is untrusted. Because nothing is + executed and NMSh classifies the result itself, an injected instruction can at worst produce a suggestion the person + sees, with NMSh's label. +- **Recording.** Ask's existing "Record Ask in transcript" setting applies to the visible request and suggestion. The + prompt sent to the provider and raw responses are never stored. Ask text never enters shell history. +- **Network.** HTTPS to the configured host only; a 15 s timeout; Esc cancels; no background or speculative requests; + no telemetry. + +## Cost and tokens + +A request is a fixed instruction of roughly 300–400 tokens plus the request (20–60 tokens); the answer is 50–150 +tokens. On `claude-haiku-5-5` at $0.10 per million input and $0.50 per million output tokens (current list price for +prompts up to 100K tokens), that is about $0.0001–0.0003 per suggestion, or $0.10–0.30 per thousand, including some +thinking at low effort. Prompt caching does not apply (the fixed prefix is below the minimum cacheable size) and is not +needed. Ask can show a suggestion's token use. Local providers cost nothing beyond the machine's own resources; on an 8 +GB Mac a model large enough to write commands competes with everything else, which is a reason the local path is never +started implicitly. + +## Not in scope + +No agent loop, tool use, file reading, multi-step plans or automatic execution; no conversation memory beyond one +refinement; no new assistant surface; no replacement of the deterministic resolver; no change to how commands run. + +## Tests + +Fake providers only (no network in tests): schema validation; adversarial outputs (escape sequences, control +characters, multi-line, oversize, `rm -rf ~`, `curl … | sh`, a command hidden after a newline); NMSh's label overriding +any model wording; insert puts the exact text in the composer and runs nothing; Esc and timeout cancel; missing key, +unreachable endpoint and model refusal each produce a clear message and the deterministic result; nothing secret in +transcripts, journals or settings; narrow width, `NO_COLOR` and Safe glyphs. + +## Decisions (2026-10-18) + +1. **Remote providers:** allowed as an optional, Off-by-default bring-your-own-key choice, in a separate PR after a + security review. Consumer subscriptions (ChatGPT, Claude) are never used as API credentials; no browser cookies or + session tokens. +2. **Order:** the local model first (#440); Anthropic and OpenAI-compatible providers later. +3. **Entry point:** explicit requests through `/btw` and Ask only. A composer Ask mode can follow after the keybinding + and input-ownership review. +4. **Credentials:** an environment-variable name only, for the remote slice. +5. **Dependency:** decided with the remote slice. + +Behavior in every slice: display only; Insert is deliberate and only Enter runs; NMSh owns the risk label; nothing is +sent silently (files, transcripts, environment, secrets, output); lightweight on an 8 GB Mac. + +**The first slice (#440):** the `suggested` outcome, the local provider through the shared model service, Copy and +Insert, NMSh's label and the validation rules above. Not yet: refinement turns, a composer mode, remote providers. diff --git a/docs/development/v022-ledger.md b/docs/development/v022-ledger.md new file mode 100644 index 00000000..fbaed11a --- /dev/null +++ b/docs/development/v022-ledger.md @@ -0,0 +1,178 @@ +# v0.22 ledger + +Durable state of the v0.22 candidate (#431) so a later session can resume without rediscovering it. Design and +acceptance: [v0.22 design](../design/v022-daily-use.md). Automated results here are not physical-terminal validation. + +## Baseline (verified 2026-10-18) + +- `dev` = `a8b383c2` (v0.21 integrated, plus #425, #427, #428). `master` = `6af5cbab` (v0.20.0 line), an ancestor of + `dev`. `release/v0.21.0` = `d9e08393` (`dev` at `433af8fd` plus the candidate commit). +- PR #430 (`dev` → `release/v0.21.0`) has merged: merge commit `86bb3a6c` (dated 22:02 IST), which contains `dev` + `a8b383c2`. GitHub still reported it open when this session first checked. Draft PR #429 is the v0.21 release. +- **v0.22 PRs stay draft against `dev` until v0.21 ships**: a fix v0.21's QA needs lands on `dev` and reaches the + release by another `dev` → release merge, which would carry anything v0.22 merged earlier. Nothing in v0.22 + touches `release/v0.21.0`, its QA evidence, `master` or publishing. +- Worktree: `/Volumes/T7Tmp/ai-dev/worktrees/notMyShell/v022` (APFS volume `T7Tmp`, UUID + `62D4E8A9-087C-4927-A62F-25BB88B7F1C7`); unfixed comparison worktree `…/v022-base` (detached at `a8b383c2`). + Scratch and evidence: `/Volumes/T7Tmp/ai-dev/scratch/v022/`. Tests run with `NMSH_TEST_TMPDIR=/Volumes/T7Tmp/ai-dev`. + +## Slices + +| # | Slice | Issues | Branch / PR | State | +| --- | --- | --- | --- | --- | +| 0 | Design, ledger, NL proposal | #431, #432 | `docs/v022-design` / #435 (this ledger's PR) | Draft PR | +| R1 | Input ownership under load | #426 | `fix/v022-input-ownership-look` / #433 | Draft PR; redesigned after review (held keys never fall through on a timeout) | +| R3 | Type-ahead eaten after Ctrl+C or a slow prompt hook | #436 | `fix/v022-ctrlc-type-ahead` / #437, based on #433 | Draft PR; two mechanisms reproduced and fixed, the race family not declared solved; carries the combined #433 + #437 fix | +| T1 | Live tests wait for NMSh's journal, not screen text | — | `fix/v022-queue-live-sync` / #442 | Draft PR, test only; CI green | +| R2 | Fish keys after a handed-over read | #356 | — | Not reproduced (32/32 at 8×); left open, deferred | +| — | Fish 3.7 on Linux: output and markers from Fish sometimes lost | #443 | — | Seen 3 times in CI; not reproduced; cause unverified | +| U1 | `/view` → open in editor | #366 | `feature/v022-view-open-editor` / #434 | Draft PR | +| D1 | History browsing, one step per entry | #389 | `feature/v022-history-browse` / #438 | Draft PR (approved as D1) | +| D1b | `History n/N` on a free composer rule | #389 | `feature/v022-history-position` / #441, based on #438 | Draft PR | +| D3 | Choose a terminal editor | #366 | `feature/v022-editor-choice` / #439 | Draft PR (approved as D3) | +| D2 | Ask command suggestions, local model only | #432 | `feature/v022-ask-command-suggestions` / #440 | Draft PR (approved as D2); remote BYOK is a later PR after a security review | + +## Decisions + +- **Maintainer decisions (2026-10-18):** D1 (#389) approved, with the position indicator as a separate slice that uses + an existing composer edge. D2 (#432) approved as optional: deterministic Ask stays the default, suggestions are Off + by default, the local model comes first, remote Anthropic/OpenAI-compatible keys (by environment-variable name only) + come later after a security review, consumer subscriptions are never API credentials, and suggestions are display + only (Insert is deliberate, Enter runs). D3 (#366) approved: discover installed terminal editors, keep explicit + preferences first, never install anything. +- **#426: held keys wait for the owner; a timeout is not a route.** The first version routed held text after 5 s and + on a failed probe, so a password typed for a slow `read -s` or `sudo` could reach the shell as a command. Now: + - After 5 s without an answer, NMSh looks once more. If it still cannot tell, the keys stay held and the composer + says so: Ctrl+S sends them to the program, Tab shows them, Esc discards them. + - Ctrl+C discards held keys, and the command ending routes them as typed. + - A probe that fails or is unavailable counts as "cannot tell", never as "the shell". + - Echo-off canonical input is a hidden request. + + 12 adversarial regressions fail on the first version (`6958c277`). +- **#426: a runnable first sample is inconclusive.** One sample of a process on a CPU with nothing to compare no longer + ends a look; when the shell holds its terminal the follow-up decides. Every other evidence path is unchanged. +- **macOS has no kernel read report.** `ps -o wchan` prints `-` for every process on macOS 27 (checked: zsh/bash + `read`, `cat`, `sleep`, a busy loop, Python `select`, Node stdin), so the two-sample rule stays on macOS. +- **#436: two mechanisms, both reproduced and explained.** + 1. A late input request from the interrupted command arrived after Ctrl+C and took the next typed command as its + answer. A request older than the stop is now ignored, but only while typing still counts as type-ahead (2 s); + after that it is re-applied if it is still the latest state. + 2. A slow prompt hook (zsh `precmd`, Bash `PROMPT_COMMAND`, Fish `fish_prompt`) looked like the finished command + reading input. Each shell now emits an authenticated `done` marker before its other hooks, and it ends the + look. + + The first hypothesis (the shell's read swallowed the line) was wrong; temporary instrumentation found both, and was + removed. The wider family of input-timing races is not declared solved. +- **D2: NMSh's label is its own and conservative.** A review found `echo $(rm -rf ~)` labelled "reads only". + - "Reads only" now needs every part to be plainly read-only: no substitution, no wrapper (`env`, `xargs`…), no + `find` action, and nothing the `/track` gate questions. + - The note leads with NMSh's verdict and quotes the model, attributed. An explanation that mentions NMSh is + rejected. +- **Every new setting is reachable from Setup.** The Setup coverage audit caught D2's and D3's new rows missing from + Setup; both were added (D3 as a route to the editor chooser). +- **Paused/deferred issues stay as they are** (#330–#332, #337–#340, #333–#335, #341, #398, #399, #304, #296, #15). + +## Verification notes + +Full canonical `npm run verify` (local, macOS 27, T7 scratch), with the load average at the time: + +| PR | Head | Result | +| --- | --- | --- | +| #433 | `e566a1e6` | 3,161 tests, 3,159 pass, 2 skipped, 0 failed | +| #437 | `9aae2f03` | 3,149 tests, 3,147 pass, 2 skipped, 0 failed | +| #437 | `83135298` | 6 live tests timed out at load 20–23; all 6 files 46/46 on rerun. A clean full run is pending | +| #434 | `5d81d93e` | 3,148 tests, 3,146 pass, 2 skipped, 0 failed (load 4–13) | +| #438 | `29d73a69` | 3,143 pass, 1 live timeout (load 9–20); 16/16 rerun at load 5 | +| #439 | `4ab9f44a` | 3 Setup coverage failures (fixed in `898fbc33`), 2 live timeouts (16/16 rerun); a fresh full run is pending | +| #437 | `83135298` (again) | 9 failures, none in #437's logic: 7 startup or teardown timeouts (including #437's own Bash slow-hook test, which stalled before the shell started), 1 frontend that did not exit, and the known #426 queued answer (this branch does not contain #433). Not clean evidence; a rerun is owed | +| #440 | `c1a30017` | not valid evidence: 113 failures under memory exhaustion (about 100 MB free, heavy swap), almost all startup timeouts plus starved benchmarks; a rerun is owed | +| #441 | `48f70e96` | pending; the new tests and nearby suites pass | +| #437 | `e2776680` (with #433) | combined: deterministic input suites 61/61, live `promptHookLive` 2/2 and `inputOwnershipLive` 6/6 (load 15–30, under 100 MB free); full run not repeated under memory pressure | +| #442 | `815ecc52` | `commandQueueLive` 19/19, `typingAfterRead` 3/3, forced-delay variants pass; CI green | + +- **CI:** + - #434, #437, #438, #439 and #440: green. + - #433's latest run (`e566a1e6`) failed one Ubuntu Fish live test: a queued `printf` got `write: Interrupted + system call`. The diff changes no signal or resize path, and the test types nothing that would be held. The + failed jobs were rerun; the result is recorded on #433. +- **R1 live:** 4 copies of `tests/inputOwnershipLive.test.ts` in parallel. The unfixed baseline reproduced #426 + under memory pressure, both as a queued answer and as a drawn hidden reply. +- **CPU contention alone does not reproduce #426:** at load 64 a probe (`stty` + `ps`) took 10 ms p50 and 39 ms + max. The slow looks came with 5–7 GB of swap. +- **#436:** reproduced 6/8 and 4/8 on `dev` with live runs, and deterministically with a slow prompt hook at load 3. + `tests/ctrlCTypeAhead.test.ts` and `tests/promptHookLive.test.ts` (live zsh and Bash) fail before the fix. +- **R2 (#356):** Fish 4.9.3, 8 copies in parallel, four rounds: 32/32. A 16-copy run failed at startup in every + copy (environmental). +- **Environment, not product:** + - A Finder window on the T7 makes `LiveSandbox.dispose`'s `lsof +D` time out. + - Spotlight indexing (`mds_stores`) and load above about 20 stall live-test startup on every build. + - Live tests started while another full run is going time out waiting for the first prompt. + +- **#441 CI (`48f70e96`)** failed two live tests. Both are explained; neither is caused by #441. + - **macOS:** `inputOwnershipLive` zsh, the answer to a read queued: #426's signature, and #441's base does not contain #433. The same failure appears in CI on `feature/v021-themes` (`7d8e1529`) and `stabilize/v0.19.0` (`bcc8e760`). + - **Ubuntu:** `commandQueueLive` never saw `Running cat`. A race in the test, reproduced on `a8b383c2` and on #441: its wait matched the queued command's text in the queue list before that command's prompt returned (53→83 ms on `dev`, 701→901 ms on #441), so `cat` was queued and the queue row replaced `Running cat`. Fixed in #442. +- **#442 found the same class of race in `typingAfterRead`** (macOS CI): the wait matched before NMSh's journal recorded the command (6/6 rounds on `dev`), reproduced with a read whose output comes 1 s late. #442 makes both tests wait for the journal record; CI green on `815ecc52`. +- **#437 CI on the combined head (`e2776680`)** failed `typingAfterRead` (the race #442 fixes) and Ubuntu Fish `inputOwnershipLive` (#443: Fish's reader marker never arrived; the typed answer stayed held, safely). +- **#443:** three Ubuntu Fish 3.7.0 failures where output or NMSh's markers from Fish never reached NMSh, one with `write: Interrupted system call`. Not reproduced with Fish 4.9.3 on macOS. Unverified hypothesis: `SIGWINCH` from NMSh's terminal resizes interrupting Fish 3.7 writes. + +## Integration plan (after v0.21 ships; nothing merges before) + +**Order:** #435 → #433 → #437 → #442 → #438 → #441 → #434 → #439 → #440. +- **#437 now contains #433** (merged into it, `4556dfab`). Its combined fix (`e2776680`) settles held typing as typing for a command that ended when the shell says it is done, and keeps evidence of hidden input; see #437. +- **#441 is based on #438.** Retarget it to `dev` once #438 is in. +- **#442 goes early,** so that later CI runs are meaningful. +- **#439 after #434.** Then add the combined test kept in scratch (`zzViewChosenEditor`): `o` in `/view` composes the chosen terminal editor's command. It passes on the candidate. + +**Candidate, built 2026-10-19:** `integration/v022-candidate` = `a8b383c2` plus the nine merges in that order. First head `56da3c56`; after #437's two follow-ups (merged in again), head `567cf533`, tree `2b23866e`. Pushed, with no PR. +- **Conflicts:** the CHANGELOG only, for #439 and #440. Rule: keep every line, and put the incoming bullets at the end of the section they belong to. +- **Verified on the candidate:** + - type check: clean; + - deterministic suites: 61/61 (input ownership, including the combined tests), 83/83 (history, indicator, Keep Awake, composer, viewer, editors) and 199/199 (Ask with its coverage audit, Setup, Settings, providers); + - live: `promptHookLive` 2/2, `typingAfterRead` 3/3, `inputOwnershipLive` 6/6, `commandQueueLive` 19/19; + - the combined `/view` editor check: passes. +- **First CI run** ([39196110708](https://github.com/raiseCatError/notMyShell/actions/runs/39196110708), `56da3c56`): all jobs green except Ubuntu shard 2. `typingAfterRead` zsh timed out waiting for the journal record of `echo NEXT-ahead3` after a `read -s`. + - **Not a flake:** a regression in #437's combined fix. Type-ahead after answering a hidden question was discarded, because a look still saw the answered question. + - **Fixed in #437 (`e391bcb0`);** the automated security review then found that grace unbounded (a second hidden question's reply could run), fixed in `807d3c03`. + - Regression tests for all three cases; input suites 64/64. +- **Second CI run** ([39197269209](https://github.com/raiseCatError/notMyShell/actions/runs/39197269209), `567cf533`): **all green**. Every job passed: Detect, Quality, Node 22, Tests on macOS and Ubuntu (both shards), the Fedora subset, timing smoke on both, and Verify 22.x and 26.x. +- **Not run:** a full local `npm run verify`, since this 8 GB machine already panicked under memory pressure (see below). + +**When v0.21 ships:** +1. Rebuild the candidate from the released `dev`. `[Unreleased]` becomes `[0.21.0]`, and v0.22's entries move into a new `[Unreleased]`. +2. Run `npm run verify` and `verify:release` on its head at reasonable memory (several GB free, no swap storm), CI, and physical QA passes 1–6. +3. Land the PRs in the same order, with the same resolutions. +4. Before announcing, confirm that `dev`'s tree equals the tested head's tree (`git rev-parse HEAD^{tree}`). Any difference means another pass. + +**Separate, later slice: remote bring-your-own-key providers for Ask (#432).** These are Anthropic and OpenAI-compatible, configured by environment-variable name only, after a security and credential-handling review. They are not part of the combined pass above. + +## Crash recovery (2026-10-19) and findings for the test-resource pass + +The machine (8 GB) had a kernel panic: a watchdog timeout with compressed memory full, ten swapfiles and low swap. No v0.22 work was lost. +- **Recovered:** + - every branch matched its remote; + - the candidate was intact; + - this ledger's edits were uncommitted but intact; + - the scratch evidence was complete, with nothing written after the reboot; + - no test of this workstream was running at the panic; + - `git fsck --connectivity-only` found no missing objects. + +Findings for the planned performance and test-resource pass: +- **Local full runs on 8 GB are not trustworthy beside other agents and Chrome.** Two of them in this session were wrecked by memory exhaustion (113 and 29 failures, almost all startup timeouts and starved benchmarks). Focused files, one at a time, plus CI gave clean results. +- **Running focused tests while a full run is going** starved both. Runs should be strictly sequential, and a full run should refuse to start below a memory threshold. +- **Live tests spawn a frontend, a session service and a shell each.** With the runner's concurrency on top, they are the main memory cost; a cap tied to free memory would help. +- **Screen-text waits are fragile under load.** The journal (`Frontend.completions`) is the deterministic signal; other screen-text waits should move to it. +- **Orphans:** processes from other worktrees' live sandboxes (two days old, reparented to `launchd`) were found before the panic, so interrupted runs can leave sandboxes behind. Sandbox teardown after a crash needs an owner-checked cleanup. +- **Storage:** all scratch, caches and worktrees were on the T7. The internal SSD was at 96%, which leaves little room for swap. + +## Physical QA still required + +Checklist: [v0.22 physical QA](../testing/v022-physical-qa.md), one pass per PR. Nothing in v0.22 has had a physical +pass. + +## Open risks + +- v0.21 unreleased: v0.22 cannot integrate into `dev` yet (see the rule above). All v0.22 PRs stay draft. +- #441 is based on #438 and must merge after it. +- #439 and #434 both use the host-actions bridge; whichever merges second picks up the other. +- #436 fixes two explained mechanisms; other timing races in input ownership may remain. +- The changelog entries sit in `dev`'s `[Unreleased]` beside v0.21's and move at reconciliation. diff --git a/docs/testing/v022-physical-qa.md b/docs/testing/v022-physical-qa.md new file mode 100644 index 00000000..f5815f32 --- /dev/null +++ b/docs/testing/v022-physical-qa.md @@ -0,0 +1,135 @@ +# v0.22 physical QA + +## Status + +- **Candidate slices, not integrated.** Each pass belongs to a draft PR held until v0.21 ships; see the + [ledger](../development/v022-ledger.md). Nothing below has been run yet. +- **What does not count:** headless PTY fixtures, synthetic keys, captures of a window driven that way, and green CI. + Each item needs a person at a real terminal with a real keyboard. +- **Build under test:** the PR head (or the later integration commit), built from a clean checkout (`npm ci && npm run + build`); record `node dist/index.js --version` in every row. +- **Isolation:** `scripts/qa/v021-fixtures.sh ` (unchanged, from v0.21) makes a throwaway folder with its + own NMSh configuration and session service, so the QA build neither reads your settings nor attaches to your running + sessions. Your shell's own startup files still run. + +## Results + +| Pass | Host and version | OS | Shell and version | Commit | Result | Notes | +|---|---|---|---|---|---|---| +| 1 Input ownership under load (#426, #433) | Ghostty | macOS | zsh | | Not run | | +| 1 Input ownership under load (#426, #433) | Terminal.app | macOS | Bash | | Not run | | +| 1 Input ownership under load (#426, #433) | Ghostty | macOS | Fish | | Not run | | +| 2 `/view` → your editor (#366, #434) | Ghostty | macOS | zsh | | Not run | | +| 2 `/view` → your editor (#366, #434) | Terminal.app | macOS | Fish | | Not run | | +| 2 `/view` → your editor (#366, #434) | Zed or VS Code terminal | macOS | zsh | | Not run | | +| 3 Type-ahead after Ctrl+C and slow hooks (#436, #437) | Ghostty | macOS | zsh | | Not run | | +| 3 Type-ahead after Ctrl+C and slow hooks (#436, #437) | Terminal.app | macOS | Bash | | Not run | | +| 3 Type-ahead after Ctrl+C and slow hooks (#436, #437) | Ghostty | macOS | Fish | | Not run | | +| 4 History browsing and History n/N (#389, #438, #441) | Ghostty | macOS | zsh | | Not run | | +| 4 History browsing and History n/N (#389, #438, #441) | Terminal.app | macOS | Fish | | Not run | | +| 5 Terminal editor choice (#366, #439) | Ghostty | macOS | zsh | | Not run | | +| 5 Terminal editor choice (#366, #439) | Zed or VS Code terminal | macOS | zsh | | Not run | | +| 6 Ask command suggestions (#432, #440) | Ghostty | macOS | zsh | | Not run | | + +## Walkthrough + +**Prepare once,** in a plain terminal window (not inside NMSh), from the checkout of the build under test: + +```sh +node dist/index.js --version +scripts/qa/v021-fixtures.sh /Volumes/T7Tmp/ai-dev/q22 +``` + +Launch with `/Volumes/T7Tmp/ai-dev/q22/nmsh` (zsh), `… nmsh bash` or `… nmsh fish`; quit with `exit`. + +### Pass 1: input ownership (#426) + +1.1 `read -r reply; echo got=$reply` (Fish: `read reply; echo got=$reply`). Type `hello` and Enter **at once**, without +waiting: `got=hello`, the status row never shows `queued`. +1.2 The same after a two-second pause: same result. +1.3 `read -s` (no prompt; Bash: `read -rs`; Fish: `read --silent reply`). Type `QSECRETQ` and Enter at once: the text +never appears anywhere on screen and nothing is queued. Then `sudo -k true` (Cancel with Ctrl+C after typing if you +prefer not to authenticate): the password is never shown in the composer. +1.4 `sleep 5`, then type `echo next` and Enter at once: it is queued immediately (`≡ 1 queued`) and runs after `sleep`. +1.5 Load: in a second window run something heavy (a large build or test suite, or several at once), then repeat 1.1 and +1.3 a few times. The composer may briefly say `Checking whether … is waiting for input · your typing is kept`; either +way the answer reaches the read and nothing is shown or queued. Note whether the note appeared and roughly how long. +1.6 Ctrl+C right after typing during 1.5's note: the command is interrupted and **the held typing is discarded** (it is +neither sent to the shell nor shown); the composer is empty. +1.7 If NMSh still cannot tell after about five seconds (heavy load, swap), the activity row says `Can't tell whether … +is reading your typing · held, not shown` and the composer offers `Ctrl+S sends it to … · Tab shows it · Esc discards +it`. Try each: Ctrl+S sends the held text to the program (the read gets it); Tab puts it in the composer, visible and +unsent; Esc discards it. Nothing held is ever run as a shell command without one of these. If the state never appears, +note that; it is expected on a lightly loaded machine. + +### Pass 2: `/view` → your editor (#366) + +Make a long file first: `seq 1 500 > long.txt`. + +2.1 With `EDITOR=nvim` (or `vim`) exported in your shell startup or the launching window: `/view long.txt`, scroll to +about line 200 (`:200` Enter), press `o`. The viewer closes and the composer holds `nvim +200 /…/long.txt` (nothing +runs). Enter opens the editor at line 200; quitting it returns to NMSh's composer. +2.2 While a command runs: `sleep 20`, then `/view long.txt`, `o`. The editor command is placed in the composer; +Enter queues it (`≡ 1 queued`) and the editor opens after `sleep` ends. (A draft already in the composer is never +replaced; that case is covered by automated tests, since opening `/view` empties the composer.) +2.3 A name with spaces: `printf 'x\n' > 'my notes.txt'`, `/view my notes.txt` (the rest of the line is the path, no +quotes), `o`: the composer command quotes the path as one argument. +2.4 With `VISUAL` and `EDITOR` unset, outside any editor: `o` says no editor is known and how to set one. +2.5 Inside Zed's or VS Code's integrated terminal: `o` opens the file at the line in that editor; the viewer stays. +2.6 Markdown: `/view README.md` (rendered), `o` opens at the start; after `m` (source) it opens at the line shown. +2.7 Narrow window (about 46 columns): the footer wraps and `Esc close` stays visible. + +### Pass 3: type-ahead after Ctrl+C and a slow prompt hook (#436) + +3.1 `sleep 30`, press Ctrl+C, then **at once** type `echo after` and Enter: `after` is printed exactly once, as its own +command. Nothing is queued forever and nothing is taken as an answer. +3.2 A program that asks just before it is interrupted: `read -r x` (Fish: `read x`), Ctrl+C, then at once `echo after` +Enter: same result. +3.3 A slow prompt hook. zsh: `slow() { sleep 1 }; precmd_functions+=(slow)`. Bash: `PROMPT_COMMAND='sleep 1'`. Fish: +`function slow --on-event fish_prompt; sleep 1; end`. Then run `true` and, while the next prompt is still coming, type +`echo after` and Enter: it runs once, after the hook, and is never shown as an answer to a waiting program. Repeat a few +times, also under load (as in 1.5). + +### Pass 4: history browsing and `History n/N` (#389) + +Run a multi-line command (Ctrl+J between lines, for example `echo one` Ctrl+J `echo two`) and a few one-line ones. + +4.1 From an empty composer, Up recalls the newest; each further Up moves one command, passing the multi-line one in one +press; Down comes back the same way; Down past the newest restores what you had typed. +4.2 Recall the multi-line command, press Left (or type a character), then Up: the caret moves within it. +4.3 Narrow window (about 46 columns) with a long one-line command: one press per entry. +4.4 (#441) While stepping, the bottom composer rule shows `History 1/N` for the newest, counting up as you go back. It +disappears when you type, move the caret or go past the newest. No row is added and nothing moves. +4.5 (#441) With `/keep-awake display` running, Keep Awake keeps its place; the label is on the other rule or not shown. +In a window of about 30 columns, and with Settings → Composer dividers Off, no label appears and nothing wraps. +4.6 (#441) Light and dark themes: the label is readable. + +### Pass 5: terminal editor choice (#366) + +Launch with `VISUAL` and `EDITOR` unset: `env -u VISUAL -u EDITOR /Volumes/T7Tmp/ai-dev/q22/nmsh`. + +5.1 `/open README.md:3` explains that no editor is set and names the terminal editors installed here. +5.2 Settings → Sessions → Terminal editor (also Setup → Sessions & alerts → Terminal editor): installed editors are +marked; the highlighted one previews the exact command. Choosing one that is not installed is refused, and nothing is +installed. Choose one that is installed. +5.3 `/open README.md:3:2` puts its command in the composer at the right line (and column where the editor supports +it); nothing runs until Enter. Quitting the editor returns to NMSh. +5.4 In `/view README.md`, `o` uses the chosen editor (once #434 and #439 are both in the build under test). +5.5 Inside Zed's or VS Code's integrated terminal, the surrounding editor still wins. + +### Pass 6: Ask command suggestions (#432) + +Needs a local model set up in `/llm`. + +6.1 With Settings → Ask → Command suggestions Off (the default), `/btw convert every png here to webp` behaves as +before: no model is asked. +6.2 Set Command suggestions to Local model (Settings, or Setup → Ask & local understanding). The same request shows +one suggested command with `NMSh's check: …` first, then the model's words in quotes with its name. There is Copy and +Insert, never Run. +6.3 Insert puts the command in the composer, unsent; Enter runs it. Copy runs nothing. +6.4 `/btw delete all my files` stays a refusal; no suggestion is offered. +6.5 With local understanding Off in `/llm`, the suggestion attempt says how to turn it on. Ask's own suggestions are +still offered. +6.6 Activity Monitor: no new long-lived process appears beyond the shared model service while suggestions are on. + +**Reporting.** One line per pass, for example `Pass 1 · Ghostty 1.3.1 · zsh 5.9: PASS (note seen once, ~1 s)`.