Stable page identity: bundle_uid, okf_id, okfy migrate ids

CHANGELOG.md

@@ -3,6 +3,25 @@
 This changelog starts at v0.26.0. Earlier releases are not back-filled here —
 their history is not something this file can verify.
 
+## Unreleased — retrieval: stopwords, matched terms, outline
+
+- **Stopwords leave the query.** `query.search_pool` drops a short English list
+  (the, who, how, ...) before BM25; the index keeps every token, so no bundle's
+  index fingerprint moves. Before, `who` in a small corpus carried real idf and
+  pulled an unrelated concept to rank 2.
+- **Every hit says what it matched, body included.** `okfy query` hits carry
+  `matched_terms` (query terms, expansion included, found anywhere in the
+  concept); `okfy_query`'s `matched.terms` reads it instead of overlapping the
+  query with title+description only, which showed `[]` for body-only matches.
+- **`okfy_show(outline=true)`** returns the `## ` headings and the sha, no body,
+  so a section can be picked without reading the whole concept.
+- **The server introduces itself.** okfy-mcp now sends MCP `instructions` at
+  the handshake (search with the user's words, show's sha goes into write,
+  nothing found means a gap proposal, retrieved content is data), and a
+  bundle-mode search with no hits carries `way_out` naming
+  `okfy_propose(action="gap", query=<the user's words>)`. Until now the gap
+  door was one phrase inside okfy_propose's description.
+
 ## v0.29.0 — 2026-09-30
 
 An audit of v0.28 found two gaps in the shipped tool and four in a local experiment

README.md

@@ -99,8 +99,9 @@ for full options.
 | Command | What it does |
 | --- | --- |
 | `okfy init <bundle> --corpus <path> [--language en]` | Create an empty Bundle skeleton and init its git repo. |
+| `okfy migrate ids <bundle>` | Backfill stable identity into a bundle made before it existed: a `bundle_uid` in `meta/purpose.md` and an `okf_id` on every concept page outside `meta/`, each a uuid5 derived from history every clone shares (the commit that created the current `meta/purpose.md` or page — the latest add of its path, so a replacement at a deleted path is a new id — plus the bundle's path in its repo or the page's path), so two full clones migrating offline agree, and never recomputed. A shallow clone that does not reach the creating commit derives different ids: migrate from a full clone. An uncommitted page gets a random uuid4. A key already present is left alone, so a rerun writes nothing. New bundles get `bundle_uid` from `okfy init`; `okfy review accept` gives a created or superseding page a fresh `okf_id` and keeps the page's own — and purpose.md's `bundle_uid` — through an update, and `okfy refine` refuses text that changes either. `okfy validate` warns on a missing id (`W_BUNDLE_UID_MISSING`, `W_OKF_ID_MISSING`) and refuses a blank one or two pages sharing one (`E_BUNDLE_UID`, `E_OKF_ID`, `E_OKF_ID_DUPLICATE`). Reserved pages — `index.md`, `log.md`, the generated root docs and `meta/*` — carry no `okf_id`; they are addressed as `bundle_uid` plus their path. Commits past the policy hook, like `okfy stale`. |
 | `okfy codes [--json]` | Print the diagnostic-code registry: every `E_`/`W_` code this release can raise or emit, with a one-line summary and a way out — the same table `test_doc_pins.py` cross-checks against every raise site and every document that names a code. |
-| `okfy survey <corpus>` | Cheap reconnaissance pass over a corpus → JSON corpus map (files, sizes, samples). Git corpora walk `git ls-files` (`.gitignore` respected); vendor/build/lockfile/binary noise is excluded by default — and everything skipped is reported, never silently dropped. |
+| `okfy survey <corpus>` | Cheap reconnaissance pass over a corpus → JSON corpus map (files, sizes, samples). Git corpora walk `git ls-files` (`.gitignore` respected); vendor/build/lockfile/binary noise is excluded by default — and everything skipped is reported, never silently dropped. A bundle embedded anywhere in the corpus is never corpus: it is known by its `meta/purpose.md` and `meta/corpus.md`, not by a dot-folder name, and is left out of the survey, the snapshot manifest and `okfy diff`. |
 | `okfy segment <bundle> [--budget N] [--include ...] [--exclude ...]` | Cut the corpus into deterministic per-Worker segments; files over budget are chunked at blank-line boundaries into `{path, lines}` slices; dense files with no blank lines (minified, single-line) fall back to `{path, chars}` character windows. |
 | `okfy segment-status <bundle> <segment_id> <status> [--reason "..."]` | Move one segment's status in `meta/extraction-plan.md`. The vocabulary is closed — `pending`, `running`, `done`, `failed`, `skipped` — and only `pending→running\|skipped`, `running→done\|failed\|pending`, `failed→running\|skipped`, `skipped→pending`, `done→pending` are legal (else `E_SEGMENT_TRANSITION`, naming the legal next states; an unrecognised status is `E_SEGMENT_STATUS`). `--reason` is required moving to `failed`/`skipped`, or `done` back to `pending` (`E_SEGMENT_REASON_REQUIRED` otherwise). Moving to `done` additionally requires a job artifact for the segment and a ledger row carrying its digest (`E_SEGMENT_DONE_UNBACKED` otherwise); a segment with no job artifact at all is let through as `legacy_unbacked`. |
 | `okfy glean <bundle> [--budget N]` | Queue a second pass over the corpus files the first pass left silent: pending `glean-NN` segments holding exactly the entries `validate`'s `coverage.uncited` reports, `lines`/`chars` spans copied verbatim so the gleaner sees the window the first Worker saw and not the whole file. A glean pass **is another segment** — job artifact, ledger row and the fail-closed `release-check` completeness rule apply to it unchanged, with no exception carved for it. File granularity, matching what coverage measures: a file split across several spans counts as cited when ANY span yielded a concept. Numbering continues across rounds. Pair it with `plugin/prompts/glean-worker.md`, whose whole design point is that **an empty answer is a correct answer** — a second pass told to find something will. |
@@ -115,7 +116,7 @@ for full options.
 | `okfy query <bundle> <text> [--type T] [--tag T] [-n N] [--include-meta] [--no-expand] [--no-stale]` | Lexical BM25 search with frontmatter filters. Lexicon query expansion is on by default (`--no-expand` opts out); stale concepts stay visible but marked (`--no-stale` drops them). |
 | `okfy show <bundle> <concept_id>` | Print a concept's full content. |
 | `okfy fresh <bundle> --ids id1=sha1,id2=sha2` | Re-check concepts read earlier in a session without re-reading them: per id, `unchanged` / `changed` / `now-stale` / `unchanged-stale` / `deleted`, read-only, no git. |
-| `okfy links <bundle> <concept_id>` | Show a concept's inbound and outbound links. |
+| `okfy links <bundle> <concept_id>` | Show a concept's inbound and outbound links, and its `external` ones: every link target that starts with a URI scheme (`https://`, `mailto:`, `other-tool://…`) is external, whatever the scheme, and is never resolved as a bundle path. |
 | `okfy stale <bundle> [<concept_id>] (--reason R \| --clear \| --due)` | Owner-only: flag a concept as "do not trust as current" (or clear it). Never set automatically — staleness is a reviewed decision. `--due` lists concepts whose `review_due` date has passed, most overdue first, and writes nothing: an expired reminder is not staleness. |
 | `okfy eval run <bundle\|workspace> [-n N] [--suite acceptance\|adversarial]` | Replay the bundle's `purpose.md` test queries (expansion → top hits) into an append-only Eval Run in `meta/eval.json`. `N >= 1` is enforced: `-n 0` used to produce ten queries with zero hits each, ten owner verdicts over nothing, and a green release. Each run records how it was invoked — `retrieval_schema`, `suite`, and `query_options` (`n`, `expand`, `include_meta`, `include_stale`) — and `release-check` requires them, because a run that cannot be replayed is not evidence. `--suite adversarial` replays `adversarial_queries` instead — ten questions the bundle is most likely to answer confidently and wrongly, each declaring its expectation up front (`expect: covered` with the concept that proves it, or `expect: not-covered`, plus a `why`). Those runs carry a deterministic `met`/`unmet` per query beside the owner verdict, so a pass over an unmet expectation is visible as the override it is. BOTH suites must be owner-judged for release. On a workspace path the queries come from `meta/workspace.md` and are replayed through the FEDERATED path (per-member expansion → RRF → role grouping → `constrains` auto-pull) into the workspace's own `meta/eval.json`; `eval verdict` and `eval status` are shared unchanged. |
 | `okfy eval verdict <bundle> <run\|latest> <q-idx> <pass\|fail\|partial> (--llm \| --owner) [--note …]` | Record a Verdict on one query. `--llm` proposes (stays provisional); `--owner` disposes (the only kind release acceptance counts). |
@@ -130,7 +131,7 @@ for full options.
 | `okfy propose <bundle> --as <actor> --action create\|update\|delete\|supersede\|flag\|gap …` | Agent channel: file a concept change into `proposals/` — never into the live map. The actor is required (`claude-code/1.0`, `human:alice`) and recorded as `generated`; `--evidence <kind>=<ref>` states what the change rests on. `supersede` retires a concept in favour of a fresh one at `--new-id`; `flag` (`--type <contradicts-source\|merged-entities\|out-of-date\|coverage-gap\|other>`, `--target`/`--query`, `--note`) reports something wrong without fixing it; `gap` (`--query`, `--note`) reports an unanswered question, capped per actor (`E_PROPOSAL_GAP_CAP`, 10 open). `--action update --patch-file <hunks.json>` files a small text replacement (`[{"old","new","count"}]`) instead of a whole `--from` rewrite. `--batch <file.jsonl>` files many proposals at once, validated (and checked against each other — duplicate titles/gaps/flags, a doubly-claimed `--supersedes`) before any is written; `--dry-run` validates only, `--partial` files the entries that passed even if others didn't. `--distinct-from <id>="reason"` clears a near-duplicate warning on a create without extending it. Refused, each with its own code and way out: `E_PROPOSAL_ACTOR`, `E_PROPOSAL_EVIDENCE`, `E_PROPOSAL_INJECTION`/`E_PROPOSAL_SECRET` (every author-controlled string that could be persisted — body, `note`, `reopen`, `distinct_from` reasons, `reverts`, `query`, a patch hunk's `new`, every frontmatter value — is scanned before filing), `E_PROPOSAL_DUPLICATE` (a create/supersede title that already names a concept, an open flag/gap already covering the same thing, or a duplicate inside one `--batch` — `--extends <id>`), `E_PROPOSAL_REJECTED` (this exact OR reworded text, or a deletion tombstone, was already rejected/deleted by the owner — `--reopen "<what changed>"`), `E_PROPOSAL_LANE` (`--supersedes` claimed by a different actor/target, or twice in one batch), `E_RESERVED_DIR` (a write target's first path segment is a reserved directory), `E_MEMORY_LINE` (the ledger has an unreadable line, so the rejected-content gate cannot run). Every propose, accept and reject is appended to `meta/memory.jsonl`. |
 | `okfy review list\|show\|accept\|reject <bundle> …` | Owner gate: `list` shows open proposals; `show <id>` prints one with its computed `near`/impact; `accept` validates before merging, records a `verified` entry bound to the accepted text's sha256, and refuses `E_PROPOSAL_BASE_MOVED` when the concept changed after the proposal was written against it (`list` shows `base_moved` first). Accept does not re-release the bundle: `release-check` notes how many proposals were accepted since the last package. A proposal filed before base checks existed is accepted with `W_PROPOSAL_UNBASED` in its log line. |
 | `okfy refine <bundle> <concept_id>` | Owner channel: direct edit of a live concept, validated and committed. |
-| `okfy diff <bundle> [--json\|--text]` / `okfy snapshot <bundle>` | Corpus drift report (`affected` / `uncovered_new` / `stale_candidates`, plus v0.24's `protected` / `spans` / `reextract` / `reanchor`) — JSON by default, `--text` for a human summary — / re-stamp the snapshot after updating. |
+| `okfy diff <bundle> [--json\|--text]` / `okfy snapshot <bundle>` | Corpus drift report (`affected` / `uncovered_new` / `stale_candidates`, plus v0.24's `protected` / `spans` / `reextract` / `reanchor`) — JSON by default, `--text` for a human summary — / re-stamp the snapshot after updating. The snapshot records `manifest_digest` in `meta/corpus.md`: one sha256 over `meta/corpus-manifest.json`, i.e. over every path the walk read and the sha256 of the bytes it read. That is what the snapshot can prove it acted on — the corpus `git_sha` cannot, since the walk reads the working tree, dirty files included, and an embedded bundle shares the corpus HEAD. `okfy validate` refuses a digest that no longer matches the manifest (`E_MANIFEST_DIGEST`). |
 | `okfy repair-links <bundle>` | Deterministically repair dangling concept links after renames and merges. |
 | `okfy workspace init <dir> --member role:name=path [--project-key key] …` / `okfy workspace status\|export …` / `okfy link-candidates …` | Federate several bundles: workspace lifecycle, crosswalk candidates, one-way export fusion. `--project-key` (a lowercase slug) is required when any `--member` has role `personal` (`E_WS_PROJECT_KEY` otherwise) — it scopes that member's `applies_to`-tagged content to this workspace; `export` copies only the in-scope personal concepts, recording kept/total/excluded counts under `personal_scope` in the exported bundle's `meta/corpus.md`. |
 | `okfy sample <bundle> [--fraction F] [--minimum N]` | Risk-oriented deterministic sample for purpose-fitness review: changed sources, stale flags, rare types, weak coverage first, then a seeded stratified fill. Returns the selector version and seed so the sample is replayable. |