Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/fleet-page.yml
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,7 @@ jobs:
path: .gap-contract
persist-credentials: false
- run: python scripts/gaps/build.py --claw-root .gap-contract --check
- run: node --test tests/gap-search.test.js
- run: node --test tests/gap-search.test.js tests/gap-embedding.test.js
- uses: ruby/setup-ruby@b3aaceaef7e0ee231b499e7b5f49c03fc1d32199 # v1
with:
ruby-version: '3.3'
Expand Down
136 changes: 136 additions & 0 deletions _gaps/EMBEDDING_MAP.md

Large diffs are not rendered by default.

24 changes: 21 additions & 3 deletions _gaps/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Maintaining the knowledge-gap browser

The [original implementation plan](IMPLEMENTATION.md) covers the published native-discussion browser and shared export contract (website #582 and CLAW #1215). The [term explorer plan](TERM_EXPLORER.md) extends it with entity frequencies and co-occurrence, tracked in website #583.
The [original implementation plan](IMPLEMENTATION.md) covers the published native-discussion browser and shared export contract (website #582 and CLAW #1215). The [term explorer plan](TERM_EXPLORER.md) extends it with entity frequencies and co-occurrence, tracked in website #583. The [embedding-map contract and refresh commands](EMBEDDING_MAP.md) cover the text-and-record map in website #609.

The browser reads a retained native-data snapshot. It makes no visitor-time GitHub API calls and runs no literature scans. It does not change scientific records or infer gaps from absent fields. Default results are active knowledge gaps and human/model mismatches; all discussions and review-required candidates are separate views. Recorded does not mean scientifically validated.

Expand All @@ -9,15 +9,21 @@ The magenta section links below the title preserve search filters and support di
## Inputs and provenance

- `sources.json` declares the sixteen public corpus repositories, immutable revisions, source roots, adapters, schema/license digests and publication holds. Support and permission are separate. Unsupported or held roots have unknown counts, never zero.
- Public source names and repositories must belong to the committed `_fleet/data/manifest.json` allowlist projected by CLAW's `FleetManifest.public()` API. Membership validation applies before acquisition and again when retained artifacts are published; no private-name denylist is maintained.
- `snapshot.json` retains normalized public items and measured coverage. Record identity includes its source path: distinct documents can share a semantic identifier. Discussion identity also includes the native discussion ID. Source renames need an explicit migration decision.
- `toolchain.json` pins the CLAW normalizer commit and module/schema bytes. CI checks out that commit, validates the snapshot and reproduces the public assets byte for byte.
- `term-vocabulary.json` retains candidate names, identifiers, unqualified native aliases and exact typed synonyms from every publishable pinned record root, with complete acquisition counts and source provenance. Lexical candidates include colliding identities; shared canonical entities also retain their other native owners even when an owner's local name is absent from the discussion prose. Source roots without native gap fields can still supply vocabulary. The AIMech maintained-concept publication hold also applies here.
- `embedding-inputs.json` retains each discussion's complete permitted prose followed by its accepted matched records' identifiers, names and native definitions/descriptions. Every native owner keeps its source attribution even when identical text blocks are deduplicated. `embedding-layout.json` retains model/runtime, chunk coverage, fixed coordinates, original-vector cosine neighbors and projection diagnostics. `embedding-vectors.npy` retains the normalized float32 vectors in discussion-ID order; browsers do not download it.
- `../assets/gaps/manifest.json` points to hashed search, term and detail assets. Large Mechs are partitioned by stable item-ID order into numbered shards below 500 KiB compressed; index rows identify their detail shard. The search worker covers the complete index; details load on demand. JSON and CSV exports include every match, not just the current page, including exact term/pair filters.

`scripts/gaps/terms.py` performs dictionary-based entity recognition over prompt, rationale, evidence explanations, proposed experiment prose and resolution text. It excludes the owning record's metadata, handles Unicode with original-text spans, keeps canonical identifiers separate from document addresses, and leaves ambiguous names unlinked. The generated term sidecar records the policy, vocabulary digest, source coverage and every accepted occurrence. This is conservative lexical recognition, not contextual semantic disambiguation or an assertion of biological relationships. Cloud size uses raw occurrences; the adjacent counts distinguish discussions, and co-occurrence counts a pair once per discussion. Chart counts follow all ordinary search filters before applying the selected entity or pair. Source Mech restricts the questions; Term vocabulary independently restricts entity ownership. Shared entities can belong to multiple vocabularies without double-counting within a view.

Recognition policy version 3 retains the version-2 numeric, compact-symbol and source-scoped precision protections, and adds a canonical case rule for `PFAM:PF02597`: `ThiS family` must preserve the symbol's native case, while `family` remains case-insensitive. Ordinary `this family` prose is excluded across both vocabulary owners; explicit Pfam identifiers remain available. Bare `ThiS` remains excluded by the existing `this` stop-form rule. See the [precision review](reviews/terms-precision-20261115.md) for source definitions, baseline evidence and known recall limits. Scientific identities are never merged from a shared label alone.

The retained embedding inputs cover all 3,830 indexed discussions: 2,976 include at least one native description, 240 have matched record identifiers/names without a native description, and 614 contain gap text alone. The 2,624 accepted entities resolve to 3,380 native owners; 2,971 owners supply descriptions and 409 do not. MediaIngredientMech has no native definition/description field at its pin, so its 217 matched owners are explicitly `unsupported_field`. Optional fields absent from other records are `missing_native_field`. Neither status is filled from curation notes or generated definitions. These counts describe input coverage, not inference or biological validation; the [field mapping](EMBEDDING_MAP.md#native-description-fields-and-input-coverage) records the source distinctions.

The map uses one combined text embedding per discussion, with long inputs chunked in full. Search and entity filters select existing points without recomputing coordinates. Its two-dimensional distances are an exploratory approximation; the inspectable neighbor scores use the original normalized vectors. Labels, source coverage and repeated wording can influence either representation.

Native `research_gaps` in CofactorMech and AssayMech map to `CURATION_TODO`, with unspecified lifecycle/review and no inherited record evidence. Their native target corpus/scope provides the discussion key. Scanner origin, current attribution, review state and native storage are separate contract fields. A scanner-like ID alone establishes none of them.

Dedicated evidence snippets and unconstrained notes are excluded; retained prompts and explanations can still contain attributed material and remain subject to source terms. Every item links its pinned source. Source licenses, contributor attribution and projection changes are recorded in `sources.json`.
Expand All @@ -44,14 +50,24 @@ Verify persisted counts, hashes and source links; inspect a corrected scanner-or
```sh
python scripts/gaps/snapshot.py --claw-root "$CLAW_CHECKOUT" --repos-root "$MECH_CHECKOUTS" --output _gaps/snapshot.json
python scripts/gaps/vocabulary.py --repos-root "$MECH_CHECKOUTS" --output _gaps/term-vocabulary.json
python scripts/gaps/build.py --claw-root "$CLAW_CHECKOUT" --without-embeddings --output .temp/gaps-embedding-refresh-assets --summary .temp/gaps-embedding-refresh-summary.html
python scripts/gaps/build.py --claw-root "$CLAW_CHECKOUT" --without-embeddings --output .temp/gaps-embedding-refresh-assets --summary .temp/gaps-embedding-refresh-summary.html --check
```

The normal non-pilot build now requires matching retained embedding inputs, layout and vectors by default. `--without-embeddings` produces the intermediate term bundle needed when snapshot, vocabulary or recognition changes make the previous embeddings stale. Keep that bundle under `.temp/`; it is not a publication substitute. Set `GAP_TERM_ASSETS=.temp/gaps-embedding-refresh-assets` and complete the [native-record acquisition and encoder canaries](EMBEDDING_MAP.md#refresh-commands), then their full runs. If the snapshot and term asset have not changed, reuse the existing retained embeddings and skip inference. Finish with the normal publication checks:

```sh
python scripts/gaps/build.py --claw-root "$CLAW_CHECKOUT"
python scripts/gaps/build.py --claw-root "$CLAW_CHECKOUT" --check
python scripts/gaps/embedding_inputs.py --check
python -m unittest discover -s tests -v
node --test tests/gap-search.test.js
node --test tests/gap-search.test.js tests/gap-embedding.test.js
python scripts/fleet/assemble_page.py --check
bundle exec jekyll build --destination .temp/ci-site
```

To check inputs before the normal build installs refreshed public assets, pass `--assets "$GAP_TERM_ASSETS"`. Normal CI needs only the ordinary fleet dependencies: it validates the retained input/layout/vector bindings and reproduces the public map asset byte for byte, without downloading a model, running inference or rerunning PaCMAP. Only an explicit embedding refresh needs the separate environment in `scripts/gaps/requirements-embeddings.txt` and the pinned model already cached locally.

Acquisition validates every supported, publishable record. A parse, schema, provenance or boundary failure prevents replacing the previous snapshot. Vocabulary acquisition separately scans all publishable roots and preserves explicit coverage. Vocabulary and discussion snapshot hashes must agree. Asset installation stages all outputs and rolls back on failure. Search and term payloads each have a 2 MiB compressed budget; detail shards have a 500 KiB budget. Inspect a failure rather than quietly dropping a source or relaxing its count.

Normal vocabulary acquisition closes shared canonical ownership after lexical selection. To repair a retained version-1 vocabulary without repeating its full YAML extraction, use the pinned identity-field closure path. Run the CofactorMech canary first, inspect its saved ownership/provenance, then run the full repair:
Expand All @@ -72,8 +88,10 @@ Retain receipts under `reviews/`; browser screenshots and downloads stay under `

The term-specific helper, `scripts/adhoc/check_gap_terms_20261115.py`, independently counts the published postings and verifies the cloud, top-twenty pairs, source/view filters, exact term/pair downloads, keyboard/history/cold URLs and missing-index retry. Run `--canary` and inspect its receipt before the full desktop/mobile pass. Supply `--site` for actual Jekyll output. Its screenshots and downloads stay under `.temp/gap-terms-20261115/`. The live release verifier also exercises the term view when the deployed manifest declares it.

`scripts/adhoc/check_gap_embedding_20261115.py` checks the map against the complete filtered ID set, including results beyond the current page. It verifies fixed coordinates and colors, matched descriptions, discussion history, keyboard selection, pan/zoom, term/pair filters, complete JSON/CSV exports, empty states and isolated download failure/retry. Use the same `--site`, `--assets`, `--summary`, `--report` and optional `--browser-executable` arguments. Run `--canary`, inspect the saved receipt, then repeat without it for desktop/mobile and localhost failure injection. Artifacts remain under `.temp/gap-embedding-20261115/`. The live verifier compares the declared map bytes with the release commit and exercises map interactions without injecting failures on production.

Review each PR independently and read-only; file findings as issues, fix in scope, then re-review. Merge the shared contract first, then the website, only after explicit authorization for those PRs. Pinning the contract protects reproducibility before and after its merge. Verify the deployed Pages revision, manifest and populated browser before calling publication complete. Delete only the merged task branches and owned clean worktrees; preserve unrelated local work.

For this rollout, `scripts/adhoc/verify_live_gap_browser_20261114.py` accepts `--expected-website-sha` and a `--report` path inside `.temp/`. It requires Playwright and an installed browser; `--browser-executable` can select one. Run `--canary` first, verify its saved receipt, then repeat without that flag. The verifier checks the successful Pages build before and after, compares live manifest/data/UI bytes with the exact website commit, follows the hub link and exercises search, full exports, detail shards and mobile navigation. It retains response headers, downloaded bodies, screenshots and measured results without changing the published site.

To roll back published data, revert the snapshot/config/toolchain/generated-assets commit together through a PR. Do not mix assets from different build IDs. Hashed assets and client verification turn a mixed or missing deployment into a visible retryable failure rather than misleading empty results.
To roll back published data, revert the snapshot/config/toolchain/vocabulary/embedding/generated-assets changes together through a PR. Do not mix assets from different build IDs or term/input hashes. Hashed assets and client verification turn a mixed or missing deployment into a visible retryable failure rather than misleading empty results.
1 change: 1 addition & 0 deletions _gaps/embedding-inputs.json

Large diffs are not rendered by default.

1 change: 1 addition & 0 deletions _gaps/embedding-layout.json

Large diffs are not rendered by default.

Binary file added _gaps/embedding-vectors.npy
Binary file not shown.
1 change: 1 addition & 0 deletions _gaps/reviews/embedding-asset-measurements.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"budget_comparisons":{"2_mib":{"budget_bytes":2097152,"headroom_bytes":-2247740,"utilization_percent":207.181},"3_mib":{"budget_bytes":3145728,"headroom_bytes":-1199164,"utilization_percent":138.12},"4_mib":{"budget_bytes":4194304,"headroom_bytes":-150588,"utilization_percent":103.59},"5_mib":{"budget_bytes":5242880,"headroom_bytes":897988,"utilization_percent":82.872}},"compression":{"default_compresslevel":9,"http_transfer_note":"These are reproducible gzip estimates, not measured negotiated HTTP transfer sizes.","python":"3.13.8","zlib":"1.2.12"},"counts":{"contexts":3830,"neighbor_references":57450,"points":3830,"records":3380},"full_public_document":{"gzip_by_level_bytes":{"1":4809181,"6":4376663,"9":4344892},"gzip_level9_bytes":4344892,"gzip_level9_mib":4.143612,"json_bytes":13902258},"independent_components":{"additional_bundle_metadata":{"gzip_level9_bytes":565,"gzip_level9_mib":0.000539,"json_bytes":2252},"context_matched_terms_only":{"gzip_level9_bytes":615231,"gzip_level9_mib":0.58673,"json_bytes":2097750},"context_record_references_only":{"gzip_level9_bytes":296145,"gzip_level9_mib":0.282426,"json_bytes":915146},"discussion_contexts":{"gzip_level9_bytes":863507,"gzip_level9_mib":0.823504,"json_bytes":3422704},"layout_metadata_without_points":{"gzip_level9_bytes":2737,"gzip_level9_mib":0.00261,"json_bytes":5505},"layout_points":{"gzip_level9_bytes":2732725,"gzip_level9_mib":2.60613,"json_bytes":7271860},"native_descriptions_only":{"gzip_level9_bytes":198831,"gzip_level9_mib":0.18962,"json_bytes":636439},"native_records":{"gzip_level9_bytes":745329,"gzip_level9_mib":0.710801,"json_bytes":3199909},"native_source_provenance_only":{"gzip_level9_bytes":226206,"gzip_level9_mib":0.215727,"json_bytes":1460188},"point_neighbors_only":{"gzip_level9_bytes":2298870,"gzip_level9_mib":2.192373,"json_bytes":6260353},"retained_layout":{"gzip_level9_bytes":2735933,"gzip_level9_mib":2.609189,"json_bytes":7277374}},"inputs_sha256":"0626e8174943f26456b2cdf42669a665442eeafc28968adba9b91885b68cdf25","interpretation":"Component compression is not additive. Removal variants diagnose byte cost only; preserve native definitions, provenance and exact original-space neighbor evidence.","marginal_removal_measurements":{"without_context_matched_terms":{"gzip_level9_bytes":3944725,"gzip_level9_bytes_saved":400167,"gzip_level9_mib":3.761983,"json_bytes":11743230},"without_contexts":{"gzip_level9_bytes":3481811,"gzip_level9_bytes_saved":863081,"gzip_level9_mib":3.320514,"json_bytes":10479543},"without_native_descriptions":{"gzip_level9_bytes":4072925,"gzip_level9_bytes_saved":271967,"gzip_level9_mib":3.884244,"json_bytes":13218501},"without_native_source_provenance":{"gzip_level9_bytes":4080032,"gzip_level9_bytes_saved":264860,"gzip_level9_mib":3.891022,"json_bytes":12411652},"without_point_neighbors":{"gzip_level9_bytes":1995308,"gzip_level9_bytes_saved":2349584,"gzip_level9_mib":1.902874,"json_bytes":7595947},"without_records":{"gzip_level9_bytes":3599636,"gzip_level9_bytes_saved":745256,"gzip_level9_mib":3.43288,"json_bytes":10702339}},"method":"Exact embedding_bundle.build_document result and strict public_bytes serializer; canonical UTF-8 JSON; gzip mtime=0","public_document_sha256":"2e3975f95d55b6f62f511503ace87368f6f4f75bd03c05994b81c99253190598","retained_layout_sha256":"07741db3c02fea5160f78bf86fe83d50511310523bb391bfaf361e5d61223778","retained_vectors_sha256":"d80aff4cb9b470bf9de79b265425cb2a3965fed97afc6a1929feb7d06362f3a9","schema_version":1}
66 changes: 66 additions & 0 deletions _gaps/reviews/embedding-browser-jekyll-canary.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
{
"complete": true,
"canary": true,
"views": [
{
"width": 1280,
"initial_mapped": 1187,
"layout": {
"width": 1280,
"scrollWidth": 1280,
"h1": 1,
"main": 1
},
"marker_diameter_css_px": 8,
"pointer_point_opens_discussion": true,
"marker_size_stable_after_zoom_and_resize": true,
"pagination_does_not_change_map": true,
"off_page_point_populated_context_and_history": true,
"keyboard_point_opens_discussion": true,
"fixed_coordinates_and_colors_under_filtering": true,
"pair_export": {
"points": 52,
"bytes": 286845,
"sha256": "fd84b1f444b2742e5ff51f255a8cc337e3b1e0d5a33e2913911e908e962a19e7"
},
"pair_csv": {
"points": 52,
"bytes": 236454,
"sha256": "9993ff6ef56f2c24d75b8471939b01f3b6d2c8429bcb75608c24506c79982248"
},
"all_export": {
"points": 3830,
"bytes": 19756212,
"sha256": "38c81eac5ea0936489a084a38ea1dbef6109610b15eb0a9b775f44c1c524fb00"
},
"dark_layout": {
"width": 1280,
"scrollWidth": 1280,
"h1": 1,
"main": 1
},
"point_scope_filters_and_empty_states": true,
"page_errors": []
}
],
"started_at": "2026-11-16T03:18:06.660091+00:00",
"artifacts": ".temp/gap-embedding-20261115/embedding-jekyll-canary-artifacts",
"provenance": {
"method": "Actual supplied Jekyll build, served locally",
"site": ".temp/gap-embedding-20261115/ci-c91f903",
"page_sha256": "9aa67357a054a9afd219c41cc02e125a50ce8e0cf91dace278db54d60f0f5647",
"ui_sha256": {
"browser.js": "5a24c35ca3aba47218a8d415f9325486b61c96e1af47127e9645db50e7cfb8b2",
"search-core.js": "44a0351a5c9179a1da266274f338154da2aa5076767cdc352cdb4f59b50f1832",
"search-worker.js": "88356aee113e6d6cae79da7d5e220097c20f1a6ce1702309d26f665b8f0099ae",
"browser.css": "aeac93415fa90fd87c32ceb9a6a5c7477e79a9e95545f0f30ae5f637d2507d19",
"embedding-map.js": "f7d43d2bb3cd13203623082e4fa1ea260724a3efd73efd8d8515d89d485aca7e"
},
"main_css_sha256": "f861c07e386d770d235e0a7d5674e3b851103960f1b675d5ac614b01f881e02e",
"custom_css_sha256": "5efa39711927e49297d1a5bda4c6d4ec8b9c48b0daa89a29b7fa6ee6e75df616"
},
"build_id": "33c56cd4ba45f871e80cf3c4e12b19f9ab1cd662cc1d648185e80f2d0a75a864",
"embedding_sha256": "2e3975f95d55b6f62f511503ace87368f6f4f75bd03c05994b81c99253190598",
"browser_version": "157.0.8103.51",
"finished_at": "2026-11-16T03:18:22.551524+00:00"
}
Loading
Loading