Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions _gaps/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,29 @@ The [original implementation plan](IMPLEMENTATION.md) covers the published nativ

The browser reads a retained native-data snapshot. It makes no visitor-time GitHub API calls and runs no literature scans. It does not change scientific records or infer gaps from absent fields. Default results are active knowledge gaps and human/model mismatches; all discussions and review-required candidates are separate views. Recorded does not mean scientifically validated.

The magenta section links below the title preserve search filters and support direct fragment URLs. The summary tiles count indexed `KNOWLEDGE_GAP` and `HUMAN_MODEL_MISMATCH` entries across all lifecycle states, excluding candidate entries. These snapshot-wide counts do not change with search filters. Selecting a count clears other filters and opens the corresponding gaps; unsupported or partially covered sources retain an explicit coverage label rather than claiming a measured zero.
The magenta section links below the title preserve search filters and support direct fragment URLs. The summary tiles count indexed `KNOWLEDGE_GAP` and `HUMAN_MODEL_MISMATCH` entries across all lifecycle states, excluding candidate entries, across all sixteen public corpus Mechs plus any public non-corpus fleet infrastructure (`EdgeMech`, declared in `_fleet/infrastructure.json`). These snapshot-wide counts do not change with search filters. Selecting a corpus count clears other filters and opens the corresponding gaps; unsupported, partially covered, or non-corpus infrastructure members link to explicit scope details rather than claiming a measured zero.

## Inputs and provenance

- `sources.json` declares the sixteen public corpus repositories, immutable revisions, source roots, adapters, schema/license digests and publication holds. Support and permission are separate. Unsupported or held roots have unknown counts, never zero.
- `sources.json` declares the sixteen public corpus repositories, immutable revisions, source roots, adapters, schema/license digests and publication holds. `_fleet/infrastructure.json` declares non-corpus public fleet infrastructure (`EdgeMech`) and is validated against `_fleet/data/manifest.json` so all seventeen public fleet members are accounted for without inventing corpus counts or website URLs. Support and permission are separate. Unsupported or held roots have unknown counts, never zero.
- Public source names and repositories must belong to the committed `_fleet/data/manifest.json` allowlist projected by CLAW's `FleetManifest.public()` API. Membership validation applies before acquisition and again when retained artifacts are published; no private-name denylist is maintained.
- `snapshot.json` retains normalized public items and measured coverage. Record identity includes its source path: distinct documents can share a semantic identifier. Discussion identity also includes the native discussion ID. Source renames need an explicit migration decision.
- `toolchain.json` pins the CLAW normalizer commit and module/schema bytes. CI checks out that commit, validates the snapshot and reproduces the public assets byte for byte.
- `term-vocabulary.json` retains candidate names, identifiers, unqualified native aliases and exact typed synonyms from every publishable pinned record root, with complete acquisition counts and source provenance. Lexical candidates include colliding identities; shared canonical entities also retain their other native owners even when an owner's local name is absent from the discussion prose. Source roots without native gap fields can still supply vocabulary. The AIMech maintained-concept publication hold also applies here.
- `embedding-inputs.json` retains each discussion's complete permitted prose followed by its accepted matched records' identifiers, names and native definitions/descriptions. Every native owner keeps its source attribution even when identical text blocks are deduplicated. `embedding-layout.json` retains model/runtime, chunk coverage, fixed coordinates, original-vector cosine neighbors and projection diagnostics. `embedding-vectors.npy` retains the normalized float32 vectors in discussion-ID order; browsers do not download it.
- `../assets/gaps/manifest.json` points to hashed search, term and detail assets. Large Mechs are partitioned by stable item-ID order into numbered shards below 500 KiB compressed; index rows identify their detail shard. The search worker covers the complete index; details load on demand. JSON and CSV exports include every match, not just the current page, including exact term/pair filters.
- `../assets/gaps/manifest.json` points to hashed search, term, embedding, synthesis-analytics and detail assets. Large Mechs are partitioned by stable item-ID order into numbered shards below 500 KiB compressed; index rows identify their detail shard and carry compact facet projections (`posed_date`, `resolved_date`, `has_experiments`, `evidence_stances`, and normalized `attach_layers`). The search worker covers the complete index; details load on demand. JSON and CSV exports include every match, not just the current page, including exact term/pair, schema-layer, stance, reference, experiment and month filters.

## Cross-Mech analyses and synthesis sidecar

`assets/gaps/search-core.js` computes live filter-responsive summaries across the matching discussion scope (before entity-chip narrowing for entity charts):
- **Fleet-wide discussion breakdown:** Lists all sixteen corpus Mechs and `EdgeMech` infrastructure, stacking matching discussions by `kind`, `status`, `evidence_stance`, or normalized `attach_layer` (`attaches_to` path prefix before `#`, `.`, `[`, or `/`, with CURIEs grouped under `entity_curie` and external HTTP(S) URIs under `external_uri`). Zero-count corpus rows distinguish complete zero extractions (`CultureMech`, `HabitatMech`, `TaxonMech`) from unsupported schemas (`CMMMech`, `PathwayMech`), held roots (`AIMech`), and non-corpus infrastructure (`EdgeMech`).
- **16 × 16 Source Mech × Term vocabulary cross-pollination matrix:** Rows are discussion source Mechs and columns are Mech entity vocabularies matched in prose, separating within-Mech diagonal mentions from off-diagonal cross-Mech reuse.
- **Posed vs. resolved timeline and reference silos:** Aggregates monthly ISO `posed_date` and `resolved_date` counts alongside canonicalized `DOI:` and `PMID:` citations (folding both CURIE prefixes and `doi.org` / `pubmed.ncbi.nlm.nih.gov` URLs) and cross-Mech citation overlap.
- **Co-occurrence network and specificity metrics:** Top-20 entity pairs report shared discussions, Jaccard overlap $|A \cap B| / |A \cup B|$, and normalized pointwise mutual information $\text{NPMI}(A,B) = \ln(P(A,B)/(P(A)P(B))) / (-\ln P(A,B))$ over the active discussion scope $N$ (with $\text{NPMI}=0$ when $|A \cap B| = N$).

`scripts/gaps/analytics.py` builds the hashed `analytics.<sha16>.json` sidecar (512 KiB compressed budget, bound by SHA-256 to `snapshot.json`, `terms.*.json`, and `embedding-inputs.json`), precomputing corpus-wide synthesis views without browser-side vector math:
- **Cross-Mech semantic bridges:** Unordered original-vector cosine nearest-neighbor pairs crossing Mech boundaries (diversified to at most two featured bridges per discussion ID), Mech-to-Mech neighbor edge summaries, duplicate-input hash clusters, and per-entity neighborhood cohesion (share of top-15 neighbors sharing the entity).
- **Proposed-experiments backlog and unmatched blind-spots:** Structured `proposed_experiments` across Mechs, plus frequent bigrams and unigrams in discussions (and recorded knowledge gaps) with zero accepted dictionary entity matches. All phrases and neighbor scores are exploratory aids, not biological assertions.

`scripts/gaps/terms.py` performs dictionary-based entity recognition over prompt, rationale, evidence explanations, proposed experiment prose and resolution text. It excludes the owning record's metadata, handles Unicode with original-text spans, keeps canonical identifiers separate from document addresses, and leaves ambiguous names unlinked. The generated term sidecar records the policy, vocabulary digest, source coverage and every accepted occurrence. This is conservative lexical recognition, not contextual semantic disambiguation or an assertion of biological relationships. Cloud size uses raw occurrences; the adjacent counts distinguish discussions, and co-occurrence counts a pair once per discussion. Chart counts follow all ordinary search filters before applying the selected entity or pair. Source Mech restricts the questions; Term vocabulary independently restricts entity ownership. Shared entities can belong to multiple vocabularies without double-counting within a view.

Expand Down
8 changes: 6 additions & 2 deletions _includes/gaps-summary.html
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<p>3,830 recorded discussion items across 16 public Mechs; 8 are candidates. Counts describe indexed records, not the extent of scientific knowledge. Unsupported roots are not empty results.</p>
<p>3,830 recorded discussion items across 16 public Mechs (plus 1 public infrastructure member without a scientific corpus: EdgeMech); 8 are candidates. Counts describe indexed records, not the extent of scientific knowledge. Unsupported roots are not empty results.</p>
<ul class="gap-static-coverage">
<li><a href="https://github.com/CultureBotAI/HabitatMech/tree/61edb3d0ae912e8f041378e669928051fbbd509e">HabitatMech</a>: complete; 0 indexed items. <a href="https://github.com/CultureBotAI/HabitatMech/blob/61edb3d0ae912e8f041378e669928051fbbd509e/LICENSE-DATA">CC-BY-4.0</a>. Observed 2026-11-14.<ul>
<li>canonical-records: complete — 3,209 records inspected; 0 items</li>
Expand Down Expand Up @@ -50,4 +50,8 @@
<li><a href="https://github.com/CultureBotAI/AssayMech/tree/2c39f17bfcd940c15c6f10dfdc26b4ce2d6e734e">AssayMech</a>: complete; 28 indexed items. <a href="https://github.com/CultureBotAI/AssayMech/blob/2c39f17bfcd940c15c6f10dfdc26b4ce2d6e734e/LICENSE-DATA">CC-BY-4.0</a>. Observed 2026-11-14.<ul>
<li>canonical-records: complete — 9 records inspected; 28 items</li>
</ul></li>
</ul><p>Selected structured fields are normalized for browsing; third-party quotation snippets are omitted. Credit each linked Mech and its contributors. Original records and cited sources remain authoritative.</p>
</ul>
<div class="gap-static-infrastructure"><h3>Shared curation infrastructure (no scientific corpus)</h3><ul>
<li><a href="https://github.com/CultureBotAI/EdgeMech">EdgeMech</a>: Curation infrastructure under development for exact KG-Microbe release-edge accounting and inherited native-review scope, with bulk rows and histories in external Parquet artifacts. No published scientific corpus or Pages browser is included in this audit. Edge occurrences and inherited-review matches are not independent scientific approvals. The sixteen-corpus census, record totals, graph and historical repository statistics exclude this infrastructure project. Checked 2026-11-11 (<a href="https://github.com/CultureBotAI/culturebotai-claw/blob/7233987c85dcae9c248c7a2dbd445065e5a6c1b8/src/kg_microbe_fleet/fleet.yaml">scope evidence</a>).</li>
</ul></div>
<p>Selected structured fields are normalized for browsing; third-party quotation snippets are omitted. Credit each linked Mech and its contributors. Original records and cited sources remain authoritative.</p>
Loading
Loading