Skip to content

Browse knowledge gaps in an embedding map with matched Mech definitions - #613

Merged
realmarcin merged 4 commits into
mainfrom
feat/gap-embedding-map-20261115
Nov 16, 2026
Merged

realmarcin merged 4 commits into
mainfrom
feat/gap-embedding-map-20261115

Conversation

@realmarcin

@realmarcin realmarcin commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds an embedding map to the knowledge-gap browser. Each indexed discussion combines its complete gap text with identifiers, names and native definitions/descriptions of the entries accepted by the existing term recognizer. Clicking a point opens the discussion, evidence, matched records and nearest discussions from the original embedding vectors.

The map uses the existing source, lifecycle, search and term/pair filters across all matching discussions, independently of result pagination. Coordinates and colors remain stable under filtering. It includes keyboard navigation, pan/zoom, a seventh magenta section link, and complete filtered JSON/CSV exports. Map loading failures preserve ordinary search and offer retry.

All 3,830 discussions and 3,380 matched native owner records are retained. 2,976 discussions include native descriptions; 240 use matched names only and 614 use gap text alone. The local, pinned BGE-large-en-v1.5 encoder chunks long inputs without truncation. Retained vectors, fixed PaCMAP coordinates, original-vector neighbors and diagnostics are versioned; CI validates and packages those artifacts without model downloads or inference. The map is an exploratory view of text similarity, with projection limitations explained in the interface.

The fixed PaCMAP graph excludes self by row identity even for duplicate vectors; all nearest, mid-near and far pairs are validated before fitting and checked unchanged afterward. Vector bytes and original cosine-neighbor lists are retained unchanged. The optional map download is 4.14 MiB gzip under a separate 5 MiB limit; its size is shown before loading.

Validation: 385 Python tests and 33 JavaScript tests pass in CI, including deterministic public-asset regeneration and production Jekyll build. Real-data browser checks cover desktop/mobile filtering, off-page point selection, matched definitions, history, keyboard/pointer access, stable marker size after zoom/resize, complete exports, empty states and isolated download recovery. Section-navigation/statistics checks cover 1280px, 620px and 390px, including no-JavaScript and failure/loading states.

Adversarial review and browser checks produced six findings, all fixed and independently re-reviewed: fresh canary execution, auxiliary model-file pins, numeric CSV coordinates, strict diagnostic serialization, duplicate-vector graph self exclusion, and mobile marker visibility. Method, refresh commands and retained evidence are under _gaps/EMBEDDING_MAP.md and _gaps/reviews/embedding-implementation-review.md.

Closes #609. Fixes #610. Fixes #611. Fixes #612. Fixes #614. Fixes #615. Fixes #616.

@realmarcin
realmarcin marked this pull request as ready for review November 16, 2026 03:19
@realmarcin
realmarcin merged commit 09a2fee into main Nov 16, 2026
1 check passed
@realmarcin
realmarcin deleted the feat/gap-embedding-map-20261115 branch November 16, 2026 03:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment