[Epic] Retrieval Eval v2 — Phase 1 closeout #192

Phase 1 closeout decision (2026-10-01)

This epic is now bounded to Retrieval Eval v2 Phase 1. Stop expanding retrieval evaluation. The original broad proposal is preserved below as historical context, not the current closure gate.

Closeout checklist

• Scope frozen: no more K / threshold / hybrid / retrieval-metric tuning under this epic.
• Generator evaluation split into [Eval] Generator evaluation — groundedness and answer completeness #213 (groundedness, correctness/completeness).
• Citation evaluation split into [Eval] Citation evaluation — precision, recall and entailment #214 (precision, recall, entailment).
• Integrate the stacked feat(eval): hit rate, MAP, and per-query-type metrics #195–feat(eval): fifteen near-miss unanswerable questions — closes Phase 1 #208 work into main, preserving subsequent product fixes, via chore(eval): integrate Retrieval Eval v2 and close Phase 1 #215 (merge 6cf23ea). Their earlier merged PR status alone did not mean the work reached main.
• Refresh retrieval-v1.6 against the current 34/44 split; 518 tests, typecheck, lint (0 errors), build and Linux/macOS/Windows CI passed.
• Integration chore(eval): integrate Retrieval Eval v2 and close Phase 1 #215 merged; this epic is closed as Retrieval Eval v2 Phase 1 complete.

Phase 1 delivered scope (integrated into main via #215)

• Production-config baseline; separate candidate depth and context window (fix(eval): run the harness on Windows and keep the baseline POSIX #193–feat(eval): separate candidateK from contextK, and run the production config #194).
• Hit rate / MAP / per-query-type reporting and deterministic context precision/recall (feat(eval): hit rate, MAP, and per-query-type metrics #195, feat(eval): measure the context window, and name what still needs a model #198).
• Validation/test split, validation-only threshold/strategy selection, saturation-aware adoption rule (feat(eval): derive the similarity threshold on validation, report on test #196–feat(eval): adopt on the first metric with headroom, not a saturated one #197, feat(eval): unanswerable questions, and an explicit split manifest #202, fix(eval): select the strategy on validation, and stop calling candidates passages #204).
• Bounded sweeps, dense-channel threshold tracing and depth validation (feat(eval): bounded parameter sweep with a trade-off dashboard #199–fix(eval): refuse a context window wider than the retrieval depth #201).
• Cross-lingual and confusion-cluster corpus, score diagnostics, paired deltas and near-miss unanswerable cases (feat(eval): nine cross-lingual questions, and the corpus stops being saturated #203, feat(eval): confusion clusters, and the corpus reaches 53 chunks #205–feat(eval): fifteen near-miss unanswerable questions — closes Phase 1 #208).

Limits and production outcome

The current first-party dataset is 21 documents / 53 chunks / 78 questions (53 answerable, 25 unanswerable). It does not meet the original ~300-chunk target; dataset scaling is explicitly deferred rather than claimed complete. Quote/block grounding is retained. The corpus is diagnostic evidence, not proof of broad real-world performance.

Dense remains the production default (candidateK=20, contextK=3, threshold=0.5). Validation does not authorize hybrid adoption. Deterministic context metrics are not generator metrics. Similarity-threshold limitations on unanswerable questions are a documented negative result, not a reason for another tuning campaign here.

Removed from this closure gate

• Generator and citation metrics → [Eval] Generator evaluation — groundedness and answer completeness #213 / [Eval] Citation evaluation — precision, recall and entailment #214.
• Public QASPER / MIRACL-zh suite → deferred, not started.
• Offline cross-encoder / reranker ladder → deferred until a model can be pinned; feat(retrieval): fusion candidate layer, BM25/hybrid strategies, and the retrieval experiment #170 is a historical merged PR, not an open blocker issue.
• Further corpus scaling, context-token accounting beyond the existing character proxy, and additional abstention mechanisms → deferred, not prerequisites.

No new public-suite, reranker, corpus-growth or abstention project is started by this closeout.

Original proposal (historical scope; superseded by the Phase 1 boundary above)

Theme

One sentence: turn the retrieval regression test into a real RAG benchmark — one that runs the production configuration, measures the generator and the citations, and can actually discriminate between candidate algorithms.

The harness we built in v1.4/v1.5 is not the problem. dense + BM25 + RRF, Recall@K / MRR / nDCG@10, p50/p95 latency, a chunking A/B test, a pinned embedding model, and a reproducible offline run are already the hard part of the work. What we do not yet have is an evaluation whose numbers can be trusted to represent the product.

This epic is about fixing the conditions of the experiment, not replacing the algorithm.

Why the current numbers do not mean what they look like

1. The benchmark is saturated

eval/corpus/ is 13 documents / 19 chunks / 30 questions, evaluated at topK = 10. Retrieving 10 of 19 chunks per question means the harness hands the ranker ~53% of the entire knowledge base before scoring. docs/eval/retrieval-v1.5.md already says it plainly:

The rule's Recall@5 condition is saturated on this corpus: dense already scores 1.0000, so no strategy can improve it and the rule can therefore never be met here.

So the v1.5 result — hybrid better on Recall@1 (0.8667 vs 0.8333), MRR (0.9444 vs 0.9278) and nDCG@10 (0.9561 vs 0.9437) — is recorded as inconclusive not because hybrid is not better, but because the adoption rule ("Recall@5 must improve") cannot move on a corpus where Recall@5 is already perfect. Any future retrieval change is blocked by the same rule. That is a benchmark defect, not a retrieval result.

2. The harness does not run the product

We are benchmarking a retriever the user never runs. A chunk at rank 4 with cosine 0.47 is a hit in the eval and does not exist in production. Recall@5 = 100% can be reported while the shipped path silently drops the evidence. This mismatch is more consequential than the value of topK itself.

3. Candidate K and context K are conflated

There is one topK in the contract. It is doing two different jobs:

• how wide the first stage searches (recall-oriented), and
• how much text is assembled into the prompt (precision/budget-oriented).

HybridRetriever.ts fuses and then slices at the same number:

hits = rrfFuse([denseHits, sparseHits]).slice(0, topK)

With topK = 3 in chat, dense contributes 3 and BM25 contributes 3 — the fusion pool is at most 6, then truncated back to 3. That is not a two-stage retrieval; it is a single narrow search with fusion applied inside it.

The standard shape separates these: retrieve a wide candidate pool, then narrow it.

                  ┌→ Dense  candidateK=20 ─┐
Query → Retrieval │                        ├→ RRF → rerank → contextK=3..8 → LLM
                  └→ BM25   candidateK=20 ─┘

NVIDIA's RAG blueprint sets VDB top-k at 100 and reranker top-k at 10 for exactly this reason; the Datawhale advanced-RAG path retrieves top-20 then reranks to top-5.

4. Two things are simply not evaluated

docs/eval/retrieval-v1.5.md records that reranking is not evaluated (no offline cross-encoder could be pinned), and the harness stops at the retriever. There is no generator metric and no citation metric — while citation is the product's core differentiator and has shipped since #155/#156/#162/#164.

Scope

A. Evaluation dataset

Two suites, deliberately separate.

A1. In-domain — KnowNote Eval (the one that sets production defaults)

Public datasets test their own distribution, not the one users upload. Target scale, staged:

stage 1:  ~20 documents / ~300 chunks  / ~80 questions
stage 2:  50-100 docs   / 500-3000 chunks / 200-500 questions

Source types: PDF, Markdown, Web, DOCX, PPTX. Query types deliberately covered, because these are what separate algorithms:

exact keyword · semantic · paraphrase · single-hop · multi-chunk
multi-document · entity/number/version · hard negative · unanswerable
source-scoped · citation-required

Hard negatives are mandatory — e.g. two chunks for "GPT-4 context window" and "GPT-4 Turbo context window" where dense cross-contaminates. Unanswerable questions are mandatory: today the harness has no way to measure whether we say "not in your sources".

Ground truth extends the existing relevant[] schema with graded relevance, a reference answer, and claims, so nDCG can use graded gains and the generator/citation layers have a target:

{
  "id": "kn-001",
  "question": "...",
  "answerable": true,
  "type": "multi-source",
  "referenceAnswer": "...",
  "claims": ["...", "..."],
  "evidence": [
    { "document": "...", "page": 12, "block": 21, "quote": "...", "relevance": 3 }
  ]
}

relevance: 3 essential / 2 useful / 1 supporting / 0 irrelevant.

Ground truth must stay quote-and-location based (block/page), not chunk-id based — otherwise re-chunking invalidates the dataset and chunk-size experiments become unmeasurable. The chunk↔block mapping from #108 is what makes this work.

LLM-authored questions are allowed as candidates but must be reviewed, and the hard categories (paraphrase, hard negative, multi-hop, multi-source, unanswerable) authored deliberately. A fully synthetic set reproduces the current failure: the source sentence and the generated question share vocabulary, dense scores 1.0, and we are back to a saturated benchmark.

A2. Public — sanity and comparability

Kept small and pinned; a reproducible subset is enough. Priority order:

The public suite answers "is our retriever competitive?" The in-domain suite answers "did the product get better?" Only the second one sets defaults. Public results must always be reported separately and labelled as out-of-distribution.

B. Config parity — one config, both paths

• Split the contract: candidateK (first stage, per channel) and contextK (assembled into the prompt). Keep threshold explicit as a separate concern.
• The eval harness must run the production configuration as its baseline run, and any deviation must be a named experiment.
• HybridRetriever must fuse a wide pool and truncate once, after fusion — not slice each channel to the final K.
• Re-derive threshold from the validation set instead of hand-picking 0.5. Cosine ~0.5 has no universal meaning; it varies with model, language, query type, chunk length and domain. Sweep 0.3 / 0.4 / 0.5 / 0.6 and watch recall, precision and the no-result rate. First stage may legitimately run threshold = off and let a reranker do the filtering.

C. Retrieval metrics

Keep Recall@K, MRR, nDCG@K. Add HitRate@K, Precision@K and MAP. Report per query type, not only aggregate — a strategy that helps entity queries and hurts paraphrase queries should not average to "no change".

D. Generator evaluation

The retriever finding the evidence is not the answer using it. Add:

Context Recall · Context Precision      (did retrieval cover the needed facts?)
Faithfulness / Groundedness             (is every claim supported by context?)
Answer Correctness / Completeness       (is the answer right and complete?)
Noise Sensitivity                       (does extra context hurt?)

RAGAS defines the context/generator split this way, and Microsoft's end-to-end RAG evaluation separates groundedness, relevance and completeness. Claim-level checking (RAGChecker-style atomic claims) fits this codebase better than answer-vs-reference embedding similarity.

E. Citation evaluation — P0 for this product

Citation is the differentiator, so it deserves first-class metrics rather than being implied by retrieval:

Citation Precision     cited spans that actually support their claim
Citation Recall        factual claims that carry a citation at all
Citation Entailment    claim ↔ evidence: cited ≠ correctly cited

This extends the coverage work already shipped in #156/#164. Page/block matching is deterministic; claim↔evidence entailment needs a judge. This is where KnowNote can be measurably better than a generic RAG stack, so it should not wait for a later epic.

F. Experiments

Ordered, cheapest-and-most-informative first. Fixed chunking within a sweep; never full Cartesian product.

1. candidateK sweep      dense and hybrid:  5 / 10 / 20 / 40     (contextK fixed)
2. contextK sweep        hybrid:            3 / 5 / 8           (candidateK fixed)
3. strategy ladder       dense → BM25 → hybrid RRF → hybrid + reranker
4. threshold sweep       0.3 / 0.4 / 0.5 / 0.6  (+ off)
5. chunking re-run       only after the above, on the new corpus

Every run records p50/p95 latency, index size and context tokens alongside quality. A quality gain that triples p95 is a product decision, not an automatic adoption — the existing rule already says this and should survive into v2.

G. Reporting

• Dashboard of independent metrics, not a single scalar "KnowNote RAG score".
• Every experiment reported as a delta against the production-config baseline.
• Public-suite and in-domain numbers never averaged together.
• Latency, index size and token cost are columns in the same table as the quality metrics.

                 Dense    Hybrid   Hybrid+Rerank
Recall@5          0.82      0.91       0.94
MRR               0.74      0.79       0.84
nDCG@10           0.77      0.83       0.88
Context Precision 0.56      0.62       0.79
Faithfulness      0.88      0.90       0.92
Citation Recall   0.76      0.84       0.90
p95               18 ms     34 ms     120 ms

H. Evaluation methodology

deterministic metrics  →  retrieval + citation location matching
LLM judge              →  faithfulness, completeness, entailment
small human audit      →  30-50 sampled cases per release

LLM-as-judge is non-deterministic and must not be the only evidence; a deterministic core plus a bounded human sample keeps release decisions reproducible.

Non-goals

• No GraphRAG, Agentic RAG, HyDE, Self-RAG, ColBERT yet. Hybrid + RRF + reranker is the next real step for private-document QA with citations; anything more elaborate is premature until the benchmark can measure it.
• No larger embedding model as the fix. Changing the model on a saturated 19-chunk corpus teaches nothing.
• No single aggregate score.
• No wholesale dataset replacement. The current first-party corpus stays as a smoke corpus; it is not deleted, it is demoted.

Known blocker

Reranking (#170) is blocked on a cross-encoder that can be pinned and run offline, the same way multilingual-e5-small@761b726 is pinned in eval:prepare. Until then, stage 1 is dense candidateK=20 + BM25 candidateK=20 → RRF → contextK=5, which needs no new model and is already a strictly better shape than the current slice(0, topK).

Acceptance criteria

1. The harness runs the production configuration by default, and candidateK / contextK are separate named parameters in both the contract and the eval config.
2. In-domain corpus is at least stage 1 scale, with hard-negative, unanswerable, multi-chunk and multi-document categories present and counted.
3. Ground truth is quote/block-based and survives a re-chunk without re-labelling.
4. Retrieval metrics are reported per query type; HitRate@K and MAP are added.
5. Answer-level metrics (context precision/recall, faithfulness, completeness) exist and run on the same dataset.
6. Citation precision / recall / entailment exist and are reported.
7. The dense → hybrid → hybrid + reranker ladder is reported with candidateK and contextK sweeps, each with p50/p95, index size and context tokens.
8. A public suite subset (at minimum QASPER + MIRACL-zh) runs reproducibly offline and is reported separately from in-domain numbers.
9. docs/eval/ states, for the corpus in use, whether Recall@K is saturated — and the adoption rule is amended so saturation is detected rather than silently blocking every change.

Suggested child issues

Build order

1 (config parity)  →  2 (corpus v2)  →  3 (metrics v2)
                                        ↓
                              4 (threshold)  →  7 (sweeps/dashboard)
                                        ↓
                              5 (generator)  →  6 (citation)
                                        ↓
                              8 (public)     →  9 (reranker)

Config parity and the corpus come first because every number produced before them is unrepresentative of the product.

References

• LlamaIndex RetrieverEvaluator — hit rate, MRR, precision, recall, AP, nDCG
• RAGAS — context precision, context recall, response relevancy, faithfulness, noise sensitivity
• RAGChecker — atomic-claim precision/recall, claim recall, context utilization, hallucination
• Datawhale All-in-RAG — hybrid retrieval, RRF, cross-encoder/ColBERT, context relevance & faithfulness
• Microsoft — parameter sweep for algorithm/top-k/chunk-size; groundedness, relevance, completeness; LLM-judge non-determinism
• NVIDIA RAG Blueprint — separate VDB top-k (100) and reranker top-k (10)
• BEIR, MIRACL, QASPER, HAGRID, CRUD-RAG, RAGBench