Skip to content

feat(docscout): report values missing from selected sections with their own cause - #71

Closed
wolfy-j wants to merge 9 commits into
mainfrom
feat/docscout-scoped-missing
Closed

wolfy-j wants to merge 9 commits into
mainfrom
feat/docscout-scoped-missing

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Stacked on #62 (verified semantics, which introduces the missing cause; it carries #61, #59, #55, #53). Review the last commit; the diff against main includes those PRs until they merge.

Problem

A field with search_within: sections reads passages from its selected sections only. When the model finds no value there, the value carries the same missing cause as a field that read the whole document, so a caller cannot tell "absent from the document" from "absent from the sections routed to this field".

Cause

extract_call builds every not-found value the same way, whatever region the field read, and verify then assigns missing to any missing value without a cause.

Fix

  • A not-found value of a sections field records cause missing_in_sections with the reason "Not stated in its selected sections". Verify keeps it, as it keeps any recorded cause.
  • missing stays true, so consumers reading missing, or counting summary.unverified per cause (an open map), keep working; a consumer that treats unknown causes as missing sees no change.
  • The missing field description in the extraction, Block and tool schemas, and the README cause lists, name the new cause.

Test

  • engine_test "tells a value missing from a field's sections apart from one missing from the document": a sections field and a whole-document field both come back not found; the first reports ["missing_in_sections"] with its reason, the second ["missing"] with none.

Red before the fix, green after. make lint clean. Suite on SQLite and PostgreSQL (runtime v0.3.59a): 196 passed, 0 failed.

KickSide added 9 commits November 10, 2026 16:04
A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.
The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.
…y item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.
A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.
# Conflicts:
#	docscout/src/engine/shapes.lua
#	docscout/src/engine/stages.lua
Model calls were assembled without an input budget: passages were neither
ranked nor capped, verification and section choice sent everything in one
call and section summaries went in one call over all sections, so a long
document overflowed the window and the refused call failed its fields.

Every call now fits the model card's context window less its reserved
output, estimated as the llm module estimates a prompt before admitting it.
Extraction and scout calls allocate passages per field from pool
provenance (citations, then the field's own hits by rank) and record cut
passages per field; support checks, section choices, summaries, outline
windows, highlights and generated contexts are batched to fit; Deep reads
the passages that fit in the order found.
verified mixed absence with confirmation: a value the document does not
state came back verified, a found value without a quote passed when its
field did not require evidence, and the support check judged excerpts
without the field's meaning, so a quote holding some date supported a
wrong-role date, and nothing judged whether verbatim text was complete.

verified now holds only for a found value with a quote whose checks pass.
The support check reads each field's type and description, judges whether
the cited text states this value for this field, may answer
insufficient_evidence, and for text and list values reads the source
region around the citations and returns a separate completeness verdict,
exposed as `complete`.

BREAKING CHANGE: missing values are returned with verified false, and
found values without a quote are never verified.
A constrained decoder only admits the verdict strings, but the prompt did
not name supported or not_supported, so the model chose among labels it
could not see: a replayed batch returned not_supported for every value
while each reason said the quote states the value. The prompt now names
each verdict and completeness label with its meaning, built from the same
constants as the schema.
…ir own cause

A field that searches only its selected sections reads passages from those
sections alone. When the model finds no value there, the value carried the
same `missing` cause as a field that read the whole document, so callers
could not tell a value absent from the document from one absent from the
sections routed to the field.

Such a value now records cause `missing_in_sections` with the reason "Not
stated in its selected sections". It stays `missing: true`, so consumers
reading `missing` see no change.
@wolfy-j

wolfy-j commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #78, merged and released.

@wolfy-j wolfy-j closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant