Repository navigation
Conversation
added 9 commits
November 10, 2026 16:04
A trace step reported only step totals, so a change in accuracy or cost could not be traced to the call, prompt sources or failure that caused it. Each step now lists its model calls with stage, model, prompt and completion tokens (absent when the provider reported none), latency, finish reason, error code, and the chunk ordinals and field paths the call carried. Run results and stored runs carry a summary of field coverage and of unverified fields per cause.
The support check asked for an array of verdicts bounded only above, so a constrained decoder could return an empty or partial list and every field without a verdict came back unverified. The reply is now an object with one required verdict per field path; a reply that omits a field fails schema validation as a retryable structured-output error.
…y item Scout evidence and notes, section summaries and highlighter passages were asked for as arrays a constrained decoder may legally leave empty or partial. An empty scout evidence list beside a drop list pruned the pool, missing summaries went unnoticed, and an empty highlighter reply dropped every passage of the batch. Each reply is now an object with one required key per field, section or passage. An explicitly empty list or note stays legal and means none; the scout step reports how many fields had no cited evidence. The support verdict schema uses the same keyed shape.
A scout drop removed a passage for every field and blacklisted it, so a passage another field needed or a later targeted search found again could not return, and scout citations were discarded after pruning. The pool now records per passage the fields and queries that found it, its best rank, the fields a review cited it for and the fields it is excluded for. A drop excludes a passage only for the reviewed fields that did not cite it, a field's own later search re-admits it for that field, and each extraction call reads the passages still visible to its fields.
# Conflicts: # docscout/src/engine/shapes.lua # docscout/src/engine/stages.lua
Model calls were assembled without an input budget: passages were neither ranked nor capped, verification and section choice sent everything in one call and section summaries went in one call over all sections, so a long document overflowed the window and the refused call failed its fields. Every call now fits the model card's context window less its reserved output, estimated as the llm module estimates a prompt before admitting it. Extraction and scout calls allocate passages per field from pool provenance (citations, then the field's own hits by rank) and record cut passages per field; support checks, section choices, summaries, outline windows, highlights and generated contexts are batched to fit; Deep reads the passages that fit in the order found.
verified mixed absence with confirmation: a value the document does not state came back verified, a found value without a quote passed when its field did not require evidence, and the support check judged excerpts without the field's meaning, so a quote holding some date supported a wrong-role date, and nothing judged whether verbatim text was complete. verified now holds only for a found value with a quote whose checks pass. The support check reads each field's type and description, judges whether the cited text states this value for this field, may answer insufficient_evidence, and for text and list values reads the source region around the citations and returns a separate completeness verdict, exposed as `complete`. BREAKING CHANGE: missing values are returned with verified false, and found values without a quote are never verified.
A constrained decoder only admits the verdict strings, but the prompt did not name supported or not_supported, so the model chose among labels it could not see: a replayed batch returned not_supported for every value while each reason said the quote states the value. The prompt now names each verdict and completeness label with its meaning, built from the same constants as the schema.
…ir own cause A field that searches only its selected sections reads passages from those sections alone. When the model finds no value there, the value carried the same `missing` cause as a field that read the whole document, so callers could not tell a value absent from the document from one absent from the sections routed to the field. Such a value now records cause `missing_in_sections` with the reason "Not stated in its selected sections". It stays `missing: true`, so consumers reading `missing` see no change.
Contributor
Author
|
Superseded by #78, merged and released. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #62 (verified semantics, which introduces the
missingcause; it carries #61, #59, #55, #53). Review the last commit; the diff againstmainincludes those PRs until they merge.Problem
A field with
search_within: sectionsreads passages from its selected sections only. When the model finds no value there, the value carries the samemissingcause as a field that read the whole document, so a caller cannot tell "absent from the document" from "absent from the sections routed to this field".Cause
extract_callbuilds every not-found value the same way, whatever region the field read, and verify then assignsmissingto any missing value without a cause.Fix
missing_in_sectionswith the reason "Not stated in its selected sections". Verify keeps it, as it keeps any recorded cause.missingstaystrue, so consumers readingmissing, or countingsummary.unverifiedper cause (an open map), keep working; a consumer that treats unknown causes as missing sees no change.missingfield description in the extraction, Block and tool schemas, and the README cause lists, name the new cause.Test
engine_test"tells a value missing from a field's sections apart from one missing from the document": a sections field and a whole-document field both come back not found; the first reports["missing_in_sections"]with its reason, the second["missing"]with none.Red before the fix, green after.
make lintclean. Suite on SQLite and PostgreSQL (runtime v0.3.59a): 196 passed, 0 failed.