Repository navigation
Conversation
added 2 commits
November 10, 2026 17:04
verified mixed absence with confirmation: a value the document does not state came back verified, a found value without a quote passed when its field did not require evidence, and the support check judged excerpts without the field's meaning, so a quote holding some date supported a wrong-role date, and nothing judged whether verbatim text was complete. verified now holds only for a found value with a quote whose checks pass. The support check reads each field's type and description, judges whether the cited text states this value for this field, may answer insufficient_evidence, and for text and list values reads the source region around the citations and returns a separate completeness verdict, exposed as `complete`. BREAKING CHANGE: missing values are returned with verified false, and found values without a quote are never verified.
A constrained decoder only admits the verdict strings, but the prompt did not name supported or not_supported, so the model chose among labels it could not see: a replayed batch returned not_supported for every value while each reason said the quote states the value. The prompt now names each verdict and completeness label with its meaning, built from the same constants as the schema.
This was referenced Nov 11, 2026
Contributor
Author
|
Superseded by #79, merged and released. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #61 (call budgets, on #59 and #55, with #53). Review the last commit; the diff against
mainincludes those PRs until they merge.Cause
verifiedmixed absence with confirmation, and the support check judged excerpts without field semantics:verified = not failed and #reasons == 0marked it verified, while the same outcome from "no passage" was unverified;evidence_requiredwas false;Fix
verifiedholds only for a found value. Missing values returnverified: falsewith causemissing(no reason text); the step message counts missing values separately.supported,not_supportedorinsufficient_evidence(causesupport_insufficient).no_quote) whateverevidence_requiredsays, and gets no support call.retrieval.section_budget, else the cited chunk and the next) and returns a separate completeness verdict (complete,incomplete,insufficient_evidence). Fields carrycomplete(boolean, absent when not judged);incompletemakes the value unverified (causeincomplete). The support batches fit the model window (fix(docscout): assemble every model call within its context window #61).completeand the meaning ofverified.Contract change (approved): verified rates drop where missing values were counted as verified;
shapes.verified(tool and Block top-levelverified) is false when any field is missing.The support prompt names every verdict and completeness label with its meaning (built from the same constants as the schema): a constrained decoder admits only those strings, and without the names a replayed batch on the target model returned
not_supportedfor all 11 values while each reason said the quote states the value; with the names it returnedsupportedfor all 11.Tests
engine_testVerification: a missing value is unverified with causemissing; a quote-less found value is unverified with or withoutevidence_required; the support line carries type, description and source; the support schema requires a completeness verdict exactly for text and list fields; rejected, incomplete and insufficient verdicts fail their values with their causes; supported text values reportcomplete; the support prompt names every label the reply schema admits.missing).Red before, green after.
make lintclean. Suite on SQLite and PostgreSQL: 195 passed, 0 failed.test/check-package-bindings.pyfails onmainas well (unrelated).