Skip to content

fix(docscout)!: verify only found values and judge support per field - #62

Closed
wolfy-j wants to merge 2 commits into
fix/docscout-call-budgetsfrom
fix/docscout-verified-semantics
Closed

wolfy-j wants to merge 2 commits into
fix/docscout-call-budgetsfrom
fix/docscout-verified-semantics

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Stacked on #61 (call budgets, on #59 and #55, with #53). Review the last commit; the diff against main includes those PRs until they merge.

Cause

verified mixed absence with confirmation, and the support check judged excerpts without field semantics:

  • a missing value had no reason, so verified = not failed and #reasons == 0 marked it verified, while the same outcome from "no passage" was unverified;
  • a found value with no quote passed the quote check vacuously when evidence_required was false;
  • the support prompt saw only path, value and quotes, so a quote holding a date "supported" a wrong-role date, while "support its value exactly" rejected quotes holding the value inside more text;
  • nothing judged whether a verbatim text or list value was complete, and the verifier only saw the extractor's chosen excerpts.

Fix

  • (a) verified holds only for a found value. Missing values return verified: false with cause missing (no reason text); the step message counts missing values separately.
  • (b) The support check reads each field's type and description and judges whether the cited text states this value for this field, reformatting allowed. Its verdict is supported, not_supported or insufficient_evidence (cause support_insufficient).
  • (c) A found value without a quote is unverified (no_quote) whatever evidence_required says, and gets no support call.
  • (d) For text and list values the check also reads the source region around the citations (the cited chunk's whole section when it fits retrieval.section_budget, else the cited chunk and the next) and returns a separate completeness verdict (complete, incomplete, insufficient_evidence). Fields carry complete (boolean, absent when not judged); incomplete makes the value unverified (cause incomplete). The support batches fit the model window (fix(docscout): assemble every model call within its context window #61).
  • Contract, Block and tool field schemas declare complete and the meaning of verified.

Contract change (approved): verified rates drop where missing values were counted as verified; shapes.verified (tool and Block top-level verified) is false when any field is missing.

The support prompt names every verdict and completeness label with its meaning (built from the same constants as the schema): a constrained decoder admits only those strings, and without the names a replayed batch on the target model returned not_supported for all 11 values while each reason said the quote states the value; with the names it returned supported for all 11.

Tests

  • engine_test Verification: a missing value is unverified with cause missing; a quote-less found value is unverified with or without evidence_required; the support line carries type, description and source; the support schema requires a completeness verdict exactly for text and list fields; rejected, incomplete and insufficient verdicts fail their values with their causes; supported text values report complete; the support prompt names every label the reply schema admits.
  • Updated assertions that encoded the old contract (missing verified, quote-less value verified when evidence was optional, run summary now counts missing).

Red before, green after. make lint clean. Suite on SQLite and PostgreSQL: 195 passed, 0 failed. test/check-package-bindings.py fails on main as well (unrelated).

KickSide added 2 commits November 10, 2026 17:04
verified mixed absence with confirmation: a value the document does not
state came back verified, a found value without a quote passed when its
field did not require evidence, and the support check judged excerpts
without the field's meaning, so a quote holding some date supported a
wrong-role date, and nothing judged whether verbatim text was complete.

verified now holds only for a found value with a quote whose checks pass.
The support check reads each field's type and description, judges whether
the cited text states this value for this field, may answer
insufficient_evidence, and for text and list values reads the source
region around the citations and returns a separate completeness verdict,
exposed as `complete`.

BREAKING CHANGE: missing values are returned with verified false, and
found values without a quote are never verified.
A constrained decoder only admits the verdict strings, but the prompt did
not name supported or not_supported, so the model chose among labels it
could not see: a replayed batch returned not_supported for every value
while each reason said the quote states the value. The prompt now names
each verdict and completeness label with its meaning, built from the same
constants as the schema.
@wolfy-j

wolfy-j commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #79, merged and released.

@wolfy-j wolfy-j closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant