Skip to content

fix(docscout): make completeness judgments opt-in per extractor - #79

Merged
wolfy-j merged 7 commits into
mainfrom
fix/docscout-opt-in-completeness
Nov 11, 2026
Merged

wolfy-j merged 7 commits into
mainfrom
fix/docscout-opt-in-completeness

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Verify adds an opt-in per-field support judgment and completeness judgment for text and list fields. stages.verify.check_completeness defaults off. Existing configurations retain the support prompt, schema, model requests, call count, verification outcomes, summaries and persisted results.

When enabled, the request includes the full field contract and source passages. Complete, incomplete and unknown remain distinct in current and stored results and in the Test and Runs views. If the enhanced contract or any optional source context cannot fit, or optional source acquisition fails, Verify sends the existing support request for all fields in that call and leaves completeness unknown. The policy adds input tokens without an extra model call.

Validation uses the official v0.3.59a runtime and refreshed dependencies:

  • Regression tests fail before the admission, chunk-read and source-region fixes and pass afterward on SQLite and PostgreSQL. Source-budget guards compare captured requests and usage with the existing support request for single and multiple fields, in both field orders, and require supported fields to stay verified with completeness unknown.
  • Captured request bodies from main are compared exactly for one extractor containing accepted, rejected, missing and failed fields, with Verify enabled and disabled and with the policy absent or false. Usage derives from serialized requests and replies. Summaries and persisted fields also match.
  • Full Docscout suites pass 303 tests per engine; 136 UI tests, TypeScript and lint pass. The make target rebuilds static assets.
  • Browser guards enable the policy through the Builder, assert all three visible completeness states and stored results, and cover oversized optional inputs on both engines. The companion guard PR https://github.com/kicksideai/kickside-internal/pull/315 includes the coverage map and issue audit.

Configuration and result fields are optional and additive. Existing data needs no migration. The change preserves the informational field codes and scoped-absence behavior on main.

KickSide added 7 commits November 11, 2026 01:07
A constrained decoder only admits the verdict strings, but the prompt did
not name supported or not_supported, so the model chose among labels it
could not see: a replayed batch returned not_supported for every value
while each reason said the quote states the value. The prompt now names
each verdict and completeness label with its meaning, built from the same
constants as the schema.
@wolfy-j
wolfy-j merged commit a1cfbc9 into main Nov 11, 2026
16 checks passed
@github-actions github-actions Bot mentioned this pull request Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant