Repository navigation
Conversation
added 3 commits
November 10, 2026 16:04
A trace step reported only step totals, so a change in accuracy or cost could not be traced to the call, prompt sources or failure that caused it. Each step now lists its model calls with stage, model, prompt and completion tokens (absent when the provider reported none), latency, finish reason, error code, and the chunk ordinals and field paths the call carried. Run results and stored runs carry a summary of field coverage and of unverified fields per cause.
# Conflicts: # docscout/src/engine/shapes.lua # docscout/src/engine/stages.lua
Model calls were assembled without an input budget: passages were neither ranked nor capped, verification and section choice sent everything in one call and section summaries went in one call over all sections, so a long document overflowed the window and the refused call failed its fields. Every call now fits the model card's context window less its reserved output, estimated as the llm module estimates a prompt before admitting it. Extraction and scout calls allocate passages per field from pool provenance (citations, then the field's own hits by rank) and record cut passages per field; support checks, section choices, summaries, outline windows, highlights and generated contexts are batched to fit; Deep reads the passages that fit in the order found.
This was referenced Nov 10, 2026
Contributor
Author
|
Superseded by #83, merged and released. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #59 (pool provenance, itself on #55) and includes a merge of #53 (call records) so budget exclusions land in the per-call trace. Review the last commit; the diff against
mainincludes those PRs until they merge.Cause
Every model call was assembled without an input or output budget. Passages were not ranked or capped (the pool grows with fields x queries x chunks), the support check and section choice sent every value or field in one call, section summaries went in one call over all sections, and the highlighter and generated-context batches had fixed sizes. The only guard was the llm module refusing an oversized prompt, which failed every field of the call.
Fix
max_tokens). The reserved output is the stage's output limit, else the card'soutput_tokens. The estimate is the llm module's own admission estimate (encoded messages and schema, 4 characters per token), so DocScout cuts before the provider refuses (ai.estimate,ai.fitter).allocategives each field its share of the passages still visible to it, in order of its scout citations, then its own search hits by rank, then position, taken in turns while the request fits. Cut passages are recorded per field as"k of n passages cut by budget"with causebudget_cut(unverified), and per call asexcludedin the call record. A call whose candidates are all cut produces missing values with that reason and no model call.budget_cutinstead of a call), section choice (outline in every batch, fields batched), section summaries (windowed like the outline), outline windows (also bounded by the 150-chunk window), highlighter (also at most 12 per batch) and generated context (also at most 8 per batch). Context and highlighter calls pass their reserved output asmax_tokens.prompt_exceeds_budget, naming what to lower.shapes(CHARS_PER_TOKEN,tokens), used by processing and stages.No configuration change. Default prompts are unchanged when they fit.
Tests
engine_test"Call budgets": per-field allocation order (citations, own hits by rank, turns) and cut counts; extraction cut to the window with per-field loss and callexcluded; a call whose fields alone exceed the window fails with the settings to lower; support checks batched with a verdict per value; summaries windowed and section choices batched with a summary per section and a region per field. A passage that alone exceeds the window is left out with its loss recorded (the previous test asserted the provider refusal for this case).Red before, green after.
make lintclean. Suite on SQLite and PostgreSQL: 192 passed, 0 failed.test/check-package-bindings.pyfails onmainas well (unrelated).