Skip to content

fix(docscout): assemble every model call within its context window - #61

Closed
wolfy-j wants to merge 3 commits into
fix/docscout-pool-provenancefrom
fix/docscout-call-budgets
Closed

wolfy-j wants to merge 3 commits into
fix/docscout-pool-provenancefrom
fix/docscout-call-budgets

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Stacked on #59 (pool provenance, itself on #55) and includes a merge of #53 (call records) so budget exclusions land in the per-call trace. Review the last commit; the diff against main includes those PRs until they merge.

Cause

Every model call was assembled without an input or output budget. Passages were not ranked or capped (the pool grows with fields x queries x chunks), the support check and section choice sent every value or field in one call, section summaries went in one call over all sections, and the highlighter and generated-context batches had fixed sizes. The only guard was the llm module refusing an oversized prompt, which failed every field of the call.

Fix

  • One rule for every call: the request fits when its estimated prompt plus the reserved output stays inside the model card's context window (max_tokens). The reserved output is the stage's output limit, else the card's output_tokens. The estimate is the llm module's own admission estimate (encoded messages and schema, 4 characters per token), so DocScout cuts before the provider refuses (ai.estimate, ai.fitter).
  • Extraction and scout: allocate gives each field its share of the passages still visible to it, in order of its scout citations, then its own search hits by rank, then position, taken in turns while the request fits. Cut passages are recorded per field as "k of n passages cut by budget" with cause budget_cut (unverified), and per call as excluded in the call record. A call whose candidates are all cut produces missing values with that reason and no model call.
  • Batched to fit: support check (a value too large alone gets budget_cut instead of a call), section choice (outline in every batch, fields batched), section summaries (windowed like the outline), outline windows (also bounded by the 150-chunk window), highlighter (also at most 12 per batch) and generated context (also at most 8 per batch). Context and highlighter calls pass their reserved output as max_tokens.
  • Deep analysis reads the passages that fit, in the order it found them.
  • A call whose fixed prompt alone exceeds the window fails with prompt_exceeds_budget, naming what to lower.
  • The token estimate constant moves to shapes (CHARS_PER_TOKEN, tokens), used by processing and stages.

No configuration change. Default prompts are unchanged when they fit.

Tests

engine_test "Call budgets": per-field allocation order (citations, own hits by rank, turns) and cut counts; extraction cut to the window with per-field loss and call excluded; a call whose fields alone exceed the window fails with the settings to lower; support checks batched with a verdict per value; summaries windowed and section choices batched with a summary per section and a region per field. A passage that alone exceeds the window is left out with its loss recorded (the previous test asserted the provider refusal for this case).

Red before, green after. make lint clean. Suite on SQLite and PostgreSQL: 192 passed, 0 failed. test/check-package-bindings.py fails on main as well (unrelated).

KickSide added 3 commits November 10, 2026 16:04
A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.
# Conflicts:
#	docscout/src/engine/shapes.lua
#	docscout/src/engine/stages.lua
Model calls were assembled without an input budget: passages were neither
ranked nor capped, verification and section choice sent everything in one
call and section summaries went in one call over all sections, so a long
document overflowed the window and the refused call failed its fields.

Every call now fits the model card's context window less its reserved
output, estimated as the llm module estimates a prompt before admitting it.
Extraction and scout calls allocate passages per field from pool
provenance (citations, then the field's own hits by rank) and record cut
passages per field; support checks, section choices, summaries, outline
windows, highlights and generated contexts are batched to fit; Deep reads
the passages that fit in the order found.
@wolfy-j

wolfy-j commented Nov 12, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #83, merged and released.

@wolfy-j wolfy-j closed this Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant