Skip to content

fix(docscout): meter failed replies and report length cuts as truncation - #69

Merged
wolfy-j merged 69 commits into
mainfrom
feat/docscout-call-trace
Nov 11, 2026
Merged

wolfy-j merged 69 commits into
mainfrom
feat/docscout-call-trace

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

Brings the change reviewed in #64 to main. #64 was stacked on the call-trace branch and merged into that branch rather than main.

Failed model replies that carry usage are metered, and a reply cut by its output limit is reported as truncation, using the failure details wippy/llm supplies (details.tokens, details.finish_reason). With an older llm module that supplies none, behaviour is unchanged. Tests cover the structured, generate and agent failure paths and the older-module case.

Wolfy-J added 30 commits November 10, 2026 11:59
The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.
A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.
The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.
These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.
…y item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.
Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.
A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.
A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.
A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.
The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.
KickSide and others added 28 commits November 10, 2026 19:44
…ion (#64)

* fix(docscout): meter failed replies and report length cuts as truncation

A structured reply cut at the output limit failed in the provider as
invalid JSON, without usage, so truncation looked like a model error and
its tokens were missing from the trace. The llm module now carries a
failed reply's usage and finish reason in the error details; model calls
meter that usage, record the finish reason, and report a length cut as
structured_reply_truncated.

* test(docscout): cover failed generation and cumulative agent usage

* chore(docscout): resolve current published harness modules

* fix(docscout): retain reasons for unverified saved values (#66)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve schema defaults in UI creation payloads

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>

* feat(docscout): record model call outcomes and run summaries (#53)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* feat(docscout): record every model call and a run summary in the trace

A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve call outcomes and known request attribution

* fix(docscout): preserve schema defaults in UI creation payloads

* docs(docscout): identify native model references precisely

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

* chore(docscout): refresh published harness dependencies

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>

---------

Co-authored-by: KickSide <dev@kickside.ai>
@wolfy-j
wolfy-j merged commit 55bca58 into main Nov 11, 2026
16 checks passed
@github-actions github-actions Bot mentioned this pull request Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant