Skip to content

feat(docscout): record model call outcomes and run summaries - #53

Merged
wolfy-j merged 67 commits into
mainfrom
feat/docscout-call-trace
Nov 11, 2026
Merged

wolfy-j merged 67 commits into
mainfrom
feat/docscout-call-trace

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Trace records retain model call outcomes and known request attribution without changing prompts or extraction policy. Selected-agent records consume published native Dataflow outcomes, including request failures, truncation, latency and field attribution. Outline, summary, highlight and support records carry known source ordinals and fields; malformed embeddings retain their validation failure and reported usage.

The branch merges current main and the saved-verification-reason fix without rewriting existing commits. The static bundle is rebuilt from the merged source; all 118 UI tests and typecheck pass. The harness resolves dependencies fresh, including Dataflow 0.7.37 and LLM 0.5.18, with no source replacement for either dependency.

Validation with official v0.3.59a:

  • All 227 Docscout tests pass on SQLite and PostgreSQL, including selected-agent failure/truncation, attribution, malformed embeddings and persistence.
  • Lint and package-binding checks pass.
  • A real browser run shows the failed selected-agent Scout step, paid tool-round usage and persisted run total on both engines; its call records retain the native failure metadata.

For C reported requests, N document chunks and F <= 50 configured fields, additional metadata has at most C records, CN source references and CF field references plus scalar outcomes. Records contain no prompts, replies or vectors. Native agent-internal tool and memory requests remain outside primary-step telemetry; source ordinals stay absent when the native contract does not report them. Summaries count produced field outcomes rather than all requested fields.

Compatibility: call records and summaries are optional additions to existing JSON payloads. Older saved runs remain readable. There is no new setting, table, migration, retry or retention mechanism. At run admission rate r and effective retention T, retained run rows approach rT, 10rT and 100r*T at base, 10x and 100x load; metadata increases bytes per row within the documented input-dependent bound.

Wolfy-J added 10 commits November 10, 2026 11:59
The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.
A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.
The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.
These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.
…y item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.
Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.
A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.
A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.
A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.
The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.
KickSide added 25 commits November 10, 2026 19:45
@wolfy-j
wolfy-j merged commit 8619ee8 into main Nov 11, 2026
16 checks passed
@github-actions github-actions Bot mentioned this pull request Nov 11, 2026
wolfy-j added a commit that referenced this pull request Nov 11, 2026
…ion (#64)

* fix(docscout): meter failed replies and report length cuts as truncation

A structured reply cut at the output limit failed in the provider as
invalid JSON, without usage, so truncation looked like a model error and
its tokens were missing from the trace. The llm module now carries a
failed reply's usage and finish reason in the error details; model calls
meter that usage, record the finish reason, and report a length cut as
structured_reply_truncated.

* test(docscout): cover failed generation and cumulative agent usage

* chore(docscout): resolve current published harness modules

* fix(docscout): retain reasons for unverified saved values (#66)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve schema defaults in UI creation payloads

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>

* feat(docscout): record model call outcomes and run summaries (#53)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* feat(docscout): record every model call and a run summary in the trace

A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve call outcomes and known request attribution

* fix(docscout): preserve schema defaults in UI creation payloads

* docs(docscout): identify native model references precisely

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

* chore(docscout): refresh published harness dependencies

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>

---------

Co-authored-by: KickSide <dev@kickside.ai>
wolfy-j added a commit that referenced this pull request Nov 11, 2026
…ion (#69)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* feat(docscout): record every model call and a run summary in the trace

A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve call outcomes and known request attribution

* fix(docscout): preserve schema defaults in UI creation payloads

* docs(docscout): identify native model references precisely

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

* chore(docscout): refresh published harness dependencies

* fix(docscout): meter failed replies and report length cuts as truncation (#64)

* fix(docscout): meter failed replies and report length cuts as truncation

A structured reply cut at the output limit failed in the provider as
invalid JSON, without usage, so truncation looked like a model error and
its tokens were missing from the trace. The llm module now carries a
failed reply's usage and finish reason in the error details; model calls
meter that usage, record the finish reason, and report a length cut as
structured_reply_truncated.

* test(docscout): cover failed generation and cumulative agent usage

* chore(docscout): resolve current published harness modules

* fix(docscout): retain reasons for unverified saved values (#66)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve schema defaults in UI creation payloads

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>

* feat(docscout): record model call outcomes and run summaries (#53)

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* feat(docscout): record every model call and a run summary in the trace

A trace step reported only step totals, so a change in accuracy or cost
could not be traced to the call, prompt sources or failure that caused it.

Each step now lists its model calls with stage, model, prompt and
completion tokens (absent when the provider reported none), latency,
finish reason, error code, and the chunk ordinals and field paths the call
carried. Run results and stored runs carry a summary of field coverage and
of unverified fields per cause.

* fix(docscout): require one support verdict per field

The support check asked for an array of verdicts bounded only above, so a
constrained decoder could return an empty or partial list and every field
without a verdict came back unverified. The reply is now an object with one
required verdict per field path; a reply that omits a field fails schema
validation as a retryable structured-output error.

* fix(docscout): default map, scout, extract and verify to temperature 0

These stages had no configured temperature by default, so new extractors
sampled at the provider default and the same passages could read
differently from run to run. New configurations now carry temperature 0
for each of them; an explicitly omitted temperature still leaves the
provider default. The UI bundle is rebuilt with the new schema defaults.

* fix(docscout): key scout evidence, section summaries and highlights by item

Scout evidence and notes, section summaries and highlighter passages were
asked for as arrays a constrained decoder may legally leave empty or
partial. An empty scout evidence list beside a drop list pruned the pool,
missing summaries went unnoticed, and an empty highlighter reply dropped
every passage of the batch.

Each reply is now an object with one required key per field, section or
passage. An explicitly empty list or note stays legal and means none; the
scout step reports how many fields had no cited evidence. The support
verdict schema uses the same keyed shape.

* fix(docscout): fail free-text replies cut by their output limit

Deep analysis and agent scout replies were read without checking
finish_reason, so a reply cut at the output limit was handed to extraction
as if complete, and neither Deep nor the highlighter could set an output
limit. ai.generate and the agent loop now fail a length-cut reply with
model_reply_truncated. Deep accepts max_tokens next to its temperatures,
and the highlighter accepts temperature and max_tokens.

* fix(docscout): keep the built outline on the processed copy

A built outline lived only in the run's memory, so every run of an
unheaded document paid for the outline calls again.

Map now stores the outline it builds on the processed copy under an
identity of the map model card, outline prompt version and outline
window, and reuses it when the identity matches. Map outline "build"
still builds on every run.

* fix(docscout): verify dates against their declared format

A declared expected_format reached the model only as prompt text, so a
date in another format or an impossible date passed verification.

Verify now parses a date value with the pattern its expected_format
declares (YYYY, YY, MMMM, MMM, MM, M, DD, D and separators, any letter
case), in lists and objects too, and reports a mismatch as a reason. A
format written in prose stays a model hint. The README states that an
extractor reads one document per input and that cross-field rules
belong to the consumer.

* fix(docscout): keep per-field provenance in the evidence pool

A scout drop removed a passage for every field and blacklisted it, so a
passage another field needed or a later targeted search found again could
not return, and scout citations were discarded after pruning.

The pool now records per passage the fields and queries that found it, its
best rank, the fields a review cited it for and the fields it is excluded
for. A drop excludes a passage only for the reviewed fields that did not
cite it, a field's own later search re-admits it for that field, and each
extraction call reads the passages still visible to its fields.

* fix(docscout): mark a length-cut free-text reply as not retryable

The same call stops at the same output limit again, so replaying it cannot
succeed; truncation is not a transport failure.

* docs(docscout): scope truncation guarantee to the built-in scout

* test(docscout): establish incomplete support reply failures

* test(docscout): exercise keyed stage omission and empty replies

* test(docscout): expose pruning transitions and unused highlights

* fix(docscout): apply current evidence reviews and highlight usable passages

* fix(docscout): leave month-name dates unverified without a language

* test(docscout): preserve localized dates through enabled and disabled verification

* fix(docscout): key cached outlines by sampling and model-card inputs

* docs(docscout): define outline reuse and invalidation behavior

* fix(docscout): preserve call outcomes and known request attribution

* fix(docscout): preserve schema defaults in UI creation payloads

* docs(docscout): identify native model references precisely

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* test(docscout): refresh harness dependencies from the registry

* fix(docscout): retain reasons for unverified saved values

* chore(docscout): refresh published harness dependencies

* chore(docscout): refresh published harness dependencies

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>

---------

Co-authored-by: KickSide <dev@kickside.ai>

---------

Co-authored-by: Wolfy-J <dev@kickside.ai>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant