Repository navigation
fix(agents): count the tokens a failed call already billed - #1205
Merged
Merged
Conversation
A 4xx body, or a stream that dies after its last chunk, can carry usage numbers -- tokens the provider charged and reported. The adapter recorded the call's wall-clock and dropped the tokens, so the run's totals (and the max_tokens meter) under-reported what was spent. Both body-reading sites now read the body once, before record(), and parse counts from the untruncated text while detail stays cut at 400 bytes. The accumulation happens in the same record() call that adds to the totals, so the failed-with-usage column cannot drift from total_tokens. TokenSink mirrors the counters only, so a spec run's provider attempts are visible without counting anything twice. A failure with no usage records 0 tokens and is only a failed attempt; no estimation tightens a limit.
`_usage_counts` and `_usage_from_body` return `Optional[Tuple[int, int]]`, but `Tuple` was never imported: the typing line stopped at `Sequence, runtime_checkable`. With `from __future__ import annotations` the paste does not fail at import, so the tests and CI stayed green, and the gap only showed when the annotations were resolved -- `typing.get_type_hints` on either function raised `NameError: name 'Tuple' is not defined`.
Collaborator
|
Review (Birfy) Tip SHA: b21b06c
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1202.
What this changes
A failed model call today reports its wall-clock but not the tokens the provider already billed.
Usagesplits wall-clock failures (failures,failure_seconds) and counts them insummary(), but a call that failed after the provider had metered tokens adds nothing toprompt_tokens/completion_tokens/total_tokens— so a reader sees "3 failed" beside a token total that silently excludes them, and the token meter runs behind what the provider will charge.This adds two fields and records what the provider actually reported:
Usage.failed_usage_attempts— failed provider attempts that reported usage.Usage.failed_usage_tokens— the tokens those attempts reported.Both default to
0, so an old payload or fixture that lacks them loads unchanged; nothing inagents.pyserializesUsage, andevolution.py's--jsonusage dict gains the two keys only (names byte-identical to the fields).Decision: failed calls with reported usage count toward totals; failed calls without usage count only as failed_callsThe rejected alternative was a reporting-only column that left the totals and the limit untouched — that keeps the limit underestimating what the provider charges, which is the defect.
Why it lands in
max_tokensand the governor, notmax_usdmax_usdis charged per call, not per token:budget.py:112-135reservesn * usd_per_callbefore the call,_takenonly grows, and there is no refund. A failed call is therefore already counted on the dollar side today — what was missing is only the token side. Adding these tokens moves themax_tokensmeter (async_evolve.py:1109) and the governor's spend (:1223); it does not changemax_usd.How counts are recorded
record(...), outside theusage is not Noneguard, and parse counts from the whole body;detailstill stops atstrip()[:400]and is byte-identical to today's text._read_ssewrites each chunk's usage into aseendict, so a stream that dies after the provider billed still counts those tokens._usage_countsaccepts{"usage": {...}}or top-level, both wire shapes, integers only, non-negative, booleans ignored. Atotal_tokens-only body is treated as no evidence — splitting it would be a guess, and the decision is to count on evidence only.with self._lockblock asprompt_tokens/completion_tokens, so the failure column is a subset of the total by construction, not a parallel tally.TokenSink.record(..., reported_usage=...)forwards tokens as before and then callsUsage.note_failed_usage(...), which increments only the two new counters — noprompt_tokens, nocalls, nothing double counted.Invariants the tests pin
failed_usage_tokens <= total_tokenson both paths.failed_usage_attempts <= callson the direct path, where attempts and calls are the same thing.TokenSinkpath the column counts usage-bearing attempts, one actor call can be retried, so it may exceedcalls; that path asserts only the token subset, and the docstring says so.Red first
At the live
mainthis branch merges (e43e847a206cf20206d5460c2e27756df92fac65),clean worktree, only
tests/test_failed_call_usage.pyadded:The two passers pin unchanged behaviour (a
usage=Nonecaller keeps the same error text; the subprocess and SDK paths add no tokens), so they are not part of the red count. Typical failures:TypeError: Usage.record() got an unexpected keyword argument 'reported_usage',AttributeError: 'Usage' object has no attribute 'failed_usage_attempts', andassert 0 == 100where the base never parses the error body at all.Green at the tip (18 tests): totals and the new column both move for a usage-bearing failure; a usage-less failure moves
failed_callsonly; an old five-key payload round-trips with both fields0; the body-padded row pins "parse untruncated,detailunchanged at 400 bytes"; the streaming row pins a stream that dies after usage; the reconciliation row keeps one usage-bearing + one usage-less failure at2 failed, 1 with usage; the meter row shows a usage-bearing failure can end the run atstop_reason == "max_tokens"where the same cap without it does not; the sink rows pin the counters-only path and a retried call reporting several attempts.The tip is a plain merge of
mainate43e847ainto the branch (no rebase, noforce-push); its tree is byte-identical to the merge this change was verified on
before the push, and
docs/api.mdneeded no regeneration after it.Scope
agents.py(theUsagefields,record(),note_failed_usage(),TokenSink.record(), the two HTTPError sites,_read_sse,_usage_counts,summary()), theevolution.pyusage whitelist dict (two additive keys only),docs/api.mdregenerated by the tool, one changelog fragment, and the new test file. No engine behaviour change, no newRoundInfofield, noshowprojection change, no default value changed.