Skip to content

fix(agents): count the tokens a failed call already billed - #1205

Merged
Birfy merged 3 commits into
mainfrom
agent/failed-usage-tokens
Nov 21, 2026
Merged

Birfy merged 3 commits into
mainfrom
agent/failed-usage-tokens

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Fixes #1202.

What this changes

A failed model call today reports its wall-clock but not the tokens the provider already billed. Usage splits wall-clock failures (failures, failure_seconds) and counts them in summary(), but a call that failed after the provider had metered tokens adds nothing to prompt_tokens / completion_tokens / total_tokens — so a reader sees "3 failed" beside a token total that silently excludes them, and the token meter runs behind what the provider will charge.

This adds two fields and records what the provider actually reported:

  • Usage.failed_usage_attempts — failed provider attempts that reported usage.
  • Usage.failed_usage_tokens — the tokens those attempts reported.

Both default to 0, so an old payload or fixture that lacks them loads unchanged; nothing in agents.py serializes Usage, and evolution.py's --json usage dict gains the two keys only (names byte-identical to the fields).

Decision: failed calls with reported usage count toward totals; failed calls without usage count only as failed_calls

The rejected alternative was a reporting-only column that left the totals and the limit untouched — that keeps the limit underestimating what the provider charges, which is the defect.

Why it lands in max_tokens and the governor, not max_usd

max_usd is charged per call, not per token: budget.py:112-135 reserves n * usd_per_call before the call, _taken only grows, and there is no refund. A failed call is therefore already counted on the dollar side today — what was missing is only the token side. Adding these tokens moves the max_tokens meter (async_evolve.py:1109) and the governor's spend (:1223); it does not change max_usd.

How counts are recorded

  • Both cross-module HTTP error paths read the response body once, before record(...), outside the usage is not None guard, and parse counts from the whole body; detail still stops at strip()[:400] and is byte-identical to today's text.
  • _read_sse writes each chunk's usage into a seen dict, so a stream that dies after the provider billed still counts those tokens.
  • _usage_counts accepts {"usage": {...}} or top-level, both wire shapes, integers only, non-negative, booleans ignored. A total_tokens-only body is treated as no evidence — splitting it would be a guess, and the decision is to count on evidence only.
  • The new counters are added inside the same with self._lock block as prompt_tokens/completion_tokens, so the failure column is a subset of the total by construction, not a parallel tally.
  • TokenSink.record(..., reported_usage=...) forwards tokens as before and then calls Usage.note_failed_usage(...), which increments only the two new counters — no prompt_tokens, no calls, nothing double counted.

Invariants the tests pin

  • failed_usage_tokens <= total_tokens on both paths.
  • failed_usage_attempts <= calls on the direct path, where attempts and calls are the same thing.
  • On the spec/TokenSink path the column counts usage-bearing attempts, one actor call can be retried, so it may exceed calls; that path asserts only the token subset, and the docstring says so.

Red first

At the live main this branch merges (e43e847a206cf20206d5460c2e27756df92fac65),
clean worktree, only tests/test_failed_call_usage.py added:

python3 -m pytest -o addopts='' -p no:cacheprovider -q tests/test_failed_call_usage.py
16 failed, 2 passed in 7.68s

The two passers pin unchanged behaviour (a usage=None caller keeps the same error text; the subprocess and SDK paths add no tokens), so they are not part of the red count. Typical failures: TypeError: Usage.record() got an unexpected keyword argument 'reported_usage', AttributeError: 'Usage' object has no attribute 'failed_usage_attempts', and assert 0 == 100 where the base never parses the error body at all.

Green at the tip (18 tests): totals and the new column both move for a usage-bearing failure; a usage-less failure moves failed_calls only; an old five-key payload round-trips with both fields 0; the body-padded row pins "parse untruncated, detail unchanged at 400 bytes"; the streaming row pins a stream that dies after usage; the reconciliation row keeps one usage-bearing + one usage-less failure at 2 failed, 1 with usage; the meter row shows a usage-bearing failure can end the run at stop_reason == "max_tokens" where the same cap without it does not; the sink rows pin the counters-only path and a retried call reporting several attempts.

python3 -m pytest -o addopts='' -p no:cacheprovider -q tests/test_failed_call_usage.py
18 passed in 7.55s
python3 -m pytest -o addopts='' -p no:cacheprovider -q \
    tests/test_failed_call_usage.py tests/test_agents.py tests/test_usage.py \
    tests/test_token_budget.py tests/test_spec_usage.py tests/test_call_budget.py \
    tests/test_budget_governor.py tests/test_backward_compat.py
115 passed, 2 warnings in 21.27s
python3 -m tools.gen_api_docs --check      # docs/api.md is up to date
python3 -m tools.changelog_fragments check # 13 changelog fragment(s) OK

The tip is a plain merge of main at e43e847a into the branch (no rebase, no
force-push); its tree is byte-identical to the merge this change was verified on
before the push, and docs/api.md needed no regeneration after it.

Scope

agents.py (the Usage fields, record(), note_failed_usage(), TokenSink.record(), the two HTTPError sites, _read_sse, _usage_counts, summary()), the evolution.py usage whitelist dict (two additive keys only), docs/api.md regenerated by the tool, one changelog fragment, and the new test file. No engine behaviour change, no new RoundInfo field, no show projection change, no default value changed.

A 4xx body, or a stream that dies after its last chunk, can carry usage
numbers -- tokens the provider charged and reported. The adapter recorded
the call's wall-clock and dropped the tokens, so the run's totals (and the
max_tokens meter) under-reported what was spent.

Both body-reading sites now read the body once, before record(), and parse
counts from the untruncated text while detail stays cut at 400 bytes. The
accumulation happens in the same record() call that adds to the totals, so
the failed-with-usage column cannot drift from total_tokens. TokenSink
mirrors the counters only, so a spec run's provider attempts are visible
without counting anything twice. A failure with no usage records 0 tokens
and is only a failed attempt; no estimation tightens a limit.
`_usage_counts` and `_usage_from_body` return
`Optional[Tuple[int, int]]`, but `Tuple` was never imported: the
typing line stopped at `Sequence, runtime_checkable`. With
`from __future__ import annotations` the paste does not fail at
import, so the tests and CI stayed green, and the gap only showed
when the annotations were resolved -- `typing.get_type_hints` on
either function raised `NameError: name 'Tuple' is not defined`.
@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller agent-pr Pull request opened by an agent labels Nov 21, 2026
@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: b21b06c
Ran:

  • .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 1205 → T3 (528 lines / 4 files; agents.py, evolution.py)
  • pytest -o addopts="" -p no:cacheprovider -q tests/test_failed_call_usage.py tests/test_spec_usage.py tests/test_metrics.py tests/test_agents.py tests/test_usage.py at this tip → 75 passed
  • python -m tools.changelog_fragments check → 13 fragments OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Required CI at this tip → green (build, test (3.9)–test (3.13), otel-wire; deploy skipped; risk tier / reviewed tip report-only)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • mkdocs build --strict locally — docs/api.md is check-clean and the CI docs build is already green
    Findings:
  • No actionable defects. A 4xx body or a stream that dies after its last usage chunk now records the provider's tokens in the ordinary totals (so max_tokens sees the bill) and in failed_usage_attempts / failed_usage_tokens. Both fields default to 0. HTTP error paths read the body once, parse usage from the full text, and keep detail at 400 bytes. TokenSink forwards the column via note_failed_usage without double-counting tokens; on that path the column counts attempts, which may exceed calls. Subprocess and SDK failure paths are unchanged (no tokens). Integers only; a total_tokens-only body is not split.
    Verdict: PASS

@Birfy
Birfy merged commit 2dff12b into main Nov 21, 2026
12 checks passed
@Birfy
Birfy deleted the agent/failed-usage-tokens branch November 21, 2026 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(agents): a failed model call reports its wall-clock but not the tokens the provider billed

2 participants