Skip to content

fix(spend): a streamed call counts its prompt once when the first event carries the whole prompt - #6704

Open
PetrBulanek wants to merge 6 commits into
mainfrom
fix/6597-stream-prompt-counted-twice
Open

PetrBulanek wants to merge 6 commits into
mainfrom
fix/6597-stream-prompt-counted-twice

Conversation

@PetrBulanek

@PetrBulanek PetrBulanek commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

A streamed model call now counts each prompt token once.
Some upstreams put the whole prompt as uncached input in the first event of a stream, and give the real split only in the last event.
CurveBender's RITS models do this today, and LiteLLM does it when an operator turns on its prompt estimate.
DAM kept the highest value of each count, so it counted such a prompt two times, once at the full input price.
After this change, a streamed call costs the same as the same call without a stream.
A spend limit then stops an agent at the right time, and the token metrics show the real counts.

Closes #6597

Sub-issues

  • 01 — The gateway meter uses the last event's counts
  • 02 — The Model Router counts the same way
  • 03 — Only a finished call sets the day's most expensive call
  • 04 — Three outdated doc lines

Reviewer guide

What changes in substance

  • In a Messages stream, the last event (message_delta) decides each count that it sends as a number. Before, the highest value won. A missing or null count keeps the first event's value.
  • Only the top-level usage of message_delta can lower a count. Every other usage object keeps "the highest value wins", because in a JSON reply the model can write a usage object in tool input.
  • A stream that stops before message_delta keeps the counts of message_start.
  • The Model Router uses the same rule, so its token metrics match the meter.
  • Only a call's final record sets the day's most expensive call. The api-server uses that value to estimate calls without usage. A day whose most expensive call is still 0 falls back to the last seven days, as a day without rows does.

Where to look closely

  • The meter's Lua tracks which event holds a usage object, across chunk boundaries. An error here can lower real counts or keep the double count.
  • The meter now merges each usage object once. Without this, the 256-byte tail that each chunk scans again merges message_start again after message_delta lowered a count. (open)
  • The meter uses the new rule only when type comes before usage in an event. Anthropic, LiteLLM and Bedrock send them in this order.
  • A final record that changes no amount still writes the day's most expensive call.

Ruled out

  • Replace only the uncached input when the last event reports input: it fails again if LiteLLM starts to estimate the cache counts too.
  • Replace the whole input group, or treat 0 as "no change": the protocol sends running totals, and no known provider sends 0 for "no change".
  • Ignore the first event's input on LiteLLM Connections for stopped streams: it records less than LiteLLM bills in every lab test.
  • Skip the ledger write for a final record without a change (the one-line fix): the day's most expensive call then stays at 0.

Gates you changed

  • model-router-usage.test.ts, "merges Messages' start and delta usage": the expected cacheReadTokens changes from 300 to 0, because 0 in message_delta now replaces the count. Its two scenario comments become one.

Scope

  • The gateway meter (Lua in the controller), the Model Router's usage scanner, and the api-server spend ingest (settle, and the reference cost of an estimate).
  • Docs: spend-limits.md (the stream rule, "most expensive finished call"), harness-config.md (the router follows the meter's stream rule; the second model listing), connection-presets.md (the second model listing), persistence.md (the price tables).
  • The plan files: the first commit adds them, and the last commit deletes them.
  • Out: the price of each token kind, and stopped streams that are counted too low or too high (not filed; it waits for PROD data).

Testing

  • mise run test: controller, agent-runtime (1,084 tests) and api-server (4,700 tests) pass. On macOS, cli, vm-runner and agents:test:browser-launch fail for environment reasons. This PR does not touch them, and CI runs them on Linux.
  • agent-runtime:check, api-server:check, docs:check and check:comment-types pass.
  • Meter: an offline Lua 5.1 harness ran 20 stream cases at every chunk split and in 300 random chunkings. The cases cover Anthropic and Bedrock invoke streams, null and 0 counts, stopped streams, and forged usage in model text and in JSON replies. The old meter fails exactly the 7 replace cases.
  • AC-1, through a real agent gateway and the shared CurveBender Connection (rits/deepseek-ai/DeepSeek-V4.1-Flash-1017, cached prompt): the streamed call recorded 63 uncached plus 22,976 cache-read tokens, the same as the call without a stream. Before the fix: 23,039 plus 22,976. On the Usage view, "Tokens in" grew by 23.1K for the streamed call and by 23.0K for the other call.
  • Regression: an IBM LiteLLM call to aws/claude-haiku-5-5 recorded the same counts and cost with and without a stream.
  • Model Router: real CurveBender and IBM LiteLLM streams, captured and run through the old and the new scanner, give the same counts as the meter. This check did not run inside a rebuilt agent image.
  • Spend ingest: a throwaway script drove the ingest with Redis and an in-memory ledger. Periodic records no longer raise the day's most expensive call. A lower final record sets it, and so does a final record that changes no amount.
  • Not run on staging.

Refs #6597

Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
…elta

- Each count that the top-level usage of message_delta sends as a number replaces the count so far, also with a lower value.
- A missing or null count keeps its value, and a stream that stops before message_delta keeps the counts of message_start.
- Every other usage object keeps "highest wins".
- Each usage object merges once, so the re-scanned tail cannot raise a replaced count again.

Refs #6597

Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
- Each count that message_delta sends as a number replaces the count of message_start, also with a lower value.
- A missing or null count keeps its value, so a Bedrock invoke stream keeps its input counts.
- The router's call records now match the gateway meter's for the same stream.

Refs #6597

Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
- A periodic record passes 0 as the call cost, so it never raises the day's most expensive call.
- A final record passes its cost, also when it changes no amount.
- A day whose most expensive call is still 0 falls back to the last seven days for an estimate, as a day without rows does.

Refs #6597

Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
- The model catalog asks the second listing whenever the first gives no model list, not only on a refusal.
- The spend entry of persistence names the price tables.

Refs #6597

Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
@PetrBulanek
PetrBulanek marked this pull request as ready for review November 19, 2026 14:56
@PetrBulanek PetrBulanek added the code-guardian:review Add this label to trigger an automatic review by code-guardian. label Nov 19, 2026

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian 🛡️ — ✅ Code Review @ d5089d6

PR #6704: fix(spend): a streamed call counts its prompt once when the first event carries the whole prompt

Author: PetrBulanek | Branch: fix/6597-stream-prompt-counted-twice → main | Changes: +73 −37 (8 files)

Summary

In a Messages stream, the gateway meter and the Model Router now take each numeric count from message_delta. Only final records now set the day's most expensive call, with a seven-day fallback when it is 0.

Limits: 1 skill(s) did not run: react-ui-engineering.

2 suggestion(s) dropped under the 🟢 budget.

Findings

  • ✅ Looks good: I replayed the new meter Lua in LuaJIT 2.1 at every chunk split. 300 random chunkings gave the same counts. An Anthropic event stream and a Bedrock invoke stream now record 63 uncached input tokens instead of 23,039. The merge-once guard stops a second merge of message_start from the 256-byte tail. A type after usage, a null count and forged usage in stream text keep highest-wins, as on main.

Documentation Check (doc-drift)

✅ No findings.

Issue Fit

✅ No findings.

TypeScript Engineering Review

✅ No findings.

Verdict

APPROVE — No blocking finding: the meter, the Model Router and the ingest follow the stated rules.


Review by Code Guardian 🛡️ · automated code guardian · ibm/aws/claude-opus-5-5 on pi

@dam-code-protector dam-code-protector Bot removed the code-guardian:review Add this label to trigger an automatic review by code-guardian. label Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A Messages stream counts the prompt twice when its first event carries the whole prompt

1 participant