Repository navigation
fix(spend): a streamed call counts its prompt once when the first event carries the whole prompt - #6704
Open
PetrBulanek wants to merge 6 commits into
Open
fix(spend): a streamed call counts its prompt once when the first event carries the whole prompt#6704PetrBulanek wants to merge 6 commits into
PetrBulanek wants to merge 6 commits into
Conversation
Refs #6597 Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
…elta - Each count that the top-level usage of message_delta sends as a number replaces the count so far, also with a lower value. - A missing or null count keeps its value, and a stream that stops before message_delta keeps the counts of message_start. - Every other usage object keeps "highest wins". - Each usage object merges once, so the re-scanned tail cannot raise a replaced count again. Refs #6597 Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
- Each count that message_delta sends as a number replaces the count of message_start, also with a lower value. - A missing or null count keeps its value, so a Bedrock invoke stream keeps its input counts. - The router's call records now match the gateway meter's for the same stream. Refs #6597 Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
- A periodic record passes 0 as the call cost, so it never raises the day's most expensive call. - A final record passes its cost, also when it changes no amount. - A day whose most expensive call is still 0 falls back to the last seven days for an estimate, as a day without rows does. Refs #6597 Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
- The model catalog asks the second listing whenever the first gives no model list, not only on a refusal. - The spend entry of persistence names the price tables. Refs #6597 Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
Signed-off-by: Petr Bulánek <bulanek.petr@gmail.com>
PetrBulanek
marked this pull request as ready for review
November 19, 2026 14:56
There was a problem hiding this comment.
🛡️ Code Guardian 🛡️ — ✅ Code Review @ d5089d6
PR #6704: fix(spend): a streamed call counts its prompt once when the first event carries the whole prompt
Author: PetrBulanek | Branch: fix/6597-stream-prompt-counted-twice → main | Changes: +73 −37 (8 files)
Summary
In a Messages stream, the gateway meter and the Model Router now take each numeric count from message_delta. Only final records now set the day's most expensive call, with a seven-day fallback when it is 0.
Limits: 1 skill(s) did not run: react-ui-engineering.
2 suggestion(s) dropped under the 🟢 budget.
Findings
- ✅ Looks good: I replayed the new meter Lua in LuaJIT 2.1 at every chunk split. 300 random chunkings gave the same counts. An Anthropic event stream and a Bedrock invoke stream now record 63 uncached input tokens instead of 23,039. The merge-once guard stops a second merge of
message_startfrom the 256-byte tail. Atypeafterusage, anullcount and forged usage in stream text keep highest-wins, as onmain.
Documentation Check (doc-drift)
✅ No findings.
Issue Fit
✅ No findings.
TypeScript Engineering Review
✅ No findings.
Verdict
APPROVE — No blocking finding: the meter, the Model Router and the ingest follow the stated rules.
Review by Code Guardian 🛡️ · automated code guardian · ibm/aws/claude-opus-5-5 on pi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A streamed model call now counts each prompt token once.
Some upstreams put the whole prompt as uncached input in the first event of a stream, and give the real split only in the last event.
CurveBender's RITS models do this today, and LiteLLM does it when an operator turns on its prompt estimate.
DAM kept the highest value of each count, so it counted such a prompt two times, once at the full input price.
After this change, a streamed call costs the same as the same call without a stream.
A spend limit then stops an agent at the right time, and the token metrics show the real counts.
Closes #6597
Sub-issues
Reviewer guide
What changes in substance
message_delta) decides each count that it sends as a number. Before, the highest value won. A missing ornullcount keeps the first event's value.message_deltacan lower a count. Every other usage object keeps "the highest value wins", because in a JSON reply the model can write a usage object in tool input.message_deltakeeps the counts ofmessage_start.Where to look closely
message_startagain aftermessage_deltalowered a count. (open)typecomes beforeusagein an event. Anthropic, LiteLLM and Bedrock send them in this order.Ruled out
0as "no change": the protocol sends running totals, and no known provider sends0for "no change".Gates you changed
model-router-usage.test.ts, "merges Messages' start and delta usage": the expectedcacheReadTokenschanges from 300 to 0, because0inmessage_deltanow replaces the count. Its two scenario comments become one.Scope
settle, and the reference cost of an estimate).spend-limits.md(the stream rule, "most expensive finished call"),harness-config.md(the router follows the meter's stream rule; the second model listing),connection-presets.md(the second model listing),persistence.md(the price tables).Testing
mise run test: controller, agent-runtime (1,084 tests) and api-server (4,700 tests) pass. On macOS,cli,vm-runnerandagents:test:browser-launchfail for environment reasons. This PR does not touch them, and CI runs them on Linux.agent-runtime:check,api-server:check,docs:checkandcheck:comment-typespass.nulland0counts, stopped streams, and forged usage in model text and in JSON replies. The old meter fails exactly the 7 replace cases.rits/deepseek-ai/DeepSeek-V4.1-Flash-1017, cached prompt): the streamed call recorded 63 uncached plus 22,976 cache-read tokens, the same as the call without a stream. Before the fix: 23,039 plus 22,976. On the Usage view, "Tokens in" grew by 23.1K for the streamed call and by 23.0K for the other call.aws/claude-haiku-5-5recorded the same counts and cost with and without a stream.