Skip to content

perf(aikit): parse streamed tool-call arguments incrementally - #143

Open
sanchitrk wants to merge 1 commit into
canaryfrom
claude/cod-199-stream-args-3c0338
Open

sanchitrk wants to merge 1 commit into
canaryfrom
claude/cod-199-stream-args-3c0338

Conversation

@sanchitrk

@sanchitrk sanchitrk commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Summary

On every tool-input-delta, aikit appended the delta to the raw text and re-parsed the whole thing, so streaming large arguments (a big write or edit) cost quadratic CPU and memory. The raw text also rode on every partial message as partialJson.

  • New utils/streamjson.ts: an incremental, tolerant JSON parser. It reads each delta once and returns snapshots that are never mutated after being handed out; finished strings and containers are shared, and only the open path is copied. Copy credit keeps total work linear even for a huge open container.
  • stream.ts: a per-run Map<callID, ToolCallArgumentsStream> replaces partialJson; each delta sets block.arguments = args.append(delta). Final arguments still come from the AI SDK's parsed tool-call input.
  • transform.ts: StreamingToolCallBlock, updateToolCallFromInput and the parseStreamingJson call are removed. jsonparse.ts is kept.

No wire or persisted format change.

Args (64-char deltas) Before After
100 KB 1.2 s 0.9 ms
200 KB 5.2 s 1.4 ms
500 KB 36 s 4.2 ms

Partial snapshots differ from the old parser only on edge cases: a trailing lone backslash is held back until its escape completes, __proto__ stays an own key, a missing comma is tolerated, and an array root shows as {}.

Test

test/e2e/streaming.tool.args.e2e.test.ts replays an Anthropic Messages SSE stream from a local server (no API key): about 600 KB of write arguments in 9,220 deltas. It asserts the final arguments, the delta and snapshot counts, that no snapshot is mutated, no partialJson on parts, and a 2 s bound (about 170 ms locally), then writes __artifacts__/streaming-tool-args.json.

vp check and vp run typecheck pass.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

Each tool-input delta was appended to the raw text and the whole text
re-parsed, so large arguments (a big write) cost quadratic CPU and
memory: 500 KB took about 36 s. An incremental, tolerant JSON parser now
reads each delta once and hands out immutable snapshots, about 4 ms for
the same input. The raw text no longer rides on partial messages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant