Skip to content

fix(responses): preserve cached input token usage - #221

Merged
caigee-cmd merged 2 commits into
caigee-cmd:mainfrom
gruffwy:fix/responses-cache-usage
Sep 21, 2026
Merged

caigee-cmd merged 2 commits into
caigee-cmd:mainfrom
gruffwy:fix/responses-cache-usage

Conversation

@gruffwy

@gruffwy gruffwy commented Sep 20, 2026

Copy link
Copy Markdown
Contributor

Problem

The Responses compatibility layer omits cached input token usage even when providers return it. Streaming providers may expose the value either as top-level cache_read_tokens or as prompt_tokens_details.cached_tokens.

Behavior

  • Add standard usage.input_tokens_details.cached_tokens to streaming and non-streaming Responses output when the value is present.
  • Preserve explicit zero values.
  • Prefer top-level cache_read_tokens, falling back to prompt_tokens_details.cached_tokens only when absent.
  • Keep the field absent when cache usage is unknown.
  • Do not add cached tokens to total_tokens a second time.
  • Do not expose the non-standard cache-write value in the Responses schema.

Tests

  • go test ./internal/gateway
  • GOFLAGS=-buildvcs=false go test ./internal/app -run 'TestImportConstraints|TestDutyBoundaries' -count=1
  • git diff --check

@caigee-cmd
caigee-cmd merged commit 4ba84f7 into caigee-cmd:main Sep 21, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants