Skip to content

fix(llm_gateway): make token breakdown fields optional and support float penalties - #261

Open
pylrn wants to merge 1 commit into
AssemblyAI:masterfrom
pylrn:fix/llm-gateway-token-details-optional
Open

pylrn wants to merge 1 commit into
AssemblyAI:masterfrom
pylrn:fix/llm-gateway-token-details-optional

Conversation

@pylrn

@pylrn pylrn commented Sep 24, 2026

Copy link
Copy Markdown
Contributor

Summary

When LLM Gateway proxies models that return partial token details (such as OpenAI reasoning models returning completion_tokens_details: {"reasoning_tokens": 128} without audio/prediction fields, or Claude models returning prompt_tokens_details: {"cached_tokens": 512} without audio fields), Pydantic raised a ValidationError because all breakdown fields were declared as required integers.

This PR makes token detail breakdown fields optional with a default of None, adds thinking delta support for streamed reasoning models, and ensures model penalty parameters support float values.

Changes

  • assemblyai/llm_gateway/v1/models.py:
    • In LLMGatewayCacheCreation, LLMGatewayPromptTokensDetails, and LLMGatewayCompletionTokensDetails, declare breakdown fields as Optional[int] = None so partial breakdowns from upstream providers parse cleanly.
    • In LLMGatewayChunkDelta, add thinking: Optional[str] = None for parity with LLMGatewayResponseMessage.thinking during streamed completions with reasoning/thinking models.
    • In LLMGatewayDefaultParameters, update frequency_penalty to Optional[float] = None and add presence_penalty: Optional[float] = None.
  • tests/unit/test_llm_gateway.py:
    • Add test_chat_completions_tolerates_partial_token_details covering partial prompt caching and reasoning token breakdowns.
    • Add test_stream_chunk_exposes_thinking_delta verifying streamed reasoning delta parsing.
    • Add test_list_models_tolerates_float_penalties_and_presence_penalty verifying float model parameter parsing.
  • assemblyai/__version__.py:
    • Bump to 1.6.1 to satisfy CI version bump check.

Backwards Compatibility

All new/modified fields are Optional with None defaults or widen integer types to floats, preserving full compatibility with existing code and responses.

…oat penalties

- Make token detail breakdown fields in LLMGatewayCacheCreation, LLMGatewayPromptTokensDetails, and LLMGatewayCompletionTokensDetails Optional[int] = None so partial breakdowns (e.g. reasoning_tokens without audio_tokens) parse without validation errors
- Add thinking: Optional[str] = None to LLMGatewayChunkDelta for streaming reasoning model parity
- Support float frequency_penalty and optional presence_penalty in LLMGatewayDefaultParameters
- Add unit tests for partial token details, thinking delta, and float penalties
- Bump version to 1.6.1 for version bump check
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant