Skip to content

fix(llm): default an unset max_tokens to the card output limit - #273

Draft
wolfy-j wants to merge 1 commit into
masterfrom
fix/llm-card-output-default
Draft

wolfy-j wants to merge 1 commit into
masterfrom
fix/llm-card-output-default

Conversation

@wolfy-j

@wolfy-j wolfy-j commented Nov 16, 2026

Copy link
Copy Markdown
Contributor

Cause

route.plan_output applies the model card's output_tokens only as a cap on a requested max_tokens. A request without one ignores the card: OpenAI-family drivers send no limit, so the server's remaining context window is the only bound, and Claude/Bedrock send their fixed default_max_tokens (2000) even when the card allows far more.

Fix

An unset max_tokens is planned as a request for the card's output_tokens. The driver default applies only when the card declares no limit. A requested value is unchanged: min(requested, card). Budget-thinking planning runs on the same effective value, so an unset request on a 64000 card with effort 100 sends 64000 with a 24000 budget.

Admission reserves the planned output, so an unset request now reserves the card's output_tokens instead of 0 (OpenAI-family) or 2000 (Claude/Bedrock). That matches what is sent on the wire.

Tests

  • route_test: unset + card limit sends the card limit (with and without budget thinking); requested below the card stays; requested above is lowered; unset with no card limit keeps the driver behavior (nil / 2000 / budget raise).
  • replay_safety_test (effective output): OpenAI unset + card sends max_output_tokens = 65536; requested 4096 stays; requested 100000 is lowered and reported in metadata.adjusted; Claude without a card limit still sends 2000. The Bedrock case that asserted the 2000 default under a 64000 card now asserts 64000.
  • llm_test (context admission): the case asserting an unset request stays unbounded under a card ceiling now asserts the card limit is sent and reserved.

Red on master: the three "unset + card limit" cases fail (expected 64000/65536, got 2000/nil). Green: full llm suite 2922 tests, 9 existing skips, 0 failures; wippy lint --level error clean.

kicksideai/providers#117 (llm-ocr stops hard-coding max_tokens) depends on this.

The route planner applied the card's output_tokens only as a cap on a
requested max_tokens. A request without one sent no limit on
OpenAI-family drivers and the fixed 2000 driver default on Claude and
Bedrock, regardless of the card. The planner now treats an unset
max_tokens as a request for the card limit; the driver default applies
only when the card declares no limit.
@wolfy-j
wolfy-j marked this pull request as draft November 16, 2026 00:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant