Skip to content

feat: stop replaying thinking after the prompt cache expires - #116

Closed
msmakouz wants to merge 1 commit into
masterfrom
feat/drop-expired-thinking
Closed

msmakouz wants to merge 1 commit into
masterfrom
feat/drop-expired-thinking

Conversation

@msmakouz

Copy link
Copy Markdown
Contributor

What was changed

Opt-in. When a user message opens a turn after the session was idle longer than the prompt cache lifetime, thinking blocks of earlier turns are no longer replayed. The cache has expired by then, so no cached prefix is lost, and the rewrite and later reads are smaller by those blocks. Off by default. Only drivers that replay thinking_blocks (Claude, Bedrock) are affected.

Usage

Agent or trait:

agent_options:                                                                                                                                                                
  prompt_cache:                                                                                                                                                               
    ttl: 1h                      # optional, 5m when unset                                                                                                                    
    drop_expired_thinking: true                                                                                                                                               

Model card, first route:

  providers:                                                                                                                                                                    
    - id: wippy.llm.claude:provider                                                                                                                                             
      provider_model: claude-sonnet-5                                                                                                                                           
      thinking: adaptive                                                                                                                                                        
      drop_expired_thinking: true                                                                                                                                               
      options:                                                                                                                                                                  
        cache_ttl: 1h              # optional                                                                                                                                   

A boolean on the agent wins over the card, so drop_expired_thinking: false switches it off for one agent.

Behaviour

  • Lifetime: agent prompt_cache.ttl, else the card route's options.cache_ttl, else 5m, plus 60 s. Known values: 5m, 1h. If the provider reports tokens.cache_write_by_ttl, the longer lifetime applies; an unknown reported lifetime expires nothing.
  • Idle time: since the last stored message or the last model step, whichever is later.
  • The opening user message gets metadata.expires_thinking before the first request of the turn. The prompt builder omits thinking_blocks of assistant rows before the newest effective mark, so later requests extend the first one.
  • A mark is effective only if its message has text and the previous turn ended with a text answer. Otherwise the blocks leave at the next mark that qualifies.
  • Removed blocks never return, also after the option is turned off.
  • Session meta gains model_step_at and model_cache_ttl.
  • message_handlers now imports wippy.llm:llm (card lookup through llm.resolve_model).

@msmakouz

Copy link
Copy Markdown
Contributor Author

Closed by: #117

@msmakouz msmakouz closed this Nov 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant