Conversation
Demonstrates Anthropic prompt caching as a simple illustration of the KV cache reuse mechanism described in Ramp's Latent Briefing paper. Uses AnthropicBedrock client with cache_control to show how shared context is computed once and reused across multiple queries. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
No summary was generated for this pull request. All finding details can be found in the DryRun Security Dashboard. |
There was a problem hiding this comment.
Pull request overview
Adds a new “exercise-25” under scripts/extras/ to demonstrate Anthropic prompt caching (as a practical KV-cache reuse example aligned with Ramp’s Latent Briefing concept) using the AnthropicBedrock client, and documents how to run/interpret the demo.
Changes:
- Added a runnable Python demo that caches a large “security policy” system block and runs 3 queries while printing cache creation vs cache read usage metrics.
- Added exercise instructions explaining the concept mapping to Latent Briefing and how to observe caching metrics.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
| scripts/extras/exercise-25/kv_cache_demo.py | Implements the KV-cache/prompt-caching demo using AnthropicBedrock and prints cache usage metrics across multiple calls. |
| scripts/extras/exercise-25/INSTRUCTIONS.md | Documents the objective, background, and how to run the demo and interpret the caching metrics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+18
to
+20
| client = AnthropicBedrock( | ||
| aws_region=os.environ.get("AWS_DEFAULT_REGION", "us-east-1"), | ||
| ) |
Comment on lines
+41
to
+44
| - Remove the `cache_control` block and compare — all calls pay full input token cost | ||
| - Add more queries to observe continued cache hits within the 5-minute TTL | ||
| - Increase the policy size and observe how cache savings scale | ||
| - Try setting cache TTL to longer durations for persistent caching |
Prompt caching does not work with us.anthropic.* inference profiles since requests route across regions. Updated script output and INSTRUCTIONS.md to explain this and note alternatives. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.
Suppressed comments (2)
scripts/extras/exercise-25/kv_cache_demo.py:184
- These TAKEAWAY lines currently imply that calls 2 and 3 will show cache-read metrics, but with the default
us.anthropic.*cross-region model ID this will not happen (cache metrics remain 0). Align the messaging to be conditional on using a caching-enabled model/region so the output matches what users will actually observe by default.
print("TAKEAWAY: When prompt caching is active, calls 2 and 3 show")
print("cache_read tokens instead of cache_creation tokens. The model")
print("reuses pre-computed KV representations — the same mechanism")
print("Ramp's Latent Briefing paper uses to share context between")
print("orchestrator and workers.")
scripts/extras/exercise-25/kv_cache_demo.py:9
- The docstring states that the policy is cached on the first call and reused on subsequent calls, but with the default cross-region model ID in this script prompt caching is not supported (as noted at the end of the script). This makes the docstring misleading; please phrase this as conditional on using a caching-enabled model/region.
This issue also appears on line 180 of the same file.
The large security policy is cached on the first call. Subsequent calls reuse
the cached KV representations, avoiding redundant computation.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
scripts/extras/exercise-25/demonstrating Anthropic prompt caching as a simple illustration of KV cache reuse from Ramp's Latent Briefing paperAnthropicBedrockclient withcache_controlon a large security policy, then sends 3 different queries showing cache creation vs cache read metricsanthropicSDK (via Bedrock wrapper)Test plan
🤖 Generated with Claude Code