Repository navigation
feat(captures): refuse to promote turns that claim tool calls they never made - #72
Merged
Merged
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ever made Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed tool claim Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g overrides Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Promises ("let me check", "I'll search your notes once you confirm") are
how a model asks a clarifying question before acting -- exactly the turn
where calling no tool is right -- so they no longer count as claims. A
reported result counts only as "my" search/lookup/query: "the query
didn't include a WHERE clause" is as often about the user's query.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A turn that recorded a tool result, a retrieval (a RAG answer did search), or more than one model call (an agent loop: a second call only follows something fed back, even in a pre-2.1.0 capture with no requested_tool_calls) is no longer flagged as an unbacked tool claim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A blank final_output event is no reply: the claim check (and history reconstruction, which shares the rule) now falls back to the last non-blank model-call output. With nothing to fall back to, the blank final still counts as the reply, so reconstruction keeps its empty assistant message as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reply is backed when its turn -- from one user message to the next -- holds any tool row or assistant tool_calls, before or after it, so "I searched your calendar." followed by the call no longer warns. A recorded history with no tool rows at all (stripped by the app when it resent history) is undecidable and yields no warning, instead of flagging every past claim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iewed cases `capture sync --repair` refusing a case whose rebuild is now an unbacked tool claim names the flag that lets the repair run. The stale-case hint for a newly blocked capture no longer asks to delete a case a human has marked provenance: reviewed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Promises no longer count as claims, tool results / retrieval / agent loops back a turn, history backing is per turn and skips tool-stripped history, and the repair and reviewed-case behaviour of sync. The spec gets an amended-after-review note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
Capture promotion now refuses a turn whose reply claims a tool call that never happened.
captures/claims.py): the check looks for a first-person claim, in English, that a tool action already happened ("I searched", "I've checked", "my notes search didn't"). Promises ("let me check", "I'll search once you confirm") and bare tool names don't trigger it. A turn with any other sign that it acted counts as backed and isn't flagged: a tool call, a tool result, a requested tool call, a retrieval event, or more than one model call.capture promoteexits 1 on such a turn and tells you to re-run with--allow-unbacked-claims.capture syncskips such turns and counts them on their own line. If an earlier sync already promoted the case, the skip line names the case file to delete, unless the case is markedprovenance: reviewed. When--repairrefuses a case, it suggests--allow-unbacked-claims. A refused turn also drops out of later turns' reconstructed history.--allow-unbacked-claims/--block-unbacked-claimsandcaptures.allow_unbacked_claimsinevalshift.yaml. The flag wins, then the YAML key, and the default is to block. An allowed case is promoted with a warning and never assertsexpected_no_tools.initscaffolds the key commented out.DOCS.md,llms-full.txt,docs/{agents,configuration,conversations}.md, plus a spec and a plan underdocs/superpowers/.Why
A turn that was offered tools, called none, and still replied "my notes search didn't find anything" used to be promoted as ground truth. The made-up reply became
expected, and the case assertedexpected_no_tools: true, so the golden suite rewarded the hallucination.Test plan
pytestpasses: 3505 passed. The only local failure wastest_online_sync_says_it_ignores_limits_it_cannot_read. That's a macOS temp-path wrapping artefact (Rich wraps the long/private/var/folders/...path at 80 columns), not caused by this branch. It passes with a shortTMPDIR, and the pre-pushmake cipassed that way.ruff check,ruff format --checkcleanmypy --strict src/evalshift_cliclean## [Unreleased]Upgrade note: older CLIs reject the
captures.allow_unbacked_claimskey, so bump any CIevalshift-version:pin before adding it to a config.🤖 Generated with Claude Code