Skip to content

feat(captures): refuse to promote turns that claim tool calls they never made - #72

Merged
babaliauskas merged 15 commits into
mainfrom
feat/unbacked-tool-claims
Nov 1, 2026
Merged

babaliauskas merged 15 commits into
mainfrom
feat/unbacked-tool-claims

Conversation

@babaliauskas

Copy link
Copy Markdown
Collaborator

What this changes

Capture promotion now refuses a turn whose reply claims a tool call that never happened.

  • Detection (captures/claims.py): the check looks for a first-person claim, in English, that a tool action already happened ("I searched", "I've checked", "my notes search didn't"). Promises ("let me check", "I'll search once you confirm") and bare tool names don't trigger it. A turn with any other sign that it acted counts as backed and isn't flagged: a tool call, a tool result, a requested tool call, a retrieval event, or more than one model call.
  • capture promote exits 1 on such a turn and tells you to re-run with --allow-unbacked-claims.
  • capture sync skips such turns and counts them on their own line. If an earlier sync already promoted the case, the skip line names the case file to delete, unless the case is marked provenance: reviewed. When --repair refuses a case, it suggests --allow-unbacked-claims. A refused turn also drops out of later turns' reconstructed history.
  • Overrides: --allow-unbacked-claims / --block-unbacked-claims and captures.allow_unbacked_claims in evalshift.yaml. The flag wins, then the YAML key, and the default is to block. An allowed case is promoted with a warning and never asserts expected_no_tools. init scaffolds the key commented out.
  • History warning: promotion warns when a case's recorded history contains an assistant reply making such a claim with nothing in its turn to back it. Claims are judged per turn, and history stripped of tool rows isn't checked.
  • Docs: DOCS.md, llms-full.txt, docs/{agents,configuration,conversations}.md, plus a spec and a plan under docs/superpowers/.
  • README: links the getting-started video on YouTube at the top.

Why

A turn that was offered tools, called none, and still replied "my notes search didn't find anything" used to be promoted as ground truth. The made-up reply became expected, and the case asserted expected_no_tools: true, so the golden suite rewarded the hallucination.

Test plan

  • pytest passes: 3505 passed. The only local failure was test_online_sync_says_it_ignores_limits_it_cannot_read. That's a macOS temp-path wrapping artefact (Rich wraps the long /private/var/folders/... path at 80 columns), not caused by this branch. It passes with a short TMPDIR, and the pre-push make ci passed that way.
  • ruff check, ruff format --check clean
  • mypy --strict src/evalshift_cli clean
  • CHANGELOG.md updated under ## [Unreleased]
  • Docs updated

Upgrade note: older CLIs reject the captures.allow_unbacked_claims key, so bump any CI evalshift-version: pin before adding it to a config.

🤖 Generated with Claude Code

babaliauskas and others added 15 commits October 27, 2026 23:58
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ever made

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ed tool claim

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…g overrides

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Promises ("let me check", "I'll search your notes once you confirm") are
how a model asks a clarifying question before acting -- exactly the turn
where calling no tool is right -- so they no longer count as claims. A
reported result counts only as "my" search/lookup/query: "the query
didn't include a WHERE clause" is as often about the user's query.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A turn that recorded a tool result, a retrieval (a RAG answer did
search), or more than one model call (an agent loop: a second call only
follows something fed back, even in a pre-2.1.0 capture with no
requested_tool_calls) is no longer flagged as an unbacked tool claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A blank final_output event is no reply: the claim check (and history
reconstruction, which shares the rule) now falls back to the last
non-blank model-call output. With nothing to fall back to, the blank
final still counts as the reply, so reconstruction keeps its empty
assistant message as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A reply is backed when its turn -- from one user message to the next --
holds any tool row or assistant tool_calls, before or after it, so "I
searched your calendar." followed by the call no longer warns. A
recorded history with no tool rows at all (stripped by the app when it
resent history) is undecidable and yields no warning, instead of
flagging every past claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…iewed cases

`capture sync --repair` refusing a case whose rebuild is now an unbacked
tool claim names the flag that lets the repair run. The stale-case hint
for a newly blocked capture no longer asks to delete a case a human has
marked provenance: reviewed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Promises no longer count as claims, tool results / retrieval / agent
loops back a turn, history backing is per turn and skips tool-stripped
history, and the repair and reviewed-case behaviour of sync. The spec
gets an amended-after-review note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@babaliauskas
babaliauskas merged commit 035855d into main Nov 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant