[babaliauskas]

Why

--rounds was a flag only, so teacher-forced replay had to be passed on every capture sync. Under the default (first), an agent that answers after a tool call is scored on the call alone, and semantic and llm_judge never see its answer. On a real converter suite they scored 2 of 17 rows: the 15 turns that called convert_currency before answering had no text to score. A durable setting belongs in evalshift.yaml.

What changes

• captures.rounds: first | all (default first) in evalshift.yaml sets the rounds capture sync promotes with.
  • capture sync --rounds still overrides it for one run.
  • capture promote reads no config and takes only the flag, as before.
  • An unreadable config falls back to first, the same way sync's other captures: settings fall back.
• It never rewrites existing cases. The setting applies to cases promoted from then on, because sync never rewrites a promoted case unasked.
• When the setting is all, sync counts cases it just declined to rewrite.
  • A case counts only if it is named after its capture (the case --force would rewrite) and still replays one round although its capture now builds a teacher-forced case.
  • Reviewed cases are left out, because --force would drop the mark. Stale cases are left out, because the planner may retire them in the same run. Cases promoted under another name (capture promote --as) and repair candidates are never reached, so they aren't counted either.
  • It names the command that would upgrade them, with --rounds all and --suite <S> added when the run used them, and says what --force costs: ⚠ 15 promoted case(s) replay only their first round, although their captures carry the recorded tool results to replay every round. Sync never rewrites a promoted case unasked: evalshift capture sync --force once re-promotes each case whose capture is still local, which drops hand edits and provenance: reviewed marks (reviewed cases are not counted here).
• An unreadable config now also says sync is running without captures.rounds (replaying the first round only).
• init scaffolds # rounds: all commented out, with a one-line explanation.
• Docs: docs/configuration.md (the captures table plus a note on existing cases and CI pins), docs/agents.md, DOCS.md, llms-full.txt and the CHANGELOG.

Compatibility: an older CLI rejects the new key (extra="forbid"), so a CI evalshift-version: pin has to be updated before the key is added. The docs and CHANGELOG say so. The hosted bundle doesn't carry the captures config.

Test plan

• New tests, each failing first:
  • the config accepts all, defaults to first and rejects anything else;
  • sync with no flag takes rounds: all from the config, giving a case with tool_result_fixtures;
  • --rounds first overrides rounds: all;
  • after a first-round sync, switching the config to all makes sync print the hint and leave the case alone, sync --force re-promotes it with fixtures, and the next sync is quiet;
  • the hint in a bounded suite;
  • the hint naming --rounds all when the flag asked for it, and --suite when one was given;
  • no hint for a reviewed case or a promote --as case;
  • sync --rounds some exits 2;
  • the unreadable-config warning naming captures.rounds.
• An independent review (approve with fixes). Its probes for every scenario above now pass, and the bounded-path probe shows the hint, then none after --force.
• ruff check, ruff format --check and mypy --strict are clean. pytest: 3303 passed, 1 skipped (the existing evalshift-server skip), above the 90% coverage gate. pre-commit run --all-files passes.

🤖 Generated with Claude Code