Skip to content

feat(bench): re-grade stored runs when a scenario's checks change - #621

Merged
Teakowa merged 2 commits into
mainfrom
fix/614-bench-regrade
Oct 11, 2026
Merged

Teakowa merged 2 commits into
mainfrom
fix/614-bench-regrade

Conversation

@e54-bot

@e54-bot e54-bot commented Oct 10, 2026

Copy link
Copy Markdown
Collaborator

Fixes #614 (SPEC-414 REQ-007).

Summary

A scenario's check change turns its grader hash over, but stored runs kept their old verdict — report --regrade only flagged unstable graders and never wrote grades back (and left regrade-1/regrade-2 scratch dirs inside each trial, 162 in one report).

  • New agent_bench.py regrade <dirs> --wright <binary> --out <dir>: copies every trial into <out>/<run-dir>/<trial path> — input directories are never modified — and re-grades only trials whose recorded grader.hash differs from the current scenario's.
  • A re-graded result.json keeps everything the agent produced (transcript, trace, usage, identity) and records the earlier hash as grader.regradedFrom. Trials at the current hash are copied without invoking the grader.
  • A stored invalid caused solely by grader unavailability clears when the new grade has every grader available; agent-side invalid reasons are preserved.
  • The report prints each re-graded scenario's hash transition and the count of changed usable verdicts.
  • report --regrade now grades into a TemporaryDirectory — no scratch directories inside the run directories on either path. regrade grades into <out>/<trial>/grading, matching run_trial's layout.
  • --out must be disjoint from every input directory and must be empty or absent.
  • docs/agent-benchmark-howto.md gained a "Re-grade runs after a scenario's checks change" section: when to re-grade and where the output goes.

Verification

  • python3 -m unittest test_agent_bench — 103 tests; 2 pre-existing environmental failures (test_scenarios_are_solvable_and_not_vacuous needs the unpinned oracle install; test_matrix_skips_... fails identically on the clean checkout), 3 oracle-skipped, all others pass including 3 new RegradeTest cases:
    • test_a_changed_check_regrades_the_copy_and_preserves_the_original — check change → new verdict + regradedFrom + byte-identical original.
    • test_a_current_hash_is_copied_without_grading — matching hash → copy, no grading/ directory.
    • test_report_regrade_leaves_no_scratch_inside_the_run_directory — regrade_notes leaves nothing behind.
  • End-to-end smoke: agent_bench.py regrade on a fabricated run with a stale hash re-graded the repair-runaway-loop seed (usable true→false: no-unbounded-loop + root-cause-named fail), recorded stale-hash-forced → d7eb57b9…, and left zero files in the input tree.
  • git diff --check clean.

Fixes #614

A changed scenario check turns the grader hash over, but stored runs kept their old verdict because nothing wrote new grades back. agent_bench.py regrade <dirs> --out <dir> copies every trial into a run-shaped output directory — inputs are never modified — and re-grades only the trials whose recorded grader hash differs, recording the earlier hash as grader.regradedFrom. Trials at the current hash are copied without grading. report --regrade also no longer leaves regrade-*/ scratch directories inside the run directories; both paths grade into scratch outside the archived runs.
Refs #614

Archives written before grader.hash existed carry no recorded hash, so regrade_trial set regradedFrom=None and regrade() then crashed building the transition summary (sorted() on (None, str) tuples, and old[:12] on None). Pre-hash trials now re-grade cleanly, keep regradedFrom: null in the copied result, and read <none> in the transition report; a regression test covers the grader-less archive path.

run_status and INFRA_EXIT move to bench_report so agent_bench and bench_regrade share one definition instead of a duplicated status_of with a hardcoded 75; the unused tempfile import goes with it.
@e54-bot

e54-bot commented Oct 11, 2026

Copy link
Copy Markdown
Collaborator Author

Review fixes in 652de1e:

  • Pre-hash archives no longer crash the transition summary — regradedFrom: null is preserved in the copied result and renders <none> in the report; new test test_a_pre_hash_archive_regrades_and_reports_the_transition covers the grader-less path.
  • Unused tempfile import removed; run_status + INFRA_EXIT moved to bench_report so agent_bench and bench_regrade share one definition instead of a duplicated status_of with a hardcoded 75.

Note: test_scenarios_are_solvable_and_not_vacuous and the matrix interruption test fail locally — every .opy scenario's upstream-valid check fails because the pinned OverPy oracle toolchain isn't installed in this environment (ana-paintball-ws, which needs no oracle, passes), and the matrix test hits local adapter limits (pi-m missing, provider outages). Both are environmental — the grading/status logic the diff touches is unchanged.

@Teakowa
Teakowa merged commit b77f80a into main Oct 11, 2026
17 checks passed
@Teakowa
Teakowa deleted the fix/614-bench-regrade branch October 11, 2026 03:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Development

Successfully merging this pull request may close these issues.

bench: re-grade stored runs when a scenario's checks change

2 participants