Repository navigation
feat(bench): re-grade stored runs when a scenario's checks change - #621
Merged
Merged
Conversation
Fixes #614 A changed scenario check turns the grader hash over, but stored runs kept their old verdict because nothing wrote new grades back. agent_bench.py regrade <dirs> --out <dir> copies every trial into a run-shaped output directory — inputs are never modified — and re-grades only the trials whose recorded grader hash differs, recording the earlier hash as grader.regradedFrom. Trials at the current hash are copied without grading. report --regrade also no longer leaves regrade-*/ scratch directories inside the run directories; both paths grade into scratch outside the archived runs.
Refs #614 Archives written before grader.hash existed carry no recorded hash, so regrade_trial set regradedFrom=None and regrade() then crashed building the transition summary (sorted() on (None, str) tuples, and old[:12] on None). Pre-hash trials now re-grade cleanly, keep regradedFrom: null in the copied result, and read <none> in the transition report; a regression test covers the grader-less archive path. run_status and INFRA_EXIT move to bench_report so agent_bench and bench_regrade share one definition instead of a duplicated status_of with a hardcoded 75; the unused tempfile import goes with it.
Collaborator
Author
|
Review fixes in
Note: |
This was referenced Oct 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #614 (SPEC-414 REQ-007).
Summary
A scenario's check change turns its grader hash over, but stored runs kept their old verdict —
report --regradeonly flagged unstable graders and never wrote grades back (and leftregrade-1/regrade-2scratch dirs inside each trial, 162 in one report).agent_bench.py regrade <dirs> --wright <binary> --out <dir>: copies every trial into<out>/<run-dir>/<trial path>— input directories are never modified — and re-grades only trials whose recordedgrader.hashdiffers from the current scenario's.result.jsonkeeps everything the agent produced (transcript, trace, usage, identity) and records the earlier hash asgrader.regradedFrom. Trials at the current hash are copied without invoking the grader.invalidcaused solely by grader unavailability clears when the new grade has every grader available; agent-side invalid reasons are preserved.usableverdicts.report --regradenow grades into aTemporaryDirectory— no scratch directories inside the run directories on either path.regradegrades into<out>/<trial>/grading, matchingrun_trial's layout.--outmust be disjoint from every input directory and must be empty or absent.docs/agent-benchmark-howto.mdgained a "Re-grade runs after a scenario's checks change" section: when to re-grade and where the output goes.Verification
python3 -m unittest test_agent_bench— 103 tests; 2 pre-existing environmental failures (test_scenarios_are_solvable_and_not_vacuousneeds the unpinned oracle install;test_matrix_skips_...fails identically on the clean checkout), 3 oracle-skipped, all others pass including 3 newRegradeTestcases:test_a_changed_check_regrades_the_copy_and_preserves_the_original— check change → new verdict +regradedFrom+ byte-identical original.test_a_current_hash_is_copied_without_grading— matching hash → copy, nograding/directory.test_report_regrade_leaves_no_scratch_inside_the_run_directory—regrade_notesleaves nothing behind.agent_bench.py regradeon a fabricated run with a stale hash re-graded therepair-runaway-loopseed (usable true→false:no-unbounded-loop+root-cause-namedfail), recordedstale-hash-forced → d7eb57b9…, and left zero files in the input tree.git diff --checkclean.