Repository navigation
Calibrate judges against an answer key; tune the over-triage note - #173
Merged
Merged
Conversation
simpleaudit.answer_key turns scenarios with a decision block and an
accepted answer into a calibration set with known labels. planted_answers
writes one reply per option, replayed_answers takes the replies of a
stored single-turn run, and both come with a target that plays the replies
back through the ordinary audit path, so the judge sees descriptions,
expectations and judge notes as in a real audit. The labels are the
verdicts of triage_match, and calibrate() scores a judge against them.
reveal="key" and reveal="none" hide the directions or the whole key from
the judge, which measures what the judge model knows by itself.
On the test sittings of normedqa_triage every judge tried failed every
wrong answer when shown the key. With the key hidden, the default judge on
gpt-4.1-mini passed 16 of 25 wrong answers that gpt-4o-mini had argued
for. The pack README has the tables ("Judges on this pack").
The pack's judge note for over-triage is reworded. With "rate it low" the
default judge rated 12 of 54 over-triage answers low on gpt-4.1-mini and
14 on gpt-5.4-mini; with the new wording 45 and 40. The wording was chosen
on the dev sitting and the counts are from the test sittings.
triage_match and choice_match now record where the designed severity came
from (designed_severity_source), as the checklist judge does. Without it,
rejudge() with the checklist judge graded a stored run of either judge
against the default ceiling (medium), whatever the scenario's severity.
Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code,
with Claude Opus 5.5 subagents.
This was referenced Nov 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uses the answer key of closed questions to check whether an LLM judge is right, with no labelling, and reports what that shows for three judges on the
normedqa_triagepack. Follows #172.What is in it
simpleaudit/answer_key.py(new), exported asplanted_answers,replayed_answersandAnswerKeySet.planted_answers(scenarios)writes one reply per option of every scenario that has adecisionblock with an accepted answer.replayed_answers(run, scenarios)takes the replies of a stored single-turn run, which carry the target's own reasoning.targetthat plays the replies back, areferenceforcalibrate(), andverdicts_by_outcome(), which tabulates what the judge said about right, under-triage and over-triage replies.Auditor(planted.target, judge=...)), so the judge sees the description, expectations and judge notes it sees in a real audit.rejudge()could not do this: it cannot reapply judge notes.triage_match: the accepted option passes, a wrong one gets the severity its direction calls for.reveal="key"shows the judge the right option and nothing about the wrong ones.reveal="none"shows no key, so the judge has to know the answer.calibrate()already accepted aRater; its docs now name the answer key as a third kind of reference next to human labels and a cross-family judge. No change in behaviour.examples/answer_key_calibration.py: the check for a judge and judge model of your choice.normedqa_triage(see "A tuned note").triage_matchandchoice_matchrecorddesigned_severity_source(see "Severity source"). This is the fix pasientsikkerhet_no: record where the scenario's ceiling came from #170 makes forpasientsikkerhet_no, and it is not part of the calibration work.What it shows (test sittings, 95 Bokmål questions, one run per row, 2026-11-20)
Shown the key, every judge fails every wrong answer. 380 planted replies: 96 right, 120 under-triage, 54 over-triage, 110 other wrong.
lowchecklistchecklistpasientsikkerhet_nopasientsikkerhet_nochecklistderives severity in code and does not use the notes, which is why it rates most over-triage answersmedium.Real replies. On the 380 replies the four baseline targets gave to the same questions (329 right, 51 wrong), the default judge passed 314 of the right ones on gpt-6-luna and 325 on gpt-4.1-mini, and failed all 51 wrong ones on both.
Without the key, the judge cannot be trusted. Default judge, key and notes hidden:
gpt-4.1-mini passed 16 of gpt-4o-mini's 25 wrong answers, among them 6 of the 7 that give the patient less care than the key. With the key in the expectations it failed all 25.
Smaller findings are in the pack README: the key alone is enough for pass or fail; an added claim of authority did not get a wrong answer passed; a right answer that leaves out the option letter is failed two times in three; Nynorsk made no difference to the judge.
A tuned note
The pack's judge note for over-triage read "rate it low". The default judge followed it on gpt-6-luna (54 of 54) and mostly did not on gpt-4.1-mini (12 of 54) and gpt-5.4-mini (14 of 54). The note now says why over-triage is the lesser error and that the verdict is
lowalthough an expectation is not met. With it the counts are 54, 45 and 40.The wording was chosen on the
devsitting (gpt-4.1-mini: 7 to 35 of 40) and the counts above are from thetestsittings. Innormedqa_triage.pythis is 7 added lines and 1 removed, in the_scenariofunction;_ITEMSis untouched.Severity source
triage_matchandchoice_matchstoreddesigned_severityin the judgment without saying where it came from.rejudge()reuses a stored ceiling only when the judgment names its source, so a run graded by either judge lost its ceiling when it was graded again by a judge model.I checked it with a scenario of severity
low, run with each judge and then re-graded withchecklist. Before, the re-grade used the default ceilingmediumand warned about it. Now it useslow, recorded asstored, with no warning.Both judges write the source the way
checklist,fact_checkand the composed judges do: the caller'sseverity_sourcewhen there is one, elsescenario, anddefaultwhen the scenario has no severity. The stored judgment gains one key, sotests/snapshots/core/choice_match.jsongains one line.Limits
replayed_answersneeds a single-turn run and leaves out replies whose option the key cannot read; it counts them inleft_out.Tests
pytest: 3,861 passed, 3 skipped (Python 3.12, all extras installed).tests/test_answer_key.py: 15 tests with invented scenarios and fake judges. They cover the labels per option, the reply styles, what eachrevealmode puts in the judge's prompt, scenarios without a decision id, a run throughAuditorscored withcalibrate, a failed judge call, and whatreplayed_answerskeeps and leaves out.python scripts/check_scenario_pack.py normedqa_triage: 0 ERROR, the same 2 WARN as in Add normedqa_triage pack, triage_match judge and NorMedQA loader #172.tests/test_triage_match_judge.pyandtests/test_choice_match_judge.pyfor the severity source: the three values it can take, and thatrejudge()'s reader accepts the ceiling.Other open PRs
#169 and #170 touch none of these 14 files. #60 also edits
simpleaudit/__init__.pyand #140 also editsREADME.md, so whichever merges second may need a rebase.Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code, with Claude Opus 5.5 subagents.