Skip to content

Calibrate judges against an answer key; tune the over-triage note - #173

Merged
kelkalot merged 1 commit into
devfrom
feat/answer-key-calibration
Nov 20, 2026
Merged

kelkalot merged 1 commit into
devfrom
feat/answer-key-calibration

Conversation

@kelkalot

Copy link
Copy Markdown
Collaborator

Uses the answer key of closed questions to check whether an LLM judge is right, with no labelling, and reports what that shows for three judges on the normedqa_triage pack. Follows #172.

What is in it

  • simpleaudit/answer_key.py (new), exported as planted_answers, replayed_answers and AnswerKeySet.
    • planted_answers(scenarios) writes one reply per option of every scenario that has a decision block with an accepted answer.
    • replayed_answers(run, scenarios) takes the replies of a stored single-turn run, which carry the target's own reasoning.
    • Both return a set with a target that plays the replies back, a reference for calibrate(), and verdicts_by_outcome(), which tabulates what the judge said about right, under-triage and over-triage replies.
    • The run goes through the ordinary audit path (Auditor(planted.target, judge=...)), so the judge sees the description, expectations and judge notes it sees in a real audit. rejudge() could not do this: it cannot reapply judge notes.
    • The labels are the verdicts of triage_match: the accepted option passes, a wrong one gets the severity its direction calls for.
    • reveal="key" shows the judge the right option and nothing about the wrong ones. reveal="none" shows no key, so the judge has to know the answer.
  • calibrate() already accepted a Rater; its docs now name the answer key as a third kind of reference next to human labels and a cross-family judge. No change in behaviour.
  • examples/answer_key_calibration.py: the check for a judge and judge model of your choice.
  • Pack README: a new section, "Judges on this pack", with the results below. A paragraph in the main README's calibration section.
  • One reworded judge note in normedqa_triage (see "A tuned note").
  • triage_match and choice_match record designed_severity_source (see "Severity source"). This is the fix pasientsikkerhet_no: record where the scenario's ceiling came from #170 makes for pasientsikkerhet_no, and it is not part of the calibration work.

What it shows (test sittings, 95 Bokmål questions, one run per row, 2026-11-20)

Shown the key, every judge fails every wrong answer. 380 planted replies: 96 right, 120 under-triage, 54 over-triage, 110 other wrong.

Judge Judge model Wrong answers failed Under-triage given the key's severity Over-triage rated low No usable verdict
default gpt-6-luna 284 of 284 120 of 120 54 of 54 0
default gpt-4.1-mini 284 of 284 120 of 120 45 of 54 0
default gpt-5.4-mini 284 of 284 120 of 120 40 of 54 0
checklist gpt-6-luna 284 of 284 112 of 120 23 of 54 0
checklist gpt-4.1-mini 281 of 281 115 of 120 23 of 53 5
pasientsikkerhet_no gpt-6-luna 284 of 284 120 of 120 54 of 54 0
pasientsikkerhet_no gpt-4.1-mini 284 of 284 111 of 120 34 of 54 0

checklist derives severity in code and does not use the notes, which is why it rates most over-triage answers medium.

Real replies. On the 380 replies the four baseline targets gave to the same questions (329 right, 51 wrong), the default judge passed 314 of the right ones on gpt-6-luna and 325 on gpt-4.1-mini, and failed all 51 wrong ones on both.

Without the key, the judge cannot be trusted. Default judge, key and notes hidden:

Replies Judge model Wrong answers failed Right answers passed
380 planted gpt-6-luna 278 of 284 69 of 96
380 planted gpt-4.1-mini 266 of 284 60 of 96
380 planted gpt-5.4-mini 233 of 284 92 of 96
95 real, gpt-4o-mini's gpt-6-luna 23 of 25 54 of 70
95 real, gpt-4o-mini's gpt-4.1-mini 9 of 25 67 of 70

gpt-4.1-mini passed 16 of gpt-4o-mini's 25 wrong answers, among them 6 of the 7 that give the patient less care than the key. With the key in the expectations it failed all 25.

Smaller findings are in the pack README: the key alone is enough for pass or fail; an added claim of authority did not get a wrong answer passed; a right answer that leaves out the option letter is failed two times in three; Nynorsk made no difference to the judge.

A tuned note

The pack's judge note for over-triage read "rate it low". The default judge followed it on gpt-6-luna (54 of 54) and mostly did not on gpt-4.1-mini (12 of 54) and gpt-5.4-mini (14 of 54). The note now says why over-triage is the lesser error and that the verdict is low although an expectation is not met. With it the counts are 54, 45 and 40.

The wording was chosen on the dev sitting (gpt-4.1-mini: 7 to 35 of 40) and the counts above are from the test sittings. In normedqa_triage.py this is 7 added lines and 1 removed, in the _scenario function; _ITEMS is untouched.

Severity source

triage_match and choice_match stored designed_severity in the judgment without saying where it came from. rejudge() reuses a stored ceiling only when the judgment names its source, so a run graded by either judge lost its ceiling when it was graded again by a judge model.

I checked it with a scenario of severity low, run with each judge and then re-graded with checklist. Before, the re-grade used the default ceiling medium and warned about it. Now it uses low, recorded as stored, with no warning.

Both judges write the source the way checklist, fact_check and the composed judges do: the caller's severity_source when there is one, else scenario, and default when the scenario has no severity. The stored judgment gains one key, so tests/snapshots/core/choice_match.json gains one line.

Limits

  • One run per row, and all judge models are OpenAI's.
  • A planted reply has one stock sentence as its justification. Judges that weigh the justification mark some right planted replies down, so the pass rate for right answers should be read from replayed answers. The module's docs say so.
  • The under and over labels the severities rest on are model-derived, as Add normedqa_triage pack, triage_match judge and NorMedQA loader #172 describes. The pass-or-fail numbers rest on the exam's key alone.
  • replayed_answers needs a single-turn run and leaves out replies whose option the key cannot read; it counts them in left_out.

Tests

  • pytest: 3,861 passed, 3 skipped (Python 3.12, all extras installed).
  • tests/test_answer_key.py: 15 tests with invented scenarios and fake judges. They cover the labels per option, the reply styles, what each reveal mode puts in the judge's prompt, scenarios without a decision id, a run through Auditor scored with calibrate, a failed judge call, and what replayed_answers keeps and leaves out.
  • python scripts/check_scenario_pack.py normedqa_triage: 0 ERROR, the same 2 WARN as in Add normedqa_triage pack, triage_match judge and NorMedQA loader #172.
  • One new test in each of tests/test_triage_match_judge.py and tests/test_choice_match_judge.py for the severity source: the three values it can take, and that rejudge()'s reader accepts the ceiling.
  • An independent review run found one real bug before this was opened (replies keyed by a decision id that is not unique when scenarios have none); it is fixed and tested.

Other open PRs

#169 and #170 touch none of these 14 files. #60 also edits simpleaudit/__init__.py and #140 also edits README.md, so whichever merges second may need a rebase.

Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code, with Claude Opus 5.5 subagents.

simpleaudit.answer_key turns scenarios with a decision block and an
accepted answer into a calibration set with known labels. planted_answers
writes one reply per option, replayed_answers takes the replies of a
stored single-turn run, and both come with a target that plays the replies
back through the ordinary audit path, so the judge sees descriptions,
expectations and judge notes as in a real audit. The labels are the
verdicts of triage_match, and calibrate() scores a judge against them.
reveal="key" and reveal="none" hide the directions or the whole key from
the judge, which measures what the judge model knows by itself.

On the test sittings of normedqa_triage every judge tried failed every
wrong answer when shown the key. With the key hidden, the default judge on
gpt-4.1-mini passed 16 of 25 wrong answers that gpt-4o-mini had argued
for. The pack README has the tables ("Judges on this pack").

The pack's judge note for over-triage is reworded. With "rate it low" the
default judge rated 12 of 54 over-triage answers low on gpt-4.1-mini and
14 on gpt-5.4-mini; with the new wording 45 and 40. The wording was chosen
on the dev sitting and the counts are from the test sittings.

triage_match and choice_match now record where the designed severity came
from (designed_severity_source), as the checklist judge does. Without it,
rejudge() with the checklist judge graded a stored run of either judge
against the default ceiling (medium), whatever the scenario's severity.

Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code,
with Claude Opus 5.5 subagents.
@kelkalot
kelkalot merged commit f449837 into dev Nov 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant