Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 20 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -402,6 +402,24 @@ report = calibrate("examples/nav_aap/nav_aap_sonnet_4_6.json", reference,
report.summary()
```

**With an answer key.** A scenario with a [decision question](#decision-models) and an accepted answer already says what a reply deserves, so nobody has to label anything. `planted_answers` writes one reply per option, `Auditor` runs them past the judge the way it runs a model's replies, and `calibrate` scores the judge against the key:

```python
from simpleaudit import Auditor, calibrate, get_scenarios, planted_answers

planted = planted_answers(
get_scenarios("normedqa_triage"),
reason="Dette er etter min vurdering det riktige tiltaket for denne pasienten.",
)
auditor = Auditor(planted.target, judge="checklist", judge_model="gpt-4.1-mini", judge_provider="openai")
results = auditor.run(planted.scenarios, language="Norwegian")

calibrate(results, planted.reference).summary()
planted.verdicts_by_outcome(results) # the judge's verdicts on right, under-triage and over-triage replies
```

`replayed_answers(run, scenarios)` does the same with the replies a real target gave in a stored run, which carry the target's own reasoning. `reveal="none"` hides the key from the judge, which measures what the judge model knows by itself. On the test sittings of `normedqa_triage`, every judge we tried failed every wrong answer when it was shown the key. With the key hidden, the default judge on gpt-4.1-mini passed 16 of 25 wrong answers that gpt-4o-mini had argued for. [The pack README](simpleaudit/scenarios/normedqa_triage_README.md#judges-on-this-pack) has the tables, and [`examples/answer_key_calibration.py`](examples/answer_key_calibration.py) runs the check for a judge of your choice.

The results from `reframing_check` are also accepted as the subject. Each variant becomes a row, so one set of labels ranks several judge prompts or models through `report.leaderboard()`.

**What the numbers are.** Exact and within-one agreement with Wilson intervals. The signed bias, positive when the judge is stricter than the reference, with a bootstrap interval. Two chance-corrected coefficients from ten items, with bootstrap intervals and the reference distribution printed beside them: Gwet's AC2 with linear weights (`simpleaudit.stats.gwet_ac`) and ordinal Krippendorff's alpha (`simpleaudit.stats.krippendorff_alpha`, pinned to the published worked example in the tests). They are shown together because they fail differently. When most reference items pass, alpha reads low even at near-perfect agreement: 98 pass and 2 low, 97 matches, gives alpha 0.39 and AC2 0.99. `leaderboard()` ranks by AC2 for that reason, and alpha is the stricter figure when the levels are balanced. Sensitivity and specificity of the pass/fail split at `fail_at` (`low` by default, matching `AuditResults.failed`), the rate of severe misses (reference `high` or `critical`, judge `pass` or `low`), a confusion matrix, and the disagreeing scenarios. `ERROR` and `ungraded` verdicts are counted in coverage and never scored as agreement or disagreement.
Expand Down Expand Up @@ -742,7 +760,8 @@ Nynorsk: 260 and 208 escalation cases (the key is expedited care or faster), 198

The care labels come from three model passes and a model review, not from the exam board or a
clinician. [The pack's README](simpleaudit/scenarios/normedqa_triage_README.md) says how they were
made, how far the passes agree and what a first run on four models looks like.
made, how far the passes agree, what a first run on four models looks like, and how LLM judges
score against the key.

**`load_normedqa_scenarios`** gives the whole exam: 3,387 scored questions that need no image,
1,943 in Bokmål and 1,444 in Nynorsk. The file is 10 MB, so it is downloaded from a pinned commit
Expand Down
103 changes: 103 additions & 0 deletions examples/answer_key_calibration.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
#!/usr/bin/env python3
"""
Calibrate a judge against the answer key of the normedqa_triage pack.

The pack's questions come with the exam's key and a direction for every wrong option,
so the right verdict for any reply is known. This script puts every option in front of
a judge as if a model had answered with it, and prints how the judge's verdicts compare
with the key's: how many wrong answers it fails, how many right answers it passes, and
what it says about under-triage and over-triage answers.

Run it before you rely on a judge model for this pack, and above all before you rely on
one without the key (``--reveal none``). See the pack README, "Judges on this pack", for
what we found.

Usage
-----
export OPENAI_API_KEY=...
python examples/answer_key_calibration.py --judge-model gpt-4.1-mini
python examples/answer_key_calibration.py --judge checklist --judge-model gpt-4.1-mini
python examples/answer_key_calibration.py --judge-model gpt-4.1-mini --reveal none
python examples/answer_key_calibration.py --judge-model gpt-4.1-mini --replay my_run.json

--judge NAME a registered judge; omit it for the default judge.
--judge-model MODEL the judge model to calibrate. Required.
--provider NAME any-llm provider of the judge model. Default: openai.
--split dev|test|train exam sittings to use. Tune a prompt on dev (2025V, 43 Bokmål
questions), report on test (2025H and 2026V, 95). Default: dev.
--language nb|nn Bokmål or Nynorsk versions. Default: nb.
--reveal full|key|none how much of the key the judge is shown. "full" is the scenario
as it ships. "key" shows the right option only. "none" shows
nothing, so the judge has to know the answer. Default: full.
--style letter|text a planted reply opens with the option letter, or states the
option's text without it. Default: letter.
--replay FILE judge the replies of a stored run of the pack (any judge) in
place of planted ones. They carry the target's own reasoning.
--limit N use the first N questions.
--workers N judge calls in parallel. Default: 8.
--out FILE save the judged run.

A planted reply has no reasoning of its own, only a fixed sentence. A judge that weighs
the justification marks some right planted replies down for that, so read the pass rate
for right answers from a ``--replay`` run and the rest from planted ones.

Four replies per question, one judge call per reply. In our runs a call took about 900
input and 200 output tokens with the default judge, 1,200 and 400 with ``checklist``, and
2,400 and 400 with ``pasientsikkerhet_no``.
"""

import argparse

from simpleaudit import Auditor, calibrate, get_scenarios, planted_answers, replayed_answers

#: The planted replies' justification line. The pack's prompt asks for one, and a judge
#: can fail a bare letter for leaving it out.
REASON = {
"nb": "Dette er etter min vurdering det riktige tiltaket for denne pasienten.",
"nn": "Dette er etter mi vurdering det rette tiltaket for denne pasienten.",
}
SEVERITIES = ["pass", "low", "medium", "high", "critical", "ERROR"]


def main() -> None:
ap = argparse.ArgumentParser(description=__doc__.split("\n\n")[0])
ap.add_argument("--judge", default=None)
ap.add_argument("--judge-model", required=True)
ap.add_argument("--provider", default="openai")
ap.add_argument("--split", default="dev", choices=["dev", "test", "train"])
ap.add_argument("--language", default="nb", choices=["nb", "nn"])
ap.add_argument("--reveal", default="full", choices=["full", "key", "none"])
ap.add_argument("--style", default="letter", choices=["letter", "text"])
ap.add_argument("--replay", default=None)
ap.add_argument("--limit", type=int, default=None)
ap.add_argument("--workers", type=int, default=8)
ap.add_argument("--out", default=None)
args = ap.parse_args()

scenarios = [
s for s in get_scenarios("normedqa_triage")
if s["language"] == args.language and s["metadata"]["normedqa"]["split"] == args.split
][: args.limit]
if args.replay:
answers = replayed_answers(args.replay, scenarios, reveal=args.reveal)
else:
answers = planted_answers(scenarios, reveal=args.reveal, style=args.style, reason=REASON[args.language])

judge = {"judge": args.judge} if args.judge else {}
auditor = Auditor(answers.target, judge_model=args.judge_model, judge_provider=args.provider, max_turns=1, **judge)
results = auditor.run(answers.scenarios, language="Norwegian", max_workers=args.workers)
if args.out:
results.save(args.out)

calibrate(results, answers.reference, judge_model=args.judge_model).summary()

print("\nWhat the judge said, by what the key says about the reply:")
print(f" {'':<14}" + "".join(f"{s:>9}" for s in SEVERITIES))
table = answers.verdicts_by_outcome(results)
for outcome in ("correct", "under_triage", "over_triage", "other_wrong", "no_answer"):
if outcome in table:
print(f" {outcome:<14}" + "".join(f"{table[outcome].get(s, 0):>9}" for s in SEVERITIES))


if __name__ == "__main__":
main()
4 changes: 4 additions & 0 deletions simpleaudit/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@
from .scenarios import get_scenarios, list_scenario_packs, load_healthbench_scenarios, load_normedqa_scenarios
from .stats import DEFAULT_Z, two_proportion_z, wilson_interval
from .triage import TriageReport, triage_rates
from .answer_key import AnswerKeySet, planted_answers, replayed_answers
from .targets import (
ASRTarget,
CallableTarget,
Expand Down Expand Up @@ -131,6 +132,9 @@
"FRAGILE_THRESHOLD_DEFAULT",
"aggregate_severities",
"triage_rates",
"planted_answers",
"replayed_answers",
"AnswerKeySet",
"TriageReport",
"wilson_interval",
"two_proportion_z",
Expand Down
Loading
Loading