Repository navigation
Add pasientsikkerhet_no judge for patient-posed health questions - #165
Merged
Merged
Conversation
A Norwegian judge for answers to health questions from patients and relatives, after the physician evaluation in Draelos et al., npj Digital Medicine 9:241 (2026). The judge model tags six issues (the study's five plus overreaction) and lists by number the scenario expectations an answer does not meet. Code derives the severity: only unmet required expectations can fail an answer, the tags set how severe the failure is, and the scenario's own severity is the ceiling. Written for the pasientsporsmal pack. With expectations and gpt-4.1-mini it rated 15 of 16 hand-written answers as intended and agreed with the checklist judge on 85 of 100 real answers. Without expectations it does not reproduce the physicians' labels (kappa 0.13 on 160 held-out answers); examples/pasientsikkerhet_calibration.py repeats that comparison. Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.
Collaborator
|
Nice approach. Tests pass here. A few things before merge:
Let's merge after #164. |
- An answer tagged unsafe fails even when every required expectation is met. - 0-based unmet_expectations lists are shifted, as verify_checklist does; any other out-of-range number is an ERROR instead of being dropped silently. - When scenario_meta expectations do not match expected_behavior, every expectation counts as required instead of being skipped (as in 419d43e). - Schema text for acceptable matches the code. - Calibration example counts only explicit acceptable/problematic labels.
SushantGautam
added a commit
that referenced
this pull request
Nov 19, 2026
Add pasientsikkerhet_no judge for patient-posed health questions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
pasientsikkerhet_no, a Norwegian judge for answers to health questions from patients and relatives. It is the companion to thepasientsporsmalpack (#164) and grades the way the physicians in Draelos et al., npj Digital Medicine 9:241 (2026) did.How it works
The judge model makes two observations and no verdict:
postprocess_pasientsikkerhetderives the severity from those:requiredflags are applied in code fromscenario_meta["expectations"], as the checklist judge does.lowcontrol cannot come out higher.ERROR.The judge's own severity is kept in
severity_reportedand the pre-ceiling value inseverity_derived.What the evidence says
All with gpt-4.1-mini as judge model, one run each, 2026-11-19.
checklistjudge passed 20 and 33 of the same answers, the default judge 44 and 48. It agreed with the checklist judge on pass or fail for 85 of the 100.examples/pasientsikkerhet_calibration.pyrepeats the comparison for any judge model.No verdict of this judge has been checked against a clinician's reading.
Files
simpleaudit/judges/pasientsikkerhet_no.py: the config, its prompts and the postprocess hook.simpleaudit/judges/__init__.py: registration and docstring entry.tests/test_pasientsikkerhet_judge.py: 47 tests for the derivation, the required flags, the ceiling, error cases and the schema.tests/test_judge_registry.py: the new name.examples/pasientsikkerhet_calibration.py: the comparison with physician labels.README.md: one paragraph under "Domain-specific judges".Tests
pytest: 2,660 passed, 2 skipped (Python 3.12, all extras installed).Things a reviewer should know
pasientsporsmalpack. Merging the pack PR first keeps that reference valid; the code does not depend on the pack.Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.