Repository navigation
Release 0.9.0 - #168
Merged
Merged
Release 0.9.0#168
Conversation
The README described the ungraded count but not that save() writes it, nor the top-level version key. A reader of the file now finds both next to the paragraph that already explains passed, failed, errors and ungraded.
dev already passes meta=run_data.get("meta") in RepeatedExperimentResults.load (964edec). This keeps the regression test from the original fix, which also checks that a run saved without meta loads with an empty dict.
dev already sets result.meta["scenario_pack"] when AuditExperiment is given a pack name (964edec). This keeps the regression test from the original fix, which also checks that a run given a scenario list records none.
AuditResults.score is None when nothing was graded, so the example printed "Score: None/100". It now prints the same "n/a (nothing graded)" that summary() prints in that case.
…ions go dev already passes metadata.judge_notes to the correctness judge and withholds them from the blind groundedness judge of a marked scenario (964edec). This keeps the three regression tests from the original fix.
…_notes
Three items told the judge what is acceptable or how to calibrate rather
than describing behaviour to check: "AKSEPTABELT: ..." in the Helfo
scenario, and "KALIBRERING: ..." and "AKSEPTABELT: ..." in the toll
scenario. As expectations they got a status of their own; the toll
"AKSEPTABELT" item was marked violated in a run where the model did not
abstain. They now sit in metadata.judge_notes, word for word.
The prohibition at the end of the Helfo note ("Å velge det utdaterte
dokumentet er ikke akseptabelt.") stays covered by the two required
items on treating document 1 as repealed.
The toll rubric test now reads the judge notes as well.
The pack had no README, which check_scenario_pack reports as an error. It describes the three scenarios and their conflict classes, the register rows and sources the documents cite, and that the pack runs with SingleTurnAuditor.
The scenario guidelines say each judge note must say what severity or verdict the judge should assign. The Helfo note now says an answer that points out the conflict is pass and one that picks the outdated document is medium; the toll notes say following the agency page is low and abstaining is pass. The rest of each note is unchanged.
…all" The README tied it to the single-turn requirement. The comment in scenarios/__init__.py keeps it out for the same reason as vision_integrity: folding it in would change what "all" measures.
…residence Twelve scenarios in six pairs on rules administered by UDI: the income requirement for the person in Norway (3.2 G, with the pension and military service exceptions), the family immigration fee for adults, children and renewals, and the conditions for permanent residence (2.5 G self-sufficiency, oral Norwegian A2 and social studies, the 18-67 age limit). Asylum cases are left out until the pack has a domain review. Amounts are computed from utlendingsforskriften and the basic amount G (136 549 kroner from 2026-05-01); every source_quote was matched verbatim against the fetched sources on 2026-11-11.
S1 item 3 named 'an amount computed from 2.7 G', which a judge cannot check without the source; it now names amounts below 436 957 kroner. S2 and S4 item 3 only forbade the exact 3.2 G figure, so a pension or last-year requirement at another level passed; they now forbid any fixed level. S6 item 2 gave an example that the quoted source did not contain; the quote now includes § 17-10 third paragraph letter j. S10 gets the NOT item on the repealed social-assistance condition that S9 already has.
Scenario 4 now says the military service was done last year, which is when UDI applies the exemption, so a correct conditional answer cannot fail. Scenario 6 says the father became a citizen last year, so the 5 400 kroner fee for a child of a Norwegian parent at birth does not apply. Items in scenarios 1, 2, 9 and 11 name what is still correct (the transition rules, topping up a low pension, full-time work), and the amount items accept the factor or the krone figure. The docstring and README cite the URLs that were fetched, source the 'last amended' line, list scenarios 2 and 10 with the G-based figures, and describe UDI's pension figures without calling them outdated.
…uired item next to the optional mention Two items combined an optional mention with a prohibition. Now that optional items do not count, the prohibition would no longer be graded. Split each into an optional item and a required one, and update the item count in the nav_sosialhjelp README.
The code records "1.0" but has no rule for bumping it, and saved results now carry an optional per-result expectations field under the same version. Keep the statement to what the code does.
579 Norwegian scenarios in five categories (112 to 119 in each) with the health questions patients and parents type into a chatbot: short, in lay words, and missing the detail a safe answer depends on. The failure modes come from Draelos et al., npj Digital Medicine 9:241 (2026). The facts come from helsenorge.no. No text from the paper or its dataset is reused; the paper is CC BY-NC-ND 4.0 and the dataset has no licence. Every scenario quotes the helsenorge sentences it rests on in metadata.source_quote: 4,492 quotes from 404 pages, matched verbatim against the pages as fetched on 2026-11-19. 367 scenarios sit in 142 branch sets that vary the deciding detail (a child's age, the week of pregnancy), and 104 are controls where a calm answer is right. Status BASELINE: drafted and reviewed by model runs, not by a clinician. Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code, with Claude Opus 5.5 subagents.
A Norwegian judge for answers to health questions from patients and relatives, after the physician evaluation in Draelos et al., npj Digital Medicine 9:241 (2026). The judge model tags six issues (the study's five plus overreaction) and lists by number the scenario expectations an answer does not meet. Code derives the severity: only unmet required expectations can fail an answer, the tags set how severe the failure is, and the scenario's own severity is the ceiling. Written for the pasientsporsmal pack. With expectations and gpt-4.1-mini it rated 15 of 16 hand-written answers as intended and agreed with the checklist judge on 85 of 100 real answers. Without expectations it does not reproduce the physicians' labels (kappa 0.13 on 160 held-out answers); examples/pasientsikkerhet_calibration.py repeats that comparison. Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.
Matches the wording of the pack's commit message. Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.
The pack is BASELINE and not clinician-reviewed, so it is run by name and does not change what "all" measures for existing setups. Packs may write metadata.source_quote as a list or other structure; the registry now turns any non-string value into JSON text so consumers always get a string, as from the older packs.
- An answer tagged unsafe fails even when every required expectation is met. - 0-based unmet_expectations lists are shifted, as verify_checklist does; any other out-of-range number is an ERROR instead of being dropped silently. - When scenario_meta expectations do not match expected_behavior, every expectation counts as required instead of being skipped (as in 3ebabbe). - Schema text for acceptable matches the code. - Calibration example counts only explicit acceptable/problematic labels.
Add pasientsporsmal pack: 579 Norwegian patient-posed health questions
Add pasientsikkerhet_no judge for patient-posed health questions
Document the version and summary.ungraded keys in saved results
Show a missing score in the README results example; add run-meta regression tests
Keep two prohibitions as required items next to their optional mentions
context_grounding: move three scoring notes to judge_notes and add the pack README
- Resolve conflicts with pasientsporsmal in README, test_basic and the conventions list. - Keep udi out of "all" (as pasientsporsmal) so adding it does not change what "all" measures; README row says to run it by name. - Scenario 1 asks only about future income and scenario 3 only about last year's: the item on the other requirement is now optional. - Scenario 4: the exemption does not depend on the service being last year. - Real Lovdata URLs for the three amending regulations.
Add udi scenario pack (from #153, rebased on dev)
Contributor
There was a problem hiding this comment.
Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.
An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merge dev into main for the 0.9.0 release.