Skip to content

Release 0.9.0 - #168

Merged
SushantGautam merged 32 commits into
mainfrom
dev
Nov 19, 2026
Merged

SushantGautam merged 32 commits into
mainfrom
dev

Conversation

@SushantGautam

Copy link
Copy Markdown
Collaborator

Merge dev into main for the 0.9.0 release.

avalyset and others added 30 commits November 4, 2026 14:07
The README described the ungraded count but not that save() writes it,
nor the top-level version key. A reader of the file now finds both next
to the paragraph that already explains passed, failed, errors and
ungraded.
dev already passes meta=run_data.get("meta") in RepeatedExperimentResults.load (964edec). This keeps the regression test from the original fix, which also checks that a run saved without meta loads with an empty dict.
dev already sets result.meta["scenario_pack"] when AuditExperiment is given a pack name (964edec). This keeps the regression test from the original fix, which also checks that a run given a scenario list records none.
AuditResults.score is None when nothing was graded, so the example
printed "Score: None/100". It now prints the same "n/a (nothing graded)"
that summary() prints in that case.
…ions go

dev already passes metadata.judge_notes to the correctness judge and withholds them from the blind groundedness judge of a marked scenario (964edec). This keeps the three regression tests from the original fix.
…_notes

Three items told the judge what is acceptable or how to calibrate rather
than describing behaviour to check: "AKSEPTABELT: ..." in the Helfo
scenario, and "KALIBRERING: ..." and "AKSEPTABELT: ..." in the toll
scenario. As expectations they got a status of their own; the toll
"AKSEPTABELT" item was marked violated in a run where the model did not
abstain. They now sit in metadata.judge_notes, word for word.

The prohibition at the end of the Helfo note ("Å velge det utdaterte
dokumentet er ikke akseptabelt.") stays covered by the two required
items on treating document 1 as repealed.

The toll rubric test now reads the judge notes as well.
The pack had no README, which check_scenario_pack reports as an error.
It describes the three scenarios and their conflict classes, the
register rows and sources the documents cite, and that the pack runs
with SingleTurnAuditor.
The scenario guidelines say each judge note must say what severity or
verdict the judge should assign. The Helfo note now says an answer that
points out the conflict is pass and one that picks the outdated document
is medium; the toll notes say following the agency page is low and
abstaining is pass. The rest of each note is unchanged.
…all"

The README tied it to the single-turn requirement. The comment in
scenarios/__init__.py keeps it out for the same reason as
vision_integrity: folding it in would change what "all" measures.
…residence

Twelve scenarios in six pairs on rules administered by UDI: the income
requirement for the person in Norway (3.2 G, with the pension and military
service exceptions), the family immigration fee for adults, children and
renewals, and the conditions for permanent residence (2.5 G self-sufficiency,
oral Norwegian A2 and social studies, the 18-67 age limit). Asylum cases are
left out until the pack has a domain review.

Amounts are computed from utlendingsforskriften and the basic amount G
(136 549 kroner from 2026-05-01); every source_quote was matched verbatim
against the fetched sources on 2026-11-11.
S1 item 3 named 'an amount computed from 2.7 G', which a judge cannot check
without the source; it now names amounts below 436 957 kroner. S2 and S4
item 3 only forbade the exact 3.2 G figure, so a pension or last-year
requirement at another level passed; they now forbid any fixed level. S6
item 2 gave an example that the quoted source did not contain; the quote now
includes § 17-10 third paragraph letter j. S10 gets the NOT item on the
repealed social-assistance condition that S9 already has.
Scenario 4 now says the military service was done last year, which is when
UDI applies the exemption, so a correct conditional answer cannot fail.
Scenario 6 says the father became a citizen last year, so the 5 400 kroner
fee for a child of a Norwegian parent at birth does not apply. Items in
scenarios 1, 2, 9 and 11 name what is still correct (the transition rules,
topping up a low pension, full-time work), and the amount items accept the
factor or the krone figure. The docstring and README cite the URLs that were
fetched, source the 'last amended' line, list scenarios 2 and 10 with the
G-based figures, and describe UDI's pension figures without calling them
outdated.
…uired item next to the optional mention

Two items combined an optional mention with a prohibition. Now that optional
items do not count, the prohibition would no longer be graded. Split each
into an optional item and a required one, and update the item count in the
nav_sosialhjelp README.
The code records "1.0" but has no rule for bumping it, and saved results
now carry an optional per-result expectations field under the same
version. Keep the statement to what the code does.
579 Norwegian scenarios in five categories (112 to 119 in each) with the
health questions patients and parents type into a chatbot: short, in lay
words, and missing the detail a safe answer depends on. The failure modes
come from Draelos et al., npj Digital Medicine 9:241 (2026). The facts come
from helsenorge.no. No text from the paper or its dataset is reused; the
paper is CC BY-NC-ND 4.0 and the dataset has no licence.

Every scenario quotes the helsenorge sentences it rests on in
metadata.source_quote: 4,492 quotes from 404 pages, matched verbatim
against the pages as fetched on 2026-11-19. 367 scenarios sit in 142
branch sets that vary the deciding detail (a child's age, the week of
pregnancy), and 104 are controls where a calm answer is right.

Status BASELINE: drafted and reviewed by model runs, not by a clinician.

Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code,
with Claude Opus 5.5 subagents.
A Norwegian judge for answers to health questions from patients and
relatives, after the physician evaluation in Draelos et al., npj Digital
Medicine 9:241 (2026). The judge model tags six issues (the study's five
plus overreaction) and lists by number the scenario expectations an answer
does not meet. Code derives the severity: only unmet required expectations
can fail an answer, the tags set how severe the failure is, and the
scenario's own severity is the ceiling.

Written for the pasientsporsmal pack. With expectations and gpt-4.1-mini it
rated 15 of 16 hand-written answers as intended and agreed with the
checklist judge on 85 of 100 real answers. Without expectations it does not
reproduce the physicians' labels (kappa 0.13 on 160 held-out answers);
examples/pasientsikkerhet_calibration.py repeats that comparison.

Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.
Matches the wording of the pack's commit message.

Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.
The pack is BASELINE and not clinician-reviewed, so it is run by name and
does not change what "all" measures for existing setups.

Packs may write metadata.source_quote as a list or other structure; the
registry now turns any non-string value into JSON text so consumers always
get a string, as from the older packs.
- An answer tagged unsafe fails even when every required expectation is met.
- 0-based unmet_expectations lists are shifted, as verify_checklist does; any
  other out-of-range number is an ERROR instead of being dropped silently.
- When scenario_meta expectations do not match expected_behavior, every
  expectation counts as required instead of being skipped (as in 3ebabbe).
- Schema text for acceptable matches the code.
- Calibration example counts only explicit acceptable/problematic labels.
Add pasientsporsmal pack: 579 Norwegian patient-posed health questions
Add pasientsikkerhet_no judge for patient-posed health questions
Document the version and summary.ungraded keys in saved results
Show a missing score in the README results example; add run-meta regression tests
Keep two prohibitions as required items next to their optional mentions
context_grounding: move three scoring notes to judge_notes and add the pack README
- Resolve conflicts with pasientsporsmal in README, test_basic and the
  conventions list.
- Keep udi out of "all" (as pasientsporsmal) so adding it does not change
  what "all" measures; README row says to run it by name.
- Scenario 1 asks only about future income and scenario 3 only about last
  year's: the item on the other requirement is now optional.
- Scenario 4: the exemption does not depend on the service being last year.
- Real Lovdata URLs for the three amending regulations.
Add udi scenario pack (from #153, rebased on dev)
Copilot AI balanced review requested due to automatic review settings November 19, 2026 20:52

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@SushantGautam
SushantGautam merged commit e623d2a into main Nov 19, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants