Repository navigation
Add pasientsporsmal pack: 579 Norwegian patient-posed health questions - #164
Merged
Merged
Conversation
579 Norwegian scenarios in five categories (112 to 119 in each) with the health questions patients and parents type into a chatbot: short, in lay words, and missing the detail a safe answer depends on. The failure modes come from Draelos et al., npj Digital Medicine 9:241 (2026). The facts come from helsenorge.no. No text from the paper or its dataset is reused; the paper is CC BY-NC-ND 4.0 and the dataset has no licence. Every scenario quotes the helsenorge sentences it rests on in metadata.source_quote: 4,492 quotes from 404 pages, matched verbatim against the pages as fetched on 2026-11-19. 367 scenarios sit in 142 branch sets that vary the deciding detail (a child's age, the week of pregnancy), and 104 are controls where a calm answer is right. Status BASELINE: drafted and reviewed by model runs, not by a clinician. Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code, with Claude Opus 5.5 subagents.
Matches the wording of the pack's commit message. Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code.
Collaborator
|
Great work. Checker and tests pass here. Before merge:
|
The pack is BASELINE and not clinician-reviewed, so it is run by name and does not change what "all" measures for existing setups. Packs may write metadata.source_quote as a list or other structure; the registry now turns any non-string value into JSON text so consumers always get a string, as from the older packs.
SushantGautam
added a commit
that referenced
this pull request
Nov 19, 2026
Add pasientsporsmal pack: 579 Norwegian patient-posed health questions
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
pasientsporsmal: 579 Norwegian scenarios with the health questions patients and parents type into a chatbot. They are short, in everyday words, and leave out the detail a safe answer depends on.Where it comes from
expected_behaviorlist rests on a sentence quoted inmetadata.source_quote.The study has 222 questions and the pack has 579 scenarios, so the study supplies topics for a minority. By each scenario's
source.inspiration: 167 rest on a study question where the physicians found the failure, 11 on a study question that drew no such finding, 297 on topics the study has no question on, and 104 are controls. The README says this in the same words.What is in it
pair_type: "branch_set") that ask one situation several times with the deciding detail changed: a baby of six weeks, eight months or no stated age; week 8 or week 33.How it was built and checked
The most common review changes: a detail made optional; a required route (113, legevakt) removed where it rested only on a page's general contact box; a
source.inspirationcorrected where it claimed a study finding the study does not have. The review of my own seed found a prompt that was nearly a translation of a study question, and it was rewritten.No clinician has reviewed the pack. Status is BASELINE.
Tests
python scripts/check_scenario_pack.py pasientsporsmal: 0 ERROR, 0 WARN.pytest: 2,608 passed, 2 skipped (Python 3.12, all extras installed).CONFORMING_PACKS.Baseline (one run, 2026-11-19, OpenAI API,
max_turns=1, judge model gpt-4.1-mini)checklistjudge 20; on gpt-4.1's 58 answers, 48 and 33. Both put gpt-4.1 well ahead. Neither was checked against a clinician.Result files are not committed.
Things a reviewer should know
ung.pyis 1.6 MB). It is generated from reviewed fragments by a script that is not in this PR; scenarios can be edited in place.allgrows from 1,424 to 2,003. That changes aggregate scores fromauditor.run("all")and what such a run costs. Say so if you would rather keep the pack out ofall.udi) touches the same registration files (scenarios/__init__.py, the README table, the twoallsum tests,CONFORMING_PACKS). Whichever merges second needs a rebase and a newallcount.pasientsikkerhet_no, is in Add pasientsikkerhet_no judge for patient-posed health questions #165. Neither PR depends on the other.Built with support from Claude Fable 5.2 (claude-fable-5-2) in Claude Code, with Claude Opus 5.5 subagents.