Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -586,6 +586,7 @@ SimpleAudit includes pre-built scenario packs:
| `nb_kryss_ordning` | 13 | National Library cross-scheme transfer: ISBN/ISSN/ISMN format rules, ISBN series thresholds, legal-deposit copy counts, unchanged-reprint consequences, jurisdiction. Six matched pairs — each outlier probe has a majority twin with character-identical wording |
| `healthbench_behaviours` | 17 | One scenario per HealthBench consensus category: emergency referrals (including over-referral), context seeking, hedging, communication with clinicians and laypeople, response depth, global health, health data tasks |
| `context_grounding` | 3 | Retrieval-context grounding: does the target use the supplied documents and ignore a planted superseded/low-authority chunk? **Requires `SingleTurnAuditor`** (one fixed question over a fixed document set), so it is **not** part of `all` |
| `pasientsporsmal` | 579 | Norwegian patient-posed health questions, 100+ in each of five categories, built on the failure modes in Draelos et al. (2026): missing warnings, accepting the user's own explanation, answers that depend on a detail the user left out, unsafe remedies and doses, the emergency step placed late, and controls where a calm answer is right. 142 branch sets vary the deciding detail (a child's age, the week of pregnancy). Facts quoted from helsenorge.no. Not clinician-reviewed yet, so **not** part of `all`: run it by name |
| `all` | 1424 | All scenarios combined |

</div>
Expand Down
24 changes: 24 additions & 0 deletions simpleaudit/scenarios/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -36,12 +36,16 @@
- context_grounding: Marked retrieval context — counterfactual, superseded and
lower-authority chunks (3 scenarios, requires SingleTurnAuditor; not part of 'all')
- healthbench_behaviours: One scenario per HealthBench consensus category (17 scenarios)
- pasientsporsmal: Norwegian patient-posed health questions in five categories; failure
modes from Draelos et al. 2026, facts from helsenorge.no (579 scenarios; not
clinician-reviewed yet, so not part of 'all')
- all: All scenarios combined

HealthBench is not a built-in pack: OpenAI asks that its examples are not reposted in
plain text, so load_healthbench_scenarios() downloads it and builds scenarios at run time.
"""

import json
import re
from collections import Counter
from datetime import date, datetime
Expand Down Expand Up @@ -77,9 +81,16 @@
from .nb_kryss_ordning import NB_KRYSS_ORDNING_SCENARIOS
from .context_grounding import CONTEXT_GROUNDING_SCENARIOS
from .healthbench_behaviours import HEALTHBENCH_BEHAVIOURS_SCENARIOS
from .pasientsporsmal import PASIENTSPORSMAL_SCENARIOS
from .healthbench_loader import load_healthbench_scenarios


def _to_text(value: Any) -> Any:
"""A ``source_quote`` as text: strings and None unchanged, anything else as JSON."""
return value if value is None or isinstance(value, str) else json.dumps(value, ensure_ascii=False)



SCENARIO_PACKS = {
"safety": SAFETY_SCENARIOS,
"rag": RAG_SCENARIOS,
Expand Down Expand Up @@ -115,6 +126,10 @@
"nb_kryss_ordning": NB_KRYSS_ORDNING_SCENARIOS,
"healthbench_behaviours": HEALTHBENCH_BEHAVIOURS_SCENARIOS,

# BASELINE pack, not clinician-reviewed yet. Kept out of "all" so that
# adding it does not change what "all" measures for existing setups.
"pasientsporsmal": PASIENTSPORSMAL_SCENARIOS,

# Scored under a fixed pack, and the scores only hold if the documents
# reach the target in the ranking the author gave them. The multi-turn
# loop regenerates the probe from turn 1, so this pack needs
Expand Down Expand Up @@ -142,6 +157,15 @@
}


# A pack may write metadata.source_quote as a list or other structure for
# readability; consumers always get a string, as from the older packs.
for _pack in SCENARIO_PACKS.values():
for _scenario in _pack:
_meta = _scenario.get("metadata")
if isinstance(_meta, dict) and "source_quote" in _meta:
_meta["source_quote"] = _to_text(_meta["source_quote"])


def get_scenarios(pack_name: str) -> List[Dict]:
"""
Get scenarios from a built-in pack.
Expand Down
Loading
Loading