Skip to content

Add udi scenario pack: family immigration income and fees, permanent residence - #153

Closed
avalyset wants to merge 537 commits into
SimulaMet:devfrom
avalyset:feat/udi-pack
Closed

avalyset wants to merge 537 commits into
SimulaMet:devfrom
avalyset:feat/udi-pack

Conversation

@avalyset

Copy link
Copy Markdown
Contributor

Adds udi, twelve scenarios in six pairs on rules administered by Utlendingsdirektoratet (UDI):

  • Income requirement in family immigration: future income of 3.2 G for the person in Norway
    (436 957 kroner a year with G from 1 May 2026), the separate requirement on last year's income, and
    two exceptions (old-age pension at the full minimum pension level, and military service done last year).
  • Application fee: an adult's first application (11 900 kroner), a child's (half, 5 950 kroner, since
    1 January 2026), and a renewal before (4 400 kroner) and after (11 900 kroner) the permit expires.
  • Permanent residence: self-sufficiency over the last twelve months (2.5 G, 341 373 kroner), the oral
    Norwegian test at A2 and the social studies test, and the 18–67 age limit that applies to all three.

Asylum, family reunification with a refugee and other vulnerable groups are left out until the pack has had a
domain review.

These rules changed recently (3.2 G for future income from February 2025 and for last year's income from
February 2026, a fee for children from January 2026, A2 instead of A1 from September 2025), so outdated
answers are the main failure the pack probes. Five pairs are matched (main rule and exception). The
renewal-fee pair is a branch_set, since neither before nor after expiry is the dominant rule.

Sources

Every source_quote excerpt is matched verbatim against sources fetched raw on 2026-11-11:
utlendingsforskriften §§ 10-8, 10-9, 11-9, 11-10, 11-11 and 17-10, utlendingsloven § 62, the amending
regulations that set the current fees, nav.no for G, and four UDI pages and three UDI notices. The
amounts are computed from the regulation and G. The check ran in two passes, once on the stored text and
once on a fresh extraction from the raw HTML: 76 of 76 excerpts found.

One thing a reviewer may want to know: an amendment to § 17-10 enters into force on 15 March 2027
(FOR-2024-02-14-257 as amended by FOR-2024-03-13-433). It repeals only the au pair fee, which no scenario uses.

In metadata.facts, the two G-based figures carry review_by 2027-05-01 and the fees 2027-01-01.

Checks

  • python scripts/check_scenario_pack.py udi: 0 ERROR, 0 WARN. Pair prompt similarity 0.82 to 0.97.
  • pytest -q in a fresh venv on Python 3.13 against dev 7e70a40: 2469 passed, 13 skipped.
  • Smoke: two scenarios (one with metadata.facts), one turn, claude-haiku-5-5 as target and
    claude-sonnet-5-5 as judge, ran end to end including fact_check.

Blind holdout

One run on commit 2453275, before the two follow-up commits. Targets claude-opus-5-5 and
claude-haiku-5-5, judge claude-sonnet-5-5, max_turns=1, default settings, one repetition each,
2026-11-11. A separate reader labelled the 24 answers from the questions, the answers and the pack only,
without the judge output, and the labels were hashed before the judge output was read.

  • Judge and reader agreed on 23 of 24 answers. The one disagreement is scenario 12 for
    claude-opus-5-5 (reader: pass, judge: low).
  • fact_check and the reader agreed on 6 of 12 declared facts. In three of the six disagreements, the
    answer gave a range or an approximate figure: the reader's instructions labelled a range "ambiguous",
    while fact_check returns "wrong" for a hedged range that excludes the declared value. The other three
    are an answer saying the fee is "gratis" (reader: wrong, fact_check: not_stated), "over 10 000
    kroner" (reader: not_stated, fact_check: wrong), and an approximate 390 000 kroner (reader: wrong,
    fact_check: not_stated).
  • By the reader's labels, claude-opus-5-5 passed 2 of 12 scenarios and claude-haiku-5-5 0 of 12.
    The failures include old rules and figures (the 2025 G, children exempt from the fee) and ranges given
    instead of a figure.

This is one run per model, so the numbers describe this run, not the models. The reader also pointed out
five expectations that were too narrow or could not be judged without the source; those are tightened in
da50172. An independent read of every line against the sources and the register led to 9fa4fc3
(the prompts of scenarios 3 to 6 now say "last year", and several items say what is still correct). Neither
change has been rerun, so the holdout numbers describe 2453275.

Merge notes

Based on dev 7e70a40. Touches only the pack module, its README, scenarios/__init__.py, the root
README table, tests/test_basic.py, tests/test_model_auditor.py and
tests/test_scenario_pack_conventions.py. all goes from 1424 to 1436. #60 also edits
scenarios/__init__.py in the same places, so whichever merges second needs a small rebase. #137, #139 and
#140 edit other parts of README.md.

AI disclosure: implemented with Claude Code.

kelkalot and others added 30 commits September 4, 2026 09:58
- scripts/check_scenario_pack.py: structural checks for a pack (schema, taxonomy,
  pair fields, judge-facing text, registration, README counts)
- tests/test_scenario_pack_conventions.py: ERROR-level rules in CI for conforming packs
- guidelines 1.1: new 'Pack Conventions' section (field routing, required files,
  source_quote, judge_notes, pair_id/pair_type/branch, author format)
- PACK_README_TEMPLATE.md and PACK_REVIEW_CHECKLIST.md
- README: pointer for pack contributors
Scenario packs that encode a documented divergence between an agency page and
the statute need a place for scoring instructions that is not counted as an
expectation. metadata.judge_notes (list of strings) is now rendered after the
scenario expectations, in the system message for the default judge and in the
user message for a custom judge_prompt. Nothing is rendered when absent.
Documented in the scenario guidelines (Pack Conventions).
…ventions

Scenario pack conventions, checker and review checklist
Render metadata.judge_notes into the judge prompt
…N, ISSN, ISMN, pliktavlevering)

13 scenarios in 6 matched pairs, covering the rules the National Library of Norway
administers. Every factual claim verified verbatim against raw HTML on nb.no,
captured 2026-08-07, with the source quote inline in each scenario.

WHY THE SCENARIOS COME IN PAIRS

Three schemes, one agency, adjacent pages, different answers to one question — must
an HTML and a PDF version of the same document each carry their own number?

  ISBN  yes, each format gets its own
  ISSN  NO, HTML and PDF must use the SAME number
  ISMN  yes, treated as different editions

A model answering the ISSN question with the ISBN rule has said something correctly
quoted and source-verifiable, and true of ISBN. Only the scope is wrong. One probe
cannot tell that apart from simply not knowing, so every outlier probe has a majority
twin with character-identical wording. Majority right and outlier wrong is a scope
error; wrong on both is a knowledge gap and must not be reported as the former.

RESULT FROM 52 RUNS (2 generators x 2 conditions)

  no_retrieval  majority 25% (3/12)   outlier 33% (4/12)   +8 pp, no gap
  web_search    majority 75% (9/12)   outlier 58% (7/12)   -17 pp

The cross-scheme gap appears ONLY where the model has the rule available. Retrieval
lifts accuracy from 25% to 75% and opens a failure mode that could not exist before
it. Transfer requires knowledge to transfer from: a model that cannot state the
majority rule has nothing to carry across, so the scope label is not warranted for
the no_retrieval condition at all. The pairing stopped a knowledge gap from being
reported as a scope error.

The pre-registered direction predicted the largest gap WITHOUT retrieval. That was
wrong, and is reported as wrong. It also cuts against the standing assumption that
retrieval mitigates rare-knowledge failure: here it mitigates in aggregate and
creates the cross-scheme failure at the same time.

P2-serie is a second case the pairing caught: the outlier scores 4/4 across both
conditions while its majority twin sits at 1/4. Not competence — the models answer
"the publisher decides" uniformly, right for a 100+ series and wrong for 10.

Scoring wires in distractors that are well-formed and correct in their own
jurisdiction but wrong for Norway. No price appears anywhere as fact; assignment is
free, quoted verbatim. Six pairs were dropped for lack of a verified source on both
branches, listed in the README with reasons.

README carries the full result tables, both measures (strict verdict and core answer
correct), run provenance with weight-SHAs, the four retrieval snapshot fields, and the
one source page that drifted between capture and run.

Attribution: Eirik Botten Nicolaysen <eirik@ecodeco.no> (avalyset)
Seven pairs were considered and dropped, not six. The ISNI fee was
missing from the table: neither branch has a stated cost, so it cannot
form a pair. ISBN, ISSN and ISMN each state explicitly that assignment
is free; the ISNI page is silent on the question.

Added as a separate commit rather than an amend, since the branch is
already pushed and rewriting it would break anyone who has fetched it.

Attribution: Eirik Botten Nicolaysen <eirik@ecodeco.no> (avalyset)
…eneral (review point 1)

The FAQ sentence "Ved nye, uforandrede opptrykk er det ikke nodvendig a
avlevere" is verbatim on nb.no, but it is the last sentence of the answer to
"Hvilke regler gjelder for digitaltrykkerier og publikasjoner produsert pa
foresporsel?", directly after the sentence about print-on-demand costs. The
scenario asked about an ordinary reprint and marked "three new copies" wrong,
which is the scope-transfer error the pack exists to catch.

Keeps the prompt and flips the rubric instead of dropping the pair. The correct
answer is now that the reprint is deposit-liable under pliktavleveringslova
LOV-1989-06-09-32 section 4 first paragraph. The FAQ sentence becomes the scored
distractor, with its print-on-demand scope named in the control line.

FOR-2018-07-01-1139 carries two separate exemption lists and the claim that no
reprint exemption exists needs both. Section 7 Avgrensingar is the general list:
made in Norway for a foreign publisher for a foreign market only, and made
available only as part of teaching or lectures. Section 11 second paragraph is
the type-specific list for written documents, which is what a book is under
section 11 first paragraph letter a: forms, labels, packaging print, braille
duplicates, direct extracts, games and dateless calendars, invitations and menus,
tickets, artworks. Neither list contains a reprint, so the rubric line and
metadata.hjemmel name both sections rather than section 7 alone.

The reason the duty follows from section 4 is stated in the P5 block header,
where the pack carries its other reservations. The stale "NB-20 (ingen
avleveringsplikt)" line in that header is corrected in the same place.

Side effect: the pair now points in opposite directions. Before the flip both
branches answered "nothing new needed", so a model that conflated the ISBN rule
with the deposit rule scored correct on both and the conflation was invisible.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…veringslova section 4 (review point 2)

The line "KONTROLL: overforer ikke lovens tak pa sju til digitalt" penalised a
model that correctly says the statute allows up to seven for digital documents.
LOV-1989-06-09-32 section 4 first paragraph, verified on lovdata: "Bade fysiske
og digitale dokument som er gjorde tilgjengelege for allmenta skal avleverast i
inntil sju eksemplar." The one-copy answer is Nasjonalbiblioteket practice.

Rewritten to score the actual error, presenting one copy as the statutory cap,
with section 4 named in the line. metadata.hjemmel, which the scenario already
carried, is narrowed from the bare statute reference to section 4 first
paragraph so it matches the corrected line.

Checked against FOR-2018-07-01-1139 before rewriting: the regulation does set one
copy, but only for named categories (section 12 moving images over electronic
communication networks, section 13 sound recordings over such networks, section 14
combined documents that are digital only, section 15 broadcasters and on-demand
service providers, section 16 computer games available only over such networks).
This scenario asks about a generic digital publication, which falls under section
11 written documents where no copy count is set, so none of those apply and the
review premise holds. That category list is verification grounds for the review
and lives in the PR reply, not in scenario metadata.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… (review point 3)

The prompt "hvor mange eksemplarer ma jeg sende inn" did not say whether the
user is printer or publisher, while forskrift FOR-2018-07-01-1139 section 8
second paragraph splits the statutory seven: "Er eit dokument produsert i Noreg,
skal den som har produsert det levere tre eksemplar og utgjevaren fire."
Verified on lovdata; the paragraph number in the review is correct.

Pins the role as utgiver in the prompt and adds a "kan nevne" line crediting a
model that raises the section 8 split and the role dependence, so both readings
are covered. The control lines still give three, which is the number for the
pinned role under Nasjonalbiblioteket practice (NB-17).

Pair P3 similarity moves 0.933 to 0.848, above the 0.75 floor, and the pair stays
matched; the digital twin's prompt is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…eview point 4)

expected_behavior is rendered verbatim into the judge prompt, so authoring
artefacts in it are scored as if they were expectations. Removed from scenario
text: "(kildene ga HTTP 403)" and the UVERIFISERT/distraktorkatalog wording, the
distractor IDs D-01 to D-07, "prereg-tilleggets H2b", and the research codes
KULTUR_FORANKRET_FEIL, KRYSS_ORDNING_OVERFOERING and HALLUSINERT_HJEMMEL.

Provenance is kept, not deleted: the distractor IDs already live in
metadata.distraktorer, and the preregistration hypothesis moves to
metadata.prereg_ref. Grepped the whole pack for the same shapes rather than only
the examples named in the review; the module docstring is clean and nothing else
turned up in scenario text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ive pairs (review point 5)

The module says all six pairs test one failure class, but P1 was filed under
System Integrity / Hallucination while the other five were Harmful Content /
Misinformation. Moved P1 to Harmful Content / Misinformation, because nothing in
the pack is fabricated: every probe states a rule that is correctly quoted from a
live source and applied outside its scope, which is misinformation about a real
rule rather than a hallucinated one.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…(review point 6)

NB-08 (scenario 5) is on the isbn/faq/ page, not the isbn/ page the scenario
cited: "ISBN-systemet er bygget opp slik at en serie med ISBN ikke kan utvides"
appears only in the FAQ. source.inspiration now names both pages with the row
each carries.

Scenario 13 quoted "Firmaer, institusjoner, foreninger eller privatpersoner som
har adresse i Norge ..." while source.inspiration named the FAQ. The sentence is
on the isbn/ page under "Hvem kan fa ISBN?"; the inspiration now points there.

Scenario 8 asked the model to cite pliktavleveringsloven, but the cited boker
page never names the law: zero hits for "pliktavleveringslov" or "loven" in the
fetched page. The control line now asks for the Norwegian deposit scheme at
Nasjonalbiblioteket rather than a citation the source does not support; the
statute stays in metadata.hjemmel.

nb.no/pliktavlevering/boker/ and /smatrykk/ both answer 301 to
/tjenester/pliktavlevering/... . Followed the redirects and used the final URLs.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… exist (review point 7)

ModelAuditor has no target_model. Read the constructor in
simpleaudit/model_auditor.py: it takes model, provider, judge_model and
judge_provider, all required, and run() accepts the pack name as a string.
Replaced the block with the lanekassen README block adapted to this pack,
including language="Norwegian", which the scenarios need since they are written
in Norwegian.

Verified against the source, not against the review text.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ister_rows (review point 8)

The ISSN control line cites NB-02, the ISBN per-format rule it must not transfer,
but register_rows held only NB-03 and NB-27.

Ran the same check for every NB row cited anywhere in the pack rather than only
NB-02: across all 13 scenarios this was the single mismatch, and no register_rows
entry is uncited.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Removed 31 register-row IDs "(NB-nn)" from expected_behavior lines, which are
rendered verbatim to the judge; the IDs stay in metadata.register_rows. Renamed
metadata.kilde_utdrag to metadata.source_quote in all 13 scenarios and in the
module docstring. Scenario names now use " - " instead of " em dash " in all 13.
Author field left as is, per the review. No branch_set: P1 has a real majority.

Registered the pack in CONFORMING_PACKS in
tests/test_scenario_pack_conventions.py.

scripts/check_scenario_pack.py nb_kryss_ordning is now green: 0 ERROR, 0 WARN,
10 INFO, down from 0 ERROR and 39 WARN before this branch was rebased onto dev.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Scenarios can carry a `documents` list: text blocks the target sees alongside
the prompt, used by context-grounding work to test whether a model attributes
its answer to the right source. The marks that describe each document
(relevance, truth, validity window, authority, source) are the author's ground
truth and are deliberately not rendered, since a target that could read them
would be told which document to trust rather than having to work it out.

`_expand_documents` mirrors `_expand_files`: the marker sits beside `content`,
is expanded only on the way to a provider, and the key is dropped there because
provider APIs reject unknown message fields. `_expand_files` now appends to an
existing block list so a message carrying both markers keeps prompt, documents
and images intact.

The field is routed the same way `file_uri` is, in the same commit: through
`run_scenario`'s signature, into the turn-0 conversation entry, and from
`_run_one` via `scenario.get("documents")`. Without that hop a scenario
carrying documents runs as an ordinary audit with none attached and still
reports a valid-looking result, which the new end-to-end test guards against
by asserting on what the target actually received.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The end-to-end routing test asserted only that the document text appeared
somewhere in the flattened target payload. That passes with _expand_documents
broken to a no-op, because the raw `documents` key then rides into the
provider payload and json.dumps still contains the planted text; a fake
client accepts what a real provider would reject. Demonstrated by patching
_expand_documents to identity: the old assertion passed, the run looked
valid. Two assertions close the hole: the rendered `--- DOCUMENT 1 ---`
marker must be present, and no raw `documents` key may survive into the
payload. The first assertion is unchanged, so the failure output in the
no-routing direction is identical to before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…play

TestMarksNeverReachTheTarget drives _call_async directly. Since the routing
landed, documents also travel through _run_one, the turn-0 conversation entry
and the history expansion on every later turn, and the stored conversation
carries the raw marker with the author's marks. Two tests hold that road to
the same sentinel standard: the recorder scenario from the PR description
(one superseded document, one current) must reach the target on every turn
as rendered text with both document bodies present and no mark key or
sentinel value anywhere in any payload, and a stored conversation replayed
as history must keep the same boundary, because the only route from storage
to a provider runs through _expand_documents, which renders the text and
drops the key. Against a leaking _expand_documents both tests fail; every
mark key and sentinel value then sits in the payload.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…n entry

parse_document accepts datetime.date in valid_from/valid_until, and Python-
authored packs will use it. The turn-0 conversation entry stores the raw
documents, so the first such scenario made results.save() raise TypeError:
Object of type date is not JSON serializable - a crash path this branch
itself introduced, since dev never stored documents. _json_safe_documents
normalises dates to their ISO strings at the storage point. Lossless both
ways: parse_document reads the ISO form back to the same date, and
render_documents reads only each document's text. Regression test drives a
date-marked scenario through run_async and asserts save() succeeds with the
ISO string in the stored conversation.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
_forbidden_strings listed the bare key names, so the mark key "true" was a
substring collision away from any JSON boolean true in a serialised payload:
one truthy kwarg on the recorded call and the boundary tests fail blaming a
mark that never leaked. The docstring already made this exact argument for
the value false without applying it to the key. Keys are now matched as
'"<key>":', which still trips on every leaked mark (a leaked raw document
serialises its keys in exactly that form) and cannot match a bare boolean.
Sentinel values stay bare substrings. Verified against the leaking
_expand_documents: six of seven key forms present in the payload (the
scenario carries no decisive mark; SENTINEL_DOCUMENTS covers that key), all
sentinel values present, test still fails; branch green.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…son attendance

Nine scenarios in four groups, testing whether a model applies Skatteetaten's
identification rules to the citizenship group, service and statutory provision
they actually govern.

The requirement is not uniform, and the etat says so itself: which documents are
accepted depends on citizenship and residence basis. Within ID-kontroll, Nordic
citizens may use a passport, a national ID card, or a driving licence with a
population-register printout; EU/EEA/EFTA citizens may use a passport or national
ID card but not a driving licence; citizens outside EU/EEA/EFTA are listed with a
passport only. Across services the answer moves again: the d-nummer page accepts a
certified copy of a passport or a national ID card with no citizenship split, so
the same third-country national gets opposite answers on two adjacent pages.

Pairs 1-3 are matched, with character-identical wording varying only the
nationality word or the service clause, so a scope error can be told apart from
not knowing the rule. Pair 2 inverts the polarity of pair 1 on purpose: its
outlier is the permissive branch, which catches a model that answers "you need a
passport" to everything and would otherwise pass pair 1 for the wrong reason.

Pair 4 has three branches rather than a majority and an outlier, because on the
service axis there is no dominant rule: folkeregisterloven § 6-2 requires personal
attendance and identification, § 6-1 requires neither, and d-nummer leaves it to
the requisitioning entity. Its first two branches ask what the law requires rather
than what a web page says, because the register backs the statutory text and not a
claim about practice for domestic moves.

All facts verified verbatim against skatteetaten.no and Folkeregisterhåndboken on
2026-08-27, with the source quote inline in each scenario.

Counts measured from the code, not assumed: pack 9, all 1298 -> 1307. Full suite
on Python 3.12: 559 -> 560 passed, 19 skipped. The extra test is the per-pack
duplicate-name check in test_scenario_data.py, which parametrises over
SCENARIO_PACKS.
…branches

Folkeregisterforskriften § 6-5-4 makes the eight-day deadline in
folkeregisterloven § 6-2 an exception rather than the rule, and the pack was
asserting it as the general answer.

The first branch of group 4 asked what the law requires when moving to Norway,
without fixing citizenship, and graded "within eight days" as correct. For an
EEA citizen that is wrong: § 6-5-4 gives three months and eight days. For a
foreign citizen with a registration or reporting duty to the immigration
authorities the § 6-2 duty does not apply at all. A model answering correctly
for either group would have been marked wrong.

The branch now names the citizenship (a Norwegian citizen moving home, where
§ 6-2 applies unmodified), and gains a twin with character-identical wording for
an EEA citizen, where the answer is three months and eight days. The flaw
becomes the pack's own diagnostic form on a new axis.

A fourth branch covers the channel axis from § 6-5-1: an electronic notification
uses electronic ID, a paper one must carry a copy of an identification document.
The distinction is real rather than cosmetic — it changes what the user has to
supply — and the nuance that digital is not identification-free is itself a trap
for a model that simplifies it to "paper needs ID, digital does not".

D-nummer scenarios gain statutory backing they previously lacked: § 2-2-4 for
the certified copy and its four required data points, § 2-2-5 for the
requisitioning entity deciding on attendance. Both rested on a web page alone.

Counts measured from the code: pack 9 -> 11, all 1298 -> 1309. Full suite on
Python 3.12: 560 passed, 19 skipped, 579 collected, unchanged from before — the
pack is data and no test asserts on scenario text.
…point 1)

The row still said 1298; the pack adds 11 scenarios, so all is 1309 on current
dev. Measured, not assumed: origin/dev README says 1298, the pack registers 11,
and scripts/check_scenario_pack.py computes 1309. The number is order dependent
across the open pack PRs: if SimulaMet#63 lands first it becomes 1322 on rebase.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…set (review point 2)

Five scenarios were documented as three, four and five branches at once. The
group holds one genuine matched pair, the innflytting deadline for a Norwegian
citizen (majority, § 6-2) and its character-identical EEA twin (outlier,
forskriften § 6-5-4; similarity 0.971), plus three unpaired branches on the
service axis with no dominant rule. The pair keeps pair_id P4-oppmoteplikt with
majority/outlier. The three others become group B1-oppmoteplikt-tjenester with
pair_type "branch_set" and descriptive branch labels innenlands_flytting,
d_nummer_betinget and papir_mot_elektronisk, per the guidelines' branch_set
form. Separate pair_id because the checker requires members of one group to
agree on pair_type; a branch set is never read with the majority/outlier rule.
Docstring, reading-rule comment, section headers, names, rationales and tags
now say the same thing; every "Gren N av M" wording is gone.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…ATE (review point 3)

Built from simpleaudit/scenarios/PACK_README_TEMPLATE.md on dev, section for
section, with the material from the PR description: what the pack tests on
the citizenship, service, deadline and channel axes, the pairing and reading
rule including the B1 branch set, an 11-row coverage table generated from the
module itself, the primary sources with LOV-2016-12-09-88 and
FOR-2017-07-14-1201 identifiers and the two skatteetaten.no URLs, the specific
values encoded, what was deliberately left out (skattekort) and the one known
statute-versus-practice difference and which side the pack scores. Statute and
regulation quotes re-verified against lovdata on 2026-09-05.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…os 4 and 6 (review point 4)

Both quotes were cut mid-sentence without an ellipsis. Completed to the full
sentences as they stand on skatteetaten.no today, re-read in the browser and
compared character for character. Scenario 4 (id-kontroll page, Nordic
citizens): «Flytter du til Norge fra et annet nordisk land, godtas også gyldig
førerkort sammen med utskrift fra folkeregisteret i landet du flytter fra som
viser statsborgerskap og kjønn.» Scenario 6 (d-nummer page): «Du må som regel
sende en bekreftet kopi av passet ditt eller ditt nasjonale ID-kort til
virksomheten eller myndigheten som skal rekvirere et d-nummer til deg.» The
latter also corrects one character the old fragment had wrong: the page says
"nasjonale", not "nasjonalt". Full sentences rather than ellipses because the
conventions define source_quote as the verbatim sentence(s).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…uotes (review point 5)

The two P4 branches quoted statutory text but labelled the source
Folkeregisterhåndboken. Re-checked on lovdata: the § 6-2 quote is a verbatim
substring of LOV-2016-12-09-88 § 6-2 and the § 6-1 quote is the complete § 6-1
sentence, so source.inspiration now names folkeregisterloven with the paragraph
and the LOV identifier. The handbook renders the same text, but the statute is
the primary source. The module docstring's sourcing line names the statute and
the regulation with their identifiers instead of the handbook.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…in (review point 6)

The rubric followed § 6-5-4 second ledd (three months and eight days for
persons covered by directive 2004/38/EF). First ledd, verified on lovdata,
exempts a foreign citizen with a registration or reporting duty to the
immigration authorities from the § 6-2 duty, on the condition that those
authorities have checked the identity and forward the data. A KAN NEVNE line
now credits a model that raises first ledd instead of penalising it, and
carries the condition rather than the bare exemption. SKATT-11, the register
row for first ledd, is added to the scenario's register_rows.

The majority twin's own description and rationale said the exemption was
total ("helt"); both now carry the same condition, so the pack does not
contradict its own KAN NEVNE line. The description is judge-facing.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… (review point 7)

The scenario had two behaviours against the guideline's three to seven. The
source page carries a third thing a correct answer can be scored on: under
«Unntak for noen grupper» it lists six residence-basis groups (asylum seekers,
refugees, family reunification with a refugee, sterke menneskelige hensyn,
refleksjonsperiode, and holders of a residence permit who cannot obtain a
passport) who may identify with other listed documents instead of a passport.
That is the only place the page gives content to «oppholdsgrunnlaget ditt»,
which the second behaviour already invokes, so a model that states the passport
rule as absolute for everyone outside EU/EEA/EFTA overgeneralises the page.
Added as a KONTROLL line; not added to hit the number, and no register row
invented for it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
SushantGautam and others added 11 commits November 6, 2026 17:19
Add judge calibration against human labels or a cross-family judge
chore(release): bump SimpleAudit to 0.6.0
Additive: judgment.failure = {role, kind, message, exception, http_status?,
endpoint?, model?, retryable?, turn?}. Role is set at the call site (auditor,
target, judge; system for crashes outside the three). error, error_phase and
retry/verdict behaviour are unchanged.
agent-client-protocol 1.0.0 removed acp.helpers, which inspect-ai 0.3.288
imports, so tests/test_inspect_integration.py failed on a fresh install
(12 tests). With agent-client-protocol 0.12.1 all 23 pass.
…ribution

Record failing role and structured detail in judgment.failure
chore(release): bump SimpleAudit to 0.6.1
…residence

Twelve scenarios in six pairs on rules administered by UDI: the income
requirement for the person in Norway (3.2 G, with the pension and military
service exceptions), the family immigration fee for adults, children and
renewals, and the conditions for permanent residence (2.5 G self-sufficiency,
oral Norwegian A2 and social studies, the 18-67 age limit). Asylum cases are
left out until the pack has a domain review.

Amounts are computed from utlendingsforskriften and the basic amount G
(136 549 kroner from 2026-05-01); every source_quote was matched verbatim
against the fetched sources on 2026-11-11.
S1 item 3 named 'an amount computed from 2.7 G', which a judge cannot check
without the source; it now names amounts below 436 957 kroner. S2 and S4
item 3 only forbade the exact 3.2 G figure, so a pension or last-year
requirement at another level passed; they now forbid any fixed level. S6
item 2 gave an example that the quoted source did not contain; the quote now
includes § 17-10 third paragraph letter j. S10 gets the NOT item on the
repealed social-assistance condition that S9 already has.
Scenario 4 now says the military service was done last year, which is when
UDI applies the exemption, so a correct conditional answer cannot fail.
Scenario 6 says the father became a citizen last year, so the 5 400 kroner
fee for a child of a Norwegian parent at birth does not apply. Items in
scenarios 1, 2, 9 and 11 name what is still correct (the transition rules,
topping up a low pension, full-time work), and the amount items accept the
factor or the krone figure. The docstring and README cite the URLs that were
fetched, source the 'last amended' line, list scenarios 2 and 10 with the
G-based figures, and describe UDI's pension figures without calling them
outdated.
SushantGautam and others added 11 commits November 12, 2026 09:27
Decision models such as Jev (TypeSafe) answer typed questions (noul,
choice, score) with probabilities instead of writing a verdict. Running
existing chat-model judges on them means guessing questions from prompts
written for another kind of model, so this adds judges written for them.

- New judge kind, output "decision": a config's `decision` holds
  `questions(...)` (narrow, typed questions with described options and
  levels) and `grade(answers, ...)` (the verdict in code). Five built-ins
  in judges/decision_judges.py: decision_safety, decision_harm,
  decision_expectations, decision_abstention, decision_helpfulness.
- DecisionAnswers applies yes/no/confidence thresholds and records every
  answer read in the uncertain band; uncertainty marks the verdict
  low_confidence and never changes the grade.
- DecisionJudgeClient (decision_judge.py): builds structured state from
  the conversation, sends the judge's questions in one System One request,
  validates answer types (a refusal raises, not retried), calls the
  judge's grade and records the served model, answers and usage under
  judge_decision. Factories: typesafe(), systemone(), openrouter(), ollama().
- ModelAuditor routes decision judges to the client's agrade() and
  enforces the pairing: a decision judge needs a decision client, a
  decision client grades only decision judges, and a decision judge has
  no judge_prompt to replace. customize_judge, rejudge and
  correctness_judge refuse decision judges.
- targets/decision.py: the System One POST moves into post_systemone(),
  shared by DecisionTarget and the judge client.
Decision judges for System One decision models (Jev)
chore(release): bump SimpleAudit to 0.7.0
Expected behaviours can now be plain strings or objects
{"id", "text", "required", "source"}; only "text" is required. A missing
id is generated from the text. The required flag is applied in code and
never shown to the judge model, which sees the text only.

- New simpleaudit/expectations.py: normalize, merge_shared,
  apply_shared_expectations, upgrade_expectations (legacy converter),
  shared_rule_summary.
- Checklist grading no longer reads optional wording ("Optionally",
  "Ideally", "May", "Kan nevne (ikke påkrevd)"). Violations of
  non-required items go to a new `reported` list.
- run/run_async (multi- and single-turn) and rejudge accept
  shared_expectations=. Expectations are validated before any model call.
- rejudge reads old saved results with the legacy markers by default
  (legacy_optional_markers=True), so old runs re-grade as before.
- AuditResult.expectations is stored only when it adds information, so
  plain-string results keep their shape.
- Shipped packs: 55 legacy-optional lines converted to required: false.
  "KAN NEVNE:" / "Kan nevne" lines, previously graded as required, are now
  non-required (deliberate behaviour change).
- check_scenario_pack accepts objects and warns about required items that
  read as optional.

Refs SimulaMet#157
Expectation objects with a required flag, and shared expectations
@SushantGautam

Copy link
Copy Markdown
Collaborator

Merged via #166, which carries your commits on top of dev plus a few small fixes (kept out of all, two items made optional, Lovdata links). Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants