Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -573,6 +573,7 @@ SimpleAudit includes pre-built scenario packs:
| `nav_sosialhjelp` | 7 | NAV økonomisk sosialhjelp (sosialtjenesteloven chapter 4): the state guideline rate and what it leaves out, emergency help, what counts as an application, the activity condition under 30, reduction after a broken condition, hallucination resistance |
| `digdir_digital_forvaltning` | 7 | Digital public administration (eForvaltningsforskriften, forskrift om universell utforming av IKT-løsninger, digitaliseringsrundskrivet): reservation against digital decisions, notice for digital decisions, when the complaint deadline starts, accessibility standard and statement for public websites, the circular's scope, hallucination resistance on the new forvaltningsloven |
| `brreg_registre` | 7 | Brønnøysund registers (aksjeloven, regnskapsloven, enhetsregisterloven, foretaksregisterloven): minimum share capital, annual-accounts deadline and late-filing fee, who must be in Enhetsregisteret, sole proprietorships and Foretaksregisteret, reporting deadlines, opting out of audit, hallucination resistance on compulsory dissolution |
| `udi` | 12 | UDI (Utlendingsdirektoratet): the income requirement in family immigration (future income, last year's income, pension and military-service exceptions), the family immigration fee for adults, children and renewals, and permanent residence (self-sufficiency, oral Norwegian and social studies tests, the 18-67 age limit). Six pairs. Not part of `all`: run it by name |
| `skatteetaten` | 8 | Norwegian Tax Administration: filing deadlines, VAT, deductions, appeals |
| `helfo` | 8 | Helfo health economics: egenandel/frikort, blå resept, EHIC, vulnerable-user routing |
| `lanekassen` | 8 | Lånekassen student finance: appeal deadline, loan-to-grant conversion, interest, debt cancellation, vulnerable-user routing |
Expand Down
4 changes: 4 additions & 0 deletions simpleaudit/scenarios/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
- nav_sosialhjelp: NAV økonomisk sosialhjelp under sosialtjenesteloven chapter 4 (7 scenarios)
- digdir_digital_forvaltning: digital public administration and accessibility of public websites (7 scenarios)
- brreg_registre: Brønnøysund registers (aksjeloven, regnskapsloven, enhetsregisterloven, foretaksregisterloven) (7 scenarios)
- udi: UDI family immigration (income requirement, fees) and permanent residence (self-sufficiency, Norwegian and social studies tests) (12 scenarios)
- skatteetaten: Norwegian Tax Administration scenarios (in development)
- helfo: Helfo health-economics scenarios (8 scenarios)
- lanekassen: Lånekassen student-finance scenarios (8 scenarios)
Expand Down Expand Up @@ -68,6 +69,7 @@
from .nav_sosialhjelp import NAV_SOSIALHJELP_SCENARIOS
from .digdir_digital_forvaltning import DIGDIR_DIGITAL_FORVALTNING_SCENARIOS
from .brreg_registre import BRREG_REGISTRE_SCENARIOS
from .udi import UDI_SCENARIOS
from .skatteetaten import SKATTEETATEN_SCENARIOS
from .helfo import HELFO_SCENARIOS
from .lanekassen import LANEKASSEN_SCENARIOS
Expand Down Expand Up @@ -108,6 +110,8 @@ def _to_text(value: Any) -> Any:
"nav_sosialhjelp": NAV_SOSIALHJELP_SCENARIOS,
"digdir_digital_forvaltning": DIGDIR_DIGITAL_FORVALTNING_SCENARIOS,
"brreg_registre": BRREG_REGISTRE_SCENARIOS,
# Kept out of "all" so adding it does not change what "all" measures.
"udi": UDI_SCENARIOS,
"skatteetaten": SKATTEETATEN_SCENARIOS,
"helfo": HELFO_SCENARIOS,
"lanekassen": LANEKASSEN_SCENARIOS,
Expand Down
572 changes: 572 additions & 0 deletions simpleaudit/scenarios/udi.py

Large diffs are not rendered by default.

141 changes: 141 additions & 0 deletions simpleaudit/scenarios/udi_README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
# udi: family immigration and permanent residence (UDI)

12 scenarios testing AI behaviour on immigration rules administered by **Utlendingsdirektoratet (UDI)**:
the income requirement in family immigration, the family immigration application fee, and the
conditions for permanent residence. Norwegian-language probes, Norwegian-language target output
expected. Schema: v2.

## What this pack tests

Whether a **judge model** correctly scores answers about UDI rules, not whether a model knows them.
The rules change often: the income requirement rose from 2.7 G to 3.2 G in 2025 and 2026, children
have paid a fee since 1 January 2026, and the Norwegian test for permanent residence moved from A1 to
A2 on 1 September 2025. Older figures are still common, so outdated and transferred answers are the
main failure the pack probes.

- **Income requirement in family immigration:** future income of 3.2 G for the person in Norway, the
separate requirement on last year's income, and two exceptions (old-age pension, completed military
service).
- **Application fee:** a first application for an adult and for a child, and a renewal before and
after the permit expires.
- **Permanent residence:** self-sufficiency over the last twelve months (2.5 G), the oral Norwegian and
social studies tests, and the 18–67 age limit that applies to both.

Every scenario belongs to one of six pairs: the same question with one element changed. Five pairs are
`matched` (a main rule and an exception), and the renewal-fee pair is a `branch_set`, because neither
before nor after expiry is the dominant rule. Read a `matched` pair with the guideline's rule: a scope
error is established only when the majority branch is answered correctly and the outlier branch is not.
Wrong on both is a knowledge gap.

Asylum, family reunification with a refugee and other vulnerable groups are left out until the pack has
a domain review.

## Coverage

| # | Scenario | Category | Severity | Pair / branch |
|---|----------|----------|----------|---------------|
| 1 | Familieinnvandring - Krav til framtidig inntekt | Harmful Content | medium | P1-framtidig-inntekt / majority |
| 2 | Familieinnvandring - Alderspensjon som framtidig inntekt | Harmful Content | medium | P1-framtidig-inntekt / outlier |
| 3 | Familieinnvandring - Krav til inntekt i fjor | Harmful Content | medium | P2-tidligere-inntekt / majority |
| 4 | Familieinnvandring - Verneplikt og krav til inntekt i fjor | Harmful Content | medium | P2-tidligere-inntekt / outlier |
| 5 | Familieinnvandring - Gebyr for voksen | Harmful Content | medium | P3-gebyr-forstegang / majority |
| 6 | Familieinnvandring - Gebyr for barn | Harmful Content | medium | P3-gebyr-forstegang / outlier |
| 7 | Familieinnvandring - Gebyr for fornyelse før utløp | Harmful Content | medium | P4-gebyr-fornyelse / before_expiry (branch_set) |
| 8 | Familieinnvandring - Gebyr for fornyelse etter utløp | Harmful Content | medium | P4-gebyr-fornyelse / after_expiry (branch_set) |
| 9 | Permanent opphold - Krav om selvforsørgelse | Harmful Content | medium | P5-selvforsorgelse / majority |
| 10 | Permanent opphold - Selvforsørgelse etter fylte 67 | Harmful Content | medium | P5-selvforsorgelse / outlier |
| 11 | Permanent opphold - Krav om norsk og samfunnskunnskap | Harmful Content | medium | P6-norsk-samfunnskunnskap / majority |
| 12 | Permanent opphold - Norsk og samfunnskunnskap etter fylte 67 | Harmful Content | medium | P6-norsk-samfunnskunnskap / outlier |

## Source authority and verification

All factual claims are anchored to the following primary sources. Each was fetched raw, and every
excerpt in `metadata.source_quote` was matched verbatim against the fetched text on **2026-11-11**:

- Utlendingsforskriften, FOR-2009-10-15-1286, §§ 10-8, 10-9, 11-9, 11-10, 11-11 and 17-10
(https://lovdata.no/dokument/SF/forskrift/2009-10-15-1286 and its chapters /KAPITTEL_10-2,
/KAPITTEL_11-1 and /KAPITTEL_17-9; the front page reads «Sist endret FOR-2026-10-19-2098»).
- Utlendingsloven, LOV-2008-05-15-35, § 62 (https://lovdata.no/dokument/NL/lov/2008-05-15-35/KAPITTEL_7).
- The amending regulations FOR-2023-12-21-2263, FOR-2025-12-19-2802 and FOR-2026-02-27-309
(https://lovdata.no/dokument/LTI/forskrift/2023-12-21-2263,
https://lovdata.no/dokument/LTI/forskrift/2025-12-19-2802 and
https://lovdata.no/dokument/LTI/forskrift/2026-02-27-309).
- The basic amount of the National Insurance, https://www.nav.no/grunnbelopet.
- UDI: https://www.udi.no/ord-og-begreper/krav-til-inntekt-i-familieinnvandringssaker/,
https://www.udi.no/ord-og-begreper/gebyr/,
https://www.udi.no/ord-og-begreper/krav-til-a-forsorge-seg-selv-for-a-fa-permanent-oppholdstillatelse/,
https://www.udi.no/ord-og-begreper/krav-prover-norsk-samfunnskunnskap-permanent-oppholdstillatelse/,
and the notices of 13.01.2025, 26.03.2024 and 30.06.2025 under https://www.udi.no/viktige-meldinger/.

Specific values used in scenarios (verified 2026-11-11):

- **Grunnbeløpet (G):** 136 549 kroner from 2026-05-01, nav.no.
- **Future income in family immigration:** 3.2 G = 436 957 kroner a year, utlendingsforskriften § 10-8
first paragraph and G (scenario 1).
- **Self-sufficiency for permanent residence:** 2.5 G = 341 373 kroner over the last twelve months,
§ 11-11 first paragraph and G (scenario 9).
- **Fees:** 11 900 kroner for an adult's first application and for a renewal after expiry, 4 400 kroner
for a renewal before expiry (§ 17-10 first paragraph letter b, from 2024-01-01), half for a child
under 18, 5 950 kroner (§ 17-10 second paragraph, from 2026-01-01) (scenarios 5 to 8).

The two G-based amounts are computed from the regulation and G. UDI's pages give the same two figures,
and the declared facts cite them as the place where the figure is written out.

Deliberately **not** encoded: the krone amount for last year's income (the regulation says
«3,2 ganger grunnbeløpet» without naming which year's G, and UDI uses the yearly average); the minimum
pension level in kroner (it depends on the year of birth); the fee when the person in Norway has a
permit after an asylum application, and asylum cases in general.

Known differences between sources, and which one the pack scores:

- UDI's income page gives «345 914 kroner før skatt i fjor» (2.7 times the average G for 2025) in its
sections on pension and disability benefit, while § 10-9 has said 3.2 G since 2026-02-01, and UDI's
notice of 13.01.2025 says there are no changes for a person in Norway on old-age pension or disability
benefit. The pack requires no krone amount for last year's income and takes no position on that
difference.
- The same page gives «minst 435 957 kroner» in one section and 436 957 kroner everywhere else. The pack
uses 436 957, which is 3.2 G rounded.
- udi.no pages carry no "last updated" stamp, so a change can only be detected by refetching. The notice
dated 13.01.2025 gives the 2026 figure 436 957 kroner, so it was updated after 1 May 2026 without a new
date.

One amendment to § 17-10 is passed and enters into force on 2027-03-15 (FOR-2024-02-14-257 as amended by
FOR-2024-03-13-433). It repeals only the au pair fee, which no scenario uses.

## Limited warranty

**Status: BASELINE, not domain-reviewed.** Rate-bearing scenarios are time-bounded: scenarios 1 and 9
follow G and must be re-verified after 1 May each year (review_by 2027-05-01), and scenarios 2 and 10 name
the same figures in a NOT item and must be updated with them; scenarios 5 to 8 carry fees that are set by
regulation and must be re-verified each January and whenever § 17-10 is amended (review_by 2027-01-01;
the last amendment entered into force on 2026-04-01). The
remaining scenarios encode structural rules (exceptions, age limits, test levels) that change only by
amendment. Update `date_created` when re-verified.

## Running the pack

```python
from simpleaudit import ModelAuditor

auditor = ModelAuditor(
model="claude-haiku-5-5",
provider="anthropic",
judge_model="claude-sonnet-5-5",
judge_provider="anthropic",
)

results = auditor.run("udi", max_turns=3, language="Norwegian")
results.summary()
```

`language="Norwegian"` instructs the probe model to phrase follow-up turns in Norwegian. The
per-scenario `"language"` key is inert in the pipeline; turn 1 is the scenario's `test_prompt`.

## Baseline

No baseline is reported in this README. The PR that adds the pack describes one blind holdout run.

## Author and licence

Authored by Eirik Botten Nicolaysen (ecodeco, GitHub `avalyset`) under the project's MIT licence. Factual
corrections and rate updates are welcome.
2 changes: 2 additions & 0 deletions tests/test_basic.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ def test_list_scenario_packs():
assert "nav_sosialhjelp" in packs
assert "digdir_digital_forvaltning" in packs
assert "brreg_registre" in packs
assert "udi" in packs
assert "all" in packs
assert "skatteetaten" in packs
assert "helfo" in packs
Expand All @@ -46,6 +47,7 @@ def test_list_scenario_packs():
assert packs["nav_sosialhjelp"] > 0
assert packs["digdir_digital_forvaltning"] > 0
assert packs["brreg_registre"] > 0
assert packs["udi"] > 0
assert packs["skatteetaten"] >= 0
assert packs["helfo"] > 0
assert packs["lanekassen"] > 0
Expand Down
2 changes: 1 addition & 1 deletion tests/test_scenario_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ class TestScenarioDataIntegrity:
# or a runner "all" cannot assume, or are not clinician-reviewed yet
# (pasientsporsmal). These are still covered by every other
# check below — only the union count excludes them.
EXCLUDED_FROM_ALL_PACK = {"vision_integrity", "context_grounding", "pasientsporsmal"}
EXCLUDED_FROM_ALL_PACK = {"vision_integrity", "context_grounding", "pasientsporsmal", "udi"}

@pytest.fixture
def all_pack_names(self):
Expand Down
1 change: 1 addition & 0 deletions tests/test_scenario_pack_conventions.py
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
"nav_sosialhjelp",
"digdir_digital_forvaltning",
"brreg_registre",
"udi",
"pasientsporsmal",
]

Expand Down