An AI agent that checks payslips against Danish collective agreements.
The model reads and words things. Code decides and calculates. A person approves every rule.
Walkthrough · What it proves · Results · Quick start · How it works · Design decisions
A collective agreement (overenskomst) says what people must be paid: minimum wages by seniority, overtime, evening and night supplements, pension, holiday allowance. Overens reads an agreement's PDF into typed rules, has a payroll specialist approve them, and then checks payslips against them and answers questions about the agreement with the clause cited.
Overens is Danish for "in agreement". The repo is organised around one question: what happens when the model is wrong? A model reads the agreement and words the findings. It never calculates, never decides whether a payslip is right, and never publishes a rule. Every property claimed below has a test that tries to break it.
flowchart LR
specialist([Payroll specialist]) --> api
employer([Employer]) --> api
api[API and console] -- stores, then starts --> run
subgraph scheduler [Durable Task]
run[Ingestion run<br/>one per upload]
end
subgraph worker [Worker]
read[Read PDF<br/>into clauses] --> extract[Extract rules] --> checks[Check each rule<br/>no model]
end
run --> read
extract --> gateway[Model gateway<br/>CPR removed, budget, failover] --> model([Claude])
checks --> review{{A person<br/>approves}}
review --> publish[(Rule sets<br/>never changed)]
api -- payslip --> engine[Rule engine<br/>decimal, no model]
publish --> engine
engine --> findings[Findings<br/>with clause]
api -- question --> agent[Question agent<br/>own loop] --> answercheck[Answer check<br/>no model]
agent -- calculator tool --> engine
agent --> gateway
idp[Identity provider] -. tokens .-> api
api --> db[(Postgres<br/>row-level security)]
worker --> db
db --> audit[(Audit log<br/>written by triggers)]
The system running on one machine with Claude as the model. Every picture is the real console. On the left of each rule are its values; on the right, in print, the clauses it was read from, with the quoted sentence marked. The explanation and the answer in the last two pictures are the model's, after passing the checks.
| Layer | Choice | What it does here |
|---|---|---|
| Language | C# on .NET 10, warnings as errors | One language across services, engine, evals and tests |
| Model | Claude Sonnet and Haiku, Anthropic C# SDK | Reads agreements, words findings, answers questions |
| Model seam | Microsoft.Extensions.AI |
Redaction, failover, time limits, budget and replay are layers on one interface |
| Agent loop | Own, about a hundred lines | Five read-only tools; the answer is checked before it is shown |
| Rules | Typed records and an engine on decimal |
Every sum, with no model |
| Durable work | Durable Task Scheduler | One orchestration per upload; waits days for a reviewer; survives a dead worker |
| Data | PostgreSQL, plain-SQL schema, EF Core for access | Row-level security per company, three roles, immutable tables |
| Identity | OpenID Connect; Keycloak locally, Microsoft Entra ID when deployed | Roles and company from the token |
| PdfPig to read, MigraDoc to generate | Clauses from layout; tables from their ruled borders | |
| Telemetry | OpenTelemetry | Spans whose attributes come from a fixed list |
| Local | .NET Aspire | The whole system from one command |
| Tests | xUnit, Testcontainers, CsCheck, Playwright | Real Postgres and scheduler; property tests; a real browser |
Each row is a property and the test that attacks it. Where a control was added, a bug was planted to confirm the test fails without it.
| Property | How the test tries to break it |
|---|---|
| The model never calculates | A stand-in model adds up an increase itself; the sums are printed nowhere and are refused. Reported as an increase, code adds it |
| Money is never a float | Reflection over the rules assembly for any float or double |
| Every rule quotes its clause | A reworded quote, a clause that does not exist |
| Every number is the agreement's | The pension total read as the employer's share; a rate off by one krone; a start date printed nowhere |
| A missing rule is reported | A stand-in returns only the first year; nothing is flagged on any rule, and the gap is reported |
| A contradiction is put to a person | Two clauses give different figures; both are proposed, the conflict names both clauses, and approving both is refused |
| An explanation uses only the finding's numbers | A percentage the model worked out; a deadline it invented; "too much" said of a shortfall |
| An answer rests on the agreement | A sum the model made; a reworded quote; a number from a clause it did not cite; a number written as a word |
| Money in an answer is money a person approved | A planted clause, quoted word for word, that states a wage; its figure is not repeated |
| Half an answer is not an answer | A model that runs out of room; the run fails at once, is not asked again, and says why |
| What the agreement does not say is not answered | Twelve unanswerable questions per model; the fixed sentence is the only correct outcome |
| Property | How the test tries to break it |
|---|---|
| Nothing is published without a person's approval | The worker calls publish with no approval; tries to write an approval, a decision and a rule set directly |
| The API cannot publish | The API's database role calls the publish function |
| A flagged rule cannot be approved | Through the API, and directly in the database |
| What was approved is what is published | A decision changed after the approval and before publication |
| A published rule set cannot change | Update, delete and truncate, as each role and as the database's owner |
| A payslip is checked against the rules of its own date | A renewal from 2029; payslips on both sides of it |
| Property | How the test tries to break it |
|---|---|
| One company cannot see another's data | Another company's token on every route; another tenant's scope on every table |
| A token is checked, not trusted | Wrong key, expired, wrong audience, no role, no company |
| A company's token comes from its own directory | With an issuer per company, a token from one directory that names another company |
| An employer cannot do a specialist's work | Upload and rule sets with an employer's token |
| A CPR number never reaches a model | In a question and in a tool result; removed at the layer next to the provider |
| Pay data is not kept | Thirty payslips checked; the audit log holds counts and no employee id |
| Telemetry carries no content | Every attribute of every span of a run searched for the agreement's words |
| A company can be erased, by the owner only | Each service role tries; the owner tries another company's rows |
| Property | How the test tries to break it |
|---|---|
| Text in a document cannot give orders | Five passages addressed to AI systems, one inside the wage clause, one dressed as a protocol; both real models |
| A damaged file does not reach the model | A PDF cut short, scrambled, or only a header |
| Markup is shown as text | Typed into the console; any script or console error fails the browser check |
| The console loads only its own code | The content policy is read back and may not contain unsafe |
| Property | How the test tries to break it |
|---|---|
| A run survives its worker dying | The worker is stopped while a run waits; the review is done with nothing running; a second worker finishes without reading again |
| An upload is never lost | The scheduler is unreachable at upload; the worker finds the run and starts it once |
| A model that is down is replaced | The asked-for model fails; the other answers, and the run records which |
| One company cannot spend without limit | A daily budget of five cents; the second agreement is refused at once, and a failed attempt counts too |
| Calls at the same moment cannot pass the cap together | Two calls start together under a cap one of them fills; the second is refused |
| A slow model is not waited for | A model that never answers; the other one does, inside the time limit |
| Bad input is the caller's mistake, not a crash | Seven malformed payslips, overlapping shifts, work before the hiring date; each is a 400 with a reason |
| A line with no approved rule is not called wrong | A correct payslip checked without the savings rule; the line is reported as not checked |
| A run whose pipeline is gone does not stay open | A run handed to a pipeline the scheduler never had is ended as failed; one waiting for review is left alone |
| Every change is logged, and the log cannot be edited | Each role tries to write, change and empty it; the owner's edit breaks that company's hash chain; a rolled-back write does not |
| An agent cannot run forever | A stand-in that only ever searches |
Every document is synthetic: an agreement is generated from a rule set, so the right answer is known before any model reads it. No real agreement has been measured.
| What | Claude Sonnet | Claude Haiku |
|---|---|---|
| Rules read out of ten agreements | 121 of 121 | 121 of 121 |
| Rules read out of six long agreements | 79 of 79 | 79 of 79, and 1 extra in conflict |
| Wrong rules the model-free checks missed | 0 | 0 |
| Payslips checked correctly, ten agreements | 720 of 720 | 720 of 720 |
| Payslips checked correctly, six long agreements | 432 of 432 | 360, and 72 not checked |
| Questions answered correctly, of 48 | 48 | 48 |
| Wrong answers shown | 0 | 0 |
| Rules changed by five planted instructions | 0 | 0 |
| Cost to read one agreement, short and long | $0.08 and $0.28 | $0.005 and $0.015 |
The long agreements run to about 24 pages and 60 sections, most of them not about pay and full of figures that are not rules. Length, appendices, cross-references and mixed tables changed nothing for Sonnet. On one of them Haiku took the apprentices' pension for a second pension rule. The two were reported as in conflict, so nothing wrong passed, and until a person rejects one the pay check says of those 72 payslips that pension was not checked.
The numbers are after fixes. On first contact with documents the prompt had not seen, a wage given as "raised by 5,00 kr." broke extraction, Sonnet silently chose between two contradicting clauses, and Haiku searched thirteen times for something the agreement does not cover. The long set added two more: a pension share stated as "half of the employer's contribution", which printed no figure to check, and a mistake in the test data itself. Both are in the report.
The eval report has each failure, its fix, and what the numbers do not show. A test fails if the published results differ from what the recorded answers give.
It needs the .NET 10 SDK and Docker. The browser part of the smoke test also needs Google Chrome.
git clone https://github.com/devjoinedthechat/Overens && cd Overens
scripts/check.sh # secrets scan, build, format check, every test
scripts/smoke.sh # the whole system from nothing, one agreement from upload to checked payslips
git config core.hooksPath .githooks # once per clone: the secrets scan runs before every commitThe same smoke test runs against the container images the system is deployed as:
scripts/containers.sh build
STACK=containers scripts/smoke.shNo model key is needed for either. Tests use stand-in models, and the system replays the answers recorded for the corpus agreements.
dotnet run --project src/AppHost| Service | Address | What it is |
|---|---|---|
| Console | http://localhost:5058/app/ |
Upload and review as a specialist; check pay and ask as an employer |
| API | http://localhost:5058 |
The same, for programs. Contract at /openapi/v1.json, probes at /health and /alive |
| Keycloak | http://localhost:8081 |
Sign-in. Realm overens, admin admin / admin |
| Scheduler | http://localhost:8092 |
Each run, step by step |
| Dashboard | http://localhost:15034 |
Logs and traces of every service |
Local accounts, all with the password local-demo-password:
| User | Company | Role |
|---|---|---|
anna |
tvaerhavn-loen |
Specialist: uploads, reviews, approves |
bo |
elmekaer-regnskab |
Specialist at another company |
eva |
tvaerhavn-loen |
Employer: checks pay and asks questions |
erik |
tvaerhavn-loen |
None: signed in, and let in nowhere |
Upload data/synthetic/hard-2026/agreement.pdf as anna. Check a case from data/synthetic/hard-2026/pay.json as eva; pay-answers.json says which are wrong and how.
With ANTHROPIC_API_KEY set, Claude is called: findings are worded by a model and questions are answered. Each company has a daily budget for that, and each process a cap behind it. Without a key, findings get fixed wording and questions are refused.
dotnet run --project evals/Runner # extraction on sixteen agreements
dotnet run --project evals/Runner -- --questions # 48 questions with known answers
dotnet run --project evals/Runner -- --attacks # planted instructions and a contradiction
dotnet run --project evals/Runner -- --explain # how often a model's wording passes the guardEach replays recorded answers and spends nothing. Add --model claude-haiku-5-5 for the other model, and --record --cap 2 with a key to call the model for anything not yet recorded.
- Stored. The PDF is kept under the hash of its bytes and never changed, so the words a rule quotes can always be produced.
- Read, with no model. Text from the letters and their positions; tables from their ruled borders. Front page, table of contents, running titles and page numbers are dropped. Clauses split by a page break are joined. Every clause gets an id such as
§ 5, stk. 2. - Extracted. One request per kind of rule, each with a strict schema. The agreement is passed as data. The model returns typed rules, each naming its clause and quoting a sentence from it.
- Checked, with no model. The clause exists; the quote stands in it word for word; every number and date is printed where the rule says; values are in range. Missing kinds, uncovered days and conflicting rules are reported for the run.
- Reviewed. A specialist sees each rule beside its clause with the quote marked, and what publishing would change against the rules already published. They approve or reject each rule, then the run.
- Published. The database publishes the approved rules as the agreement's next version, and only if an approval is on record. A version is never edited; a later one takes over from the day it starts.
A payslip is then checked by arithmetic: the engine works out what the period should have paid, minute by minute, and reports every line that differs with the rule and clause behind it.
| What | Where |
|---|---|
| Eighteen agreements as PDF, text and structure, with the rules each states. Six are about 24 pages long | data/synthetic |
| Six months of timesheets and payslips for each, with nine kinds of seeded payroll mistake and the answers | data/synthetic/*/pay.json |
| The generators | src/Synthetic |
| Recorded model answers | evals/recordings |
The corpus is generated, never edited by hand: dotnet run --project tools/Corpus writes it, and a test fails if the committed files differ. The organisations and agreement names are invented, and each PDF says so on its front page. The PDFs use Liberation Serif, under the SIL Open Font License.
| Path | What is there |
|---|---|
src/Rules |
Rule records, the pay engine, the validator, versions and comparison. No dependencies, no model |
src/Documents |
The structure of an agreement: sections, clauses, text and tables |
src/Ingestion |
PDF to clauses, the extraction prompt and schemas, the checks, the pipeline's steps |
src/Agent |
The finding explainer and its guard; the question agent, its tools and the answer check |
src/Gateway |
Model access: redaction, failover, budget, usage, replay, telemetry |
src/Data |
The schema in SQL, the migrator, tenant-scoped access |
src/Worker |
The durable orchestration and the starter for runs that were never dispatched |
src/Api |
The API and the console |
src/AppHost |
The local system: Postgres, scheduler, Keycloak, worker, API |
src/Synthetic |
Generators for agreements, PDFs and pay data |
evals |
The eval runner and the recorded answers |
tests |
One project per source project |
tools |
Corpus writer, migrator and admin commands, the browser check and walkthrough recorder |
infra |
The Azure infrastructure file, the same system as containers with local stand-ins, the demo sign-in realm |
docs |
Decision records, eval report, threat model, cut list, a note on the EU AI Act |
- The model never calculates
- The rules come first and the documents are written from them
- A person publishes, and the database insists
- Every proposed rule is checked against the agreement with no model
- Durable Task for the pipeline
- Own agent loop instead of a framework
- An answer is shown only if it can be checked
- Rule sets are never changed, and the newest version takes over by date
- Pay data is checked and not kept
- The audit log is written by the database, and telemetry carries no content
- One model provider, and no real agreements
See also Running on Azure, the threat model, what was left out, and why, and the EU AI Act and this system. To report a vulnerability, see SECURITY.md; to change something, CONTRIBUTING.md.
- Synthetic documents only. A published agreement is its parties' copyrighted text. None is downloaded, read or sent to a model, so nothing here is evidence about real agreements.
- A value must be printed, or be one of two things code can work out. An increase on an earlier rate and a share stated as a part of another figure are reported by the model as printed and worked out by code. Anything else a model derived is flagged.
- Six kinds of rule. Minimum wage, overtime, inconvenient hours, pension, holiday allowance and special savings. A real agreement says more about pay than that.
- One model provider. Failover is between Claude models, which covers a model short of capacity and not the provider being down.
- Not yet run on Azure. The deployment is defined and checked without an account: the infrastructure file compiles, the images build, and the system passes its smoke test from those images. See Running on Azure. On one machine it runs with development settings: plain HTTP, fixed demo passwords and a default Keycloak admin.
- A reviewer approves or rejects. Editing a proposed rule is not built.
- A small console. One page, in English, written without a framework or a build step. Agreements, findings and answers are in Danish, and the agreement's own words are set apart from the interface as print.