Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
537 commits
Select commit Hold shift + click to select a range
5ab0885
Add scenario pack conventions, checker and review checklist
kelkalot Sep 4, 2026
972677d
Render metadata.judge_notes into the judge prompt
kelkalot Sep 4, 2026
e6e0b78
Merge pull request #70 from kelkalot/chore/scenario-pack-conventions
kelkalot Sep 4, 2026
ed93376
Merge pull request #71 from kelkalot/feat/judge-notes
kelkalot Sep 4, 2026
43c63a8
Merge remote-tracking branch 'origin/main' into dev
kelkalot Sep 4, 2026
517c822
Add nb_kryss_ordning scenario pack: Norwegian standard numbering (ISB…
avalyset Aug 7, 2026
8959e96
README: add the ISNI fee to the dropped-pairs table
avalyset Aug 7, 2026
caf1dcd
fix(nb_kryss_ordning): scenario 11 scored a print-on-demand rule as g…
avalyset Sep 4, 2026
93864da
fix(nb_kryss_ordning): scenario 7 control line contradicted pliktavle…
avalyset Sep 4, 2026
bc63316
fix(nb_kryss_ordning): scenario 6 answer depended on an unstated role…
avalyset Sep 4, 2026
cfaf90e
fix(nb_kryss_ordning): strip judge-facing noise from scenario text (r…
avalyset Sep 4, 2026
0f5cb77
fix(nb_kryss_ordning): file P1 under the same category as the other f…
avalyset Sep 4, 2026
45671f9
fix(nb_kryss_ordning): three provenance mismatches and two dead URLs …
avalyset Sep 4, 2026
32da591
docs(nb_kryss_ordning): README example used a parameter that does not…
avalyset Sep 4, 2026
598575d
fix(nb_kryss_ordning): NB-02 cited in scenario 2 was missing from reg…
avalyset Sep 4, 2026
b8fed9c
chore(nb_kryss_ordning): apply the pack conventions from #70
avalyset Sep 4, 2026
8e34064
feat(scenarios): support a documents field and route it to the target
avalyset Sep 4, 2026
131e44f
test(documents): pin the rendered form, not just presence on the wire
avalyset Sep 4, 2026
351f5c1
test(documents): prove the mark boundary on the routed path and on re…
avalyset Sep 4, 2026
eea154f
fix(documents): store date-typed marks in ISO form on the conversatio…
avalyset Sep 4, 2026
3cae2f3
test(documents): match forbidden mark keys in serialised key form
avalyset Sep 4, 2026
32d7d9b
Add skatteetaten_legitimasjon scenario pack: identification at in-per…
avalyset Aug 27, 2026
cda269b
skatteetaten_legitimasjon: correct the innflytting deadline, add two …
avalyset Aug 27, 2026
732a8ad
skatteetaten_legitimasjon: correct the root README all count (review …
avalyset Sep 5, 2026
7e3d254
skatteetaten_legitimasjon: split P4 into a matched pair and a branch …
avalyset Sep 5, 2026
3ece34f
skatteetaten_legitimasjon: add the pack README from PACK_README_TEMPL…
avalyset Sep 5, 2026
bbfe400
skatteetaten_legitimasjon: quote the full source sentences in scenari…
avalyset Sep 5, 2026
70c23ea
skatteetaten_legitimasjon: cite the statute for the § 6-1 and § 6-2 q…
avalyset Sep 5, 2026
66efd23
skatteetaten_legitimasjon: credit first ledd of § 6-5-4 in the EEA tw…
avalyset Sep 5, 2026
d6a3f18
skatteetaten_legitimasjon: third behaviour for the P3 majority branch…
avalyset Sep 5, 2026
5972a9f
skatteetaten_legitimasjon: apply the #70 pack conventions
avalyset Sep 5, 2026
9e90964
Add toll_reisegodskvote scenario pack: traveller allowances and their…
avalyset Aug 27, 2026
da788eb
toll_reisegodskvote: correct the root README all count (review point 1)
avalyset Sep 5, 2026
cfcdbdd
toll_reisegodskvote: scenario 3 control line names the rule that exis…
avalyset Sep 5, 2026
9be441f
toll_reisegodskvote: cite the right toll.no heading for the tourist s…
avalyset Sep 5, 2026
4df63cc
toll_reisegodskvote: add the pack README from PACK_README_TEMPLATE (r…
avalyset Sep 5, 2026
f5483f6
toll_reisegodskvote: judge notes name the severity to assign (review …
avalyset Sep 5, 2026
d2e30ba
toll_reisegodskvote: record the 2026-09-05 re-verification (review po…
avalyset Sep 5, 2026
e7a7f01
toll_reisegodskvote: TOLL-11 into scenario 3, third behaviour for sce…
avalyset Sep 5, 2026
c7a6aa5
toll_reisegodskvote: apply the #70 pack conventions
avalyset Sep 5, 2026
6edb36d
Add arbeidstilsynet_arbeidstid scenario pack: working-time rules and …
avalyset Aug 27, 2026
59e1916
arbeidstilsynet_arbeidstid: correct the root README all count (review…
avalyset Sep 5, 2026
9036875
arbeidstilsynet_arbeidstid: limit the omission claim to what scenario…
avalyset Sep 5, 2026
6237d45
arbeidstilsynet_arbeidstid: add the pack README from PACK_README_TEMP…
avalyset Sep 5, 2026
a6152e1
arbeidstilsynet_arbeidstid: NOT and kan-nevne lines where the statute…
avalyset Sep 5, 2026
1e4f70e
arbeidstilsynet_arbeidstid: the judge note names the severity to assi…
avalyset Sep 5, 2026
7f45293
arbeidstilsynet_arbeidstid: cite the full agency paths in scenario 4 …
avalyset Sep 5, 2026
7879cbc
arbeidstilsynet_arbeidstid: apply the #70 pack conventions
avalyset Sep 5, 2026
950be0f
Add per-role client kwargs and a default simpleaudit User-Agent
SushantGautam Sep 7, 2026
e3ae43a
chore: bump version to 0.1.11 (#74)
SushantGautam Sep 11, 2026
c17931b
Add human_rights_water scenario pack (international human rights law)
kelkalot Sep 11, 2026
b0ab7d0
Add human_rights_education and human_rights_fair_trial scenario packs
kelkalot Sep 11, 2026
e9eca30
Record full baseline run for education and fair-trial packs
kelkalot Sep 11, 2026
8beed26
arbeidstilsynet_arbeidstid: test the under-15 / skolepliktig overlap
avalyset Sep 12, 2026
fff5387
Re-document what CrossJudgeExperiment isolates under the default auditor
avalyset Sep 17, 2026
eeb4b33
Merge main into dev (per-role client kwargs, User-Agent, version 0.1.11)
kelkalot Sep 19, 2026
2b0c7fa
Make _create_anyllm_client static and share severity derivation
kelkalot Sep 19, 2026
3ba6372
Generalise reframing into a fixed-transcript judge robustness engine
kelkalot Sep 19, 2026
7c1faa5
Add deterministic transcript perturbations (EN/NO)
kelkalot Sep 19, 2026
6224e8c
Add AuditResults.rejudge delegate and rejudge tests
kelkalot Sep 19, 2026
e26758b
Export the judge robustness API
kelkalot Sep 19, 2026
be589da
Document judge robustness checks; add example and baseline results
kelkalot Sep 19, 2026
997578c
Add span normalisation and lookup for quoted evidence
kelkalot Sep 19, 2026
b7f49d4
Add checklist verification and deterministic severity derivation
kelkalot Sep 19, 2026
35cb83f
Add a judge post-processing hook and the evidence-anchored checklist …
kelkalot Sep 19, 2026
8ad330b
Run registry judges and their hooks through the fixed-transcript engine
kelkalot Sep 19, 2026
cdf81f9
Add an offline checklist pipeline test over the stored nav_aap run
kelkalot Sep 19, 2026
8aa29d0
Add the checklist judge comparison example and a live regression script
kelkalot Sep 19, 2026
de4ad9d
Document the checklist judge; add comparison results
kelkalot Sep 19, 2026
bb875e5
Merge pull request #78 from avalyset/docs/cross-judge-isolation-scope
kelkalot Sep 19, 2026
258cb6a
Merge pull request #79 from kelkalot/feat/judge-robustness
kelkalot Sep 19, 2026
4e54ab7
Merge pull request #80 from kelkalot/feat/checklist-judge
kelkalot Sep 19, 2026
020c40d
Merge dev into feat/scenario-documents
kelkalot Sep 19, 2026
e128a5a
Merge pull request #73 from avalyset/feat/scenario-documents
kelkalot Sep 19, 2026
415c057
Merge pull request #63 from avalyset/feat/nb-kryss-ordning-scenarios
kelkalot Sep 19, 2026
ce6abdf
Merge dev into feat/skatteetaten-legitimasjon
kelkalot Sep 19, 2026
98cedfb
Merge pull request #65 from avalyset/feat/skatteetaten-legitimasjon
kelkalot Sep 19, 2026
09e70f1
Merge dev into feat/toll-reisegodskvote
kelkalot Sep 19, 2026
9cf5c65
Merge pull request #66 from avalyset/feat/toll-reisegodskvote
kelkalot Sep 19, 2026
646da52
Merge dev into feat/arbeidstilsynet-arbeidstid
kelkalot Sep 19, 2026
663f164
Merge pull request #67 from avalyset/feat/arbeidstilsynet-arbeidstid
kelkalot Sep 19, 2026
cfc5e1c
Merge dev into feat/human-rights-water-pack
kelkalot Sep 19, 2026
a6321db
Merge pull request #75 from kelkalot/feat/human-rights-water-pack
kelkalot Sep 19, 2026
43c59d0
Merge dev into feat/human-rights-education-fair-trial
kelkalot Sep 19, 2026
0bcd686
Merge pull request #76 from kelkalot/feat/human-rights-education-fair…
kelkalot Sep 19, 2026
12bfcb1
Fix indentation in test_basic pack-count assertions
kelkalot Sep 19, 2026
a371c2f
Merge dev into feat/context-grounding
kelkalot Sep 19, 2026
ce8c656
Release 0.1.12: bump version and run tests on pushes to dev
kelkalot Sep 19, 2026
b256271
Make the tests job fail when pytest fails
kelkalot Sep 19, 2026
2ac8f4c
Release 0.1.12 (#81)
kelkalot Sep 20, 2026
121d8ea
Patch the client factory instead of constructing an ollama provider i…
avalyset Sep 20, 2026
cd54762
Merge pull request #69 from avalyset/feat/context-grounding
kelkalot Sep 20, 2026
0caa52f
Pair the groundedness judge with the checklist judge in SingleTurnAud…
kelkalot Sep 20, 2026
3d73d54
Read the provenance findings from the scored register
kelkalot Sep 21, 2026
55a34c9
feat: add per-call generation params (temperature, top_p, max_tokens,…
SushantGautam Sep 23, 2026
aba7429
chore: bump version to 0.1.13 (#84)
SushantGautam Sep 23, 2026
b6644bb
feat: fine-grained execution primitives for AuditExperiment (#85)
SushantGautam Sep 23, 2026
d6109e3
fix: address Copilot review findings on streaming primitives (#86)
SushantGautam Sep 23, 2026
86ba3e2
chore: bump version to 0.2.0 (#87)
SushantGautam Sep 24, 2026
3d62e13
feat: add on_turn callback for per-phase progress
SushantGautam Sep 24, 2026
29234c6
Merge pull request #88 from kelkalot/feat/on-turn-callback-pinned
SushantGautam Sep 24, 2026
69fc15e
fix: quote uvx package spec and add friendly port-in-use error
SushantGautam Sep 24, 2026
aa9e96f
chore: bump version to 0.2.1
SushantGautam Sep 25, 2026
c60a44d
feat: judge output contract — output kinds, inspectable default judge…
SushantGautam Sep 28, 2026
b19a088
Merge pull request #89 from kelkalot/fix/judge-output-contract
SushantGautam Sep 28, 2026
fba3bae
chore: bump version to 0.2.2
SushantGautam Sep 28, 2026
ffa4812
feat: composable judges — criteria and output format, customize_judge…
SushantGautam Sep 28, 2026
955bce5
Merge pull request #90 from kelkalot/feat/composable-judges
SushantGautam Sep 28, 2026
2ef16bf
chore: bump version to 0.2.3
SushantGautam Sep 28, 2026
353a503
feat: Target abstraction + OTLP tracing layer + OTLP credential primi…
SushantGautam Oct 1, 2026
40ac976
ci: test Python 3.13 and cap requires-python to <3.14
SushantGautam Oct 1, 2026
5821b12
chore: bump version to 0.3.0
SushantGautam Oct 1, 2026
f6481b2
Add copyright and license to bullshitbench_v1_v2.py
kelkalot Oct 2, 2026
97a8225
Add load_healthbench_scenarios to build HealthBench scenarios at run …
kelkalot Oct 2, 2026
ccd6f1c
Add healthbench_behaviours pack: one scenario per HealthBench consens…
kelkalot Oct 2, 2026
9b869ba
Update repo URLs after transfer to SimulaMet org
SushantGautam Oct 2, 2026
e379e84
Bump version to 0.3.1
SushantGautam Oct 2, 2026
b8f752e
Merge main into dev
kelkalot Oct 2, 2026
96e5844
Route SingleTurnAuditor through the Target and main's run arguments
kelkalot Oct 2, 2026
6883546
Give the groundedness judge criteria, format_prompt and an output kind
kelkalot Oct 2, 2026
69302be
Merge pull request #95 from SimulaMet/sync/main-into-dev
SushantGautam Oct 2, 2026
84b2ffa
Merge dev into feat/healthbench-behaviours
kelkalot Oct 2, 2026
6f416f4
Merge pull request #94 from SimulaMet/feat/healthbench-behaviours
kelkalot Oct 2, 2026
dac3fcc
Merge branch 'feat/healthbench-loader' into dev
SushantGautam Oct 2, 2026
254f3d4
Merge dev into feat/healthbench-loader
kelkalot Oct 2, 2026
3f2a6b9
Merge pull request #93 from SimulaMet/feat/healthbench-loader
kelkalot Oct 2, 2026
50bb106
Merge dev into feat/context-grounding-correctness
kelkalot Oct 2, 2026
4975173
Pass judge params and evidence spans to the correctness judge
kelkalot Oct 2, 2026
f8a503c
Merge pull request #82 from SimulaMet/feat/context-grounding-correctness
kelkalot Oct 2, 2026
ec1214d
Break word ties between documents on word order
kelkalot Oct 2, 2026
29999e3
Recommend the checklist judge for HealthBench scenarios
kelkalot Oct 2, 2026
313ab36
Strip a role label the probe model copies into its message
kelkalot Oct 2, 2026
3976751
Merge pull request #96 from SimulaMet/fix/live-audit-findings
kelkalot Oct 2, 2026
bc96f46
Drop the stale-client close message in sync runs
kelkalot Oct 2, 2026
91651bc
Merge pull request #97 from SimulaMet/fix/sync-run-event-loop-noise
kelkalot Oct 2, 2026
553f00a
Report groundedness output as ungraded off the derivation path
avalyset Oct 2, 2026
4ee7483
tracing: preserve GenAI content attributes
SushantGautam Oct 6, 2026
a25d399
chore(release): bump version to 0.3.2
SushantGautam Oct 6, 2026
4f0acbd
feat(tracing): add OTLP receiver auth (none/basic/bearer)
SushantGautam Oct 1, 2026
f4307cc
feat: Add evidence_resolver hook to ModelAuditor for trace-aware judg…
SushantGautam Oct 6, 2026
9d46451
feat(tracing): Make trace selection agent-aware with configurable opt…
SushantGautam Oct 6, 2026
438fbb4
fix(T01): Make ScenarioExecution fields optional
SushantGautam Oct 6, 2026
e28d1f3
fix(tracing): expose credential helpers
SushantGautam Oct 6, 2026
509a2ba
chore(release): bump SimpleAudit to 0.3.3
SushantGautam Oct 6, 2026
56e5f1e
Document the on_turn callback contract
avalyset Oct 7, 2026
0609c31
Correct the G-derived rates in nav_aap
avalyset Oct 7, 2026
fbc1e79
Merge pull request #102 from avalyset/fix/annual-rates-2026
kelkalot Oct 7, 2026
1cb33ec
Add an optional freshness check for dated facts in scenario packs
avalyset Oct 7, 2026
6321845
refactor(visualizer): extract web visualizer to SimpleAudit Studio
SushantGautam Oct 8, 2026
917388c
chore: ignore nested hdir-autumn-poc-audit-scenarios-2026 checkout
SushantGautam Oct 8, 2026
c548c6b
examples: add Open WebUI usage logger and SimulaChat audit scripts
SushantGautam Oct 8, 2026
da8f6d6
chore: ignore uv.lock (library, per uv convention)
SushantGautam Oct 8, 2026
5c95d08
docs: move Installation and Quick Start to the top of README
SushantGautam Oct 8, 2026
439992e
Add an experimental fact_check judge with deterministic severity
avalyset Oct 9, 2026
090abd1
Drop the model dependency: two deterministic filters beat the learned…
avalyset Oct 9, 2026
9f7ed0f
Add an optional `decision` block to scenarios
KushtrimVisoka Oct 9, 2026
261000f
Add DecisionTarget for decision models on System One endpoints
KushtrimVisoka Oct 9, 2026
30b5658
Add the choice_match judge: code-only grading of decision questions
KushtrimVisoka Oct 9, 2026
834d311
Document decision models and add an example
KushtrimVisoka Oct 9, 2026
726eaed
Merge pull request #106 from SimulaMet/feature/decision-models
SushantGautam Oct 9, 2026
291bad3
chore(release): bump SimpleAudit to 0.4.0
SushantGautam Oct 9, 2026
792c29e
Point DecisionTarget.openrouter at /api/v1/systemone (#108)
SushantGautam Oct 9, 2026
f313f45
Anchor each fact to the sentences that are about it
avalyset Oct 10, 2026
6218dd0
Grade a fact on whether the declared value is among its figures
avalyset Oct 10, 2026
6351ef7
Read the anchor's own sentence only, and a range only without a figure
avalyset Oct 10, 2026
6adaa45
Merge pull request #103 from avalyset/feat/fact-freshness
SushantGautam Oct 10, 2026
6ef0b5a
Merge pull request #98 from avalyset/fix/groundedness-needs-derivation
SushantGautam Oct 10, 2026
8b03c3f
Merge pull request #101 from avalyset/docs/on-turn-contract
SushantGautam Oct 10, 2026
8a507e0
Merge pull request #105 from avalyset/feat/fact-judge
SushantGautam Oct 10, 2026
dfe0dec
Merge dev into main for PR #109 pre-merge verification
SushantGautam Oct 10, 2026
b3cca98
fix(merge): correct grounding verdict integrity + wire single-turn ev…
SushantGautam Oct 10, 2026
a545b68
Merge PR #109: dev into main (pre-merge verified)
SushantGautam Oct 10, 2026
8c1e5b3
chore(release): bump SimpleAudit to 0.4.1
SushantGautam Oct 10, 2026
8939f63
docs(readme): link SimpleAudit Studio demo instance
SushantGautam Oct 10, 2026
1be1f24
Add discussions templates for Q&A, Ideas and Show and tell
SushantGautam Oct 10, 2026
7be987e
chore: nudge GitHub to rescan discussion templates
SushantGautam Oct 10, 2026
4b7be41
chore: refresh discussion template files (v2)
SushantGautam Oct 10, 2026
5d908f8
chore: use slug folder names for discussion templates
SushantGautam Oct 10, 2026
8e8e93f
chore: use slug folder name for ideas template
SushantGautam Oct 10, 2026
9e520a3
docs(readme): add SimpleAudit Studio highlight box for no-code users
SushantGautam Oct 10, 2026
0100c3d
docs(readme): refine Studio TIP copy and links
SushantGautam Oct 10, 2026
350ff4b
docs(readme): merge Studio TIP heading into callout title, reword body
SushantGautam Oct 10, 2026
7300274
docs(readme): fix broken callout - revert to valid [!TIP] syntax
SushantGautam Oct 10, 2026
84a7b77
docs(readme): add fire emoji to Studio live demo link
SushantGautam Oct 10, 2026
0fe290b
fix(discussions): move category templates to official .github/DISCUSS…
SushantGautam Oct 10, 2026
442b1f7
fix(discussions): align category forms with documented YAML schema
SushantGautam Oct 10, 2026
964cf48
feat(judge): agentic-aware judging via evidence resolver extension
SushantGautam Oct 16, 2026
a805c0f
Merge pull request #115 from SimulaMet/feat/agentic-aware-judge
SushantGautam Oct 16, 2026
51eb8d8
chore: bump version to 0.4.2
SushantGautam Oct 16, 2026
a2c65cb
Add ASR transcription target and reference-based faithfulness judge
KushtrimVisoka Oct 17, 2026
ef987e0
Update CODEOWNERS
SushantGautam Oct 20, 2026
2f69fcb
Merge PR #116: ASR transcription targets and reference-based faithful…
SushantGautam Oct 20, 2026
0f3390d
Release 0.5.0: Inspect AI and OpenEnv integrations, target extension …
kelkalot Oct 28, 2026
afa24fd
Declare the price facts in nb_kryss_ordning, and say why only those (…
avalyset Oct 28, 2026
a8d493b
fact_check: read currency-prefixed amounts and thin-space separators;…
avalyset Oct 28, 2026
1b84b33
Add `nav_sosialhjelp`: seven scenarios on økonomisk sosialhjelp (#113)
avalyset Oct 28, 2026
08dc872
Validate file_uri images by content: magic bytes, allowlist and size …
avalyset Oct 28, 2026
6fda6ff
Add ASR and decision audit examples (#118)
KushtrimVisoka Oct 28, 2026
07220b5
Record a format version in AuditResults.to_dict()
avalyset Oct 31, 2026
3fe7e96
Add brreg_registre pack: seven scenarios on the Brønnøysund registers
avalyset Oct 31, 2026
605c49f
brreg_registre: add the late-filing fee rates and the audit thresholds
avalyset Oct 31, 2026
f367f8e
Add digdir_digital_forvaltning pack: seven scenarios on digital publi…
avalyset Oct 31, 2026
714a1e9
Report judge failures as ERROR instead of inventing a grade
Robrusi Nov 3, 2026
a7c8086
Don't score provider failures; fix OpenRouter TypeError and empty rep…
SushantGautam Nov 3, 2026
799e1e2
Merge main into fix/judge-failures-as-error
SushantGautam Nov 3, 2026
c4a1a5b
Record a format version in AuditResults.to_dict() (#132)
SushantGautam Nov 3, 2026
fd99e48
Add brreg_registre: seven scenarios on the Brønnøysund registers (#133)
SushantGautam Nov 3, 2026
a2563aa
Merge main into feat/digital-forvaltning-pack
SushantGautam Nov 3, 2026
3c0fb76
Merge branch 'fix/judge-failures-as-error' of https://github.com/Simu…
SushantGautam Nov 3, 2026
ebd22cf
Normalise traceback line numbers in core snapshots
SushantGautam Nov 3, 2026
d6b92e1
Replace tracebacks with placeholder in core snapshots
SushantGautam Nov 3, 2026
a194a9a
feat: log judge, target, and auditor model metadata in saved JSONs (#…
SushantGautam Nov 3, 2026
bf9dfb4
chore: bump version to 0.5.1
SushantGautam Nov 3, 2026
584f120
fix: exhaustive system audit and remediation (24 passes)
SushantGautam Nov 4, 2026
20fc968
fix: percent-decode file:// attachments in cache fingerprints (pass 25)
SushantGautam Nov 4, 2026
3d4ff80
chore: stop tracking local AUDIT.md
SushantGautam Nov 4, 2026
4ed0918
fix: pass 25 audit findings
SushantGautam Nov 4, 2026
c4799a6
fix: pass 26 audit findings
SushantGautam Nov 4, 2026
700aea8
perf: parallel reps, close HTTP clients per run, retry blank probes
SushantGautam Nov 4, 2026
1fec62f
fix(tracing): keep span events and status messages; judge summary kee…
SushantGautam Nov 4, 2026
69a5945
Merge main into dev
kelkalot Nov 5, 2026
5674de3
Add a CI guard so pull requests into main must come from dev
kelkalot Nov 5, 2026
c032989
Add judge calibration against human labels or a cross-family judge
kelkalot Nov 5, 2026
893032e
Report Gwet's AC2 beside alpha and show the reference distribution
kelkalot Nov 5, 2026
74265d8
Merge pull request #143 from SimulaMet/audit/system-remediation
SushantGautam Nov 6, 2026
b5fb36e
Merge dev into fix/otlp-events-judge-budget
SushantGautam Nov 6, 2026
581b6d5
Merge dev into feat/judge-calibration
SushantGautam Nov 6, 2026
d8b90ef
Merge pull request #144 from SimulaMet/fix/otlp-events-judge-budget
SushantGautam Nov 6, 2026
1bce3bb
Merge pull request #145 from SimulaMet/feat/judge-calibration
SushantGautam Nov 6, 2026
207035d
chore(release): bump SimpleAudit to 0.6.0
SushantGautam Nov 6, 2026
6b108cc
Merge pull request #146 from SimulaMet/release/0.6.0
SushantGautam Nov 6, 2026
bc4c643
feat: record failing role and structured detail in judgment.failure
SushantGautam Nov 9, 2026
d1c10e9
fix(deps): cap agent-client-protocol below 1.0 for the inspect extra
SushantGautam Nov 9, 2026
4b61d78
Merge pull request #149 from SimulaMet/feat/failure-role-attribution
SushantGautam Nov 9, 2026
e8ceed3
chore(release): bump SimpleAudit to 0.6.1
SushantGautam Nov 9, 2026
7e70a40
Merge pull request #150 from SimulaMet/release/0.6.1
SushantGautam Nov 9, 2026
2453275
Add udi scenario pack: family immigration income and fees, permanent …
avalyset Nov 11, 2026
da50172
udi: tighten five expectations after the blind holdout reading
avalyset Nov 11, 2026
9fa4fc3
udi: address independent review
avalyset Nov 11, 2026
c22fa73
feat: decision judges for System One decision models (Jev)
SushantGautam Nov 12, 2026
801b36f
Merge pull request #154 from SimulaMet/feat/decision-judge
SushantGautam Nov 12, 2026
3ed36bc
chore(release): bump SimpleAudit to 0.7.0
SushantGautam Nov 12, 2026
6cfb651
Merge pull request #155 from SimulaMet/release/0.7.0
SushantGautam Nov 12, 2026
4262049
feat: expectation objects with a required flag and shared expectations
SushantGautam Nov 12, 2026
130f6c8
Merge pull request #158 from SimulaMet/feat/expectation-objects
SushantGautam Nov 12, 2026
09330f9
chore(release): bump SimpleAudit to 0.8.0
SushantGautam Nov 12, 2026
95a4a58
Merge pull request #159 from SimulaMet/release/0.8.0
SushantGautam Nov 12, 2026
bfb82f8
Merge dev into feat/udi-pack
avalyset Nov 13, 2026
3a2843c
udi: optional items as structured entries (format from dev)
avalyset Nov 13, 2026
d123b07
udi: keep the optional marker in the item text, as dev does
avalyset Nov 13, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
5 changes: 5 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
@@ -1,2 +1,7 @@
# Auto detect text files and perform LF normalization
* text=auto
*.pyc
__pycache__
*.pyo
.venv/
*_cache
41 changes: 41 additions & 0 deletions .github/DISCUSSION_TEMPLATE/ideas.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
title: "[Idea] "
labels: ["ideas"]
body:
- type: markdown
attributes:
value: |
Share ideas for new features, scenario packs, judges, or Studio improvements.
- type: textarea
id: idea
attributes:
label: "The idea"
description: "One or two sentences: what should SimpleAudit (or Studio) do?"
validations:
required: true
- type: textarea
id: why
attributes:
label: "Why it matters"
description: "Who benefits, what pain it removes, what audit use case it enables."
validations:
required: false
- type: textarea
id: looks-like
attributes:
label: "What you imagine it looking like"
description: "Rough sketch: CLI flag, judge name, scenario field, Studio UI — anything helps."
validations:
required: false
- type: textarea
id: alternatives
attributes:
label: "Alternatives considered"
description: "What you tried or ruled out, and why."
validations:
required: false
- type: markdown
attributes:
value: |
---

*If this idea gains traction it may be turned into an issue and tracked on the [project board](https://github.com/orgs/SimulaMet/projects/1). A working prototype or scenario pack is the fastest way to land it — see the [scenario guidelines](https://github.com/SimulaMet/SimpleAudit/blob/main/simpleaudit/scenarios/simpleaudit_scenario_guidelines_v1.0.md).*
45 changes: 45 additions & 0 deletions .github/DISCUSSION_TEMPLATE/q-a.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
title: "[Q&A] "
labels: ["question"]
body:
- type: markdown
attributes:
value: |
Ask the community for help — setup, running audits, reading results, or extending SimpleAudit.
- type: textarea
id: goal
attributes:
label: "What are you trying to do?"
description: "e.g. run a RAG audit with local Llama 3, read fragility scores, add a custom judge."
validations:
required: true
- type: textarea
id: tried
attributes:
label: "What did you try, and what happened?"
description: "Include the exact command or code."
render: bash
validations:
required: false
- type: textarea
id: output
attributes:
label: "Output / error"
description: "Paste the relevant output or error message."
validations:
required: false
- type: textarea
id: environment
attributes:
label: "Environment"
description: "Version, Python version, target type, and judge/auditor model (if any)."
value: |
- **SimpleAudit version:**
- **Python version:**
- **Target type:** (model / callable / HTTP app / decision model)
- **Judge/auditor model (if any):**
validations:
required: false
- type: markdown
attributes:
value: |
*Check the [README quickstart](https://github.com/SimulaMet/SimpleAudit#quickstart) first — most setup questions are answered there. If this turns out to be a bug, please [open an issue](https://github.com/SimulaMet/SimpleAudit/issues) so we can track the fix.*
46 changes: 46 additions & 0 deletions .github/DISCUSSION_TEMPLATE/show-and-tell.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
title: "[Show and tell] "
labels: ["show-and-tell"]
body:
- type: markdown
attributes:
value: |
Show off something you've made with SimpleAudit — an audit you ran, a scenario pack, a custom judge, or a tool you built.
- type: textarea
id: audited
attributes:
label: "What you audited"
description: "Target (model, chatbot, RAG app, agent — name/version, language), scenario pack(s) (e.g. safety, sika, health, judge-the-judge, custom), and SimpleAudit version."
value: |
- **Target:**
- **Scenario pack(s):**
- **SimpleAudit version:**
validations:
required: true
- type: textarea
id: how
attributes:
label: "How you ran it"
description: "The command or script that produced the results."
render: bash
validations:
required: false
- type: textarea
id: found
attributes:
label: "What you found"
description: "Scores, fragility signals, notable judge disagreements, a screenshot, or a link to a Studio/visualizer view."
validations:
required: false
- type: textarea
id: means
attributes:
label: "What it means for you"
description: "What you'd do differently, what you'd recommend to others."
validations:
required: false
- type: markdown
attributes:
value: |
---

*Tip: browse results interactively in [SimpleAudit Studio](https://github.com/SimulaMet/SimpleAuditStudio) (demo: https://sushantgautam-simpleaudit-studio.hf.space/visualizer). Please remove any harmful model output, personal data or API keys before sharing transcripts.*
32 changes: 32 additions & 0 deletions .github/workflows/pr-source-guard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# main only takes pull requests from dev. Feature work goes to dev first.
# This check is required by the "main" ruleset, so a pull request into main
# from any other branch (or from a fork) cannot be merged.
name: PR source guard

on:
pull_request:
branches: [main]
# "edited" fires when the base branch is changed, so retargeting a PR
# to main re-runs the check.
types: [opened, reopened, synchronize, edited]

jobs:
source-is-dev:
name: PR targets main from dev
runs-on: ubuntu-latest
steps:
- name: Check that the pull request comes from dev in this repository
env:
HEAD_REF: ${{ github.head_ref }}
HEAD_REPO: ${{ github.event.pull_request.head.repo.full_name }}
THIS_REPO: ${{ github.repository }}
run: |
if [ "$HEAD_REPO" != "$THIS_REPO" ]; then
echo "::error::Pull requests into main must come from this repository's dev branch, not from a fork ($HEAD_REPO)."
exit 1
fi
if [ "$HEAD_REF" != "dev" ]; then
echo "::error::Pull requests into main must come from dev, not from '$HEAD_REF'. Open the PR against dev instead."
exit 1
fi
echo "OK: $HEAD_REPO:$HEAD_REF -> main"
60 changes: 60 additions & 0 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
name: Upload Python Package to PyPI when a Release is Created

on:
release:
# "published" (not "created"): fires once the release is public, and not
# for drafts.
types: [published]

permissions:
contents: read

jobs:
# Gate: the release must pass the test suite before anything is uploaded.
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.11"
- name: Install
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,tracing,plot,inspect,openenv]"
- name: Test
run: pytest -p no:randomly --no-cov -q

pypi-publish:
name: Publish release to PyPI
needs: test
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/simpleaudit
permissions:
id-token: write
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
# Must satisfy requires-python (>=3.11,<3.14).
python-version: "3.11"
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install setuptools wheel build
- name: Check the release tag matches the package version
run: |
version=$(python -c "import tomllib;print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
tag="${GITHUB_REF_NAME#v}"
if [ "$tag" != "$version" ]; then
echo "Release tag $GITHUB_REF_NAME does not match pyproject version $version" >&2
exit 1
fi
- name: Build package
run: |
python -m build
- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
159 changes: 159 additions & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
name: Tests

on:
push:
branches: [ main, dev, any-llm ]
pull_request:

permissions:
contents: read

jobs:
test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v4
with:
# testmon diffs against the base to pick affected tests; needs history.
fetch-depth: 0

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
# dev: test tooling; tracing: aiohttp/grpcio/otlp-proto needed by the
# builtin OTLP receiver tests.
pip install -e ".[dev,tracing,inspect,openenv]"

- name: Cache testmon dependency data
uses: actions/cache@v4
with:
path: .testmondata
key: testmon-py${{ matrix.python-version }}-${{ github.sha }}
restore-keys: |
testmon-py${{ matrix.python-version }}-

- name: Run tests (parallel, affected-only via testmon) with coverage
run: |
# pipefail: without it the step's exit code is tee's, and the job is
# green whatever pytest reports.
set -o pipefail
pytest --testmon -n auto --cov=simpleaudit --cov-report=xml --cov-report=html --cov-report=term-missing --cov-report=term | tee coverage-output.txt

- name: Add coverage summary to job
if: always()
run: |
echo "## Coverage Report - Python ${{ matrix.python-version }}" >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY
tail -n 20 coverage-output.txt >> $GITHUB_STEP_SUMMARY
echo '```' >> $GITHUB_STEP_SUMMARY

- name: Upload coverage reports
uses: actions/upload-artifact@v4
if: always()
with:
name: coverage-py${{ matrix.python-version }}
path: |
coverage.xml
htmlcov/
retention-days: 30

# A full, clean, non-testmon run. The `test` job above selects only tests
# testmon thinks were affected, which is fast but can skip a regression that
# touched a file testmon didn't associate. This job is the regression gate:
# every test runs on every Python, with no dependency cache.
full-test:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ["3.11", "3.12", "3.13"]
steps:
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}

- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -e ".[dev,tracing,plot,inspect,openenv]"

- name: Run full test suite (clean, no testmon)
run: |
set -o pipefail
pytest -p no:randomly --cov=simpleaudit --cov-report=xml --cov-report=term-missing | tee full-test-output.txt

- name: Upload coverage report
uses: actions/upload-artifact@v4
if: always()
with:
name: full-coverage-py${{ matrix.python-version }}
path: coverage.xml
retention-days: 30

# Static analysis. Runs on the single latest interpreter; lint is version-agnostic.
lint:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python 3.13
uses: actions/setup-python@v5
with:
python-version: "3.13"

- name: Install ruff
run: |
python -m pip install --upgrade pip
pip install "ruff>=0.1.0"

- name: Check for undefined names and syntax errors
# E9 (syntax) and F821 (undefined name) are the rules that catch
# things that actually break. The wider F set (unused imports, etc.) is
# historical style debt the repo still carries, so gating on it would
# be red on day one and train people to ignore it.
run: ruff check --select E9,F821 .

# Distribution gate: the wheel must build and be importable in a clean
# interpreter, so a broken pyproject or a missing package module fails here
# instead of on PyPI.
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

- name: Set up Python 3.11
uses: actions/setup-python@v5
with:
python-version: "3.11"

- name: Build wheel
run: |
python -m pip install --upgrade pip
pip install build
python -m build --wheel

- name: Install built wheel and import it
run: |
pip install dist/*.whl
# Run from outside the checkout (and with -I) so the import resolves
# to the installed wheel, not ./simpleaudit; load a scenario pack so
# missing package data fails here too.
cd "$RUNNER_TEMP"
python -I -c "
import simpleaudit, simpleaudit.judges, simpleaudit.scenarios, simpleaudit.targets, simpleaudit.tracing
from simpleaudit.model_auditor import ModelAuditor
assert 'site-packages' in simpleaudit.__file__, simpleaudit.__file__
assert ModelAuditor.get_scenarios('safety')
print('installed', simpleaudit.__version__, simpleaudit.__file__)
"
Loading
Loading