Skip to content
@studio-11-co

falsify

Fix the pass bar before the test runs. Open spec, four reference implementations, public registry.
README.md

falsify: fix the pass bar before the test runs. Anyone can check it later.

PyPI npm CI Multi-lang Conformance OpenSSF Best Practices IANA DOI live example

PRML is an open format for writing down, before a test runs, what the test has to achieve: metric, comparator, threshold, dataset hash, seed. A verifier recomputes the SHA-256 digest from the manifest's canonical bytes and returns PASS, FAIL or TAMPERED. The public registry adds Ed25519-signed receipts, independent RFC 3161 timestamps and a Rekor transparency-log entry. A receipt shows that the bar existed no later than the timestamp and whether a later copy is byte-identical to it. It never shows that a result is true.

Live example: a pre-registered robustness evaluation of the open-source SynthID-Text watermark, with six pass bars timestamped before the main run and the results reported against them: 8 of 12 outcomes met.

Proof, not promises

4 reference implementations (Python, JavaScript, Go, Rust), byte-equivalent across 21 conformance vectors, all rejecting the same 20 negative vectors, plus a 92-vector edge suite derived from the formal grammar. MIT code, Community Specification License 1.0 for the spec, 646 tests in CI, OpenSSF Best Practices passing.

Repositories
falsify Spec + Python reference + conformance suite (Go/JS/Rust under impl/)
falsify-js JavaScript reference (npm: falsify-js)
prml-verify-action GitHub Action: gate CI on tampered or regressed eval claims
mlflow-falsify MLflow plugin: bind runs to their locked bar, verify on run end
falsify-inspect Inspect AI adapter (on the UK AISI extensions page)
falsify-giskard Giskard scenario adapter
falsify-cookbook Patterns, anti-patterns, worked examples (CC0)
falsify-integrity-index Scorecard: 27 well-known eval claims vs 9 falsifiability criteria

Running a benchmark or leaderboard? Require a receipt: no locked claim, no entry.

Falsify OÜ, Tallinn · hello@falsify.dev

Popular repositories Loading

  1. falsify falsify Public

    Fix the pass bar before the test runs. PRML is an open spec and CLI that hashes and timestamps evaluation criteria (SHA-256, RFC 3161) so anyone can check them later. Python, JS, Go, Rust.

    Python 8 3

  2. falsify-cookbook falsify-cookbook Public

    Field manual for the PRML v0.1 specification. 13 patterns, 4 anti-patterns, 4 working examples. CC0.

    Python 1 1

  3. prml-verify-action prml-verify-action Public

    Verify PRML (Pre-Registered ML Manifest) commitments in CI. Block merges on tampered or regressed eval claims.

    Python

  4. falsify-integrity-index falsify-integrity-index Public

    Public scorecard of how 27 ML eval claims meet 9 PRML falsifiability criteria. CC0 data; MIT tooling.

  5. falsify-inspect falsify-inspect Public

    PRML pre-registration adapter for Inspect AI eval logs. Listed on the UK AI Safety Institute's Inspect extensions page. MIT.

    Python 2

  6. falsify-js falsify-js Public

    JavaScript reference implementation of the PRML (Pre-Registered ML Manifest) v0.1 specification. Byte-equivalent to the Python reference across 21 conformance vectors. Zero runtime dependencies.

    JavaScript

Repositories

Showing 9 of 9 repositories

People

This organization has no public members. You must be a member to see who’s a part of this organization.

Top languages

Loading…

Most used topics

Loading…