Skip to content

Versus suite: wall-clock comparison with Spot and Oink (WP9) - #154

Merged
leonbohn merged 3 commits into
mainfrom
versus-bench
Nov 9, 2026
Merged

leonbohn merged 3 commits into
mainfrom
versus-bench

Conversation

@leonbohn

@leonbohn leonbohn commented Nov 9, 2026

Copy link
Copy Markdown
Owner

Part of #143 (WP9), builds on #153.

  • benches/versus/: Python harness (compare.py, @operation/Case/Contender) with operations dpa/equivalence, dpa/inclusion (vs Spot, fixtures) and game/solve (vs Spot, Oink zlk and ppp; random and ladder games). Median wall-clock per side, inputs never timed, each comparison records whether both sides computed the same result. README explains how to add operations (e.g. determinization).
  • tools/bench: after the other targets the engine builds the wheel (maturin), runs the script and stores the rows (tables versus, versus_settings). New dashboard page /versus (latest main, ratio history per operation) and a section in the run view. Informational only; failing/missing tools skip it. Flags --versus-script, --no-versus.
  • Nix: package.nix wraps maturin, python3, Oink and Spot; module option versusScript; default benches now include nondeterministic and game (they were never run by the service, the list replaces the CLI default).
  • python/tests/test_spot_games.py: 100 random games against spot.solve_parity_game. CI: smoke run of the script in the Spot job.

Checked locally: cargo test/clippy in tools/bench, nix flake check --no-build, an end-to-end automata-bench run (wheel build + script), pytest for the new test. Not run: ./scripts/check (library untouched), the workflows, the real service.

@leonbohn
leonbohn merged commit 3c01a86 into main Nov 9, 2026
15 checks passed
@leonbohn
leonbohn deleted the versus-bench branch November 9, 2026 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant