Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions budgets.json
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,15 @@
"metric": "mean",
"unit": "ms",
"target": 3.0,
"measured": 0,
"measured": 0.598405,
"workload": "10k entities, 2k dynamic bodies (PRD 8.1)"
},
{
"name": "sim_tick_p99",
"metric": "p99",
"unit": "ms",
"target": 5.0,
"measured": 0,
"measured": 0.615179,
"workload": "10k entities, 2k dynamic bodies (PRD 8.1)"
},
{
Expand Down
50 changes: 41 additions & 9 deletions docs/api/budget_harness.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,34 @@ if (!r.passed) { /* print r.report; fail the CI run (PRD 8.1 policy) */ }
```

The end-to-end form is the tool:
`./build/bin/laige-bench --suite=synthetic --runs=1000 --warmup=100
[--budget=<name>]` — exit code 0 on pass, 2 on a failed budget check.

```console
./build/bin/laige-bench --suite=<name> [--runs=N] [--warmup=N]
[--budget=<budget-name>]... [--budgets=<path>]
[--math=<fixed_point_16_16|float_pinned_32>]
[--report=<path>] [--list]
```

- `--suite` — required; `synthetic` (the M0 harness stand-in) or
`sim-tick` (the M1-BENCH-01 PRD §8.1 workload: 10k entities, 2k
dynamic bodies, 60 Hz, movement + per-tick state hash — its baseline:
`docs/benchmarks/baselines/m1-sim-tick.md`). `--list` prints the
suite table.
- `--budget` — **repeatable**: one run checks every named budget against
the same histogram (each entry's `metric` picks its statistic),
printing one report per entry. A duplicate name is a usage error.
The M1 gate form: `--budget=sim_tick_avg --budget=sim_tick_p99`.
- `--math` — the SimMath backend (ADR 0002) the `sim-tick` suite runs
on; `fixed_point_16_16` (the default) or `float_pinned_32`. Other
suites ignore it.
- `--budgets` — the budgets file (default: `$LAIGE_BUDGETS_PATH`, then
`budgets.json` in the working directory — the ctest gate entries set
the env var to the repo root).
- `--report` — append the printed report to a file.

Exit code 0 on pass, 2 on a failed budget check (any one of the named
budgets failing fails the run — PRD 8.1 policy: a budget regression
fails CI), 1 on usage/load errors.

## `laige::Histogram`

Expand Down Expand Up @@ -175,8 +201,10 @@ fields explicitly). Schema version **1**:
the first baseline).
- **All 15 PRD §8.1 targets** are present as named entries (sim tick,
50k-sprite scene, cold start, and the zone server each carry two
budgets). Until their subsystems exist (M1+) their `measured` values
stay 0 = not yet measured.
budgets). `sim_tick_avg`/`sim_tick_p99` carry their first recorded
values (M1-BENCH-01, `docs/benchmarks/baselines/m1-sim-tick.md` —
the worse of the two SimMath backends, canonical Debug tree); every
other entry stays 0 = not yet measured until its subsystem lands.
- **Versioning:** an incompatible schema change bumps `version` and ships
a migration note here; `loadBudgets` rejects every other version
(never guesses).
Expand Down Expand Up @@ -204,8 +232,12 @@ is smaller than the run and then claiming "all samples"; using
## Determinism

The harness measures wall-clock durations: it is *not* deterministic
simulation state and plays no part in replay/lockstep (ARCH-010). The
synthetic `laige-bench --suite=synthetic` workload is deterministic by
construction (fixed LCG constants — Marsaglia 2003 — no RNG, no
allocation), which is what makes its baseline reproducible
(`docs/benchmarks/baselines/m0-synthetic.md`, M0-EXIT-01).
simulation state and plays no part in replay/lockstep (ARCH-010). Both
suites are deterministic by construction, which is what makes their
baselines reproducible: `synthetic` (fixed LCG constants — Marsaglia
2003 — no RNG, no allocation; `docs/benchmarks/baselines/m0-synthetic.md`,
M0-EXIT-01) and `sim-tick` (fixed seed `0x1F055EED`, deterministic
index-derived initial state, no randomness in the measured path — the
run's `final_hash` line is a bit-identical fingerprint across runs,
build types, and compilers per the ADR 0002 scope;
`docs/benchmarks/baselines/m1-sim-tick.md`, M1-BENCH-01).
14 changes: 8 additions & 6 deletions docs/benchmarks/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,14 @@ Benchmark methodology, baselines, results, and the regression policy
map onto them, the baseline-file convention, the PRD §8.1 regression
policy, and workload discipline.
- [baselines/](baselines/README.md) — recorded baseline reports
(`<milestone>-<workload>.md`). Every `budgets.json` entry still has
`measured: 0`: the two baselines recorded so far
(`baselines/m0-synthetic.md`, M0-EXIT-01, and
`baselines/m1-ecs-stress.md`, M1-ECS-07) measure harness and stress
workloads that update no budget. `baselines/m1-profiler-cost.md`
lands with M1-PROF-01.
(`<milestone>-<workload>.md`). Four are recorded:
`baselines/m0-synthetic.md` (M0-EXIT-01),
`baselines/m1-ecs-stress.md` (M1-ECS-07),
`baselines/m1-profiler-cost.md` (M1-PROF-01), and
`baselines/m1-sim-tick.md` (M1-BENCH-01) — the first that updates a
`budgets.json` `measured` field (`sim_tick_avg`, `sim_tick_p99`);
every other entry still has `measured: 0` (its subsystem lands in a
later milestone).
- [determinism-matrix.md](determinism-matrix.md) — the **determinism
report** for the M1-SAMPLE-01 hello scenario (M1-DET-04): the
committed per-tick hash baselines (both SimMath backends), the CI
Expand Down
16 changes: 14 additions & 2 deletions docs/benchmarks/baselines/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,7 @@ and the exact command + commit that produced it. Baseline files are
**immutable**: superseding a baseline adds a new file and updates
`measured` in `budgets.json` — it never edits an existing baseline.

Every `budgets.json` entry still has `measured: 0` (their subsystems land
in M1+). Recorded so far:
Recorded so far:

- [m0-synthetic.md](m0-synthetic.md) (M0-EXIT-01, 2026-09-13) — the
synthetic harness workload; proves the measurement pipeline end to
Expand All @@ -27,3 +26,16 @@ in M1+). Recorded so far:
of add/remove churn): no leaks (ASan), pool high-water stable,
iteration within the documented cost, accounted ECS storage bytes.
Not a `budgets.json` workload — no `measured` field updated.
- [m1-profiler-cost.md](m1-profiler-cost.md) (M1-PROF-01,
2026-09-21) — the enabled-profiler cost gate (ON vs OFF on 10k-entity
ticks, bounded at 1%): measured +0.28% on the canonical Debug tree.
Not a `budgets.json` workload — the FR-11.1 counters are diagnostics,
not a budgeted subsystem.
- [m1-sim-tick.md](m1-sim-tick.md) (M1-BENCH-01, 2026-09-25) — the
PRD §8.1 simulation-tick workload (10k entities, 2k dynamic bodies,
60 Hz, movement + per-tick state hash, both SimMath backends):
0.59 ms mean / 0.62 ms p99 on the canonical Debug tree, both
backends PASS. **The first baseline to update `budgets.json`
`measured`** (`sim_tick_avg`, `sim_tick_p99` — the worse of the two
backends). Every other `budgets.json` entry still has
`measured: 0` (its subsystem lands in a later milestone).
Loading
Loading