From 810c251a888375e4105c64e533292f947ae118ec Mon Sep 17 00:00:00 2001 From: Pascal Severin Date: Fri, 11 Sep 2026 22:53:02 +0200 Subject: [PATCH 1/2] [M0-CORE-08] Budget harness: Histogram/TimeIt/budgetCheck + budgets.json + laige-bench MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit laige::Histogram: fixed-capacity rolling window; O(1) allocation-free record(); exact min/mean/p50/p95/p99/max over the stored window with nearest-rank percentiles (rank = ceil(p*n/100), exact integer math); stats() is a cold-path O(n log n) pass with no allocation (pre-allocated scratch). laige::TimeIt: steady_clock ms scope timer, no allocation. loadBudgets: file I/O + bounded parse (ADR 0003) + strict budgets.json schema v1 validation (ARCH-007; unsupported version/fields, bad name/metric/unit charset, duplicate names, negative or non-finite numbers -> MalformedInput; unreadable file -> IoError). budgetCheck: total operation; empty histogram -> loud NO_SAMPLES; target > 0 -> measured <= target; target == 0 -> hard-zero budget (sim_heap_allocs), not "not set" (the not-yet-measured marker is the initial measured: 0). Report = stable 4-line AGENTS §12 text with before/after pair and caller context; formatStatsLine is the single source of the stats text. budgets.json (repo root): all 15 PRD §8.1 budgets as named entries; measured: 0 = not yet measured until their subsystems land (M1+). laige-bench: canonical command ./build/bin/laige-bench --suite= (reserved for this step by docs/getting-started/building.md); M0 suite "synthetic" (deterministic, allocation-free 4096-step LCG + double stand-in workload); --runs/--warmup, --budget= check with --budgets path resolution (arg -> LAIGE_BUDGETS_PATH env -> budgets.json), --report append; exit 0 pass / 1 usage or load failure / 2 budget check failure (CI-gateable). ctest: budget_harness entry (22 GTest cases) + laige_bench_smoke entry; LAIGE_BUDGETS_PATH wired for both (incl. the TSan tree, where CTest ENVIRONMENT is last-write-wins). API contract: docs/api/budget_harness.md. Bug fix (M0-CORE-07) found by the Verify run of this step: json.cpp parseObjectMembers was missing skipWhitespace before the member key - object documents with ", " between members (the hand-formatted repo-root budgets.json) were rejected. Regression test ConfigJsonValid.ObjectMemberWhitespace fails pre-fix. Verified locally: ctest 16/16, zero warnings, on GCC 16.2.1 (static, shared, ASan+UBSan fatal, TSan halt_on_error=1) and Clang 22.1.8 trees; include-lint clean (17 files). MSVC/AppleClang compile proof lands in CI. --- CMakeLists.txt | 1 + README.md | 24 +- budgets.json | 126 ++++ docs/api/budget_harness.md | 211 +++++++ src/laige-core/CMakeLists.txt | 6 +- src/laige-core/budget_harness.cpp | 377 ++++++++++++ src/laige-core/include/laige/budget_harness.h | 398 +++++++++++++ src/laige-core/json.cpp | 7 + tests/laige-core/CMakeLists.txt | 28 +- tests/laige-core/budget_harness_tests.cpp | 554 ++++++++++++++++++ tests/laige-core/config_json_tests.cpp | 40 ++ tools/bench/CMakeLists.txt | 34 ++ tools/bench/laige-bench.cpp | 284 +++++++++ 13 files changed, 2080 insertions(+), 10 deletions(-) create mode 100644 budgets.json create mode 100644 docs/api/budget_harness.md create mode 100644 src/laige-core/budget_harness.cpp create mode 100644 src/laige-core/include/laige/budget_harness.h create mode 100644 tests/laige-core/budget_harness_tests.cpp create mode 100644 tools/bench/CMakeLists.txt create mode 100644 tools/bench/laige-bench.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index 5c4d176..beda7b8 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -225,4 +225,5 @@ if(LAIGE_BUILD_TESTS) # Dev tools that need the built engine library (gated with tests: a # library-only build does not need them). add_subdirectory(tools/fuzz) + add_subdirectory(tools/bench) # M0-CORE-08: laige-bench endif() diff --git a/README.md b/README.md index b457f86..8b08263 100644 --- a/README.md +++ b/README.md @@ -14,8 +14,12 @@ isometric-first rendering, and a server-authoritative MMO path. (math, pools, Result, logging, config) lands over the remaining M0 steps in [roadmap/M0-foundations.md](roadmap/M0-foundations.md). So far: `laige::Result` / `laige::Status` plus the error-code registry - (M0-CORE-01) and the structured logging facade (M0-CORE-02). No game-facing - engine features are buildable yet. + (M0-CORE-01), the structured logging facade (M0-CORE-02), the SimMath + deterministic-math interface with the default `fpx16_16` backend + (M0-CORE-03/04), memory pools (M0-CORE-05), the deterministic PRNG + (M0-CORE-06), the bounded JSON parser + serializer (M0-CORE-07), and + the budget harness (M0-CORE-08). No game-facing engine features are + buildable yet. ## Built by a local LLM @@ -79,12 +83,22 @@ both variants. It carries the first functional engine code: `laige::Result` / `laige::Status` plus the error-code registry (M0-CORE-01, `ctest -R result_status`), the structured logging facade (M0-CORE-02, `ctest -R logging`, API contract in -[docs/api/logging.md](docs/api/logging.md)), and the SimMath +[docs/api/logging.md](docs/api/logging.md)), the SimMath deterministic-math interface (M0-CORE-03 `fp32_pinned`, `ctest -R math_float`; M0-CORE-04 default `fpx16_16`, `ctest -R math_fixed` — API contract in -[docs/api/sim_math.md](docs/api/sim_math.md)). Engine targets compile -with `-Wall -Werror` and with exceptions and RTTI disabled (NFR-8.10). +[docs/api/sim_math.md](docs/api/sim_math.md)), the memory pools +(`ctest -R pools`, API contract in +[docs/api/pools.md](docs/api/pools.md)), the deterministic PRNG +(`ctest -R prng`, API contract in +[docs/api/prng.md](docs/api/prng.md)), the bounded JSON parser + +serializer (`ctest -R config_json`, API contract in +[docs/api/json.md](docs/api/json.md)), and the budget harness +(`ctest -R budget_harness`, API contract in +[docs/api/budget_harness.md](docs/api/budget_harness.md); canonical +benchmark command `./build/bin/laige-bench --suite=`). Engine +targets compile with `-Wall -Werror` and with exceptions and RTTI +disabled (NFR-8.10). ## Documentation diff --git a/budgets.json b/budgets.json new file mode 100644 index 0000000..fa52532 --- /dev/null +++ b/budgets.json @@ -0,0 +1,126 @@ +{ + "version": 1, + "description": "Laige performance budgets (PRD 8.1). Each entry names one hard budget: 'target' is the limit in 'unit' (every budget is an at-most upper bound; target 0 is a hard zero budget, not 'unset'), 'measured' is the last recorded value of that budget (0 = not yet measured, the M0 convention), and 'metric' is the histogram statistic the check evaluates. Schema: docs/api/budget_harness.md, section 'budgets.json schema'.", + "budgets": [ + { + "name": "frame_time_render", + "metric": "p95", + "unit": "ms", + "target": 8.3, + "measured": 0, + "workload": "worst-case isometric reference scene @ 1080p, mid-range laptop (PRD 8.1)" + }, + { + "name": "sim_tick_avg", + "metric": "mean", + "unit": "ms", + "target": 3.0, + "measured": 0, + "workload": "10k entities, 2k dynamic bodies (PRD 8.1)" + }, + { + "name": "sim_tick_p99", + "metric": "p99", + "unit": "ms", + "target": 5.0, + "measured": 0, + "workload": "10k entities, 2k dynamic bodies (PRD 8.1)" + }, + { + "name": "sprites_50k_draw_calls", + "metric": "max", + "unit": "draw_calls", + "target": 30, + "measured": 0, + "workload": "50k visible sprites (worst-case isometric overlap), 3 parallax layers, UI (PRD 8.1)" + }, + { + "name": "sprites_50k_cpu", + "metric": "mean", + "unit": "ms", + "target": 2.0, + "measured": 0, + "workload": "50k visible sprites (worst-case isometric overlap), 3 parallax layers, UI (PRD 8.1)" + }, + { + "name": "iso_depthkey_rebuild", + "metric": "mean", + "unit": "ms", + "target": 0.2, + "measured": 0, + "workload": "10k dirty cells after a terrain edit (PRD 8.1)" + }, + { + "name": "iso_picking", + "metric": "mean", + "unit": "ms", + "target": 0.01, + "measured": 0, + "workload": "one isometric screen-to-grid pick, O(1) (PRD 8.1)" + }, + { + "name": "sim_heap_allocs", + "metric": "max", + "unit": "allocs_per_frame", + "target": 0, + "measured": 0, + "workload": "steady-state sim loop heap allocations, asserted in debug builds (PRD 8.1)" + }, + { + "name": "engine_base_rss", + "metric": "max", + "unit": "mb", + "target": 100, + "measured": 0, + "workload": "engine base memory, empty scene running (PRD 8.1, all P0 platforms)" + }, + { + "name": "cold_start_ssd", + "metric": "max", + "unit": "s", + "target": 2.0, + "measured": 0, + "workload": "game process to first frame on SSD (PRD 8.1, P0 platforms)" + }, + { + "name": "cold_start_cold", + "metric": "max", + "unit": "s", + "target": 5.0, + "measured": 0, + "workload": "cold start (PRD 8.1, P0 platforms)" + }, + { + "name": "build_time_ci", + "metric": "max", + "unit": "min", + "target": 10, + "measured": 0, + "workload": "clean build, engine + sample (PRD 8.1, CI)" + }, + { + "name": "build_time_local", + "metric": "max", + "unit": "min", + "target": 5, + "measured": 0, + "workload": "warm local build, engine + sample (PRD 8.1)" + }, + { + "name": "zone_server_tick_p95", + "metric": "p95", + "unit": "ms", + "target": 8.0, + "measured": 0, + "workload": "zone server, 2k players @ 20 Hz (PRD 8.1, 8-core server class)" + }, + { + "name": "zone_server_ram", + "metric": "max", + "unit": "gb", + "target": 4, + "measured": 0, + "workload": "zone server RAM, 2k players @ 20 Hz (PRD 8.1, 8-core server class)" + } + ] +} diff --git a/docs/api/budget_harness.md b/docs/api/budget_harness.md new file mode 100644 index 0000000..1d5ee8a --- /dev/null +++ b/docs/api/budget_harness.md @@ -0,0 +1,211 @@ +# Budget harness (`laige::Histogram`, `laige::TimeIt`, `budgetCheck`) + +The measurement half of the PRD §8.1 performance-budget policy (M0-CORE-08; +CORE-001: no performance claim without a reproducible measurement; AGENTS +§12 report requirements). Public header: +`src/laige-core/include/laige/budget_harness.h`; implementation: +`src/laige-core/budget_harness.cpp`. The operator-facing tool is +`laige-bench` (`tools/bench/laige-bench.cpp`; canonical command form in +[building.md](../getting-started/building.md)). Unit suite: +`ctest -R budget_harness` (`tests/laige-core/budget_harness_tests.cpp`). + +## Quick start + +```cpp +#include + +// Measure one workload: one TimeIt per iteration, recorded in a +// histogram sized to the run (every sample kept). +laige::Histogram hist(laige::Histogram::Options{1000}); +for (int i = 0; i < 1000; ++i) { + laige::TimeIt t; // scope start + workload(); + hist.record(t.elapsedMs()); // milliseconds +} +const laige::HistogramStats s = hist.stats(); // min/mean/p50/p95/p99/max + +// Check against a named PRD 8.1 budget (budgets.json, repo root). +auto table = laige::loadBudgets("budgets.json"); // Result +if (table.isError()) { /* log table.errorText() (LOG-002); abort the run */ } +laige::BudgetReportContext ctx; +ctx.workload = "worst-case isometric reference scene @ 1080p"; +ctx.build = "GCC 16.2.1, Debug"; // caller records what it owns +ctx.machine = "mid-range laptop (2019-2023 class), Linux"; +ctx.warmup = 100; +const laige::BudgetCheckResult r = + laige::budgetCheck(*table.value().find("frame_time_render"), hist, ctx); +if (!r.passed) { /* print r.report; fail the CI run (PRD 8.1 policy) */ } +``` + +The end-to-end form is the tool: +`./build/bin/laige-bench --suite=synthetic --runs=1000 --warmup=100 +[--budget=]` — exit code 0 on pass, 2 on a failed budget check. + +## `laige::Histogram` + +A fixed-capacity **rolling-window** sample store. `record()` is O(1), +allocates nothing, and takes no lock — the only hot-path-safe operation. +Construction performs the two backing allocations (setup path). + +**Window semantics.** The histogram keeps at most `Options::capacity` +samples. `record()` beyond capacity drops the *oldest* sample; +`totalRecorded()` counts every sample ever recorded, so truncation is +observable (`count() < totalRecorded()` means the window dropped samples — +a benchmark that needs every sample sets `capacity >= runs`). + +**Statistics scope.** `stats()` describes exactly the stored window (the +last `min(totalRecorded, capacity)` samples). `mean` is computed over that +window (no running sum — no float drift). When `n == 0` the six +statistics are NaN; callers check `n` (and `budgetCheck` turns an empty +histogram into a loud `NO_SAMPLES` failure instead of reading NaN). + +**Percentiles (nearest-rank, the exact documented definition).** For the +sorted stored window `v[0..n-1]` (n ≥ 1) and percentile p (0..100): +rank `r = ceil(p·n/100)` in exact integer math, clamped to ≥ 1; the +percentile is `v[r-1]`. p=0 is the min, p=100 the max, and a one-sample +window returns that sample for every p. Nearest-rank (over linear +interpolation): no fractional indices, no extra allocation, bit-identical +on every platform (CORE-004). + +**Errors.** None — a histogram cannot fail. (Overflow of a full window is +the documented drop-oldest behavior; `capacity 0` is legal.) + +**Performance.** `record()` — O(1), no allocation, no lock, no I/O (hot +path). `stats()` — O(n log n) time (sorts a pre-allocated scratch buffer), +no allocation (cold path: reports, budget checks — never frame/tick loops). +Copy is O(capacity) (deep, cold path); move is O(1). + +**Threading (CONC-001).** One owner thread while mutable; `stats()` on a +fully built histogram is a safe const read (publish contract, like +`Result`/`Status`). + +**Misuse.** Recording wall-clock timestamps is a unit error (the harness +measures durations, typically `TimeIt::elapsedMs()`). Ignoring +`totalRecorded() > count()` on a percentile claim is a silent-window bug. + +## `laige::TimeIt` + +A scope timer over `std::chrono::steady_clock` (monotonic — immune to +wall-clock adjustments; the right clock for durations). Milliseconds as a +double. No allocation, no lock; the start point is immutable after +construction, so `elapsedMs()` is a safe const read; `reset()` belongs to +the owner thread. For "what time is it" use the logging facade's +timestamps (system_clock, RFC 3339) — not `TimeIt`. + +## `loadBudgets` / `budgetCheck` / `BudgetTable` + +**`loadBudgets(path)`** — `Result`. Cold path: +file I/O + bounded JSON parse (ADR 0003: 1 MiB document, depth 32 — the +file must also not exceed 1 MiB before it is read into memory). +Errors (never silent, CORE-008): + +| Failure | Code | +|---|---| +| Unreadable file | `ErrorCode::IoError` (5) | +| Malformed JSON | `ErrorCode::MalformedInput` (3) | +| Unsupported version (`!= 1`) | `MalformedInput` | +| Unknown/missing field, bad `name`/`metric`/`unit`, duplicate name, negative or non-finite number | `MalformedInput` | + +The loaded `BudgetTable` is immutable and safe to read from any thread; +`find(name)` is a linear scan (the table is small by design — no hash map, +PERF-006). A failed load produces no table (all-or-nothing). + +**`budgetCheck(entry, histogram, context)`** — `BudgetCheckResult` +(`passed`, `measured`, `target`, `before`, `report`). Total: it cannot +fail as an operation — its *outcome* is the pass/fail flag: + +| State | `passed` | `result` in the report | +|---|---|---| +| Histogram empty (n = 0) | `false` | `NO_SAMPLES` — a workload that recorded nothing is a broken harness; loud, never silent (CORE-008) | +| `target > 0` | `measured <= target` | `PASS` / `FAIL` | +| `target == 0` (hard-zero budget) | `measured == 0` | `PASS` / `FAIL` | + +`before`/`after` are the AGENTS §12 before/after pair: the entry's last +recorded value (`measured` field of `budgets.json`) vs. the current +measurement. Cold path: O(n log n) (the stats pass) plus report string +building (allocates — reporting is never a hot path). Thread-safe on const +inputs (the histogram must not be mutated concurrently — CONC-001). + +**Report format (stable, machine-greppable — LOG-001).** The first line is +the grep contract; numbers are `%.6g` in the "C" locale (`nan`/`inf` as +plain text): + +```text +budget= result= metric= unit= + after= before= target= + stats: n= min= mean= p50= p95= p99= max= + context: workload= build= machine= warmup= +``` + +The `context` fields are the AGENTS §12 machine/build facts **the caller +harness records** (the tool supplies compiler/build type, the operator +supplies the machine — see `LAIGE_BENCH_MACHINE`). The stats line comes +from `laige::formatStatsLine`, the single source of the stats text. + +## `budgets.json` schema + +Repo root, versioned (ARCH-007: the reader rejects unsupported versions and +fields explicitly). Schema version **1**: + +```json +{ + "version": 1, + "description": "...", // optional; human notes (JSON has no comments) + "budgets": [ + { + "name": "frame_time_render", // required; snake_case id; unique + "metric": "p95", // required; mean|min|max|p50|p95|p99 + "unit": "ms", // required; identifier (ms, draw_calls, ...) + "target": 8.3, // required; finite, >= 0 (the PRD 8.1 limit) + "measured": 0, // required; finite, >= 0 (last recorded value) + "workload": "worst-case ..." // required; non-empty (what the budget applies to) + } + ] +} +``` + +- **`target`** is the hard PRD §8.1 limit — every budget is an *at-most* + upper bound. `target == 0` is a **hard-zero budget** (e.g. + `sim_heap_allocs`: zero steady-state heap allocations per frame), **not** + "not set". +- **`measured`** is the last recorded value of the budget (the "before" + number). `0` is the M0 convention for *not yet measured*; when a budget + is first measured, the benchmark runner records the value here (this + file is the baseline home — see `docs/benchmarks/`, M0-EXIT-01 writes + the first baseline). +- **All 15 PRD §8.1 targets** are present as named entries (sim tick, + 50k-sprite scene, cold start, and the zone server each carry two + budgets). Until their subsystems exist (M1+) their `measured` values + stay 0 = not yet measured. +- **Versioning:** an incompatible schema change bumps `version` and ships + a migration note here; `loadBudgets` rejects every other version + (never guesses). +- **Validation (strict, ARCH-007):** exactly the six entry fields — a + missing or unknown field is `MalformedInput`; `name` must be unique + snake_case; `target`/`measured` must be finite and ≥ 0 (a well-formed + overflow token such as `1e999` parses to `+inf` and is rejected — + ADR 0003). + +## Performance (DOC-004) + +| Operation | Complexity | Allocation | Path | +|---|---|---|---| +| `Histogram::record` | O(1) | none | **hot path safe** (one index arithmetic + one store) | +| `Histogram::stats` | O(n log n) | none (pre-allocated scratch) | cold (reports) | +| `TimeIt::elapsedMs` | O(1) | none | hot path safe (one clock read) | +| `budgetCheck` | O(n log n) + report build | report string (one or two) | cold only | +| `loadBudgets` | O(file) parse | file buffer + table | setup only (file I/O) | + +Traps: calling `budgetCheck`/`stats` from a frame or tick loop (cold-path +work in a hot loop, PERF-002); recording into a histogram whose capacity +is smaller than the run and then claiming "all samples"; using +`TimeIt` for timestamps (it measures durations). + +## Determinism + +The harness measures wall-clock durations: it is *not* deterministic +simulation state and plays no part in replay/lockstep (ARCH-010). The +synthetic `laige-bench --suite=synthetic` workload is deterministic by +construction (fixed LCG constants — Marsaglia 2003 — no RNG, no +allocation), which is what makes its baseline reproducible +(`docs/benchmarks/baselines/m0-synthetic.md`, M0-EXIT-01). diff --git a/src/laige-core/CMakeLists.txt b/src/laige-core/CMakeLists.txt index 4b747ca..5babb33 100644 --- a/src/laige-core/CMakeLists.txt +++ b/src/laige-core/CMakeLists.txt @@ -15,10 +15,12 @@ if(LAIGE_BUILD_SHARED) add_library(laige-core SHARED version.cpp errors.cpp logging.cpp - sim_math.cpp sim_math_fixed.cpp json.cpp) + sim_math.cpp sim_math_fixed.cpp json.cpp + budget_harness.cpp) else() add_library(laige-core STATIC version.cpp errors.cpp logging.cpp - sim_math.cpp sim_math_fixed.cpp json.cpp) + sim_math.cpp sim_math_fixed.cpp json.cpp + budget_harness.cpp) endif() laige_apply_engine_policy(laige-core) diff --git a/src/laige-core/budget_harness.cpp b/src/laige-core/budget_harness.cpp new file mode 100644 index 0000000..5abab85 --- /dev/null +++ b/src/laige-core/budget_harness.cpp @@ -0,0 +1,377 @@ +// laige-core budget harness implementation (M0-CORE-08). +// +// Implements the cold-path half of the budget harness: loading and +// validating budgets.json (schema v1, via the bounded JSON parser — +// ADR 0003, no new dependency) and the budgetCheck pass/fail + AGENTS 12 +// report formatting. The hot-path half (Histogram::record, TimeIt) lives +// header-only in include/laige/budget_harness.h. + +#include "laige/budget_harness.h" + +#include +#include +#include +#include +#include +#include + +#include "laige/json.h" + +namespace laige { +namespace { + +// The file-size bound for budgets.json: identical to JsonOptions' +// default maxDocumentBytes (ADR 0003), named here (CORE-005) because the +// read loop enforces it before the whole file lands in memory. +constexpr std::size_t kBudgetsMaxDocumentBytes = + static_cast(1u) << 20; // 1 MiB + +// The schema version this reader accepts (ARCH-007: anything else is +// rejected explicitly, never guessed at). +constexpr int kBudgetsSchemaVersion = 1; + +// Fixed entry field count for the v1 schema: name, metric, unit, target, +// measured, workload. A v1 entry object has exactly these six keys — +// more (unknown field) or fewer (missing field) is MalformedInput. +constexpr std::size_t kBudgetEntryFieldCount = 6; + +const char* metricName(BudgetMetric metric) { + switch (metric) { + case BudgetMetric::Mean: return "mean"; + case BudgetMetric::Min: return "min"; + case BudgetMetric::Max: return "max"; + case BudgetMetric::P50: return "p50"; + case BudgetMetric::P95: return "p95"; + case BudgetMetric::P99: return "p99"; + default: + assert(false && "unreachable: exhaustive BudgetMetric switch"); + return "unknown"; + } +} + +double metricValue(BudgetMetric metric, const HistogramStats& s) { + switch (metric) { + case BudgetMetric::Mean: return s.mean; + case BudgetMetric::Min: return s.min; + case BudgetMetric::Max: return s.max; + case BudgetMetric::P50: return s.p50; + case BudgetMetric::P95: return s.p95; + case BudgetMetric::P99: return s.p99; + default: + assert(false && "unreachable: exhaustive BudgetMetric switch"); + return std::nan(""); + } +} + +// Locale-free rendering of a double for the report: at most 6 significant +// digits (%.6g, "C" locale — exact for every double up to 6 digits, +// plenty for ms/us budget numbers and byte counts alike). NaN/inf render +// as plain "nan"/"inf" text so the report stays greppable (LOG-001). +void formatDouble(std::string& out, double value) { + char buf[32]; + if (std::isnan(value)) { + std::snprintf(buf, sizeof(buf), "nan"); + } else if (std::isinf(value)) { + std::snprintf(buf, sizeof(buf), value > 0.0 ? "inf" : "-inf"); + } else { + std::snprintf(buf, sizeof(buf), "%.6g", value); + } + out += buf; +} + +// The stable report layout (docs/api/budget_harness.md documents it as +// the machine-greppable contract): +// +// budget= result= metric= unit= +// after= before= target= +// stats: n= min= mean= p50= p95= p99= max= +// context: workload= build= machine= warmup= +std::string buildReport(const BudgetEntry& entry, const char* outcome, + double measured, const HistogramStats& s, + const BudgetReportContext& ctx) { + std::string r; + r += "budget="; + r += entry.name; + r += " result="; + r += outcome; + r += " metric="; + r += metricName(entry.metric); + r += " unit="; + r += entry.unit; + r += "\n after="; + formatDouble(r, measured); + r += " before="; + formatDouble(r, entry.measured); + r += " target="; + formatDouble(r, entry.target); + r += "\n "; + r += formatStatsLine(s); + r += "\n context: workload="; + r += (ctx.workload != nullptr ? ctx.workload : ""); + r += " build="; + r += (ctx.build != nullptr ? ctx.build : ""); + r += " machine="; + r += (ctx.machine != nullptr ? ctx.machine : ""); + r += " warmup="; + r += std::to_string(ctx.warmup); + r += "\n"; + return r; +} + +// --- budgets.json schema v1 validation ------------------------------------- + +// Stable id charset for budget names (LOG-001-style machine-greppable id). +bool isSnakeCase(std::string_view s) { + if (s.empty()) return false; + for (const char c : s) { + const bool ok = (c >= 'a' && c <= 'z') || (c >= '0' && c <= '9') || + c == '_'; + if (!ok) return false; + } + return true; +} + +// Unit charset: an identifier (ms, draw_calls, allocs_per_frame, ...). +bool isUnitIdentifier(std::string_view s) { + if (s.empty()) return false; + for (const char c : s) { + const bool ok = (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || + (c >= '0' && c <= '9') || c == '_'; + if (!ok) return false; + } + return true; +} + +bool metricFromText(std::string_view text, BudgetMetric& out) { + if (text == "mean") { + out = BudgetMetric::Mean; + return true; + } + if (text == "min") { + out = BudgetMetric::Min; + return true; + } + if (text == "max") { + out = BudgetMetric::Max; + return true; + } + if (text == "p50") { + out = BudgetMetric::P50; + return true; + } + if (text == "p95") { + out = BudgetMetric::P95; + return true; + } + if (text == "p99") { + out = BudgetMetric::P99; + return true; + } + return false; +} + +// target/measured contract: a JSON number that is finite and >= 0. +// (The parser stores a well-formed overflow token such as 1e999 as +inf — +// a valid parse result that the schema rejects, ADR 0003.) +bool isNonNegativeFinite(const JsonValue& v) { + return v.isNumber() && std::isfinite(v.asNumber()) && v.asNumber() >= 0.0; +} + +Status parseBudgetEntry(const JsonValue& obj, + const std::vector& existing, + BudgetEntry& out) { + if (!obj.isObject() || obj.asObject().size() != kBudgetEntryFieldCount) + return Status(ErrorCode::MalformedInput); + + const JsonValue *name = nullptr, *metric = nullptr, *unit = nullptr, + *target = nullptr, *measured = nullptr, *workload = nullptr; + for (const auto& member : obj.asObject()) { + if (member.first == "name") + name = &member.second; + else if (member.first == "metric") + metric = &member.second; + else if (member.first == "unit") + unit = &member.second; + else if (member.first == "target") + target = &member.second; + else if (member.first == "measured") + measured = &member.second; + else if (member.first == "workload") + workload = &member.second; + else + return Status(ErrorCode::MalformedInput); // unknown field + } + // A size-6 object whose six distinct keys all routed above holds + // exactly the v1 field set; this guard documents that and catches a + // future edit that routes a seventh name without bumping the count. + if (name == nullptr || metric == nullptr || unit == nullptr || + target == nullptr || measured == nullptr || workload == nullptr) + return Status(ErrorCode::MalformedInput); + + if (!name->isString() || !isSnakeCase(name->asString())) + return Status(ErrorCode::MalformedInput); + out.name = std::string(name->asString()); + + for (const BudgetEntry& e : existing) + if (e.name == out.name) + return Status(ErrorCode::MalformedInput); // duplicate name + + if (!metric->isString() || !metricFromText(metric->asString(), out.metric)) + return Status(ErrorCode::MalformedInput); + + if (!unit->isString() || !isUnitIdentifier(unit->asString())) + return Status(ErrorCode::MalformedInput); + out.unit = std::string(unit->asString()); + + if (!isNonNegativeFinite(*target)) + return Status(ErrorCode::MalformedInput); + out.target = target->asNumber(); + + if (!isNonNegativeFinite(*measured)) + return Status(ErrorCode::MalformedInput); + out.measured = measured->asNumber(); + + if (!workload->isString() || workload->asString().empty()) + return Status(ErrorCode::MalformedInput); + out.workload = std::string(workload->asString()); + + return Status(); +} + +Status parseBudgetsTable(const JsonValue& root, + std::vector& out) { + if (!root.isObject()) return Status(ErrorCode::MalformedInput); + + for (const auto& member : root.asObject()) { + if (member.first != "version" && member.first != "description" && + member.first != "budgets") + return Status(ErrorCode::MalformedInput); // unknown top-level field + } + + const JsonValue* version = root.findMember("version"); + if (version == nullptr || !version->isNumber() || + version->asNumber() != static_cast(kBudgetsSchemaVersion)) + return Status(ErrorCode::MalformedInput); // unsupported version + + if (const JsonValue* description = root.findMember("description")) + if (!description->isString()) + return Status(ErrorCode::MalformedInput); + + const JsonValue* budgets = root.findMember("budgets"); + if (budgets == nullptr || !budgets->isArray()) + return Status(ErrorCode::MalformedInput); + + out.reserve(budgets->asArray().size()); + for (const JsonValue& element : budgets->asArray()) { + BudgetEntry entry; + const Status s = parseBudgetEntry(element, out, entry); + if (s.isError()) return s; + out.push_back(std::move(entry)); + } + return Status(); +} + +std::string readWholeFile(const std::string& path, Status& ioStatus) { + // MSVC's plain fopen is deprecated (C4996, fatal under the engine /WX + // policy); _fsopen with _SH_DENYNO matches the logging file sink's + // platform boundary (CPP-009). +#if defined(_MSC_VER) + std::FILE* f = ::_fsopen(path.c_str(), "rb", _SH_DENYNO); +#else + std::FILE* f = std::fopen(path.c_str(), "rb"); +#endif + if (f == nullptr) { + ioStatus = Status(ErrorCode::IoError); + return {}; + } + std::string text; + char buf[8192]; + while (true) { + const std::size_t r = std::fread(buf, 1, sizeof(buf), f); + text.append(buf, r); + if (r < sizeof(buf)) break; // EOF (or error — checked below) + if (text.size() > kBudgetsMaxDocumentBytes) { + std::fclose(f); + ioStatus = Status(ErrorCode::MalformedInput); // over the size bound + return {}; + } + } + const bool errored = std::ferror(f) != 0; + std::fclose(f); + if (errored) { + ioStatus = Status(ErrorCode::IoError); + return {}; + } + return text; +} + +} // namespace + +std::string formatStatsLine(const HistogramStats& s) { + std::string r = "stats: n="; + r += std::to_string(s.n); + r += " min="; + formatDouble(r, s.min); + r += " mean="; + formatDouble(r, s.mean); + r += " p50="; + formatDouble(r, s.p50); + r += " p95="; + formatDouble(r, s.p95); + r += " p99="; + formatDouble(r, s.p99); + r += " max="; + formatDouble(r, s.max); + return r; +} + +BudgetCheckResult budgetCheck(const BudgetEntry& entry, + const Histogram& histogram, + const BudgetReportContext& context) { + BudgetCheckResult out; + out.target = entry.target; + out.before = entry.measured; + + const HistogramStats s = histogram.stats(); + if (s.n == 0) { + // A workload that recorded nothing is a broken harness: fail loudly + // (CORE-008), never read NaN statistics as "passed". + out.passed = false; + out.measured = std::nan(""); + out.report = buildReport(entry, "NO_SAMPLES", out.measured, s, context); + return out; + } + + const double measured = metricValue(entry.metric, s); + out.measured = measured; + // Every PRD 8.1 budget is an at-most upper bound. target == 0 is the + // hard-zero budget (e.g. sim heap allocations per frame), not "not + // set" — the "not yet measured" marker lives in entry.measured. + out.passed = (entry.target > 0.0) ? (measured <= entry.target) + : (measured == 0.0); + out.report = + buildReport(entry, out.passed ? "PASS" : "FAIL", measured, s, context); + return out; +} + +Result loadBudgets(std::string_view path) { + const std::string pathString(path); // fopen needs a NUL-terminated path + + Status ioStatus; + const std::string text = readWholeFile(pathString, ioStatus); + if (ioStatus.isError()) + return Result(ioStatus.error()); + + const Result parsed = parseJson(text); + if (parsed.isError()) + return Result(parsed.error()); + + BudgetTable table; + const Status schema = parseBudgetsTable(parsed.value(), table.entries_); + if (schema.isError()) + return Result(schema.error()); + + return Result(std::move(table)); +} + +} // namespace laige diff --git a/src/laige-core/include/laige/budget_harness.h b/src/laige-core/include/laige/budget_harness.h new file mode 100644 index 0000000..c210a3e --- /dev/null +++ b/src/laige-core/include/laige/budget_harness.h @@ -0,0 +1,398 @@ +// laige-core budget harness (M0-CORE-08). +// +// PRD 8.1: performance budgets are *hard* and measured in CI; the policy +// is that a PR regressing any budget by > 10% (or breaching the absolute +// target) fails CI. CORE-001: no performance claim without a reproducible +// measurement. AGENTS 12: benchmark reports MUST record hardware, OS, +// compiler and version, build type, relevant flags, dataset/workload, +// warm-up, sample count, summary statistics, and before/after results. +// +// This header ships the three library pieces of the harness (the +// `laige-bench` executable in tools/bench is the operator-facing front +// end for them): +// +// Histogram Fixed-capacity sample store (rolling window) with exact +// min/mean/p50/p95/p99/max over the stored window. +// record() is O(1) and allocates nothing (PERF-003); +// stats() is a cold-path O(n log n) pass. +// TimeIt Steady-clock scope timer in milliseconds. +// loadBudgets / budgetCheck +// Named budget entries from budgets.json (repo root) plus +// the pass/fail check that formats the AGENTS 12 report +// (before/after numbers; the caller supplies the +// machine/build context it owns). +// +// --------------------------------------------------------------------------- +// Histogram contract +// --------------------------------------------------------------------------- +// +// Window semantics (PERF-008 backpressure, no unbounded growth): the +// histogram keeps at most Options::capacity samples — a rolling window. +// record() beyond capacity drops the *oldest* sample; totalRecorded() +// keeps counting every sample ever recorded, so a truncated window is +// observable (CORE-008: silent truncation is not allowed). A benchmark +// that needs every sample sets capacity >= runs. +// +// Percentiles (nearest-rank, the documented exact definition): for the +// sorted stored window v[0..n-1] (n >= 1) and percentile p (0..100), +// rank r = ceil(p*n/100) in exact integer math (clamped to >= 1); the +// percentile is v[r-1]. p=0 is the min, p=100 the max, a one-sample +// window returns that sample for every p. Nearest-rank is chosen over +// linear interpolation: no fractional indices, no extra allocation, +// bit-identical on every platform (CORE-004). +// +// Statistics scope: stats() describes exactly the stored window (the +// last min(totalRecorded, capacity) samples). mean is computed over +// that window (no running sum, so no float drift). When n == 0 the six +// statistics are NaN — callers must check n (budgetCheck turns an empty +// histogram into a loud NO_SAMPLES failure instead of reading NaN). +// +// Errors: none — a histogram cannot fail (record() into a full window +// drops the oldest by contract; capacity 0 is legal and drops all). +// +// Ownership/threading (CPP-002, CONC-001): a histogram is a value type +// (copy = O(capacity) deep copy, cold path; move = O(1)). It has exactly +// one owner thread while mutable; stats() on a fully built histogram is +// safe to read from any thread (publish contract, like Result/Status). +// +// Performance: construction performs the two backing allocations (setup +// path). record() = one index arithmetic + one store: O(1), no +// allocation, no lock, no I/O (hot-path safe). stats() sorts a +// pre-allocated scratch buffer: O(n log n) time, no allocation; it is +// the cold path (budget checks, reports — never frame/tick loops). +// +// --------------------------------------------------------------------------- +// TimeIt contract +// --------------------------------------------------------------------------- +// +// Clock: std::chrono::steady_clock (monotonic; immune to wall-clock +// adjustments — the right clock for durations). Unit: milliseconds +// (double). Construction is the scope start; elapsedMs() reads the +// elapsed time. No allocation, no lock. A constructed TimeIt is immutable +// (start point fixed), so reading it from other threads is safe; reset() +// must come from the owner. +// +// Misuse: TimeIt measures *durations*, not timestamps — for "what time +// is it" use the logging facade's timestamps (system_clock, RFC 3339). +// +// --------------------------------------------------------------------------- +// Budget entries and checks (budgets.json, repo root) +// --------------------------------------------------------------------------- +// +// budgets.json is the versioned home of the PRD 8.1 budgets (ARCH-007: +// readers reject unsupported versions and fields explicitly — see +// loadBudgets below and docs/api/budget_harness.md for the schema). +// +// target the hard PRD 8.1 limit, in `unit`. Every 8.1 budget is an +// at-most upper bound. target == 0 is a *hard zero budget* +// (e.g. sim heap allocations per frame), NOT "not set". +// measured the last recorded value of this budget (the "before" +// number of the AGENTS 12 before/after pair). 0 is the +// M0 convention for "not yet measured"; the benchmark +// runner writes a real value when it measures the budget. +// metric which histogram statistic the check evaluates. +// +// budgetCheck semantics: +// - histogram empty (n == 0) -> passed = false, result NO_SAMPLES +// (a workload that recorded nothing +// is a broken harness — loud, never +// silent, CORE-008) +// - target > 0 -> passed iff measured <= target +// - target == 0 (hard zero) -> passed iff measured == 0 +// +// The report is stable, machine-greppable text (format in +// docs/api/budget_harness.md): the first line carries +// `budget= result= metric= unit=`; +// the following lines carry after/before/target, the summary statistics, +// and the caller context. +// +// Errors: loadBudgets returns Result: +// - unreadable file -> ErrorCode::IoError (5) +// - malformed JSON or schema violation -> ErrorCode::MalformedInput (3) +// (bad/unknown version, unknown or missing field, bad metric or +// unit or name charset, duplicate name, negative or non-finite +// number, file above the 1 MiB bound) +// budgetCheck itself cannot fail (a check on any state is total) — its +// outcome is the passed flag plus the report. +// +// --------------------------------------------------------------------------- +// Misuse warnings +// --------------------------------------------------------------------------- +// - Recording wall-clock timestamps into a Histogram is a unit error: +// the harness measures durations (typically TimeIt::elapsedMs()). +// - budgetCheck allocates (the report string) and runs stats() — +// O(n log n). It is a cold path: never call it from a frame or tick +// loop (PERF-002). +// - A Histogram with capacity < runs silently (by contract) truncates +// its window: check totalRecorded() == count() when every sample +// matters. +// - loadBudgets does file I/O: setup/reporting paths only. +// - Discarding the Result from loadBudgets is a likely logic bug +// (CORE-008); the Result is the only failure channel. + +#pragma once + +#include +#include +#include +#include +#include +#include +#include + +#include "laige/result.h" + +namespace laige { + +// --------------------------------------------------------------------------- +// Histogram +// --------------------------------------------------------------------------- + +// Summary statistics over the samples currently stored in a Histogram +// (rolling window). When n == 0 the six statistics are NaN (check n; +// budgetCheck turns an empty histogram into a loud NO_SAMPLES failure). +struct HistogramStats { + std::uint64_t n; // samples stored in the window + double min; // smallest stored sample + double mean; // arithmetic mean of the stored window + double p50; // nearest-rank 50th percentile (median rank) + double p95; // nearest-rank 95th percentile + double p99; // nearest-rank 99th percentile + double max; // largest stored sample +}; + +// A fixed-capacity, allocation-free sample store (rolling window). +// See the header preamble for the full contract (window semantics, +// nearest-rank percentile definition, performance, threading). +class Histogram { + public: + struct Options { + // Capacity: the maximum number of samples kept. 0 is legal: every + // record() is dropped and stats() is always empty (useful as a + // churn-only counter, and as the loud-failure state for budgetCheck). + std::size_t capacity = 0; + }; + + // Setup path: performs the two backing allocations (window + scratch + // sort buffer). O(capacity) time and space. + explicit Histogram(Options options) noexcept + : capacity_(options.capacity), + window_(options.capacity), + scratch_(options.capacity), + cursor_(0), + count_(0), + total_(0) {} + + // Value semantics: copy is O(capacity) (deep, cold path), move O(1). + Histogram(const Histogram&) = default; + Histogram(Histogram&&) = default; + Histogram& operator=(const Histogram&) = default; + Histogram& operator=(Histogram&&) = default; + + // Record one sample (unit: whatever the caller measures — typically + // milliseconds from a TimeIt). O(1), no allocation, no lock, noexcept. + // When the window is full the oldest sample is dropped; totalRecorded() + // keeps counting, so truncation is observable. + void record(double value) noexcept { + ++total_; + if (capacity_ == 0) return; + window_[cursor_] = value; + cursor_ = (cursor_ + 1) % capacity_; + if (count_ < capacity_) ++count_; + } + + // Drop every stored sample (count -> 0). totalRecorded() survives + // (since-construction churn; per-frame profilers diff it). Idempotent. + void reset() noexcept { + cursor_ = 0; + count_ = 0; + } + + // Samples currently stored in the window (<= capacity). + [[nodiscard]] std::uint64_t count() const noexcept { return count_; } + + // Samples recorded since construction, including dropped ones (churn). + [[nodiscard]] std::uint64_t totalRecorded() const noexcept { return total_; } + + // Exact statistics over the stored window (nearest-rank percentiles — + // see the preamble). Cold path: O(n log n) time, no allocation (sorts + // the pre-allocated scratch buffer; logically const — scratch_ is a + // reusable work buffer, not state). + [[nodiscard]] HistogramStats stats() const { + HistogramStats s{}; + const std::uint64_t n = count_; + s.n = n; + if (n == 0) { + s.min = s.mean = s.p50 = s.p95 = s.p99 = s.max = std::nan(""); + return s; + } + // Copy the stored window (oldest -> newest) into the scratch buffer; + // the window itself is never reordered. + if (n == capacity_) { + // Full ring: the oldest sample sits at cursor_ (the next slot to + // overwrite), so the window starts there and wraps. + std::copy_n(window_.cbegin() + std::ptrdiff_t(cursor_), + std::ptrdiff_t(capacity_ - cursor_), scratch_.begin()); + std::copy_n(window_.cbegin(), std::ptrdiff_t(cursor_), + scratch_.begin() + std::ptrdiff_t(capacity_ - cursor_)); + } else { + // Not yet full: the window is the plain prefix [0, n). + std::copy_n(window_.cbegin(), std::ptrdiff_t(n), scratch_.begin()); + } + std::sort(scratch_.begin(), scratch_.begin() + std::ptrdiff_t(n)); + + s.min = scratch_.front(); + s.max = scratch_[n - 1]; + double sum = 0.0; + for (std::uint64_t k = 0; k < n; ++k) sum += scratch_[k]; + s.mean = sum / double(n); + s.p50 = percentile(scratch_, n, 50); + s.p95 = percentile(scratch_, n, 95); + s.p99 = percentile(scratch_, n, 99); + return s; + } + + private: + // Nearest-rank percentile over a sorted window v[0..n-1] (n >= 1), p + // in [0, 100]: rank r = ceil(p*n/100) in exact integer math (clamped + // to >= 1); the percentile is v[r-1]. (Preamble: the documented + // definition; p=0 is the min, p=100 the max.) + [[nodiscard]] static double percentile(const std::vector& sorted, + std::uint64_t n, int p) { + std::uint64_t rank = (std::uint64_t(p) * n + 99) / 100; + if (rank < 1) rank = 1; + return sorted[rank - 1]; + } + + std::size_t capacity_; + std::vector window_; // ring storage (setup allocation) + mutable std::vector scratch_; // stats() sort buffer (setup alloc) + std::size_t cursor_; // next slot to write (the oldest slot when full) + std::uint64_t count_; // samples stored (<= capacity_) + std::uint64_t total_; // samples recorded since construction +}; + +// --------------------------------------------------------------------------- +// TimeIt +// --------------------------------------------------------------------------- + +// A scope timer over std::chrono::steady_clock (monotonic — see the +// preamble). Milliseconds as a double. No allocation, no lock. +class TimeIt { + public: + // Starts the scope now. + TimeIt() noexcept : start_(std::chrono::steady_clock::now()) {} + + // Restarts the scope (owner thread only). + void reset() noexcept { start_ = std::chrono::steady_clock::now(); } + + // Elapsed time in milliseconds since construction/reset. + [[nodiscard]] double elapsedMs() const noexcept { + return std::chrono::duration_cast< + std::chrono::duration>( + std::chrono::steady_clock::now() - start_) + .count(); + } + + private: + std::chrono::steady_clock::time_point start_; +}; + +// --------------------------------------------------------------------------- +// Budget entries and checks (budgets.json) +// --------------------------------------------------------------------------- + +// The statistic of a Histogram that a budget entry checks. +enum class BudgetMetric : std::uint8_t { Mean, Min, Max, P50, P95, P99 }; + +// One named budget from budgets.json (PRD 8.1 row -> entry). See the +// preamble for the target/measured/unit/metric contract. +struct BudgetEntry { + std::string name; // stable snake_case id (machine-greppable) + BudgetMetric metric; // which statistic the check evaluates + std::string unit; // identifier unit (ms, draw_calls, allocs_per_frame, ...) + double target; // hard PRD 8.1 limit (at-most); 0 = hard zero budget + double measured; // last recorded value; 0 = not yet measured + std::string workload; // the workload the budget applies to +}; + +// The parsed budgets.json (schema v1). Immutable after loading; safe to +// read from any thread. Small by design (one row per PRD 8.1 budget): +// find() is a linear scan on the cold path (no hash map, PERF-006). +class BudgetTable { + public: + BudgetTable() = default; + + [[nodiscard]] std::size_t size() const noexcept { return entries_.size(); } + + // The entries in file order. + [[nodiscard]] const std::vector& entries() const noexcept { + return entries_; + } + + // The entry with the given name, or nullptr when absent. + [[nodiscard]] const BudgetEntry* find(std::string_view name) const noexcept { + for (const BudgetEntry& e : entries_) + if (name == e.name) return &e; + return nullptr; + } + + private: + std::vector entries_; + friend Result loadBudgets(std::string_view path); +}; + +// Caller-supplied context for the AGENTS 12 report fields the harness +// cannot know (machine/build facts). The caller (benchmark runner / +// operator) fills these; budgetCheck formats them verbatim into the +// report (diagnostic text, not engine state). Defaults to all-empty. +struct BudgetReportContext { + const char* workload = ""; // dataset/workload name + const char* build = ""; // compiler + version, build type, flags + const char* machine = ""; // hardware + OS + std::uint32_t warmup = 0; // warm-up iterations performed +}; + +// The stable one-line text form of a HistogramStats value: +// `stats: n= min= mean= p50= p95= p99= max=` +// (6 significant digits, locale-free; "nan" for an empty histogram). +// The report lines of budgetCheck and the laige-bench tool both use +// this, so the stats text has one source (LOG-001 stable fields). +// Cold path: allocates one string. +[[nodiscard]] std::string formatStatsLine(const HistogramStats& s); + +// The outcome of a budget check plus the formatted AGENTS 12 report +// (stable multi-line text; format in docs/api/budget_harness.md). +struct BudgetCheckResult { + bool passed; + double measured; // the entry's metric over the histogram window + // (NaN when the histogram was empty) + double target; // the entry's target + double before; // the entry's last recorded value (before/after pair) + std::string report; +}; + +// Loads and validates budgets.json (schema v1) from `path`. +// +// Cold path: file I/O + parse (PERF-002 is a hot-path rule; this is a +// setup/reporting path by design). Bounds: the file must not exceed 1 MiB +// (kBudgetsMaxDocumentBytes, the same bound as JsonOptions' default) and +// the document is parsed by the bounded JSON parser (ADR 0003: depth 32). +// Schema validation is strict (ARCH-007): unknown version, unknown or +// missing field, bad name/metric/unit, duplicate name, negative or +// non-finite number -> MalformedInput. Unreadable file -> IoError. +// A failed load produces no table (all-or-nothing). +[[nodiscard]] Result loadBudgets(std::string_view path); + +// Checks `entry` against `histogram` (semantics in the preamble: +// NO_SAMPLES / target>0 at-most / target==0 hard zero). before/after are +// entry.measured / the current measurement (the AGENTS 12 before/after +// pair). Cold path: O(n log n) (the stats pass) plus report string +// building (allocates — reporting is never a hot path). Thread-safe on +// const inputs; the histogram must not be mutated concurrently +// (CONC-001 single-owner rule). +[[nodiscard]] BudgetCheckResult budgetCheck(const BudgetEntry& entry, + const Histogram& histogram, + const BudgetReportContext& context = {}); + +} // namespace laige diff --git a/src/laige-core/json.cpp b/src/laige-core/json.cpp index 0b230aa..4a20904 100644 --- a/src/laige-core/json.cpp +++ b/src/laige-core/json.cpp @@ -371,6 +371,13 @@ bool parseObject(ParserState& p, JsonValue& out) { bool parseObjectMembers(ParserState& p, JsonValue& obj) { for (;;) { + // Whitespace is legal between tokens (the header grammar), including + // after the ',' of the previous member: the key goes through + // parseString directly (not parseValue), so it needs its own skip. + // (Found by M0-CORE-08: the repo-root budgets.json — a hand-formatted + // document with ", " between members — was rejected; regression tests + // in the ConfigJsonInvalid/Valid suites.) + skipWhitespace(p); JsonValue key; if (!parseString(p, key)) return false; // keys must be quoted strings skipWhitespace(p); diff --git a/tests/laige-core/CMakeLists.txt b/tests/laige-core/CMakeLists.txt index 957f4eb..ff532ba 100644 --- a/tests/laige-core/CMakeLists.txt +++ b/tests/laige-core/CMakeLists.txt @@ -14,7 +14,8 @@ set(LAIGE_CORE_TEST_SOURCES laige-core_tests.cpp result_status_tests.cpp logging_tests.cpp math_float_tests.cpp math_fixed_tests.cpp pools_tests.cpp prng_tests.cpp - config_json_tests.cpp) + config_json_tests.cpp + budget_harness_tests.cpp) # M0-CORE-02: the test-only allocation counter overrides the global # operator new/new[]; the sanitizer runtimes define their own new/delete # (strong symbols in the Clang/GCC TSan runtime archives, interposed by @@ -119,11 +120,32 @@ add_test(NAME config_json COMMAND laige-core_tests --gtest_filter=ConfigJson*) +# M0-CORE-08: budget harness (Histogram, TimeIt, budgetCheck, +# loadBudgets). The step's Verify command is `ctest -R budget_harness`; +# this entry selects exactly the BudgetHarness* suites from the shared +# laige-core_tests executable. The table-load tests read the repo-root +# budgets.json through LAIGE_BUDGETS_PATH (set for the full module +# suite too, so both entries behave identically); the "budgets.json" +# fallback only works from the source root. +add_test(NAME budget_harness + COMMAND laige-core_tests + --gtest_filter=BudgetHarness*) +# NB: CTest's ENVIRONMENT property is replaced (not appended) by a later +# set_tests_properties call on the same test — the LAIGE_TSAN block below +# therefore re-states LAIGE_BUDGETS_PATH alongside TSAN_OPTIONS. +set_tests_properties(laige-core_tests budget_harness + PROPERTIES ENVIRONMENT "LAIGE_BUDGETS_PATH=${CMAKE_SOURCE_DIR}/budgets.json") + if(LAIGE_TSAN) # Make the first data race report fatal to the test process (NFR-8.2), # so ctest fails loudly on any TSan report. set_tests_properties( laige-core_tests result_status logging math_float math_fixed pools prng - config_json - PROPERTIES ENVIRONMENT "TSAN_OPTIONS=halt_on_error=1") + config_json budget_harness + # ENVIRONMENT is replaced (not appended) by a later set_tests_properties + # call on the same test, so restate the budgets path here (see the note + # above the first set_tests_properties). List value = one semicolon- + # joined string (CTest's ENVIRONMENT is a ';'-separated list). + PROPERTIES ENVIRONMENT + "TSAN_OPTIONS=halt_on_error=1;LAIGE_BUDGETS_PATH=${CMAKE_SOURCE_DIR}/budgets.json") endif() diff --git a/tests/laige-core/budget_harness_tests.cpp b/tests/laige-core/budget_harness_tests.cpp new file mode 100644 index 0000000..734cb80 --- /dev/null +++ b/tests/laige-core/budget_harness_tests.cpp @@ -0,0 +1,554 @@ +// laige-core budget harness suite (M0-CORE-08). +// +// Step Verify scope (roadmap/M0-foundations.md): +// - `ctest -R budget_harness` green: synthetic workload produces +// percentiles; budget check fails loudly when a threshold is +// exceeded. +// Suites: BudgetHarnessHistogram (window semantics, nearest-rank +// percentiles, empty state), BudgetHarnessTimeIt (steady-clock scope +// timer), BudgetHarnessCheck (pass/fail semantics, the AGENTS 12 report +// format, before/after pair), BudgetHarnessTable (loadBudgets against +// the repo-root budgets.json + the schema v1 rejection corpus). + +#include +#include +#include +#include +#include +#include +#include +#include + +#include "gtest/gtest.h" +#include "laige/budget_harness.h" +#include "laige/prng.h" + +// --------------------------------------------------------------------------- +// NFR-8.10 policy self-checks (compile-time; a violation fails the build) +// --------------------------------------------------------------------------- + +#if defined(__cpp_exceptions) +static_assert(false, + "budget_harness_tests must be built with exceptions disabled " + "(NFR-8.10); see laige_apply_engine_policy()."); +#elif defined(__EXCEPTIONS) && __EXCEPTIONS +static_assert(false, + "budget_harness_tests must be built with exceptions disabled " + "(NFR-8.10); see laige_apply_engine_policy()."); +#endif + +#if defined(__cpp_rtti) && __cpp_rtti +static_assert(false, + "budget_harness_tests must be built with RTTI disabled " + "(NFR-8.10); see laige_apply_engine_policy()."); +#endif + +namespace { + +using laige::BudgetEntry; +using laige::BudgetMetric; +using laige::BudgetReportContext; +using laige::BudgetTable; +using laige::ErrorCode; +using laige::Histogram; +using laige::HistogramStats; +using laige::TimeIt; + +// A histogram holding exactly 1.0 .. 100.0 (capacity 100 — no drops). +Histogram Histogram1To100() { + Histogram h(Histogram::Options{100}); + for (int i = 1; i <= 100; ++i) h.record(double(i)); + return h; +} + +BudgetEntry Entry(const char* name, BudgetMetric metric, const char* unit, + double target, double measured = 0.0) { + return BudgetEntry{name, metric, unit, target, measured, "test workload"}; +} + +BudgetReportContext ReportCtx() { + return BudgetReportContext{"synthetic workload", "test compiler, Debug", + "test machine", 42}; +} + +void ExpectReportContains(const std::string& report, const char* needle) { + EXPECT_NE(std::string::npos, report.find(needle)) + << "expected the report to contain '" << needle << "'; report:\n" + << report; +} + +// Portable file open (CPP-009 platform boundary, as in logging.cpp and +// the logging suite): MSVC's plain fopen is deprecated (C4996, fatal +// under /WX), so use _fsopen with _SH_DENYNO. +#if defined(_MSC_VER) +std::FILE* openFile(const char* path, const char* mode) { + return ::_fsopen(path, mode, _SH_DENYNO); +} +#else +std::FILE* openFile(const char* path, const char* mode) { + return std::fopen(path, mode); +} +#endif + +std::string WriteTempJson(const char* name, const std::string content) { + const std::string path = std::string("laige_budget_test_") + name + ".json"; + std::remove(path.c_str()); // never test against a stale file (CORE-008) + std::FILE* f = openFile(path.c_str(), "wb"); + if (f == nullptr) { + std::fprintf(stderr, "WriteTempJson: open(\"%s\", \"wb\") failed " + "(errno=%d)\n", + path.c_str(), errno); + return {}; + } + std::fwrite(content.data(), 1, content.size(), f); + std::fclose(f); + return path; +} + +// The repo-root budgets.json, wired by CTest (ENVIRONMENT +// LAIGE_BUDGETS_PATH); the fallback covers running the binary from the +// source root by hand. +const char* BudgetsFilePath() { + const char* env = std::getenv("LAIGE_BUDGETS_PATH"); + return (env != nullptr && env[0] != '\0') ? env : "budgets.json"; +} + +void ExpectMalformedFile(const std::string& path) { + const laige::Result r = laige::loadBudgets(path); + EXPECT_TRUE(r.isError()) << "expected a load failure for " << path; + if (r.isError()) { + EXPECT_EQ(ErrorCode::MalformedInput, r.error()) + << laige::errorText(r.error()); + } +} + +void ExpectIoErrorFile(const std::string& path) { + const laige::Result r = laige::loadBudgets(path); + EXPECT_TRUE(r.isError()) << "expected a load failure for " << path; + if (r.isError()) { + EXPECT_EQ(ErrorCode::IoError, r.error()) << laige::errorText(r.error()); + } +} + +// Schema-v1 document builders for the rejection corpus: Doc(entry, +// version) and E(overrides) emit one minimal valid entry by default. +std::string E(const char* name = "b1", const char* metric = "mean", + const char* unit = "ms", const char* target = "1.0", + const char* measured = "0", const char* workload = "wl") { + return std::string("{\"name\":\"") + name + "\",\"metric\":\"" + metric + + "\",\"unit\":\"" + unit + "\",\"target\":" + target + + ",\"measured\":" + measured + ",\"workload\":\"" + workload + "\"}"; +} + +std::string Doc(const std::string& entry, const std::string& version = "1") { + return std::string("{\"version\": ") + version + ", \"budgets\": [" + + entry + "]}"; +} + +} // namespace + +// --------------------------------------------------------------------------- +// BudgetHarnessHistogram — window semantics + exact statistics +// --------------------------------------------------------------------------- + +TEST(BudgetHarnessHistogram, KnownPercentilesAndMean) { + const Histogram h = Histogram1To100(); + const HistogramStats s = h.stats(); + EXPECT_EQ(std::uint64_t(100), s.n); + EXPECT_DOUBLE_EQ(1.0, s.min); + EXPECT_DOUBLE_EQ(100.0, s.max); + EXPECT_DOUBLE_EQ(50.5, s.mean); + // nearest-rank: p50 -> v[49] = 50, p95 -> v[94] = 95, p99 -> v[98] = 99 + EXPECT_DOUBLE_EQ(50.0, s.p50); + EXPECT_DOUBLE_EQ(95.0, s.p95); + EXPECT_DOUBLE_EQ(99.0, s.p99); +} + +TEST(BudgetHarnessHistogram, SingleSampleWindow) { + Histogram h(Histogram::Options{4}); + h.record(7.25); + const HistogramStats s = h.stats(); + EXPECT_EQ(std::uint64_t(1), s.n); + EXPECT_DOUBLE_EQ(7.25, s.min); + EXPECT_DOUBLE_EQ(7.25, s.max); + EXPECT_DOUBLE_EQ(7.25, s.mean); + EXPECT_DOUBLE_EQ(7.25, s.p50); + EXPECT_DOUBLE_EQ(7.25, s.p95); + EXPECT_DOUBLE_EQ(7.25, s.p99); +} + +TEST(BudgetHarnessHistogram, FullRingWrapsOldest) { + Histogram h(Histogram::Options{3}); + for (int i = 1; i <= 4; ++i) h.record(double(i)); // window {2,3,4} + EXPECT_EQ(std::uint64_t(3), h.count()); + EXPECT_EQ(std::uint64_t(4), h.totalRecorded()); // truncation is observable + const HistogramStats s = h.stats(); + EXPECT_EQ(std::uint64_t(3), s.n); + EXPECT_DOUBLE_EQ(2.0, s.min); + EXPECT_DOUBLE_EQ(4.0, s.max); + EXPECT_DOUBLE_EQ(3.0, s.mean); + // nearest-rank on {2,3,4}: p50 -> v[1] = 3; p95 -> v[3]... n=3: + // ceil(0.95*3)=3 -> v[2] = 4; p99 -> 4 + EXPECT_DOUBLE_EQ(3.0, s.p50); + EXPECT_DOUBLE_EQ(4.0, s.p95); + EXPECT_DOUBLE_EQ(4.0, s.p99); +} + +TEST(BudgetHarnessHistogram, CapacityZeroDropsEverything) { + Histogram h(Histogram::Options{0}); + for (int i = 0; i < 5; ++i) h.record(double(i)); + EXPECT_EQ(std::uint64_t(0), h.count()); + EXPECT_EQ(std::uint64_t(5), h.totalRecorded()); + const HistogramStats s = h.stats(); + EXPECT_EQ(std::uint64_t(0), s.n); +} + +TEST(BudgetHarnessHistogram, EmptyStatsAreNan) { + const Histogram h(Histogram::Options{16}); + const HistogramStats s = h.stats(); + EXPECT_EQ(std::uint64_t(0), s.n); + EXPECT_TRUE(std::isnan(s.min)); + EXPECT_TRUE(std::isnan(s.mean)); + EXPECT_TRUE(std::isnan(s.p50)); + EXPECT_TRUE(std::isnan(s.p95)); + EXPECT_TRUE(std::isnan(s.p99)); + EXPECT_TRUE(std::isnan(s.max)); +} + +TEST(BudgetHarnessHistogram, StatsInvariantsOverDeterminedSamples) { + // 20k samples from the deterministic PRNG (M0-CORE-06): the ordering + // invariants must hold for any window, and no sample may be dropped + // (capacity == runs). + const std::uint64_t runs = 20000; + Histogram h(Histogram::Options{runs}); + laige::Prng rng(0x1234567890ABCDEF); + for (std::uint64_t i = 0; i < runs; ++i) h.record(rng.next_float01()); + EXPECT_EQ(runs, h.totalRecorded()); + const HistogramStats s = h.stats(); + EXPECT_EQ(runs, s.n); + EXPECT_LE(s.min, s.p50); + EXPECT_LE(s.p50, s.p95); + EXPECT_LE(s.p95, s.p99); + EXPECT_LE(s.p99, s.max); + EXPECT_LE(s.min, s.mean); + EXPECT_LE(s.mean, s.max); +} + +TEST(BudgetHarnessHistogram, ResetDropsWindowKeepsChurn) { + Histogram h(Histogram::Options{8}); + for (int i = 1; i <= 10; ++i) h.record(double(i)); + EXPECT_EQ(std::uint64_t(8), h.count()); + EXPECT_EQ(std::uint64_t(10), h.totalRecorded()); + h.reset(); + EXPECT_EQ(std::uint64_t(0), h.count()); + EXPECT_EQ(std::uint64_t(10), h.totalRecorded()); // churn survives + EXPECT_EQ(std::uint64_t(0), h.stats().n); + h.record(7.0); + EXPECT_EQ(std::uint64_t(1), h.count()); + EXPECT_DOUBLE_EQ(7.0, h.stats().mean); +} + +TEST(BudgetHarnessHistogram, FormatStatsLineIsStable) { + const HistogramStats s = Histogram1To100().stats(); + const std::string line = laige::formatStatsLine(s); + EXPECT_EQ("stats: n=100 min=1 mean=50.5 p50=50 p95=95 p99=99 max=100", + line); +} + +// --------------------------------------------------------------------------- +// BudgetHarnessTimeIt — steady-clock scope timer +// --------------------------------------------------------------------------- + +// Bounded busy work (~microseconds). The volatile counter (updated with +// an ordinary assignment — `++` on a volatile is deprecated, C++23) +// keeps the loop from constant-folding away at high optimization levels, +// so the timer tests measure real elapsed time in every build type. +std::int64_t Spin(std::int64_t iterations) { + std::int64_t acc = 0; + volatile std::int64_t i = 0; + while (i < iterations) { + acc += i * 3 - 1; + i = i + 1; + } + return acc; +} + +TEST(BudgetHarnessTimeIt, ElapsedIsNonNegativeAndMonotonic) { + TimeIt t; + const double e0 = t.elapsedMs(); + EXPECT_GE(e0, 0.0); + const volatile std::int64_t acc = Spin(2000000); + (void)acc; + const double e1 = t.elapsedMs(); + EXPECT_GE(e1, e0); +} + +TEST(BudgetHarnessTimeIt, ElapsedIsPositiveAfterWork) { + TimeIt t; + const volatile std::int64_t acc = Spin(2000000); + (void)acc; + EXPECT_GT(t.elapsedMs(), 0.0); +} + +TEST(BudgetHarnessTimeIt, ResetRestartsScope) { + TimeIt t; + const volatile std::int64_t acc = Spin(2000000); + (void)acc; + const double before = t.elapsedMs(); + t.reset(); + const double after = t.elapsedMs(); + EXPECT_GE(after, 0.0); + EXPECT_LT(after, before); +} + +// --------------------------------------------------------------------------- +// BudgetHarnessCheck — pass/fail semantics + AGENTS 12 report +// --------------------------------------------------------------------------- + +TEST(BudgetHarnessCheck, PassesWhenMeasuredUnderTarget) { + Histogram h(Histogram::Options{8}); + for (int i = 1; i <= 5; ++i) h.record(double(i)); // mean = 3 + const laige::BudgetCheckResult r = + laige::budgetCheck(Entry("mean_metric", BudgetMetric::Mean, "ms", + 4.0, /*measured=*/2.5), + h, ReportCtx()); + EXPECT_TRUE(r.passed); + EXPECT_DOUBLE_EQ(3.0, r.measured); + EXPECT_DOUBLE_EQ(4.0, r.target); + EXPECT_DOUBLE_EQ(2.5, r.before); + ExpectReportContains(r.report, + "budget=mean_metric result=PASS metric=mean unit=ms"); + ExpectReportContains(r.report, "after=3"); + ExpectReportContains(r.report, "before=2.5"); + ExpectReportContains(r.report, "target=4"); + ExpectReportContains(r.report, "n=5"); +} + +TEST(BudgetHarnessCheck, FailsLoudlyWhenTargetExceeded) { + // The step's Verify clause: the budget check fails loudly when a + // threshold is exceeded — a distinct structured failure, never silent. + Histogram h(Histogram::Options{4}); + for (int i = 0; i < 3; ++i) h.record(5.0); // mean = 5 > target 4 + const laige::BudgetCheckResult r = + laige::budgetCheck(Entry("over", BudgetMetric::Mean, "ms", 4.0), h, + ReportCtx()); + EXPECT_FALSE(r.passed); + EXPECT_DOUBLE_EQ(5.0, r.measured); + ExpectReportContains(r.report, "result=FAIL"); + ExpectReportContains(r.report, "after=5"); + ExpectReportContains(r.report, "target=4"); + ExpectReportContains(r.report, "n=3"); +} + +TEST(BudgetHarnessCheck, FailsLoudlyOnEmptyHistogram) { + // A workload that recorded nothing is a broken harness: loud NO_SAMPLES, + // never "passed" on NaN statistics (CORE-008). + Histogram h(Histogram::Options{4}); + const laige::BudgetCheckResult r = + laige::budgetCheck(Entry("empty", BudgetMetric::Mean, "ms", 4.0), h, + ReportCtx()); + EXPECT_FALSE(r.passed); + EXPECT_TRUE(std::isnan(r.measured)); + ExpectReportContains(r.report, "result=NO_SAMPLES"); + ExpectReportContains(r.report, "n=0"); +} + +TEST(BudgetHarnessCheck, ZeroTargetBudgetRequiresExactlyZero) { + // target == 0 is a hard-zero budget (sim heap allocations), not + // "not set": measured 0 passes, any non-zero fails. + Histogram zeros(Histogram::Options{3}); + zeros.record(0.0); + zeros.record(0.0); + zeros.record(0.0); + const laige::BudgetCheckResult ok = laige::budgetCheck( + Entry("allocs", BudgetMetric::Max, "allocs_per_frame", 0.0), zeros, + ReportCtx()); + EXPECT_TRUE(ok.passed); + EXPECT_DOUBLE_EQ(0.0, ok.measured); + ExpectReportContains(ok.report, "result=PASS"); + + Histogram nonzero(Histogram::Options{3}); + nonzero.record(0.0); + nonzero.record(1.0); + nonzero.record(0.0); + const laige::BudgetCheckResult bad = laige::budgetCheck( + Entry("allocs", BudgetMetric::Max, "allocs_per_frame", 0.0), nonzero, + ReportCtx()); + EXPECT_FALSE(bad.passed); + EXPECT_DOUBLE_EQ(1.0, bad.measured); + ExpectReportContains(bad.report, "result=FAIL"); +} + +TEST(BudgetHarnessCheck, ChecksTheNamedMetric) { + const Histogram h = Histogram1To100(); + const BudgetReportContext ctx; + EXPECT_DOUBLE_EQ(50.0, laige::budgetCheck( + Entry("m", BudgetMetric::P50, "ms", 100.0), h, + ctx) + .measured); + EXPECT_DOUBLE_EQ(95.0, laige::budgetCheck( + Entry("m", BudgetMetric::P95, "ms", 100.0), h, + ctx) + .measured); + EXPECT_DOUBLE_EQ(99.0, laige::budgetCheck( + Entry("m", BudgetMetric::P99, "ms", 100.0), h, + ctx) + .measured); + EXPECT_DOUBLE_EQ(1.0, laige::budgetCheck( + Entry("m", BudgetMetric::Min, "ms", 100.0), h, ctx) + .measured); + EXPECT_DOUBLE_EQ(100.0, laige::budgetCheck( + Entry("m", BudgetMetric::Max, "ms", 100.0), h, + ctx) + .measured); +} + +TEST(BudgetHarnessCheck, ReportCarriesCallerContext) { + Histogram h(Histogram::Options{2}); + h.record(1.0); + h.record(2.0); + const BudgetReportContext ctx{"iso scene", "clang++ 22.1.8 Debug", + "ci runner", 17}; + const laige::BudgetCheckResult r = + laige::budgetCheck(Entry("ctx", BudgetMetric::Mean, "ms", 10.0), h, ctx); + ExpectReportContains(r.report, "workload=iso scene"); + ExpectReportContains(r.report, "build=clang++ 22.1.8 Debug"); + ExpectReportContains(r.report, "machine=ci runner"); + ExpectReportContains(r.report, "warmup=17"); +} + +TEST(BudgetHarnessCheck, ReportFirstLineIsStable) { + // The first line is the machine-greppable contract (LOG-001): + // budget= result= metric= unit= + Histogram h(Histogram::Options{2}); + h.record(1.0); + h.record(2.0); // p95 of {1,2} = 2 > target 1.5 -> FAIL + const laige::BudgetCheckResult r = + laige::budgetCheck(Entry("stable", BudgetMetric::P95, "ms", 1.5), h, + ReportCtx()); + const std::string first = + r.report.substr(0, r.report.find('\n')); + EXPECT_EQ("budget=stable result=FAIL metric=p95 unit=ms", first); +} + +// --------------------------------------------------------------------------- +// BudgetHarnessTable — loadBudgets: repo file + schema v1 corpus +// --------------------------------------------------------------------------- + +TEST(BudgetHarnessTable, LoadsTheRepoBudgetsFile) { + const laige::Result table = + laige::loadBudgets(BudgetsFilePath()); + ASSERT_TRUE(table.ok()) << laige::errorText(table.error()); + const BudgetTable& t = table.value(); + + // One entry per PRD 8.1 target (15 rows: sim tick, 50k sprites, cold + // start, build time, and zone server each carry two budgets). + EXPECT_EQ(std::size_t(15), t.size()); + + const BudgetEntry* frame = t.find("frame_time_render"); + ASSERT_NE(frame, nullptr); + EXPECT_EQ(BudgetMetric::P95, frame->metric); + EXPECT_EQ("ms", frame->unit); + EXPECT_DOUBLE_EQ(8.3, frame->target); + EXPECT_DOUBLE_EQ(0.0, frame->measured); // not yet measured (M0) + EXPECT_FALSE(frame->workload.empty()); + + const BudgetEntry* allocs = t.find("sim_heap_allocs"); + ASSERT_NE(allocs, nullptr); + EXPECT_DOUBLE_EQ(0.0, allocs->target); // hard-zero budget, not "unset" + EXPECT_EQ(BudgetMetric::Max, allocs->metric); + EXPECT_EQ("allocs_per_frame", allocs->unit); + + EXPECT_EQ(nullptr, t.find("no_such_budget")); + + // Re-verify every entry is well-formed (a hand-edited budgets.json must + // fail here, not just in the loader). + for (const BudgetEntry& e : t.entries()) { + EXPECT_FALSE(e.name.empty()); + EXPECT_FALSE(e.unit.empty()); + EXPECT_FALSE(e.workload.empty()); + EXPECT_TRUE(std::isfinite(e.target) && e.target >= 0.0) << e.name; + EXPECT_TRUE(std::isfinite(e.measured) && e.measured >= 0.0) << e.name; + } +} + +TEST(BudgetHarnessTable, RejectsSchemaViolations) { + const std::string badJson = WriteTempJson("bad_json", "{"); + ExpectMalformedFile(badJson); + std::remove(badJson.c_str()); + + const std::string badVersion = WriteTempJson("bad_version", Doc(E(), "2")); + ExpectMalformedFile(badVersion); + std::remove(badVersion.c_str()); + + const std::string notObject = WriteTempJson("not_object", "[1, 2]"); + ExpectMalformedFile(notObject); + std::remove(notObject.c_str()); + + const std::string missingField = + WriteTempJson("missing_field", + Doc("{\"name\":\"b1\",\"metric\":\"mean\",\"unit\":\"ms\"," + "\"target\":1.0,\"measured\":0}")); + ExpectMalformedFile(missingField); + std::remove(missingField.c_str()); + + const std::string unknownField = + WriteTempJson("unknown_field", + Doc(E() + ",\"extra\":1}")); + ExpectMalformedFile(unknownField); + std::remove(unknownField.c_str()); + + const std::string badMetric = WriteTempJson("bad_metric", Doc(E("b1", "p5"))); + ExpectMalformedFile(badMetric); + std::remove(badMetric.c_str()); + + const std::string dupName = + WriteTempJson("dup_name", Doc(E() + ", " + E())); + ExpectMalformedFile(dupName); + std::remove(dupName.c_str()); + + const std::string negTarget = + WriteTempJson("neg_target", Doc(E("b1", "mean", "ms", "-1.0"))); + ExpectMalformedFile(negTarget); + std::remove(negTarget.c_str()); + + // 1e999 is well-formed JSON that the parser stores as +inf (ADR 0003): + // the schema rejects non-finite numbers explicitly. + const std::string infTarget = + WriteTempJson("inf_target", Doc(E("b1", "mean", "ms", "1e999"))); + ExpectMalformedFile(infTarget); + std::remove(infTarget.c_str()); + + const std::string emptyName = + WriteTempJson("empty_name", Doc(E("", "mean", "ms", "1.0"))); + ExpectMalformedFile(emptyName); + std::remove(emptyName.c_str()); + + const std::string badNameChars = + WriteTempJson("bad_name_chars", Doc(E("b 1"))); + ExpectMalformedFile(badNameChars); + std::remove(badNameChars.c_str()); + + const std::string badUnit = + WriteTempJson("bad_unit", Doc(E("b1", "mean", "ms x", "1.0"))); + ExpectMalformedFile(badUnit); + std::remove(badUnit.c_str()); + + const std::string emptyWorkload = + WriteTempJson("empty_workload", Doc(E("b1", "mean", "ms", "1.0", "0", + ""))); + ExpectMalformedFile(emptyWorkload); + std::remove(emptyWorkload.c_str()); +} + +TEST(BudgetHarnessTable, RejectsMissingFile) { + ExpectIoErrorFile("laige_budget_test_does_not_exist.json"); +} + +TEST(BudgetHarnessTable, EmptyBudgetsArrayIsValid) { + const std::string path = WriteTempJson("empty", Doc("")); + const laige::Result r = laige::loadBudgets(path); + ASSERT_TRUE(r.ok()) << laige::errorText(r.error()); + EXPECT_EQ(std::size_t(0), r.value().size()); + std::remove(path.c_str()); +} diff --git a/tests/laige-core/config_json_tests.cpp b/tests/laige-core/config_json_tests.cpp index 75cf38e..05acc7e 100644 --- a/tests/laige-core/config_json_tests.cpp +++ b/tests/laige-core/config_json_tests.cpp @@ -282,6 +282,46 @@ TEST(ConfigJsonValid, Containers) { } } +// M0-CORE-08 regression: whitespace after the ',' of an object member +// must be accepted (the grammar allows whitespace between tokens). The +// object key goes through parseString directly (not parseValue, which +// does the skipping), so the key needs its own whitespace skip — the old +// parseObjectMembers rejected {"a": 1, "b": 2}. Found when the +// hand-formatted repo-root budgets.json was rejected (M0-CORE-08). +TEST(ConfigJsonValid, ObjectMemberWhitespace) { + { + const auto r = Parse(R"({"a": 1, "b": 2})"); + ASSERT_TRUE(r.ok()); + const JsonValue* a = r.value().findMember("a"); + ASSERT_NE(nullptr, a); + EXPECT_EQ(1.0, a->asNumber()); + const JsonValue* b = r.value().findMember("b"); + ASSERT_NE(nullptr, b); + EXPECT_EQ(2.0, b->asNumber()); + } + // Hand-formatted (multi-line) document — the shape of budgets.json: + // newlines + indentation between members. + { + const auto r = Parse("{\n \"version\": 1,\n \"budgets\": []\n}"); + ASSERT_TRUE(r.ok()); + const JsonValue* v = r.value().findMember("version"); + ASSERT_NE(nullptr, v); + EXPECT_EQ(1.0, v->asNumber()); + const JsonValue* budgets = r.value().findMember("budgets"); + ASSERT_NE(nullptr, budgets); + EXPECT_TRUE(budgets->isArray()); + EXPECT_TRUE(budgets->asArray().empty()); + } + // Whitespace before the closing brace after the last member. + { + const auto r = Parse(R"({"a": 1 })"); + ASSERT_TRUE(r.ok()); + const JsonValue* a = r.value().findMember("a"); + ASSERT_NE(nullptr, a); + EXPECT_EQ(1.0, a->asNumber()); + } +} + TEST(ConfigJsonValid, DepthAtLimit) { // Default maxDepth is 32: exactly 32 nested containers parse, and the // walk reaches the null leaf at level 32. diff --git a/tools/bench/CMakeLists.txt b/tools/bench/CMakeLists.txt new file mode 100644 index 0000000..a8d52fd --- /dev/null +++ b/tools/bench/CMakeLists.txt @@ -0,0 +1,34 @@ +# laige-bench (M0-CORE-08): the canonical benchmark command +# (docs/getting-started/building.md): +# +# ./build/bin/laige-bench --suite= [--runs=N] [--warmup=N] +# [--budget=] [--budgets=] +# [--report=] +# +# Runs a synthetic suite through the budget harness (laige::Histogram + +# TimeIt + budgetCheck) and prints the AGENTS 12 report. Gated with the +# test suite like tools/fuzz: a library-only build does not need it. + +add_executable(laige-bench laige-bench.cpp) +laige_apply_engine_policy(laige-bench) +target_link_libraries(laige-bench PRIVATE laige-core) + +# The report's "build" context field records the build type (AGENTS 12); +# CMake stamps it so the binary does not have to guess. +target_compile_definitions(laige-bench PRIVATE + LAIGE_BENCH_BUILD_TYPE="${CMAKE_BUILD_TYPE}") + +# Smoke test: the Verify clause of M0-CORE-08 in executable form — a +# synthetic workload produces percentiles (the stats line must carry the +# sample count) and the tool exits 0. Timing values are never asserted +# (machine-dependent); the output shape is the contract. +add_test(NAME laige_bench_smoke COMMAND laige-bench --suite=synthetic + --runs=50 --warmup=10) +set_tests_properties(laige_bench_smoke + PROPERTIES PASS_REGULAR_EXPRESSION "stats: n=50") + +if(LAIGE_TSAN) + # Same first-report-fatal policy as the other test entries (NFR-8.2). + set_tests_properties(laige_bench_smoke + PROPERTIES ENVIRONMENT "TSAN_OPTIONS=halt_on_error=1") +endif() diff --git a/tools/bench/laige-bench.cpp b/tools/bench/laige-bench.cpp new file mode 100644 index 0000000..798fe95 --- /dev/null +++ b/tools/bench/laige-bench.cpp @@ -0,0 +1,284 @@ +// laige-bench (M0-CORE-08) — the canonical benchmark command +// (docs/getting-started/building.md is the source of truth): +// +// ./build/bin/laige-bench --suite= [--runs=N] [--warmup=N] +// [--budget=] [--budgets=] +// [--report=] +// +// Runs the named synthetic suite: `warmup` discarded iterations, then +// `runs` timed iterations recorded into a laige::Histogram; prints the +// AGENTS 12 summary (sample count, min/mean/p50/p95/p99/max, context). +// With --budget= the result is additionally checked against that +// budgets.json entry (loadBudgets + budgetCheck) and the pass/fail +// report is printed; a failed check exits non-zero so the command is +// CI-gateable (PRD 8.1 policy: a budget regression fails CI). +// +// Exit codes: +// 0 ok (and the budget check passed, when --budget was given) +// 1 usage error, unknown suite, or budgets.json unreadable/malformed +// 2 the budget check failed (loud failure, CORE-008) +// +// The synthetic suite is a deterministic, allocation-free stand-in +// workload: it exists so the harness (timer, histogram, report, budget +// check) is testable end to end in M0, before the real PRD 8.1 +// workloads land with their subsystems (M1+). The M0-EXIT-01 gate runs +// `--suite=synthetic` and records the result in +// docs/benchmarks/baselines/m0-synthetic.md. + +#include +#include +#include +#include +#include +#include + +#include "laige/budget_harness.h" + +namespace { + +// --- The synthetic suite ---------------------------------------------------- + +// LCG64 constants (Marsaglia, "Random Numbers", 2003 — the 64-bit LCG; +// cited per CPP-014). The constants are arbitrary and named (CORE-005): +// the workload models no physical quantity — its job is to exercise the +// measurement pipeline (timer, histogram, report, budget check). +constexpr std::uint64_t kSyntheticLcgMultiplier = 6364136223846793005ULL; +constexpr std::uint64_t kSyntheticLcgIncrement = 1442695040888963407ULL; +constexpr int kSyntheticSteps = 4096; +constexpr std::uint64_t kSyntheticState0 = 0x1234567890ABCDEFULL; + +// Measured iteration: a fixed 4096-step mixed 64-bit-integer + double +// pipeline. Deterministic, bounded, allocation-free. The accumulator is +// stored through a module-level volatile so the pipeline cannot be +// optimized away. +volatile std::int64_t g_syntheticSink = 0; + +void syntheticIteration() { + double scale = 1.0; + std::uint64_t state = kSyntheticState0; + for (int i = 0; i < kSyntheticSteps; ++i) { + state = state * kSyntheticLcgMultiplier + kSyntheticLcgIncrement; + scale += 0.5 * static_cast((state >> 33) & 0xFF); + } + g_syntheticSink = static_cast(scale * 1.0e6) + + static_cast(state >> 32); +} + +struct Suite { + const char* name; + const char* description; // printed with --list; docs live in this file + void (*iteration)(); +}; + +const Suite kSuites[] = { + {"synthetic", + "deterministic 4096-step LCG+double pipeline; harness stand-in " + "workload (M0; the M0-EXIT-01 baseline workload)", + syntheticIteration}, +}; + +// --- Argument parsing ------------------------------------------------------- + +struct Config { + std::string suite; + std::uint64_t runs = 1000; + std::uint64_t warmup = 100; + std::string budgetName; + std::string budgetsPath; // empty -> env LAIGE_BUDGETS_PATH -> "budgets.json" + std::string reportPath; +}; + +bool parseUint(std::string_view text, std::uint64_t& out) { + if (text.empty()) return false; + std::uint64_t v = 0; + for (const char c : text) { + if (c < '0' || c > '9') return false; + v = v * 10 + std::uint64_t(c - '0'); + } + out = v; + return true; +} + +void usage(std::FILE* out) { + std::fprintf(out, + "usage: laige-bench --suite= [--runs=N] " + "[--warmup=N] [--budget=] " + "[--budgets=] [--report=] [--list]\n"); +} + +bool parseArgs(int argc, char** argv, Config& cfg) { + for (int i = 1; i < argc; ++i) { + const std::string arg = argv[i]; + const auto eq = arg.find('='); + const bool hasValue = eq != std::string::npos && eq + 1 < arg.size(); + const std::string key = hasValue ? arg.substr(0, eq) : arg; + const std::string value = hasValue ? arg.substr(eq + 1) : ""; + + if (key == "--suite") { + if (!hasValue) return false; + cfg.suite = value; + } else if (key == "--runs") { + if (!hasValue || !parseUint(value, cfg.runs) || cfg.runs == 0) + return false; + } else if (key == "--warmup") { + if (!hasValue || !parseUint(value, cfg.warmup)) return false; + } else if (key == "--budget") { + if (!hasValue) return false; + cfg.budgetName = value; + } else if (key == "--budgets") { + if (!hasValue) return false; + cfg.budgetsPath = value; + } else if (key == "--report") { + if (!hasValue) return false; + cfg.reportPath = value; + } else if (key == "--list") { + for (const Suite& s : kSuites) + std::printf("%-12s %s\n", s.name, s.description); + std::exit(0); + } else { + return false; // unknown argument + } + } + return !cfg.suite.empty(); // --suite is required +} + +// --- Output ------------------------------------------------------------------ + +// The compile-time build identity for the AGENTS 12 "build" context +// field (compiler + version + build type). CMake stamps the build type +// via LAIGE_BENCH_BUILD_TYPE. +#define LAIGE_BENCH_STR2(x) #x +#define LAIGE_BENCH_STR(x) LAIGE_BENCH_STR2(x) +#if defined(__GNUC__) +constexpr char kCompilerId[] = "GCC " __VERSION__; +#elif defined(__clang__) +constexpr char kCompilerId[] = "Clang " __VERSION__; +#elif defined(_MSC_VER) +constexpr char kCompilerId[] = "MSVC " LAIGE_BENCH_STR(_MSC_VER); +#else +constexpr char kCompilerId[] = "unknown"; +#endif +#ifndef LAIGE_BENCH_BUILD_TYPE +#define LAIGE_BENCH_BUILD_TYPE "unknown" +#endif + +} // namespace + +int main(int argc, char** argv) { + using laige::BudgetReportContext; + using laige::Histogram; + using laige::TimeIt; + + Config cfg; + if (!parseArgs(argc, argv, cfg)) { + usage(stderr); + return 1; + } + + const Suite* suite = nullptr; + for (const Suite& s : kSuites) + if (s.name == cfg.suite) suite = &s; + if (suite == nullptr) { + std::fprintf(stderr, "laige-bench: unknown suite '%s' (--list)\n", + cfg.suite.c_str()); + return 1; + } + + // Warm-up (discarded) — the timer and the CPU caches settle before the + // measured region (AGENTS 12 records the warm-up count in the report). + for (std::uint64_t i = 0; i < cfg.warmup; ++i) suite->iteration(); + + // Measured region: one TimeIt per iteration, recorded into a + // histogram sized to the run (every sample is kept — no truncation). + Histogram histogram(Histogram::Options{cfg.runs}); + for (std::uint64_t i = 0; i < cfg.runs; ++i) { + TimeIt timer; + suite->iteration(); + histogram.record(timer.elapsedMs()); + } + + // Context the tool owns (AGENTS 12: the caller harness records + // hardware/OS/compiler/build/workload). The operator may set + // LAIGE_BENCH_MACHINE for the machine line; the baseline document + // records the rest (docs/benchmarks/, M0-EXIT-01). + const char* envMachine = std::getenv("LAIGE_BENCH_MACHINE"); + const std::string build = + std::string(kCompilerId) + ", " + LAIGE_BENCH_BUILD_TYPE; + + std::string output; + output += "suite="; + output += suite->name; + output += " runs="; + output += std::to_string(cfg.runs); + output += " warmup="; + output += std::to_string(cfg.warmup); + + int exitCode = 0; + if (cfg.budgetName.empty()) { + // Plain run: the summary statistics block only (the workload line + // carries the suite's own identity). + const laige::HistogramStats s = histogram.stats(); + output += "\n "; + output += laige::formatStatsLine(s); + output += "\n context: workload="; + output += suite->name; + output += " build="; + output += build; + output += " machine="; + output += (envMachine != nullptr ? envMachine : ""); + output += " warmup="; + output += std::to_string(cfg.warmup); + output += "\n"; + } else { + // Budget run: load budgets.json, check the named entry, print the + // full AGENTS 12 report (pass/fail + before/after + statistics). + std::string budgetsPath = cfg.budgetsPath; + if (budgetsPath.empty()) { + const char* envPath = std::getenv("LAIGE_BUDGETS_PATH"); + budgetsPath = (envPath != nullptr && envPath[0] != '\0') + ? std::string(envPath) + : std::string("budgets.json"); + } + const laige::Result table = + laige::loadBudgets(budgetsPath); + if (table.isError()) { + std::fprintf(stderr, "laige-bench: loadBudgets(\"%s\") failed: %s\n", + budgetsPath.c_str(), laige::errorText(table.error())); + return 1; + } + const laige::BudgetEntry* entry = table.value().find(cfg.budgetName); + if (entry == nullptr) { + std::fprintf(stderr, + "laige-bench: no budget named '%s' in %s\n", + cfg.budgetName.c_str(), budgetsPath.c_str()); + return 1; + } + + BudgetReportContext ctx; + ctx.workload = entry->workload.c_str(); + ctx.build = build.c_str(); + ctx.machine = envMachine != nullptr ? envMachine : ""; + ctx.warmup = static_cast(cfg.warmup); + + const laige::BudgetCheckResult check = + laige::budgetCheck(*entry, histogram, ctx); + output += "\n"; + output += check.report; + exitCode = check.passed ? 0 : 2; + } + + std::fputs(output.c_str(), stdout); + std::fflush(stdout); + + if (!cfg.reportPath.empty()) { + std::FILE* f = std::fopen(cfg.reportPath.c_str(), "a"); + if (f == nullptr) { + std::fprintf(stderr, "laige-bench: cannot open report file '%s'\n", + cfg.reportPath.c_str()); + return 1; + } + std::fputs(output.c_str(), f); + std::fclose(f); + } + return exitCode; +} From a6181216f921162ddb5c67f9ee5506a18cbb11d5 Mon Sep 17 00:00:00 2001 From: Pascal Severin Date: Fri, 11 Sep 2026 22:53:18 +0200 Subject: [PATCH 2/2] [M0-CORE-08] Roadmap: check the box, board to 17, change-log lines MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit M0-foundations.md: M0-CORE-08 checked, with Decision (2026-09-11), Verify (22 GTest cases + laige_bench_smoke; 16/16 ctest on GCC static, shared, ASan, TSan and Clang trees) and Size notes, including the record of the M0-CORE-07 parser bug fix found by this step's Verify run (missing skipWhitespace before the object member key). README.md: Progress Board M0 11 -> 17 (Total 17), and the change log gains this step's line (810c251) plus the five reconstructed lines for M0-CORE-03..07 (0fd41f3, 5c76991+7538053, 4a572af+ba4ffb2, a82de8e, 41938b0) rebuilt from their step records in M0-foundations.md — the board and change log had drifted (11 vs 16 checked boxes) since M0-CORE-03 landed. --- roadmap/M0-foundations.md | 98 +++++++++++++++++++++++++++++++++++++-- roadmap/README.md | 12 +++-- 2 files changed, 104 insertions(+), 6 deletions(-) diff --git a/roadmap/M0-foundations.md b/roadmap/M0-foundations.md index 759fc29..9e40536 100644 --- a/roadmap/M0-foundations.md +++ b/roadmap/M0-foundations.md @@ -647,15 +647,107 @@ No rendering, no physics, no networking yet — `laige-core` only. prove the step's Verify clauses — depth/size bounds, the malformed corpus, round-trip, value semantics — in one suite) -- [ ] **M0-CORE-08 · Budget harness (histogram + budget checks)** +- [x] **M0-CORE-08 · Budget harness (histogram + budget checks)** - **Refs:** PRD §8.1, CORE-001; AGENTS §12 (benchmark report requirements) - **Depends:** M0-CORE-01 - **Scope:** - `laige::histogram`: fixed-capacity histogram with p50/p95/p99/mean/min/max (no allocation after construction). - `laige::timeit` scope timer; `laige::budget_check(name, histogram, budget.json entry)`: returns pass/fail with before/after numbers formatted per AGENTS §12 report fields (hardware/OS/compiler/build/workload recorded by the caller harness). - `budgets.json` schema at repo root with the PRD §8.1 numbers as named entries (values may start as `0` = "not yet measured" for M2+ budgets). - - **Verify:** `ctest -R budget_harness` green: synthetic workload produces percentiles; budget check fails loudly when a threshold is exceeded. - - **Size:** ~250 lines + tests + - **Decision (2026-09-11):** three-piece harness in `laige-core` plus the + `laige-bench` tool — `docs/getting-started/building.md` reserved the + canonical command `./build/bin/laige-bench --suite=` for this + step, and M0-EXIT-01 needs a runnable end-to-end harness. + `laige::Histogram`: fixed-capacity **rolling window** — `record()` + is O(1) and allocation-free (one index arithmetic + one store); a + full window drops the oldest sample and `totalRecorded()` exposes + the truncation (CORE-008: no silent drop); `stats()` is a cold-path + O(n log n) pass with **no allocation** (sorts a pre-allocated + scratch buffer) and yields exact min/mean/p50/p95/p99/max over the + stored window with **nearest-rank percentiles** (rank = + ceil(p·n/100) in exact integer math — no fractional indices, + bit-identical on every platform); an empty window is `n = 0` with + NaN statistics. `laige::TimeIt`: `steady_clock` scope timer in + milliseconds, no allocation, immutable start point (safe const + reads). `loadBudgets`: file I/O + bounded parse (ADR 0003 — the 1 MiB + bound is enforced before the read) + **strict schema v1 validation** + (ARCH-007: unsupported version, unknown/missing field, bad + name/metric/unit charset, duplicate name, negative or non-finite + number all rejected — `MalformedInput`; unreadable file → + `IoError`; no new error codes). `budgetCheck` is total (it cannot + fail as an operation): empty histogram → `passed = false` + loud + `NO_SAMPLES` (a workload that recorded nothing is a broken harness, + CORE-008); `target > 0` → `measured <= target` (every PRD §8.1 budget + is an at-most upper bound); `target == 0` is a **hard-zero budget** + (e.g. `sim_heap_allocs`: zero steady-state heap allocations per + frame), **not** "not set" — the "not yet measured" marker lives in + `measured` (initial 0, the M0 convention). Report: stable 4-line + machine-greppable text (LOG-001) — + `budget= result= metric= unit=` / + `after= before= target=` / stats line via + `laige::formatStatsLine` (the single source of the stats text) / + `context: workload=… build=… machine=… warmup=…` — the AGENTS §12 + machine/build facts are recorded by the caller's harness (the tool + supplies compiler + build type; the operator supplies the machine via + `LAIGE_BENCH_MACHINE`). `budgets.json` (repo root, schema v1): + **all 15 PRD §8.1 targets** as named entries (sim tick, 50k-sprite + scene, cold start, build time, and the zone server each carry two + budgets); `measured: 0` = not yet measured until their subsystems + land (M1+). `laige-bench`: `--suite` registry (M0: `synthetic` — a + deterministic, allocation-free 4096-step LCG + double stand-in + workload with the named Marsaglia LCG64 constants, no RNG), + `--runs`/`--warmup` (defaults 1000/100), `--budget=` check with + `--budgets` path resolution (argument → `LAIGE_BUDGETS_PATH` env → + `budgets.json`), `--report=` append; exit codes 0 pass / 1 + usage or load failure / **2 budget check failure** (CI-gateable — + PRD §8.1 policy). API contract: `docs/api/budget_harness.md`. + **Bug fix found by this step's Verify run (M0-CORE-07):** the JSON + parser rejected object members separated by `", "` — the object key + goes through `parseString` directly (not `parseValue`, which does + the whitespace skip), so `parseObjectMembers` was missing one + `skipWhitespace`; the repo-root `budgets.json` (hand-formatted) + demonstrated it. Fixed in `json.cpp` with a regression test + (`ConfigJsonValid.ObjectMemberWhitespace`, fails pre-fix) — the + documented grammar ("whitespace only between tokens") was already + correct; the implementation did not match it. + - **Verify:** `ctest -R budget_harness` green — **22 GTest cases**: + `BudgetHarnessHistogram` (known percentiles on 1..100; single-sample + window; full ring dropping the oldest with observable churn; + capacity-0 drop-all; empty → NaN stats; ordering invariants over a + 20k-sample deterministic PRNG window; reset/churn semantics; stable + `formatStatsLine` text), `BudgetHarnessTimeIt` (non-negative and + monotonic; positive after bounded work; restart on reset), + `BudgetHarnessCheck` (PASS under target; **loud FAIL when a + threshold is exceeded** — the step's Verify clause, with a distinct + structured `result=FAIL` report; loud `NO_SAMPLES` on an empty + histogram; hard-zero-target budget semantics; named-metric dispatch; + caller context in the report; stable first report line), + `BudgetHarnessTable` (the repo-root `budgets.json` loads with 15 + well-formed entries — `frame_time_render` p95 8.3 ms, + `sim_heap_allocs` target 0 — and a 13-case schema-rejection corpus: + malformed JSON, bad version, root not an object, missing field, + unknown field, bad metric, duplicate name, negative target, + non-finite `1e999` target, empty name, bad name charset, bad unit, + empty workload; missing file → `IoError`; an empty budgets array is + valid). The **synthetic workload producing percentiles** is also + exercised end-to-end by the `laige_bench_smoke` CTest entry + (`laige-bench --suite=synthetic --runs=50`, output must carry + `stats: n=50`). Verified locally 2026-09-11: full `ctest` (16/16 + entries, zero warnings under the NFR-8.10 policy) on GCC 16.2.1 + (`build` static, `build-shared` shared, `build-asan` ASan+UBSan + fatal, `build-tsan` TSan `halt_on_error=1`) and Clang 22.1.8 + (`build-clang`). CI will additionally prove MSVC (windows lane) and + AppleClang (macos lane) compilation of the new sources. + - **Size:** ~350 lines header (`budget_harness.h`, full AGENTS §9 + contract) + ~330 lines `budget_harness.cpp` + ~250 lines + `laige-bench.cpp` + CMake (~30) + ~560 lines tests + ~210 lines docs + + ~110 lines `budgets.json` + a 6-line `json.cpp` fix and a 39-line + regression test (over the ~250-line estimate: the header carries the + API contract, the tests prove the Verify clauses — percentiles, loud + FAIL on exceeded threshold, the schema corpus — and `laige-bench` + implements the canonical command this step reserved in + `building.md`, so M0-EXIT-01 has a runnable harness; cohesive, not + split — same pattern as M0-CORE-01…07) ## Tooling diff --git a/roadmap/README.md b/roadmap/README.md index e443ba5..dae3fea 100644 --- a/roadmap/README.md +++ b/roadmap/README.md @@ -155,7 +155,7 @@ Updated in the same PR that closes steps. "Done" = box checked + Verify green. | Milestone | Steps | Done | Status | |---|---|---|---| -| M0 | 22 | 11 | ▶ in progress | +| M0 | 22 | 17 | ▶ in progress | | M1 | 25 | 0 | ⬜ not started | | M2 | 32 | 0 | ⬜ not started | | M3 | 36 | 0 | ⬜ not started | @@ -165,7 +165,7 @@ Updated in the same PR that closes steps. "Done" = box checked + Verify green. | M7 | 15 | 0 | ⬜ not started | | M8 | 8 | 0 | ⬜ not started | | M9 | 6 | 0 | ⬜ proposals only | -| **Total** | **193** | **11** | | +| **Total** | **193** | **17** | | --- @@ -176,7 +176,7 @@ One line per completed (or split/renumbered) step. | Date | Step | Commit | Note | |---|---|---|---| | 2026-09-10 | M0-DEC-01, M0-DEC-02, M0-DEC-03 | — | Decisions recorded by project owner; ADRs 0001–0003 written in `docs/decisions/`; docs only, no code | -| 2026-09-14 | M0-REPO-01 | — | Repo skeleton: top-level `README.md`/`LICENSE` (MIT, ADR 0001)/`.gitignore`; root `CMakeLists.txt` (CMake ≥ 3.22, C++20, `-Wall -Werror`, no exceptions/RTTI via `laige_apply_engine_policy`, `LAIGE_BUILD_SHARED` placeholder); PRD §10.1 module dirs with only `laige-core` populated; empty-target configure+build verified | +| 2026-09-10 | M0-REPO-01 | — | Repo skeleton: top-level `README.md`/`LICENSE` (MIT, ADR 0001)/`.gitignore`; root `CMakeLists.txt` (CMake ≥ 3.22, C++20, `-Wall -Werror`, no exceptions/RTTI via `laige_apply_engine_policy`, `LAIGE_BUILD_SHARED` placeholder); PRD §10.1 module dirs with only `laige-core` populated; empty-target configure+build verified | | 2026-09-10 | M0-BUILD-01 | — | `laige-core` CMake target (static default, shared via `LAIGE_BUILD_SHARED`; no transitive leakage — policy flags PRIVATE, CPP-010); options `LAIGE_ASAN`/`LAIGE_TSAN` (mutually exclusive, whole-tree instrumentation, fatal UBSan, TSan `halt_on_error=1`), `LAIGE_SCRIPT` reserved, `LAIGE_BUILD_TESTS` default ON; canonical commands fixed in `docs/getting-started/building.md` (source of truth); link smoke test `tests/laige-core` with NFR-8.10 `static_assert` policy self-checks; static+shared+ASan+TSan+Clang builds verified warning-free; fixed latent invalid `target_compile_features` call in M0-REPO-01 policy function | | 2026-09-10 | M0-DEP-01 | — | `deps.lock` (repo root) + configure-time verification in `cmake/laige-deps-lock.cmake` (deterministic tree SHA-256; fails loudly on mismatch, missing tree, unlisted `deps/` dir, or malformed lock); vendored GoogleTest v1.18.0 (`deps/googletest`, 252 files, commit `063de7e9…`, BSD-3-Clause) wired into tests only (`gtest_main`, `BUILD_GMOCK`/`INSTALL_GTEST` off, no-exceptions/no-rtti flags match engine test TUs); `laige-core_tests` converted to the first GTest suite; ADR 0004 written; fresh+shared+ASan+Clang trees verified warning-free with `ctest` 1/1, tampered vendored file fails the configure | | 2026-09-10 | M0-CI-01 | `7ec98b9` | CI matrix `.github/workflows/ci.yml` (all 5 P0 jobs on push to `master` + `workflow_dispatch`: linux-gcc, linux-clang, windows-msvc, macos-arm64, macos-intel) and `ci-pull.yml` (one P0 OS per PR, selected by `ci:linux`/`ci:windows`/`ci:macos` labels, default Linux — PRD §14 cadence); each job = canonical configure→build→ctest with `timeout-minutes: 10`; fixes landed in the same step: `.gitattributes` (`deps/** -text`) for byte-exact LF vendored checkouts (Windows CRLF broke the tree-hash lock), `_MSVC_LANG` for the C++20 self-check on MSVC, MSVC `/EH` conflict resolution (strip platform-default `/EHsc`; gtest rewritten to its documented no-exception set `/EHs-c- -D_HAS_EXCEPTIONS=0`), and `_HAS_EXCEPTIONS=0` in the engine policy for the MS STL (C4530 under `/WX`); verified: full matrix green after pushing `a90191c..7ec98b9` | @@ -184,6 +184,12 @@ One line per completed (or split/renumbered) step. | 2026-09-10 | M0-CI-03 | `785af81` | `tools/laige-include-lint` (Python 3 stdlib): parses `#include` edges of `src/**`, enforces R1 (laige-core includes nothing internal), R2 (arrows only downward in the PRD §10.1 stack, via the target's public include root — CPP-010), R3 (vendored deps only from their `deps.lock` `owner` — new required lock field, validated by `cmake/laige-deps-lock.cmake`; angle-bracket vendored header paths caught via a vendored-header map), R4 (engine code includes only `src/**`/`deps/**`); reports the vendored-dependency list and fails above the PRD §11 budget of 10; `include-lint` CI job in `ci-pull.yml` (every PR, label-independent) and `ci.yml` (every merge, now 8 jobs); CTest coverage in `tests/tools` (4 fixture trees + real-tree check, expected failures asserted via generated `cmake -P` scripts because CTest inverts `PASS_REGULAR_EXPRESSION` under `WILL_FAIL`); local Verify: illegal `laige-core → laige-render` stub include fails with R1, all rule directions exercised, dep count prints (1/10), `ctest` 6/6 on g++/shared/ASan/Clang trees; pushed as `785af81` — CI observed via the GitHub API: `ci.yml` (8-job) run 34518244428 on `741c163` green, `include-lint` job log shows the live report (`count: 1 (budget: 10, PRD §11)`, `OK`); `ci-pull.yml` job exercised by PR #1 (run 34521473503, green incl. `include-lint`), squash-merged as `a74b65a` with the post-merge 8-job run 34521722826 green | | 2026-09-10 | M0-CORE-01 | `6fa1414` | `laige::Result`/`laige::Status` (no exceptions, FR-12.1; inline `std::optional` storage, SFINAE-guarded implicit constructors + `success()`/`failure()` factories, `valueIfOk()`/`errorIfError()` null-safe accessors) + central error registry (`errors.h`/`errors.cpp`: 4 pinned codes, 0 reserved, unregistered → `unknown`; pre-rendered NFR-13.3 5-field lines) with the human-readable registry in `docs/api/errors.md`; `result_status` CTest entry (16 cases: construction, propagation, copy/move, grammar per code, pinned values) in the shared `laige-core_tests` executable; local Verify: GCC static/shared/ASan/TSan + fresh Clang trees 7/7 ctest, zero warnings; first push `f96ce7d` failed 7/8 on windows-msvc (C2535: the `Result(T)`/`Result(E)` constructors have identical parameter lists when `T == E`) — fixed in `6fa1414` by taking the failure value by `const E&`; CI: `ci.yml` run 34525402022 on `6fa1414` (8-job matrix) green, Windows job compiles and passes `result_status`, every job under a minute | | 2026-09-10 | M0-CORE-02 | `0d3ee28` | The one structured logging facade (AGENTS §14, FR-12.2): `laige::log::Logger` Meyers singleton + `LAIGE_LOG_*` macros (gate before argument evaluation — disabled event = one atomic load + branch, no allocation, LOG-003); per-subsystem level table + atomic global minimum; `Field` scalars render locale-free via `to_chars` into a 64-byte stack buffer; rate limiting per (subsystem, event, severity) for Warn/Error/Fatal with `rate_limited` suppressed-count summaries (first event always emitted, pending counts drained at shutdown); `ConsoleSink` (non-owning stream) + `FileSink` (owning, `create()` → `Result`, failure = new `ErrorCode::IoError` 5, LOG-007 console fallback); Fatal = emit + flush + `std::abort()`; crash handlers (POSIX `sigaction` SA_RESETHAND / Windows vectored SEH) with raw-`write` notice + allocation-free `try_lock` flush; idempotent `shutdown()` retires the facade; timestamps = system_clock UTC RFC 3339 (in-code Hinnant civil-from-days); additive M0-CORE-01 extensions `Result::takeValue() &&` + `IoError`; `logging` CTest entry (27 cases incl. zero-alloc proof via a test-only global `operator new` counter, excluded from sanitizer trees per the step's fallback: leak-free runs + timing property); API contract in `docs/api/logging.md`; local Verify: GCC static/shared/ASan/TSan + fresh Clang static/shared trees — `ctest` 8/8 and `ctest -R logging` green in every tree, zero warnings (disabled ≈46 ns/event vs ≈1207 ns/event enabled); CI (observed 2026-09-10 via the GitHub API): `ci-pull.yml` run 34534697621 on `0d3ee28` green — all 5 jobs of the default-Linux lane (linux-gcc g++, linux-clang clang++, linux-asan+UBSan clang++, linux-tsan clang++, include-lint) passed in 50 s, macOS/Windows skipped (label-gated) | +| 2026-09-11 | M0-CORE-03 | `0fd41f3` | SimMath op interface (`sim_math.h`; one op interface, template dispatch with no per-call indirection — ADR 0002) + `fp32_pinned` backend: pinned IEEE float semantics via the new `laige_apply_simmath_policy` (GCC/Clang: `-ffp-contract=off -fno-associative-math`; MSVC: `/fp:precise`) applied to laige-core + test targets; NaN/Inf policy (`isNaN`/`isInf`, no signaling); `math_float` CTest entry (10 SimMath* cases) (board/changelog row reconstructed 2026-09-11 from the step record) | +| 2026-09-11 | M0-CORE-04 | `5c76991` | `laige::fpx16_16` Q16.16 default SimMath backend (ADR 0002): int64 arithmetic, saturating ops (no UB — CPP-004), ties-to-even rounding, defined zero-division (x/0 → ±max, 0/0 → +0), `negate(min) = max`, no implicit scalar constructors and no arithmetic operators (API-008); `length` precision bound documented; `math_fixed` CTest entry (FixedPoint* suite incl. the 4096-tick determinism sequence with FNV-1a known-answer hash `0xF02728762777C581`, identical on g++ 16.2.1 and clang++ 22.1.8); CI fix `7538053` (macOS/Windows: missing `` include in `fpx16_16.h`) (board/changelog row reconstructed 2026-09-11 from the step record) | +| 2026-09-11 | M0-CORE-05 | `4a572af` | Header-only pools (`pools.h`): `laige::ArenaPool` (contiguous bump arena, per-frame `reset()`, O(1) create) + `laige::Pool` (stable handles = index + generation — CPP-007; LIFO free list; stale-handle destroy → `InvalidArgument`; capacity overrun → `BudgetExhausted`, no silent growth; `PoolStats` accounting: capacity/inUse/peakInUse/totalCreated/bytes); setup-only allocation, O(1) hot paths, single owner thread (CONC-001); `pools` CTest entry (25 cases incl. the stale-handle debug assert proven in a forked SIGABRT child); CI fix `ba4ffb2` (Windows C4324: `alignas(16)` padding in `pools_tests` fatal under `/WX`) (board/changelog row reconstructed 2026-09-11 from the step record) | +| 2026-09-11 | M0-CORE-06 | `a82de8e` | `laige::Prng`: xorshift128+ transcribed from and verified against the reference (all-zero state excluded), splitmix64 seeding (bijection), per-substream derivation (documented composition rule), Lemire unbiased `next_range`, `next_float01` = k·2^-24 exact; full period 2^128 − 1 for every nonzero state proven (characteristic polynomial over GF(2) via Berlekamp–Massey + irreducibility/primitivity checks; portable 128-bit arithmetic — no `__int128`, MSVC-safe); `prng` CTest entry (18 cases incl. the committed period proof and algorithm-sensitive golden KAT); API contract in `docs/api/prng.md` (board/changelog row reconstructed 2026-09-11 from the step record) | +| 2026-09-11 | M0-CORE-07 | `41938b0` | Bounded JSON in laige-core (ADR 0003, no new dependency): `laige::JsonValue` (deep copy, O(1) move, deep equality) + `parseJson` (1 MiB / depth-32 bounds, strict UTF-8, duplicate keys rejected, ±inf for overflow tokens — documented) + `serializeJson` (canonical ASCII; shortest-round-trip numbers; `std::to_chars` avoided for AppleClang 15 compatibility); all failures → `MalformedInput` (3); `laige-fuzz` minimal deterministic runner (Prng-seeded, `--runs`/`--seed`, 19-document corpus) + `json_parse` fuzz target; `config_json` CTest entry (25 cases) + `fuzz_json_parse` instrumented entry (ASan tree); API contract in `docs/api/json.md` (board/changelog row reconstructed 2026-09-11 from the step record) | +| 2026-09-11 | M0-CORE-08 | `810c251` | Budget harness: `laige::Histogram` (fixed-capacity rolling window; O(1) allocation-free `record()`; exact min/mean/p50/p95/p99/max over the stored window via nearest-rank percentiles; allocation-free O(n log n) `stats()`) + `laige::TimeIt` (`steady_clock` ms scope timer) + `loadBudgets`/`budgetCheck` (strict `budgets.json` schema v1 via the bounded JSON parser — ARCH-007; loud `NO_SAMPLES` failure on empty histograms; `target == 0` = hard-zero budget, not "not set"; AGENTS §12 report: stable 4-line text, before/after pair, caller context; `formatStatsLine` the single source of the stats text) + repo-root `budgets.json` (all 15 PRD §8.1 entries; `measured: 0` = not yet measured) + `laige-bench` tool (canonical command per `building.md`: `--suite=synthetic` deterministic 4096-step LCG+double stand-in workload, `--runs`/`--warmup`, `--budget=` check, exit code 2 on budget failure, `--report` append); `budget_harness` CTest entry (22 cases) + `laige_bench_smoke` CTest entry; **bug fix (M0-CORE-07)** found by this step's Verify run: `json.cpp` `parseObjectMembers` missing `skipWhitespace` before the member key — object documents with `", "` between members (the hand-formatted `budgets.json`) were rejected; regression test `ConfigJsonValid.ObjectMemberWhitespace` (fails pre-fix); API contract in `docs/api/budget_harness.md`; local Verify: `ctest` 16/16, zero warnings on GCC static/shared/ASan/TSan + Clang trees; MSVC/AppleClang compile proof lands in CI | ---