Skip to content

[X-MANIFEST] Attribute measures to their result so a multi-result report verifies (#517) - #618

Open
physics515 wants to merge 1 commit into
mainfrom
w8/manifest-measure-ownership
Open

physics515 wants to merge 1 commit into
mainfrom
w8/manifest-measure-ownership

Conversation

@physics515

Copy link
Copy Markdown
Collaborator

Closes #517.

The defect

RunManifest::verify collected every measure from every result into one flat list of metric names
and refused the manifest if any name repeated:

let mut names: Vec<&str> = self.grade.measures.iter().map(|m| m.name.as_str()).collect();
let before = names.len();
names.sort_unstable();
names.dedup();
if names.len() != before {
    problems.push(Problem::DuplicateMeasure("a measure name is repeated across adapters".to_string()));
}

latency.p50_ns and throughput.points_per_sec are the names every result grades, so a report
with two results could never verify
: a crash matrix, a query mix, any cross-adapter run. Because
write_to refuses to return Ok for a manifest that does not verify, finish --manifest exited
non-zero on honest evidence — which is why #147 and #513 split each lane and each crash-matrix
cell into its own report, and why #511 had to narrow crash-matrix.yml.

The check also broke the file's own error convention: every other Problem names the thing to go
and look at (measure \{name}` appears more than once` elsewhere in the same enum), while this one
pushed a static string that named nothing.

The fix

A measure's identity is the result it was quoted from, which BenchResult already states as
(profile, adapter, workload) and compare.rs already uses as its WorkloadKey. Measure carried
name and adapter, and adapter is the wrong disambiguator: a query mix runs one adapter across
many profiles, a cross-adapter run one profile across many adapters.

  • Measure gains profile and workload beside adapter, filled in push_result_measures from
    the same result the numbers come from — nothing re-derives a value, per the manifest's own rule.
  • Uniqueness is checked within (profile, adapter, workload), so the same metric may appear once
    per result and a repeat inside one result is still a defect.
  • Problem::DuplicateMeasure is now a struct variant carrying the name and the result identity, and
    reads: measure `latency.p50_ns` appears more than once within result `mcar-medium / timescaledb / downsample`; each result must grade each metric name exactly once.
  • MANIFEST_SCHEMA_VERSION 23 → 24, because the fields every verifier keys on are new.
  • The stale-v23 mentions are updated: docs/reference.md's "schema v23" line and the two
    "schema_version":23 fixtures in weft-server/src/bench_runs.rs.

A v23 manifest is refused, not converted

The version check stays an exact match, and the message changed from "cannot be verified by a build
that writes v{expected}" to "was written by an older build; re-run the benchmark with a build
that writes v{expected}". A manifest records provenance that only exists during the run — source
revision, dirtiness, environment — so telling someone to regenerate one from its report would invite
fabricated provenance. Accepting a list of versions would make a v23 reader's silence about
profile/workload invisible.

DECISIONS.md records both rejections (dropping the check; converting old manifests) and one thing
this PR deliberately does not do: remove the one-report-per-cell splitting from #513/#147. It
was a workaround for this bug, but it is also what keeps a crash cell's artifact attributable to that
cell, so removing it belongs in its own change.

Tests

  • a_multi_result_report_verifies_with_cross_result_name_reuse — two results, same metric names,
    captures and writes and verifies with no defects. Fails on main with the old check.
  • a_name_repeated_within_one_result_is_still_a_defect — a second latency.p50_ns inside one result
    is refused by write_to, and the error names both the metric and the result.
  • a_manifest_of_another_schema_version_is_refused_not_guessed now covers v22 and v23.

Verification

cargo +nightly fmt --all -- --check locally. The build, clippy (#![forbid(clippy::pedantic)] in
this crate), test and docs jobs run in CI on this branch; weft-xbench and bench/claims.json are
untouched, so no published figure moves.

…ort verifies (#517)

A measure was graded by metric name alone, so a report with two results looked like
a duplicated measure and finish --manifest refused honest evidence. Measures now
carry the result they were quoted from and uniqueness is checked within a result.

Signed-off-by: Justin Icenhour <creative515.justin@gmail.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

weft-bench run manifest refuses every multi-result report

1 participant