Skip to content

feat!: benchmark redesign — core and adaptive sets, multi-model runs, paired statistics, layered report (15.0.0) - #305

Merged
xjacka merged 6 commits into
mainfrom
claude/benchmark-review-analysis-653b98
Nov 13, 2026
Merged

xjacka merged 6 commits into
mainfrom
claude/benchmark-review-analysis-653b98

Conversation

@xjacka

@xjacka xjacka commented Nov 12, 2026 •

Copy link
Copy Markdown
Collaborator

What and why

The benchmark had two goals: regression of the definition and model choice. The old design served neither well. It ran one model per month on a live memory and compared one run with one run. This change implements the redesign of #304. The fixture set has a frozen core and an adaptive part. A run takes a list of models on one snapshot. The comparison is paired per defect. The report separates measured data, judge assessment and the agent's recommendation.

Reviewer guide

What changes in substance

  • No scheduled benchmark. The monthly schedule, its precheck mode and the 27-day gate are removed (major version 15.0.0). A minor or major definition update asks the operator about a regression run on benchmark_reference_model. The alternative was to keep a cheaper monthly run.
  • Core and adaptive set. The existing fixture/ set becomes core version 1 through benchmark-core.sh migrate. The core never changes piece by piece; new-version makes a new core and a bridge run. The adaptive part lives in adaptive/ and has no trend.
  • Frozen memory. A benchmark reviewer reads fixture/core-memory/, not the live work/MEMORY.md and LESSONS.md. The validator checks fixtures for leaks against that copy.
  • Multi-model run. benchmark-isolate.sh start takes --max-parallel and --snapshot. A file queue keeps at most benchmark_max_parallel (default 8) sub-agents running and completes one model before the next. Each model writes results/<ts>--<model-slug>.json with a shared run_id.
  • Statistics instead of repeats. scripts/lib/bench-stats.sh gives a paired McNemar test and a seeded bootstrap interval. A delta inside the interval is in_noise. No task runs k times.
  • Judge panel. benchmark_judge: <a>, <b> runs two judges and averages them. It reports disagreement and classifies each false positive as real, defensible or fabricated.
  • Report layers. Measured, Assessed and Recommended column groups have different colors and headings. Core and adaptive have separate tables. The report adds recall by language and a recommendation block under benchmark_decision_rule.
  • Bridge run. A delta compares runs of one core version only. A bridge run sets bridge_from: N and stores the old core as core-v<N>/<slug> fixtures with part: "bridge". Preflight emits a bridge entry field (the retired root and its fixtures) until a results file is on the new core version; the run spawns that root with its own start. The report shows the old-core index and its delta in the bridge row. (open)

Where to look closely

  • scripts/benchmark-isolate.sh — the queue in wait. A wrong count can start more sub-agents than the limit, or leave tasks queued that never start.
  • scripts/benchmark-core.sh migrate — it leaves leaking fixtures out of core v1 on a live instance. A mistake changes the regression series.
  • scripts/preflight.sh — the benchmark entry lost its gate. Check that no path still schedules or skips a run on the old 27-day rule. The run entry takes judge_panel and panel_note from benchmark-judge.sh resolve. With a CORE.json, the core list is its fixtures, never the directory listing.
  • scripts/benchmark-report.sh — the layer headers are in <thead> so that the sort script still works. The adaptive table shows no deltas.

Decisions made by the implementer (open)

  • memory_sha stays the git hash of MEMORY.md only, as provenance computed it before. LESSONS.md is copied but not hashed. (open)
  • benchmark_judge: auto keeps the single pinned judge. Only an explicit comma list makes a panel. (open)
  • The session writes the recommendation text and the comparison object into the results file. The report only renders them. (open)
  • LAST_PASS in scripts/preflight.sh is renamed to LAST_GATE_RUN. A local secret scanner read the old name as a password assignment. (open)

Scope

  • Persistence: scripts/work-backup.sh and docs/persistence.md protect benchmark/fixture/CORE.json and benchmark/fixture/core-memory/ from a persist deletion.
  • Weekly audit: the benchmark_fixtures check in scripts/preflight.sh validates the fixture/ and adaptive/ roots.
  • Benchmark scripts: benchmark-isolate.sh (a retired core root bundles its own frozen memory), benchmark-judge.sh, benchmark-report.sh, benchmark-score.sh, benchmark-validate.sh, benchmark-provenance.sh (author_models), templates/judge-brief.md.
  • New scripts: benchmark-core.sh, lib/bench-stats.sh, lib/bench-lang.sh, lib/bench-lock.sh (the run-lock TTL and age read that preflight.sh and benchmark-core.sh share; a lock of no measurable age is live); lib/sufficiency.sh gets languages_json.
  • Schedule removal: kit.yaml, ONBOARDING.md (Step 6d retired, Step 10 on-demand benchmark), scripts/precheck.sh, scripts/preflight.sh, scripts/verify-onboarding.sh, .github/workflows/ci.yml (run table).
  • Config keys in docs/config.md: benchmark_reference_model, benchmark_max_parallel, benchmark_decision_rule, the judge panel form of benchmark_judge, and the ## Benchmark languages table.
  • Docs: docs/benchmark.md (home of the change), docs/persistence.md (version-bump question), docs/runbook.md, docs/worklist.md, docs/audit.md, README.md.
  • Tests: new test_benchmark_core.sh, test_benchmark_stats.sh; extended isolate, judge, preflight, score, helpers, work-backup and verify-onboarding tests.
  • VERSION 15.0.0 and CHANGELOG.md. Upgrade: delete the code-guardian-benchmark-monthly schedule, run benchmark-core.sh migrate, offer the missing config keys.

Issue

Closes #304

Testing

  • scripts/tests/docker.sh over the full suite at the first push: ALL TESTS PASSED (4011 checks).
  • Review round 1: every test_benchmark_*.sh, test_work_backup.sh, test_audit_stats.sh and test_verify_onboarding.sh pass in Docker.
  • The CI cross-file section reference check passes locally (401 references).
  • Not run: a real benchmark run on the platform, and preflight.sh against a live work/CONFIG.md.

… paired statistics, layered report (15.0.0)

Implements #304.

- No scheduled benchmark: the monthly schedule, its precheck mode and the
  27-day gate are removed; a minor or major definition update asks the
  operator to run the regression on benchmark_reference_model.
- Core (frozen, regression) and adaptive (agent-managed) fixture sets; core
  versions with a frozen MEMORY.md/LESSONS.md copy, migration of an existing
  set to core v1 and new-version with a bridge run (benchmark-core.sh).
- Multi-model runs on one snapshot: benchmark-isolate.sh start takes
  --max-parallel (benchmark_max_parallel, default 8) with a queue, --snapshot
  reuse and adaptive/<slug> tasks.
- Scorer emits languages and fn_detail; bench-stats.sh gives paired McNemar
  and seeded bootstrap intervals with an in_noise verdict.
- Judge panel of two models (benchmark_judge: a, b) with averaged scores,
  disagreement and per-false-positive classes real/defensible/fabricated.
- Report keeps Measured, Assessed and Recommended layers apart, splits core
  and adaptive run tables, adds recall by language and the comparison and
  recommendation block under benchmark_decision_rule.
# Conflicts:
#	CHANGELOG.md
#	VERSION

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian - Code guardian repo — ❌ Code Review @ 619325c

PR #305: feat!: benchmark redesign — core and adaptive sets, multi-model runs, paired statistics, layered report (15.0.0)

Author: xjacka | Branch: claude/benchmark-review-analysis-653b98 → main | Changes: +2198 −293 (34 files)

Summary

This PR redesigns the model benchmark for #304: a core set and an adaptive set, a frozen memory copy, multi-model runs, paired statistics, a judge panel and a three-layer report. It removes the monthly schedule and the 27-day gate. The code is careful and well tested. The findings are about state a run can lose, and about instructions that do not match the new code.
Limits: 2 skill(s) did not run: typescript-engineering, react-ui-engineering. the diff is more than 2000 lines — the review covers the most critical files.
6 suggestion(s) dropped under the 🟢 budget.

Findings

  • 🔴 Critical: PROTECTED in scripts/work-backup.sh and the list in docs/persistence.md name the records a persist may never delete (scripts/work-backup.sh:54). Neither names benchmark/fixture/CORE.json or benchmark/fixture/core-memory/. This PR makes those the only copy of the core version and its frozen memory. A persist from a work/ that has lost them mirrors the deletion into the durable backup. A later benchmark-core.sh migrate then re-freezes the live memory, so the series silently starts again.
    Fix: Add benchmark/fixture/CORE.json and benchmark/fixture/core-memory to PROTECTED in the script and in the doc list. Before protecting benchmark/adaptive, give its fixture retirement an allow-delete route.
  • 🟡 Warning: After migrate, the migration step moves a leaking fixture to adaptive/ or retires it (docs/benchmark.md:71). The new core check then fails at every run gate with fixture <slug> not in the root. The "Retiring a fixture set" step has the same effect.
    Fix: Make every core-set change a staged set plus new-version, and say so in the migration and retirement steps.
  • 🟡 Warning: preflight.sh splits the raw benchmark_judge value into judge_panel (scripts/preflight.sh:306). The entry's judge is resolve's served id, and the validator requires it to equal judge_panel[0]. With the short names that docs/config.md allows, the results file fails run_extra, and step 8 discards the run.
    Fix: Build judge_panel from resolve's panel in the one place that emits it, and name that source in the docs.
  • 🟡 Warning: Step 8 validates results/<ts>.json, but a multi-model run writes only results/<ts>--<model>.json (docs/benchmark.md:517). The layout also gives <model> without the slug transform that a filename needs. A session that follows the text cannot pass its own gate.
    Fix: Validate every file the run wrote, and state the <run_id>--<model-slug>.json name and its slug transform once.
  • 🟡 Warning: run_rows picks the previous run by model, effort, harness and isolation only (scripts/benchmark-report.sh:302). After new-version, the first run on the new core is compared with a run on other fixtures. The regression delta is then between two sets. Issue decision 7 asks for a bridge run that keeps the trend.
    Fix: Require the same core_version for a delta, and add the bridge pair that links the two versions.
  • 🟡 Warning: The report marks the author model from author_models (docs/benchmark.md:596). Decision 5 forbids recommending it on a noise-sized win. No script writes the field, and the run steps never say to merge CORE.json's map. The mark and the rule stay inactive.
    Fix: Populate author_models from CORE.json while the results are assembled, and require it in the run steps.
  • 🟡 Warning: Decision 17 says the report names a one-family panel (scripts/benchmark-report.sh:175). resolve prints panel_note, but no results field carries it and the report never reads it. A single-judge run shows Assessed scores with no sign of the smaller panel.
    Fix: Record panel_note in the results file, and render one line beside Assessed when the panel is small.
  • 🟡 Warning: The weekly audit still calls the validator once per fixture directory (scripts/preflight.sh:2435). The new core check and the adaptive set never run there. A stale CORE.json or a leaking adaptive fixture is reported as benchmark_fixtures ok.
    Fix: Pass benchmark/fixture and benchmark/adaptive to the validator in the audit's integrity check.

Issue Fit

✅ No findings.

Human Review Guide

📊 Visual guide for the human reviewer — judge leaking fixtures in core v1, judge auto, and the on-request run design (about 55 min)

Documentation Check (doc-drift)

✅ No findings.

Agent Engineering

  • 🟢 Suggestion: No test names languages_json, so a broken table read would ship unseen (scripts/lib/sufficiency.sh:35).

Verdict

REQUEST_CHANGES — A persist can still delete the frozen core memory. Seven gaps in the new run, report and audit paths follow the same theme.


Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi

…, served panel, audit roots

- work-backup protects benchmark/fixture/CORE.json and core-memory/
- migrate leaves a leaking fixture out of core v1; core changes go through new-version
- the run entry takes judge_panel and panel_note from resolve's served panel
- step 8 validates every per-model file; layout states the model-slug transform
- report deltas need the same core_version; a bridge run links the versions
- provenance writes author_models from CORE.json and the adaptive manifests
- the report names a one-family panel beside its judge cell
- the weekly audit validates the fixture and adaptive roots
- isolate bundles a retired core's own frozen memory for a bridge run
@xjacka

xjacka commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Answer to review round 1 (commit ed637ac):

  • Fixed — scripts/work-backup.sh:54: PROTECTED and the list in docs/persistence.md now name benchmark/fixture/CORE.json and benchmark/fixture/core-memory. benchmark/adaptive stays unprotected, because adaptive fixtures retire. New test: persist-refuses-core-delete.
  • Fixed — docs/benchmark.md:71 and :818: migrate now leaves a leaking fixture out of CORE.json, so the core check stays clean. The retirement section says that every core change is a staged set plus new-version. Only a slug outside CORE.json moves to fixture/retired/. CHANGELOG 15.0.0 step 2 says the same.
  • Fixed — scripts/preflight.sh:306: the run entry takes judge_panel from the panel of resolve (the served ids). docs/benchmark.md and docs/worklist.md name that source. New test: benchmark_panel_is_the_served_one.
  • Fixed — docs/benchmark.md:517: step 8 validates every results/<run_id>--<model-slug>.json file. The layout and step 7 state the slug transform.
  • Fixed — scripts/benchmark-report.sh:302: a delta needs the same core_version. A bridge run sets bridge_from: N and stores the old core as core-v<N>/<slug> with part: "bridge". The bridge row shows the old-core index and its delta against the last core-vN run. The validator accepts bridge_from and part: bridge. Also, benchmark-isolate.sh start on a retired core root now bundles the frozen memory of that root. New tests: report_delta_needs_same_core_version, frozen_core_memory_of_a_retired_core.
  • Fixed — docs/benchmark.md:596: benchmark-provenance.sh writes author_models from CORE.json and each adaptive manifest. Step 7 lists it with the provenance fields.
  • Fixed — scripts/benchmark-report.sh:175: the run entry and the results file carry panel_note. The report shows it under the judge cell. The validator checks its type.
  • Fixed — scripts/preflight.sh:2435: the audit check validates the fixture root and the adaptive root, so the core and leak checks run. New test: audit_fixture_check_runs_the_roots.

Checks: all 8 review checks give their clean result. The CI cross-file reference check passes (402 references). Every test_benchmark_*.sh, test_work_backup.sh, test_audit_stats.sh and test_verify_onboarding.sh pass in Docker.
Optional items: not taken (scripts/lib/sufficiency.sh is not changed in this round).

@xjacka xjacka added the code-guardian:rereview Request a code-guardian delta re-review label Nov 12, 2026

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian - Code guardian repo — ❌ Code Review @ ed637ac

PR #305: feat!: benchmark redesign — core and adaptive sets, multi-model runs, paired statistics, layered report (15.0.0)

Author: xjacka | Branch: claude/benchmark-review-analysis-653b98 → main | Changes: +2393 −315 (36 files)

Summary

This re-review covers the fix round after the first review. Eight findings are fixed. This round adds five blocking findings and reports older gaps that the first round missed.
Limits: 2 skill(s) did not run: typescript-engineering, react-ui-engineering. the diff is more than 2000 lines — the review covers the most critical files.

Changes since last review

Previous HEAD: 619325c (2026-11-12T11:36:58Z) — verdict REQUEST_CHANGES

Changes since last review

  • ✅ Fixed: the frozen core memory and CORE.json are unprotected (scripts/work-backup.sh:54)
  • ✅ Fixed: a documented core-set change leaves CORE.json stale (docs/benchmark.md:71)
  • ✅ Fixed: the entry judge_panel is the raw config, not the served panel (scripts/preflight.sh:306)
  • ✅ Fixed: the validation step names no multi-model results file (docs/benchmark.md:517)
  • ✅ Fixed: trend deltas do not require the same core version (scripts/benchmark-report.sh:302)
  • ✅ Fixed: no script writes the author_models field (docs/benchmark.md:596)
  • ✅ Fixed: the one-family panel note never reaches the report (scripts/benchmark-report.sh:175)
  • ✅ Fixed: the audit fixture check skips the core and adaptive checks (scripts/preflight.sh:2435)
  • 🔁 Still present: the languages table parser has no test (scripts/lib/sufficiency.sh:35)
  • 🆕 New: migrate overwrites a protected frozen memory copy (scripts/benchmark-core.sh:116)
  • 🆕 New: a fixture left out of CORE.json is still scored as core (scripts/benchmark-core.sh:125)
  • 🆕 New: no step produces a bridge run (docs/benchmark.md:64)
  • 🆕 New: the migration doc names an author-model source the script never reads (docs/benchmark.md:71)
  • 🆕 New: no test covers the report layers or the recommendation (scripts/tests/test_benchmark_helpers.sh:185)
  • 🔎 Missed earlier: an unparsable run lock reads as stale (scripts/benchmark-core.sh:93) — present at 619325c
  • 🔎 Missed earlier: the CORE.json field list omits created, migrated and bridged_from (docs/benchmark.md:17) — present at 619325c
  • 🔎 Missed earlier: the creation entry description omits the run settings it carries (docs/benchmark.md:138) — present at 619325c
  • 🔎 Missed earlier: the report has no quality by cost chart (scripts/benchmark-report.sh:606) — present at 619325c
  • 🔎 Missed earlier: the min_share key changes no result (scripts/lib/sufficiency.sh:35) — present at 619325c
  • 🔎 Missed earlier: the validator does not require the mandatory recommendation (docs/benchmark.md:545) — present at 619325c
  • 🔎 Missed earlier: an edited core manifest passes every gate (docs/benchmark.md:81) — present at 619325c
  • 🔎 Missed earlier: no step creates an adaptive fixture (docs/benchmark.md:52) — present at 619325c
  • 🔎 Missed earlier: an unparsable spawn answer leaves the task queued for a second spawn (scripts/benchmark-isolate.sh:271) — present at 619325c
  • 🔎 Missed earlier: the documented rolling-window stability value is missing (scripts/benchmark-report.sh:291) — present at 619325c

Findings

  • 🔴 Critical: A run lock whose first line is empty or unparsable reads as stale. lock_live falls back to t=0, so new-version moves the live core during a run (scripts/benchmark-core.sh:93).
    Fix: Treat an unmeasurable lock as live and refuse until its line parses. Read the 6 h TTL from one constant that preflight.sh shares.
  • 🟡 Warning: migrate writes the live memory over fixture/core-memory/ before the leak check. A re-run after an interrupted migration overwrites the frozen copy that past results name. The all-leak path then deletes that copy (scripts/benchmark-core.sh:116).
    Fix: Freeze to a temporary path under the benchmark dir. Move it to core-memory/ only after the leak check passes.
  • 🟡 Warning: A leaking fixture stays in fixture/ after migrate, and preflight.sh builds the core list from that directory. A run before the operator moves it scores a fixture that CORE.json excludes (scripts/benchmark-core.sh:125).
    Fix: When CORE.json exists, take the core fixture list from its fixtures field. Never score a directory outside that list.
  • 🟡 Warning: The doc describes a bridge run but no step produces one. The run entry names one fixture root, and start refuses a slug with /. The model bridge at docs/benchmark.md:244 has no result field either (docs/benchmark.md:64).
    Fix: Add the two-root step and the entry field that names the retired root. Key its tasks core-v<N>/<slug>. Give the model bridge its own field, or remove it.
  • 🟡 Warning: The migration doc says migrate records unknown "when the backup history does not show it". The script reads the manifest, then --author, then unknown. It never reads the backup history (docs/benchmark.md:71).
    Fix: Name the real chain in the migration step: the manifest author_model, else --author, else unknown.
  • 🟡 Warning: No test asserts the Measured, Assessed and Recommended layers, a rendered recommendation, or the "too few data" row. The issue's "Done when" asks for tests of the report layers (scripts/tests/test_benchmark_helpers.sh:185).
    Fix: Add one report case that asserts the three layer headings, a recommendation with its rule and cites, and the language table's "too few data" row.

Issue Fit

Present at 619325c; the previous review did not report these.

  • 🟢 Suggestion: Decision 10 requires a quality × cost chart, but the report shows quality and cost only as columns (scripts/benchmark-report.sh:606).
  • 🟢 Suggestion: Decision 2 defines a main language by min_share, but no code uses that value. A main language with no fixture stays absent (scripts/lib/sufficiency.sh:35).
  • 🟢 Suggestion: Decision 12 describes a rolling-window stability value per model, but neither the report nor the index carries it (scripts/benchmark-report.sh:291).

Human Review Guide

📊 Visual guide for the human reviewer — accept the one-way core migration and settle four open choices the agent made (about 30 min)

Documentation Check (doc-drift)

Present at 619325c; the previous review did not report these.

  • 🟢 Suggestion: The CORE.json field list omits created, migrated and bridged_from, and gives no types (docs/benchmark.md:17).
  • 🟢 Suggestion: The creation entry description omits the run settings it now carries: reference_model, max_parallel, decision_rule, core_version and adaptive (docs/benchmark.md:138).

Agent Engineering

Present at 619325c; the previous review did not report these.

  • 🟢 Suggestion: The validator accepts a results file with no recommendation, though the doc makes it mandatory every run (docs/benchmark.md:545).
  • 🟢 Suggestion: Nothing enforces "a core manifest never changes". CORE.json stores no fixture digest, so an edited core manifest passes every gate (docs/benchmark.md:81).
  • 🟢 Suggestion: No step creates an adaptive fixture, though the doc says the agent keeps that set current (docs/benchmark.md:52).
  • 🟢 Suggestion: A spawn answer with exit 0 and no .id leaves the task queued, so the next wait spawns it again (scripts/benchmark-isolate.sh:271).
  • 🔁 1 finding from the previous review still present (see review at 619325c).

Verdict

REQUEST_CHANGES — One critical finding blocks: an unreadable run lock lets a core version change during a run.

For the human reviewer

The benchmark now has one frozen core set per version. A core upgrade moves the old set aside and lets one bridge run connect the old trend to the new one. The PR does not yet say how that two-set run runs, so the trend can restart without a link. Decide the bridge procedure before the one-way migration reaches a live instance.


Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi

@dam-code-protector dam-code-protector Bot removed the code-guardian:rereview Request a code-guardian delta re-review label Nov 12, 2026
…ate memory, CORE.json core list, bridge entry, report layer test
@xjacka

xjacka commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Answer to review round 2 (at ed637ac); the fixes are in the next commit on this branch.

  • Fixed: an unparsable run lock reads as stale. The new scripts/lib/bench-lock.sh holds BENCH_LOCK_TTL_MIN and one age read. preflight.sh and benchmark-core.sh new-version both use it. An empty or unparsable first line reads the file's mtime. A lock with no measurable age is live. The echo 0 and 21600 are gone. Tests: core_unmeasurable_lock_is_live, benchmark_live_run_lock_holds (empty line).
  • Fixed: migrate overwrites the frozen copy. migrate freezes to <bdir>/.core-memory.tmp.<pid>, runs the leak check on that copy, and moves it to fixture/core-memory/ only when one fixture passes. An all-leaking run deletes only the temporary copy. Test: core_migrate_keeps_frozen_copy_on_refusal.
  • Fixed: a fixture left out of CORE.json is still scored. With CORE.json, preflight takes the core list from its fixtures. The directory listing is used only before the migration. Test: benchmark_settings_configured (a leaky-one dir outside the list).
  • Fixed: no step produces a bridge run. The run entry has a new bridge field ({from, fixture_root, fixtures}). It is set while CORE.json has bridged_from and no results file is on the new core version. docs/benchmark.md adds the bridge root to the gate. It adds one more start per model on the retired root (own state dir, raw dir results/raw/core-v<N>), the in-pod loop, and the assembly keys core-v<N>/<slug>. The reference-model change at docs/benchmark.md:244 is now a Models run of the old and the new model, so it needs no bridge field. docs/worklist.md lists the field. Test: benchmark_bridge_entry.
  • Fixed: wrong author-model source. The migration bullet now names the real chain: the manifest's author_model, else --author, else unknown.
  • Fixed: no test of the report layers. New case report_layers_and_recommendation asserts the Measured and Assessed headings, the Recommended block with its rule, text and cites, the comparison row, and the language "too few data (2/3)" cell.

Checks: the review's git grep commands are clean, or each hit is the intended form (the directory listing only without CORE.json, rm -rf …core-memory only after the leak check). Docker tests test_benchmark_core, test_benchmark_preflight, test_benchmark_helpers, test_benchmark_isolate, test_benchmark_e2e, test_work_backup, test_audit_stats and test_check_definition pass. The cross-file reference check passes (402).
Optional items: none taken.

@xjacka xjacka added the code-guardian:rereview Request a code-guardian delta re-review label Nov 12, 2026

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian - Code guardian repo — ⚠️ Code Review @ 51b1f37

PR #305: feat!: benchmark redesign — core and adaptive sets, multi-model runs, paired statistics, layered report (15.0.0)

Author: xjacka | Branch: claude/benchmark-review-analysis-653b98 → main | Changes: +2563 −325 (37 files)

Summary

This re-review covers the fix round after the second review. Seven findings are fixed, and the bridge run now has a producing step. Two new warnings are open.
Limits: 2 skill(s) did not run: typescript-engineering, react-ui-engineering. the diff is more than 2000 lines — the review covers the most critical files.

Changes since last review

Previous HEAD: ed637ac (2026-11-12T21:13:36Z) — verdict REQUEST_CHANGES

  • ✅ Fixed: an unparsable run lock reads as stale (scripts/benchmark-core.sh:93)
  • ✅ Fixed: migrate overwrites a protected frozen memory copy (scripts/benchmark-core.sh:116)
  • ✅ Fixed: a fixture left out of CORE.json is still scored as core (scripts/benchmark-core.sh:125)
  • ✅ Fixed: no step produces a bridge run (docs/benchmark.md:64)
  • ✅ Fixed: the migration doc names an author-model source the script never reads (docs/benchmark.md:71)
  • ✅ Fixed: no test covers the report layers or the recommendation (scripts/tests/test_benchmark_helpers.sh:185)
  • ✅ Fixed: an unparsable spawn answer leaves the task queued for a second spawn (scripts/benchmark-isolate.sh:271)
  • 🔁 Still present: the CORE.json field list omits created, migrated and bridged_from (docs/benchmark.md:17)
  • 🔁 Still present: the creation entry description omits the run settings it carries (docs/benchmark.md:138)
  • 🔁 Still present: the report has no quality by cost chart (scripts/benchmark-report.sh:606)
  • 🔁 Still present: the min_share key changes no result (scripts/lib/sufficiency.sh:35)
  • 🔁 Still present: the validator does not require the mandatory recommendation (docs/benchmark.md:545)
  • 🔁 Still present: an edited core manifest passes every gate (docs/benchmark.md:81)
  • 🔁 Still present: no step creates an adaptive fixture (docs/benchmark.md:52)
  • 🔁 Still present: the documented rolling-window stability value is missing (scripts/benchmark-report.sh:291)
  • 🔁 Still present: the languages table parser has no test (scripts/lib/sufficiency.sh:35)
  • 🆕 New: the bridge gate reads a core_version that no step writes (scripts/preflight.sh:318)
  • 🆕 New: a bridge run's later models take a fresh snapshot, not the first model's (docs/benchmark.md:362)

Findings

  • 🟡 Warning: The bridge gate reads core_version from a results file, but no step writes that field (scripts/preflight.sh:318). A run after a core upgrade keeps the bridge on for every later run. An isolated run also fails validation at step 8.
    Fix: Emit core_version from fixture/CORE.json in benchmark-provenance.sh, and name it in step 7's field list.
  • 🟡 Warning: The bridge start takes no --snapshot, so every model after the first snapshots the live skills again (docs/benchmark.md:362). A skill refresh between two model starts gives each model different inputs. The paired comparison then mixes the two input sets.
    Fix: Pass --snapshot "$PSTATE-1-bridge" to every bridge start after the first model.

Issue Fit

✅ No findings.

Human Review Guide

📊 Visual guide for the human reviewer — accept the one-way core migration and confirm the bridge procedure before a live run (about 20 min)

Documentation Check (doc-drift)

✅ No findings.

Agent Engineering

✅ No findings.

Verdict

COMMENT — No critical finding remains. Two new warnings in the bridge procedure are open.

For the human reviewer

The benchmark now scores a new core set and the retired one in one run, and the report links the two. This review finds two gaps in that procedure. A core version is not written into the results, so the link can repeat on every later run. A later model can also read a different skill snapshot than the first model. Settle how one run records its core version before the first bridge run.


Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi

@dam-code-protector dam-code-protector Bot removed the code-guardian:rereview Request a code-guardian delta re-review label Nov 12, 2026
@xjacka

xjacka commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator Author

Answer to review round 3 (at 51b1f37); the fixes are in the next commit on this branch.

  • Fixed: the bridge gate reads a core_version that no step writes. benchmark-provenance.sh now prints core_version from fixture/CORE.json (null before the migration). The in-pod call and the isolated expect.json both carry it. Step 7 of docs/benchmark.md names it in the field list. Test: provenance_best_effort (null without CORE.json, 3 from CORE.json).
  • Fixed: a later model's bridge start takes a fresh snapshot. docs/benchmark.md step 2 now says that the first model's bridge start takes no --snapshot, and every later model's takes --snapshot "$PSTATE-1-bridge".

Checks: both review git grep commands now print the expected hits. Docker tests test_benchmark_helpers, test_benchmark_isolate, test_benchmark_e2e, test_benchmark_core and test_benchmark_preflight pass.
Optional items: none taken.

@xjacka xjacka added the code-guardian:rereview Request a code-guardian delta re-review label Nov 13, 2026

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian - Code guardian repo — ✅ Code Review @ afc43a4

PR #305: feat!: benchmark redesign — core and adaptive sets, multi-model runs, paired statistics, layered report (15.0.0)

Author: xjacka | Branch: claude/benchmark-review-analysis-653b98 → main | Changes: +2575 −325 (37 files)

Summary

This re-review covers the fix round after the third review. Both open warnings are fixed, and the delta adds no new finding.
Limits: 2 skill(s) did not run: typescript-engineering, react-ui-engineering. the diff is more than 2000 lines — the review covers the most critical files.

Changes since last review

Previous HEAD: 51b1f37 (2026-11-12T22:08:23Z) — verdict COMMENT

  • ✅ Fixed: the bridge gate reads a core_version that no step writes (scripts/preflight.sh:318)
  • ✅ Fixed: a bridge run's later models take a fresh snapshot, not the first model's (docs/benchmark.md:362)
  • 🔁 Still present: the CORE.json field list omits created, migrated and bridged_from (docs/benchmark.md:17)
  • 🔁 Still present: the creation entry description omits the run settings it carries (docs/benchmark.md:138)
  • 🔁 Still present: the report has no quality by cost chart (scripts/benchmark-report.sh:606)
  • 🔁 Still present: the min_share key changes no result (scripts/lib/sufficiency.sh:35)
  • 🔁 Still present: the validator does not require the mandatory recommendation (docs/benchmark.md:545)
  • 🔁 Still present: an edited core manifest passes every gate (docs/benchmark.md:81)
  • 🔁 Still present: no step creates an adaptive fixture (docs/benchmark.md:52)
  • 🔁 Still present: the documented rolling-window stability value is missing (scripts/benchmark-report.sh:291)
  • 🔁 Still present: the languages table parser has no test (scripts/lib/sufficiency.sh:35)

Findings

No new findings at this HEAD.

Issue Fit

Linked issue: #304 "Benchmark redesign: repo-specific core and adaptive sets, multi-model runs, paired statistics, layered report" (plain reference)

✅ Delivers the issue's ask; no undeclared scope.

Human Review Guide

📊 Visual guide for the human reviewer — at 51b1f37: the bridge fix changes no decision a person must settle.

Documentation Check (doc-drift)

✅ No findings.

Agent Engineering

✅ No findings.

Verdict

APPROVE — Both bridge warnings are fixed, and only suggestions from earlier rounds remain open.


Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi

@xjacka
xjacka merged commit 51f1aa2 into main Nov 13, 2026
2 checks passed
@xjacka
xjacka deleted the claude/benchmark-review-analysis-653b98 branch November 13, 2026 16:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Benchmark redesign: repo-specific core and adaptive sets, multi-model runs, paired statistics, layered report

1 participant