Repository navigation
feat(bench): long-horizon continual run with lineage (#941) - #1216
Merged
Merged
Conversation
Collaborator
Author
|
State change: draft -> ready for review. Reason: the repo page repro line is now Draft was a holding state for the repro-line fix; that fix is in. |
HugoXK
marked this pull request as ready for review
November 21, 2026 19:09
added 3 commits
November 21, 2026 19:18
Every other bench here measures one hop. RSI is a long loop: the same artifact rewritten step after step, each step conditioned on the version the previous one committed, and the usual failure is a decision that was locally right at step 4 and wrong at step 12. bench/long_horizon.py evolves one playbook across 1-15 steps (default 9) on a Ledger, with a deterministic offline proposer and DefaultAcceptance gating each rewrite on the held-out checks. The six check families conflict by construction (a later rule line overrides an earlier one, so the playbook has a history but cannot hold two answers), which lets the bench separate aggregate retention from per-family forgetting: on the 3-seed run every seed ends at the domain's reachable ceiling of 0.583 and is still verdict `forgetting`, because two families were erased on the way. Metrics are defined once: retention, TaskCurve's forgetting (mean/max/fraction), paired transfer on a fresh check set via bootstrap_ci, and cost. Seeds shuffle only the arrival order; the held-out split is hash-based and reported rather than assumed, and a run whose gate cannot see a family says so.
The issue's scoping comment specifies the shape of a step: `step(artifact, step_no, budget) -> (artifact, BenchStep)`. The loop was inline, so that contract could not be read or tested on its own, and a reader had to infer which numbers a step was responsible for. The step is now a closure over the artifacts it needs and returns the updated artifact plus a BenchStep whose score is the phase family's mean on that checkpoint and whose cost is the checks it evaluated; a raising step comes back as a BenchStep carrying the error instead of unwinding the run. The loop is a three-line driver. No measurement changed: identical manifest id, identical per-step decisions and identical retention/forgetting/transfer, verified by diffing the regenerated results file against the previous one. The new test pins the contract: the recorded score equals the phase family's mean on that checkpoint in the per-check curves, so the metric stays defined in one place.
Collaborator
|
Review (Birfy) Tip SHA: be265c8
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #941
What changed
New offline bench
bench/long_horizon.py: it evolves one playbook artifactacross sequential steps on a
Ledger, instead of one generation loop over afixed task set. Each step is
step(artifact, step_no, budget) -> (artifact, BenchStep): it takes the next family in a seed-shuffled arrivalorder, a deterministic offline proposer (no model call) emits the single rule
line that fixes that family,
DefaultAcceptancegates the rewrite on theheld-out checks, and an accepted rewrite is committed — a refused one is
recorded with the gate's own reason and the artifact is left untouched. A step
that raises returns a
BenchStepcarrying the error, so a long run keeps theevidence it already produced.
Before: nothing in
bench/measured an artifact across more than one hop, so arewrite that was locally right at step 4 and wrong at step 12 had no
measurement. After: the step-by-step record, the refusals, and per-check
forgetting are all published.
The domain makes forgetting possible: a later rule line overrides an earlier
one, and six check families conflict by construction. No assignment of rule
bodies satisfies every check (best reaches 0.583), which is why the report
prints that ceiling beside the measured retention.
Carrier:
FileTree, notAppendRules— the issue allows one artifact(
FileTreeorAppendRules).AppendRulesde-duplicates by content id anddrops a repeated line for the same rule, and repeated-override is exactly the
coverage failure this bench measures, so that carrier would erase the
phenomenon under test.
Metrics, each defined once: retention; forgetting (mean / max / fraction, reused
from
TaskCurve); paired transfer on a fresh check set viabootstrap_ci, withmin/max across seeds and the report stating that three seeds are a range, not
significance; cost (check evaluations, ledger commits, artifact bytes, wall
clock). Verdict vocabulary is
bench.rsi_bench's,flatandregressingincluded;
hackingcannot fire here and the report says why.Reproduce
python -m bench.long_horizon --steps 9 --seeds 3 \ --out bench/results/long-horizon-3seed.jsonExits 0 and writes
bench/results/long-horizon-3seed.json(the evidence file inthis PR). Expect
manifest_id 455feebf…, retention 0.583 on every seed,forgetting 0.167–0.333 (max 1.000), transfer +0.583, 14 rewrites committed and
13 refused out of 27 steps, overall verdict
forgetting. Everything exceptcost.wall_clock_sis reproducible byte for byte.Fast check of the same mechanics in three steps:
python -m bench.long_horizon --steps 3 --seeds 1 --json # exit 0, ~1.3 sVerification at the tip
Tip SHA: be265c8
Ran at this tip:
7194e1b..be265c88= 1 file,docs/long-horizon-bench.md+1/−1; no code, test or result file changed, so the readings below hold for both tipspytest tests/test_long_horizon_bench.py→ 8 passed in 3.84sbench/long_horizon.pyremoved → 1 error,ModuleNotFoundError: No module named 'bench.long_horizon'python -m bench.long_horizon --steps 3 --seeds 1 --json→ exit 0 in 1.33s, every stepokpython -m bench.long_horizon --steps 9 --seeds 3 --out bench/results/long-horizon-3seed.json→ exit 0manifest_id(cost.wall_clock_sremoved before comparing)python -m tools.changelog_fragments check→ 15 changelog fragment(s) OKpython -m tools.gen_api_docs --check→ docs/api.md is up to datedocs/, none missing, none danglingNot run:
pytest tests/test_docs_links.py— needs Python-Markdown, not installed on this box; the nav invariant it enforces was checked directly (149/149 above). CI runs it.mkdocs build --strict— mkdocs and its social/cairo dependency are absent here; CI'sbuildcheck covers it.pytest -q— collection oftests/test_docs_links.pyfails on the missing Python-Markdown, and with that module ignored the suite did not finish within 26 minutes on this box (known$HOMEissue). The focused command above is the gate for a bench-only change; CI runs the full suite on 3.9–3.13.