Skip to content

feat(bench): long-horizon continual run with lineage (#941) - #1216

Merged
Birfy merged 3 commits into
mainfrom
agent/941-long-horizon-bench
Nov 21, 2026
Merged

Birfy merged 3 commits into
mainfrom
agent/941-long-horizon-bench

Conversation

@HugoXK

@HugoXK HugoXK commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #941

What changed

New offline bench bench/long_horizon.py: it evolves one playbook artifact
across sequential steps on a Ledger, instead of one generation loop over a
fixed task set. Each step is step(artifact, step_no, budget) -> (artifact, BenchStep): it takes the next family in a seed-shuffled arrival
order, a deterministic offline proposer (no model call) emits the single rule
line that fixes that family, DefaultAcceptance gates the rewrite on the
held-out checks, and an accepted rewrite is committed — a refused one is
recorded with the gate's own reason and the artifact is left untouched. A step
that raises returns a BenchStep carrying the error, so a long run keeps the
evidence it already produced.

Before: nothing in bench/ measured an artifact across more than one hop, so a
rewrite that was locally right at step 4 and wrong at step 12 had no
measurement. After: the step-by-step record, the refusals, and per-check
forgetting are all published.

The domain makes forgetting possible: a later rule line overrides an earlier
one, and six check families conflict by construction. No assignment of rule
bodies satisfies every check (best reaches 0.583), which is why the report
prints that ceiling beside the measured retention.

Carrier: FileTree, not AppendRules — the issue allows one artifact
(FileTree or AppendRules). AppendRules de-duplicates by content id and
drops a repeated line for the same rule, and repeated-override is exactly the
coverage failure this bench measures, so that carrier would erase the
phenomenon under test.

Metrics, each defined once: retention; forgetting (mean / max / fraction, reused
from TaskCurve); paired transfer on a fresh check set via bootstrap_ci, with
min/max across seeds and the report stating that three seeds are a range, not
significance; cost (check evaluations, ledger commits, artifact bytes, wall
clock). Verdict vocabulary is bench.rsi_bench's, flat and regressing
included; hacking cannot fire here and the report says why.

Reproduce

python -m bench.long_horizon --steps 9 --seeds 3 \
    --out bench/results/long-horizon-3seed.json

Exits 0 and writes bench/results/long-horizon-3seed.json (the evidence file in
this PR). Expect manifest_id 455feebf…, retention 0.583 on every seed,
forgetting 0.167–0.333 (max 1.000), transfer +0.583, 14 rewrites committed and
13 refused out of 27 steps, overall verdict forgetting. Everything except
cost.wall_clock_s is reproducible byte for byte.

Fast check of the same mechanics in three steps:

python -m bench.long_horizon --steps 3 --seeds 1 --json    # exit 0, ~1.3 s

Verification at the tip

Tip SHA: be265c8

Ran at this tip:

  • docs-only delta 7194e1b..be265c88 = 1 file, docs/long-horizon-bench.md +1/−1; no code, test or result file changed, so the readings below hold for both tips
  • pytest tests/test_long_horizon_bench.py → 8 passed in 3.84s
  • red first, same file with bench/long_horizon.py removed → 1 error, ModuleNotFoundError: No module named 'bench.long_horizon'
  • python -m bench.long_horizon --steps 3 --seeds 1 --json → exit 0 in 1.33s, every step ok
  • python -m bench.long_horizon --steps 9 --seeds 3 --out bench/results/long-horizon-3seed.json → exit 0
  • two runs of the same inputs → identical report and identical manifest_id (cost.wall_clock_s removed before comparing)
  • python -m tools.changelog_fragments check → 15 changelog fragment(s) OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • nav coverage → 149 nav entries for 149 pages under docs/, none missing, none dangling

Not run:

  • pytest tests/test_docs_links.py — needs Python-Markdown, not installed on this box; the nav invariant it enforces was checked directly (149/149 above). CI runs it.
  • mkdocs build --strict — mkdocs and its social/cairo dependency are absent here; CI's build check covers it.
  • full pytest -q — collection of tests/test_docs_links.py fails on the missing Python-Markdown, and with that module ignored the suite did not finish within 26 minutes on this box (known $HOME issue). The focused command above is the gate for a bench-only change; CI runs the full suite on 3.9–3.13.

@HugoXK

HugoXK commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

State change: draft -> ready for review.

Reason: the repo page repro line is now --steps 9 (matches the page title and the committed 9-step artifact) at tip be265c88065620bb933284a994a2d7fc24463290 — the only delta from 7194e1b is docs/long-horizon-bench.md +1/−1, so the code, tests and committed results are unchanged. The body carries the red-first readings, the reproducibility statement (value-identical except the three cost.wall_clock_s scalars), and the AppendRules rationale.

Draft was a holding state for the repro-line fix; that fix is in.

@HugoXK
HugoXK marked this pull request as ready for review November 21, 2026 19:09
@github-actions github-actions Bot added risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller agent-pr Pull request opened by an agent labels Nov 21, 2026
AgentDescent Dev-4 added 3 commits November 21, 2026 19:18
Every other bench here measures one hop. RSI is a long loop: the same artifact
rewritten step after step, each step conditioned on the version the previous
one committed, and the usual failure is a decision that was locally right at
step 4 and wrong at step 12.

bench/long_horizon.py evolves one playbook across 1-15 steps (default 9) on a
Ledger, with a deterministic offline proposer and DefaultAcceptance gating each
rewrite on the held-out checks. The six check families conflict by construction
(a later rule line overrides an earlier one, so the playbook has a history but
cannot hold two answers), which lets the bench separate aggregate retention from
per-family forgetting: on the 3-seed run every seed ends at the domain's
reachable ceiling of 0.583 and is still verdict `forgetting`, because two
families were erased on the way.

Metrics are defined once: retention, TaskCurve's forgetting (mean/max/fraction),
paired transfer on a fresh check set via bootstrap_ci, and cost. Seeds shuffle
only the arrival order; the held-out split is hash-based and reported rather
than assumed, and a run whose gate cannot see a family says so.
The issue's scoping comment specifies the shape of a step:
`step(artifact, step_no, budget) -> (artifact, BenchStep)`. The loop was
inline, so that contract could not be read or tested on its own, and a reader
had to infer which numbers a step was responsible for.

The step is now a closure over the artifacts it needs and returns the updated
artifact plus a BenchStep whose score is the phase family's mean on that
checkpoint and whose cost is the checks it evaluated; a raising step comes back
as a BenchStep carrying the error instead of unwinding the run. The loop is a
three-line driver. No measurement changed: identical manifest id, identical
per-step decisions and identical retention/forgetting/transfer, verified by
diffing the regenerated results file against the previous one.

The new test pins the contract: the recorded score equals the phase family's
mean on that checkpoint in the per-check curves, so the metric stays defined in
one place.
@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator

Review (Birfy)

Tip SHA: be265c8
Ran:

  • .github/scripts/risk_gate.py --repo ThinkFlowLab/agentdescent --pr 1216 → T3 (5091 lines / 6 files; "diff not inspectable: bench/results/long-horizon-3seed.json" — reviewed the bench code, tests and docs, and re-ran the committed 9-step / 3-seed command)
  • pytest -o addopts="" -p no:cacheprovider -q tests/test_long_horizon_bench.py at this tip → 8 passed
  • python -m bench.long_horizon --steps 3 --seeds 1 --json → exit 0
  • python -m bench.long_horizon --steps 9 --seeds 3 --out … → exit 0; manifest_id 455feebf… matches the committed file; value-identical after stripping cost.wall_clock_s
  • python -m tools.changelog_fragments check → 15 fragments OK
  • python -m tools.gen_api_docs --check → docs/api.md is up to date
  • Required CI at this tip → green (build, test (3.9)–test (3.13); otel-wire also green)
    Not run:
  • Full offline suite locally — the CI test matrix is the line
  • mkdocs build --strict locally — CI build / docs is the line
    Findings:
  • No actionable defects. The bench evolves one FileTree playbook across sequential steps on a Ledger, with a deterministic offline proposer and DefaultAcceptance on held-out checks. Retention, forgetting (from TaskCurve), paired transfer, cost, and refusals are all reported; the committed 3-seed evidence reproduces and every seed is a forgetting verdict at the domain ceiling (0.583).
    Verdict: PASS

@Birfy
Birfy merged commit ee19497 into main Nov 21, 2026
12 checks passed
@Birfy
Birfy deleted the agent/941-long-horizon-bench branch November 21, 2026 22:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agent-pr Pull request opened by an agent risk:T3 Risk tier T3 (core loop, release, process; needs Danyang's click), set by the controller

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bench: long-horizon / continual run with lineage, 3 seeds, negative results allowed

2 participants