Repository navigation
feat: benchmark Phase 1 spawns in waves and respawns a failed task (14.11.0) - #303
dam-code-protector[bot] wants to merge 4 commits into
Conversation
…4.11.0) benchmark-isolate.sh gains run: start, wait and retry in one command. It spawns at most --waves (CG_BENCH_WAVES, default 8) tasks at a time, waits for the wave, and respawns a failed task up to --retries (default 1) times, so a platform start failure — target agent deleted, a target the owner's memory pool never admitted and the sweep deleted — costs one respawn instead of a manual retry sequence. start and wait stay for hand-scheduled runs; collect is unchanged.
There was a problem hiding this comment.
🛡️ Code Guardian - Code guardian repo — 4070a7b
PR #303: feat: benchmark Phase 1 spawns in waves and respawns a failed task (14.11.0)
Author: dam-code-protector[bot] | Branch: feat/benchmark-wave-spawn → main | Changes: +134 −16 (5 files)
Summary
This PR adds benchmark-isolate.sh run to spawn Phase 1 tasks in waves and respawn a failed task once. It also moves the Phase 1 procedure in docs/benchmark.md to that command. Two failure paths in the new loop misread the state of a task.
Limits: 3 skill(s) did not run: issue-fit, typescript-engineering, react-ui-engineering.
2 suggestion(s) dropped under the 🟢 budget.
Findings
- 🟡 Warning: A
waittimeout respawns still-running tasks and loses their result (scripts/benchmark-isolate.sh:396). Reported by Agent Engineering. - 🟡 Warning: An unvalidated
CG_BENCH_WAVESmakesrunexit 0 with no spawn (scripts/benchmark-isolate.sh:360). Reported by Agent Engineering.
Documentation Check (doc-drift)
✅ No findings.
Agent Engineering
- 🟢 Suggestion: The Isolation intro says Phase 1 runs "all at once" (
docs/benchmark.md:221). Step 2 of the same section now runs at most--wavestasks at a time.
Verdict
COMMENT — Two bounded failure paths in the new run loop; the wave logic is otherwise sound and tested.
Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi
| done | ||
| done | ||
| [ "$batch_n" -gt 0 ] || break | ||
| wait_all "$st" || [ "$?" -eq 3 ] || return 1 |
There was a problem hiding this comment.
🟡 Warning: wait_all returns 3 when its timeout expires with tasks still running. run_all then treats the wave as ended and respawns every task without a result. start deletes that target's .id file, so its result is never read. A second sub-agent then spends the run budget.
Fix: On exit 3, call wait_all again until it returns 0 or 1. Never respawn a task that holds an .id without a result.
| } | ||
|
|
||
| run_all() { # <start args…> <state-dir> <ts> <fixture-root> <slug>… — waves + retry | ||
| local waves="${CG_BENCH_WAVES:-8}" retries=1 pass_opts="" st ts root task slug ph batch_n attempts f failed_n |
There was a problem hiding this comment.
🟡 Warning: The env path does not validate CG_BENCH_WAVES. A value of 0 or 8 tasks makes the wave test fail, so the loop breaks before the first start. run then exits 0 without a spawn, and the state dir holds no ts or expect.json.
Fix: Validate waves and each value read from attempts/ as an integer in range. Exit nonzero when an expected task has no result and no exhausted attempt count.
|
🔧 Code Guardian - Code guardian repo — CI triage at validate — the VERSION and CHANGELOG equality check failed |
A wait that times out (rc 3) is waited for again; a task that holds an id and no end is never respawned. CG_BENCH_WAVES and every attempts file are validated as whole numbers, and run exits 1 naming every expected task that holds no result.
|
Answer to the review:
|
There was a problem hiding this comment.
🛡️ Code Guardian - Code guardian repo — ✅ Code Review @ feef199
Verdict: APPROVE. This account opened the pull request. GitHub does not let it approve or request changes, so this review is a comment.
PR #303: feat: benchmark Phase 1 spawns in waves and respawns a failed task (14.11.0)
Author: dam-code-protector[bot] | Branch: feat/benchmark-wave-spawn → main | Changes: +192 −17 (5 files)
Summary
This PR adds benchmark-isolate.sh run to spawn Phase 1 tasks in waves and respawn a failed task once. The re-review confirms the two earlier failure paths are fixed and the docs now describe the wave behavior.
Limits: 3 skill(s) did not run: issue-fit, typescript-engineering, react-ui-engineering.
Changes since last review
Previous HEAD: 4070a7b (2026-11-11T13:30:38Z) — verdict COMMENT
- ✅ Fixed: a wait timeout respawns running tasks and loses their result (
scripts/benchmark-isolate.sh:396) - ✅ Fixed: an unvalidated CG_BENCH_WAVES exits 0 with no spawn (
scripts/benchmark-isolate.sh:360) - ✅ Fixed: the intro says all at once, step 2 uses waves (
docs/benchmark.md:221)
Findings
No new findings at this HEAD.
Documentation Check (doc-drift)
✅ No findings.
Agent Engineering
✅ No findings.
Verdict
APPROVE — Both prior warnings are fixed at this HEAD and no new finding is present.
Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi
Scope
Phase 1 of a benchmark run spawned every task at once. On a full pool (16 tasks + parallel reviews) about half the targets never got a machine and were deleted by the platform sweep while their invocation waited — each reported
target agent deleted— and the runbook's fix was a hand-driven sequence of single-task retries.This adds
benchmark-isolate.sh run: start, wait and retry in one command.--waves(defaultCG_BENCH_WAVES, else 8) tasks at a time; the wave bound is what the owner's memory pool admits.--retries(default 1) times in a later wave, on the same snapshot, commit and modelstartenforces.collectis unchanged.start+waitstay for a hand-scheduled spawn of the whole list.CG_BENCH_WAVES,--waves,--retriesand everyattempts/file must be whole numbers;runexits 1 and names every expected task that holds no result.run; VERSION 14.12.0 (main took 14.11.0), Upgrade: nothing to run.Evidence
scripts/tests/test_benchmark_isolate.sh: 5 run cases — wave order proven from the REST stub log (no second spawn before the first ended), one respawn of a failed target with a fresh id, permanent-failure exit and message, flag andCG_BENCH_WAVESvalidation (0,8 tasks,xrefused, nothing spawned), a wait timeout that is waited for again (no respawn, the result is read), and a task with no result or a malformedattemptsfile that fails the run. Full file: ALL TESTS PASSED (139 ok); full suite in Docker: ALL TESTS PASSED.check-definition.sh --base origin/main: 0 fail, 0 warn, 2 note (kit task text defers to the doc; one usage-line match is substring-level).