Skip to content

feat: benchmark Phase 1 spawns in waves and respawns a failed task (14.11.0) - #303

Closed
dam-code-protector[bot] wants to merge 4 commits into
mainfrom
feat/benchmark-wave-spawn
Closed

dam-code-protector[bot] wants to merge 4 commits into
mainfrom
feat/benchmark-wave-spawn

Conversation

@dam-code-protector

@dam-code-protector dam-code-protector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor

Scope

Phase 1 of a benchmark run spawned every task at once. On a full pool (16 tasks + parallel reviews) about half the targets never got a machine and were deleted by the platform sweep while their invocation waited — each reported target agent deleted — and the runbook's fix was a hand-driven sequence of single-task retries.

This adds benchmark-isolate.sh run: start, wait and retry in one command.

  • Spawns at most --waves (default CG_BENCH_WAVES, else 8) tasks at a time; the wave bound is what the owner's memory pool admits.
  • Waits for the wave to end, then respawns a failed task up to --retries (default 1) times in a later wave, on the same snapshot, commit and model start enforces.
  • Exits 0 when every task holds a result, 1 when a task failed its last attempt (summary names task and platform reason). It never collects — collect is unchanged.
  • start + wait stay for a hand-scheduled spawn of the whole list.
  • A wave whose wait times out is waited for again; a task that holds an id and no end is never respawned. CG_BENCH_WAVES, --waves, --retries and every attempts/ file must be whole numbers; run exits 1 and names every expected task that holds no result.
  • docs/benchmark.md Phase 1 now prescribes run; VERSION 14.12.0 (main took 14.11.0), Upgrade: nothing to run.

Evidence

  • scripts/tests/test_benchmark_isolate.sh: 5 run cases — wave order proven from the REST stub log (no second spawn before the first ended), one respawn of a failed target with a fresh id, permanent-failure exit and message, flag and CG_BENCH_WAVES validation (0, 8 tasks, x refused, nothing spawned), a wait timeout that is waited for again (no respawn, the result is read), and a task with no result or a malformed attempts file that fails the run. Full file: ALL TESTS PASSED (139 ok); full suite in Docker: ALL TESTS PASSED.
  • check-definition.sh --base origin/main: 0 fail, 0 warn, 2 note (kit task text defers to the doc; one usage-line match is substring-level).

…4.11.0)

benchmark-isolate.sh gains run: start, wait and retry in one command. It
spawns at most --waves (CG_BENCH_WAVES, default 8) tasks at a time, waits
for the wave, and respawns a failed task up to --retries (default 1) times,
so a platform start failure — target agent deleted, a target the owner's
memory pool never admitted and the sweep deleted — costs one respawn
instead of a manual retry sequence. start and wait stay for hand-scheduled
runs; collect is unchanged.

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian - Code guardian repo — ⚠️ Code Review @ 4070a7b

PR #303: feat: benchmark Phase 1 spawns in waves and respawns a failed task (14.11.0)

Author: dam-code-protector[bot] | Branch: feat/benchmark-wave-spawn → main | Changes: +134 −16 (5 files)

Summary

This PR adds benchmark-isolate.sh run to spawn Phase 1 tasks in waves and respawn a failed task once. It also moves the Phase 1 procedure in docs/benchmark.md to that command. Two failure paths in the new loop misread the state of a task.
Limits: 3 skill(s) did not run: issue-fit, typescript-engineering, react-ui-engineering.
2 suggestion(s) dropped under the 🟢 budget.

Findings

  • 🟡 Warning: A wait timeout respawns still-running tasks and loses their result (scripts/benchmark-isolate.sh:396). Reported by Agent Engineering.
  • 🟡 Warning: An unvalidated CG_BENCH_WAVES makes run exit 0 with no spawn (scripts/benchmark-isolate.sh:360). Reported by Agent Engineering.

Documentation Check (doc-drift)

✅ No findings.

Agent Engineering

  • 🟢 Suggestion: The Isolation intro says Phase 1 runs "all at once" (docs/benchmark.md:221). Step 2 of the same section now runs at most --waves tasks at a time.

Verdict

COMMENT — Two bounded failure paths in the new run loop; the wave logic is otherwise sound and tested.


Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi

Comment thread scripts/benchmark-isolate.sh Outdated
done
done
[ "$batch_n" -gt 0 ] || break
wait_all "$st" || [ "$?" -eq 3 ] || return 1

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning: wait_all returns 3 when its timeout expires with tasks still running. run_all then treats the wave as ended and respawns every task without a result. start deletes that target's .id file, so its result is never read. A second sub-agent then spends the run budget.

Fix: On exit 3, call wait_all again until it returns 0 or 1. Never respawn a task that holds an .id without a result.

Comment thread scripts/benchmark-isolate.sh Outdated
}

run_all() { # <start args…> <state-dir> <ts> <fixture-root> <slug>… — waves + retry
local waves="${CG_BENCH_WAVES:-8}" retries=1 pass_opts="" st ts root task slug ph batch_n attempts f failed_n

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Warning: The env path does not validate CG_BENCH_WAVES. A value of 0 or 8 tasks makes the wave test fail, so the loop breaks before the first start. run then exits 0 without a spawn, and the state dir holds no ts or expect.json.

Fix: Validate waves and each value read from attempts/ as an integer in range. Exit nonzero when an expected task has no result and no exhausted attempt count.

@dam-code-protector

Copy link
Copy Markdown
Contributor Author

🔧 Code Guardian - Code guardian repo — CI triage at 4070a7b

validate — the VERSION and CHANGELOG equality check failed
VERSION holds 14.11.0. The check read the newest CHANGELOG entry as 14.10.0. The new heading ## 14.11.0 has no text after the version. The check matches only a heading with a space after the version number.
Fix: Write the heading as ## 14.11.0 — <date>, as the entry below it does.

A wait that times out (rc 3) is waited for again; a task that holds an id
and no end is never respawned. CG_BENCH_WAVES and every attempts file are
validated as whole numbers, and run exits 1 naming every expected task that
holds no result.
@xjacka

xjacka commented Nov 12, 2026

Copy link
Copy Markdown
Collaborator

Answer to the review:

  • Fixed — scripts/benchmark-isolate.sh:396 (wait timeout respawns running tasks): on rc 3, run calls wait_all again until it returns 0 or 1. A task that holds an .id and no .failed is never respawned; run waits for it, also when a wave spawns nothing. New test run_waits_again_after_a_wait_timeout (2 POSTs, the result is read).
  • Fixed — scripts/benchmark-isolate.sh:360 (unvalidated CG_BENCH_WAVES): CG_BENCH_WAVES, --waves, --retries and every attempts/ file must be whole numbers, else run exits 1 and names the value. After the last wave, run exits 1 and names every expected task without a result (failed for good or no result). The header comment and docs/benchmark.md → Isolation state the same rule.
  • Checks: git grep -nE -e 'wait_all .*-eq 3' → no hits; git grep -nE -e 'CG_BENCH_WAVES' → docs, CHANGELOG, header and tests, and line 382 validates the value. Before the edit, both had the reported hits. test_benchmark_isolate.sh 139 ok, full suite in Docker ALL TESTS PASSED, check-definition.sh 0 fail 0 warn.
  • CI validate failed because VERSION 14.11.0 had no dated CHANGELOG heading and main took 14.11.0. Main is merged; the version is now 14.12.0 — 2026-11-12.
  • Optional items taken: the docs intro says "in waves" (not "all at once"); the identical if/else around start is one call; the CHANGELOG heading has its date.

@xjacka xjacka added the code-guardian:rereview Request a code-guardian delta re-review label Nov 12, 2026

@dam-code-protector dam-code-protector Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛡️ Code Guardian - Code guardian repo — ✅ Code Review @ feef199

Verdict: APPROVE. This account opened the pull request. GitHub does not let it approve or request changes, so this review is a comment.

PR #303: feat: benchmark Phase 1 spawns in waves and respawns a failed task (14.11.0)

Author: dam-code-protector[bot] | Branch: feat/benchmark-wave-spawn → main | Changes: +192 −17 (5 files)

Summary

This PR adds benchmark-isolate.sh run to spawn Phase 1 tasks in waves and respawn a failed task once. The re-review confirms the two earlier failure paths are fixed and the docs now describe the wave behavior.
Limits: 3 skill(s) did not run: issue-fit, typescript-engineering, react-ui-engineering.

Changes since last review

Previous HEAD: 4070a7b (2026-11-11T13:30:38Z) — verdict COMMENT

  • ✅ Fixed: a wait timeout respawns running tasks and loses their result (scripts/benchmark-isolate.sh:396)
  • ✅ Fixed: an unvalidated CG_BENCH_WAVES exits 0 with no spawn (scripts/benchmark-isolate.sh:360)
  • ✅ Fixed: the intro says all at once, step 2 uses waves (docs/benchmark.md:221)

Findings

No new findings at this HEAD.

Documentation Check (doc-drift)

✅ No findings.

Agent Engineering

✅ No findings.

Verdict

APPROVE — Both prior warnings are fixed at this HEAD and no new finding is present.


Review by Code Guardian - Code guardian repo · automated code guardian · cb/rits/deepseek-ai/DeepSeek-V4.1-Flash-1017 on pi

@dam-code-protector dam-code-protector Bot removed the code-guardian:rereview Request a code-guardian delta re-review label Nov 12, 2026
@xjacka

xjacka commented Nov 13, 2026

Copy link
Copy Markdown
Collaborator

This PR is closed. PR #305 replaces it. PR #305 adds a parallel cap to benchmark-isolate.sh (--max-parallel, default 8). The cap starts a queued task when a running task ends. An automatic respawn of a failed task can come in a new PR on top of 15.0.0.

@xjacka xjacka closed this Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant