Skip to content

fix(workflow): spawn run Bots with bounded concurrency (#3286) - #3287

Merged
Birfy merged 1 commit into
mainfrom
dev/3286-run-bot-60-s-5-6-wf-1cdf6ec5
Nov 21, 2026
Merged

Birfy merged 1 commit into
mainfrom
dev/3286-run-bot-60-s-5-6-wf-1cdf6ec5

Conversation

@Birfy

@Birfy Birfy commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Written-by: Dev
Fixes #3286
Owner: Dev
Reviewer: Reviewer

What changed

A run created its node Bots one at a time, so a six-node line could not fit the 60s batch deadline (SPAWN_DEADLINE_MS) and a five-node line was a coin flip. spawnAll now creates up to SPAWN_CONCURRENCY = 4 at once (the width is a parameter tests can inject). A refused spawn starts no further Bot, the ones already in flight are awaited and still land in error.spawned (creation order), and the returned map keeps the graph's node order whatever order the host answers in. The deadline sentence's "needs about Ns" estimate now counts waves, and progress names the last-started node the host has not answered, so #3282's node name and reason still work with several spawns in flight.

Before / after

  • Before: serial 6 × ~11s ≈ 66s > 60s (wf-1cdf6ec5 failed three times); 5 × ~11s ≈ 55s, ~5s of slack (wf-7ed167b6 failed once at 10:45Z).
  • After: ceil(6/4) = 2 waves × ~11s ≈ 22s; ceil(5/4) = 2 waves ≈ 22s. The new test drives a fake clock with 10s per spawn: six nodes finish in 20s, with at most 4 in flight.

Verification at the tip

Tip SHA: 534c2c8
Ran:

  • cd plugins/workflow && DSH_HARNESS=/opt/flybot node --import ./test/resolve.mjs --test "test/*.test.js" — 544 tests, 544 pass, 0 fail.
  • node core/scripts/build-client.mjs --check — ok (no client change).
  • node scripts/ds-bot-patches.mjs check — ok, 10 patches.
  • node .github/scripts/changelog.mjs check — ok.
  • node scripts/lint-plugins.mjs and oxlint over core — 0 errors (warnings are pre-existing).
    Not run (and why):
  • Full npm run verify could not complete on this box: core here has no dev/harness oxlint install and arch-docs needs git + origin/main, and this host has no git. The affected workflow suite, the client check, the patch check, the changelog check and lint were run as above; core/ is untouched.

Red first

Source unchanged, new test only (plugins/workflow/test/parallel-spawn.test.js): 1 of 3 pass, 2 fail — never more than four spawns in flight: 1 (the serial loop) and two spawns are in flight: actual ["node-1"]. The same two tests pass after the change.

Checkpoint

{
  "pr": 3287,
  "claim": "Dev",
  "issue": 3286,
  "summary": "spawnAll creates run Bots four at a time instead of one after another, so a six-node lane needs two waves (~22s) and fits the 60s batch deadline. A refused spawn starts no new Bot, awaits the ones in flight, and error.spawned lists every Bot made in creation order. The returned map follows the graph's node order, and the timeout sentence's estimate counts waves.",
  "files": ["plugins/workflow/src/ephemeral.js", "plugins/workflow/src/service.js", "plugins/workflow/test/parallel-spawn.test.js", "docs/subsystems/workflow.md", "changelog.d/3286.fixed.md"],
  "checks": { "ran": "cd plugins/workflow && DSH_HARNESS=/opt/flybot node --import ./test/resolve.mjs --test 'test/*.test.js'", "result": "green", "at": "534c2c860eb0c8470cb2082ee437b5d75042cc6f" },
  "next": "none — ready for review"
}

A run created its node Bots one at a time. A host answers a spawn in
about ten seconds, so a six-node lane needed ~66s and missed the 60s
batch deadline; a five-node lane had ~5s of slack (#3286). Spawn four at
a time so the same lane takes two waves (~20s). A refused spawn stops new
starts, lets the ones in flight land, and error.spawned still lists every
Bot that was made; the result keeps the graph nodes in their own order.

Fixes #3286
Written-by: Dev
@Birfy Birfy added the agent-pr Pull request opened by an agent label Nov 21, 2026
@Birfy Birfy self-assigned this Nov 21, 2026
@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at 534c2c8
Written-by: Reviewer
Attestation: 534c2c8 role=Reviewer
Reviewed-tip: 534c2c8

Ran at this tip:

$ cd plugins/workflow && DSH_HARNESS=/opt/flybot node --import ./test/resolve.mjs --test "test/*.test.js"
# tests 544  # pass 544  # fail 0
$ tar -xzf /data/dsh/bot/workspace/i3286-impl/main.tar.gz -O ThinkFlowLab-FlyBot-747e009/plugins/workflow/src/ephemeral.js > src/ephemeral.js
$ cd plugins/workflow && DSH_HARNESS=/opt/flybot node --import ./test/resolve.mjs --test test/parallel-spawn.test.js
# tests 3  # pass 1  # fail 2
$ cp /tmp/ephemeral.tip.js src/ephemeral.js
$ cd plugins/workflow && DSH_HARNESS=/opt/flybot node --import ./test/resolve.mjs --test test/parallel-spawn.test.js test/late-spawn-release.test.js test/spawn-failure-message.test.js
# tests 8  # pass 8  # fail 0

Review notes (tip 534c2c8, base 747e009):

  • Bounded concurrency is real: six nodes run with at most 4 spawns in flight and finish in two waves (20s on the test's 10s-per-spawn fake clock, 2/3 waves bound); default SPAWN_CONCURRENCY = 4 is exported and injected as the optional last argument. No de-facto serial fallback: when I restored the base serial spawnAll, the concurrency test failed with never more than four spawns in flight: 1, and the failure test failed with only ["node-1"] in flight (2 of 3 red). Restored tip: 8/8 green.
  • Failure semantics hold: a refusal stops taking new indexes at once, the in-flight spawns are awaited, and error.spawned carries every Bot that was actually made (landing order), so releaseAll(error.spawned) in service.js:1438 plus the deadline-late branch (boundedSpawn, workflow: a late spawn failure leaks the Bots it already made #2785) release everything. late-spawn-release.test.js (2/2) and spawn-failure-message.test.js (2/2) stay green; the new failure test proves node-5/node-6 are never asked for and node-4 still lands and is reported.
  • Result order is stable: spawnAll keys ephemeral by the graph's node order via byIndex, not by answer order; regression test 3 passes, and the whole-suite run has no order/ephemeral.test.js regressions.
  • service.js deadline change is contained: the "needs about Ns" estimate now counts ceil(total / SPAWN_CONCURRENCY) waves, and progress names the last-started unanswered node via a pending map; no existing test asserts the old one-node sentence (grep needs about|past the finds none in plugins/workflow/test), and deadline.test.js / spawn-timeout paths stay green in the 544-test run. Docs example updated to the wave math.
  • Scope matches base..tip: exactly the 5 files in the brief (ephemeral.js +46/-12, service.js +16/-10, new parallel-spawn.test.js +125, docs, changelog fragment); all five blob SHAs equal the tip tree, and the 90-file plugins/workflow tree also matches the tip exactly.
  • Check-runs at this tip: only enqueue-guard fails (missing verdict — this comment); tests-ok, merge-gate, risk tier, workflow/package/policy checks all succeeded. No hold, agent-pr present, body has Reviewer: Reviewer and a valid checkpoint block (928 chars, summary 364).

@Birfy
Birfy added this pull request to the merge queue Nov 21, 2026
@github-actions github-actions Bot added the pass label Nov 21, 2026
Merged via the queue into main with commit 5cc2dee Nov 21, 2026
19 of 21 checks passed
@Birfy
Birfy deleted the dev/3286-run-bot-60-s-5-6-wf-1cdf6ec5 branch November 21, 2026 11:59
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

run Bot 串行创建 + 60 s 整批超时:5–6 节点的线随机起不来(wf-1cdf6ec5 必挂)

1 participant