Skip to content

Name the node and the reason when a workflow start's run Bots are not created (#3282) - #3284

Merged
Birfy merged 3 commits into
mainfrom
dev/3282-the-issue-fast-line-wf-1cdf6ec5-cannot-c
Nov 21, 2026
Merged

Birfy merged 3 commits into
mainfrom
dev/3282-the-issue-fast-line-wf-1cdf6ec5-cannot-c

Conversation

@Birfy

@Birfy Birfy commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #3282

Written-by: Dev
Owner: Dev
Reviewer: Reviewer
Tier: T2 (plugins/** is T2 in .github/agent/risk-rules.json; docs/** and changelog.d/** are T0/T1) — no gate file, so no Policy-sign: line
Base: 59ee395

What changed

Before: a start whose node Bots the host did not finish creating answered {"ok":false,"error":"the host did not create the run Bots within 60s"} — no node, no reason — and the Bots API delivered that inside a success envelope, {"ok":true,"value":{"ok":false,…}}, so a caller that read the envelope first saw a failed dispatch as a started run.

  1. The refusal names where it stopped. The spawn loop now reports the node it is on and the Bots the host already answered (plugins/workflow/src/service.js, startLive/boundedSpawn), so the deadline sentence reads e.g. the host did not create the run Bots within 60s: 5 of 6 node Bots were created ("Triage" (triage) in 11.0s, …); node "取证" (attest) still had no Bot; a lane of 6 nodes needs about 66s on this host at 11.0s per Bot, past the 60s deadline. A node Bot the host refuses keeps the node and the host's own words, classified model-missing / name-taken / no-slot / no-answer / refused: node "Claim" (claim) could not create its Bot (model-missing): Unknown model opencsitool/Nope-1017. The 60 s deadline itself is unchanged.
  2. The envelope fails when the dispatch failed. workflow/assign is registered through the new service.assignEndpoint(), which throws a refused answer instead of returning it, so api/bot answers {"ok":false,"error":{"code":"bot/refused","message":"…","details":{"workflowId","taskId","nodeId","nodeName","reason"}}} (core/src/host/channel.js ok()/fail()). assign() itself is untouched: { ok: false, error } in-process, never thrown.
  3. Call sites that follow the envelope change (the only ones): the canvas dialog confirmAssign in plugins/workflow/client.js now reads both assign paths through stageCall/settledCall, which turns a refused ok:false answer into the same { ok: false, error } it already drew. In-process callers need no change — the drain, ingest, runNamed, the tools and the plugin's tests keep reading assign(). Nothing else in the repo calls workflow/assign (code search; docs/subsystems/workflow.md now says so).

Why wf-1cdf6ec5 does not start (located, not changed)

Read from the live store ($DSH_HOME/bot/flybot-workflow/workflows.json, read-only; the lane itself is untouched):

  • It is the only 6-node lane (triage, claim, gate, enqueue, escalate, attest); the two controls are 5 and 3 nodes. All six nodes name the same model the working lanes name (opencsitool/DeepSeek-V4.1-Flash-1017), so no model id is missing. Node names are short and the host's runnerName appends a 4-hex suffix and retries on "already exists", so no Bot name is blocking. Its triage node is the only one of the three that carries a seated botId, which only feeds prompt/role/model fallbacks.
  • Node Bots are created serially before the run row exists, and each one costs about 10–12 s on this host: six sessions appear at 06:40:44, :53, 06:41:03, :13, :25, :35 for the first attempt, while the 5-node Triage line took 43 s (06:44:53→06:45:36) and the 3-node implementation line 23 s (06:46:01→06:46:24) and 22 s (07:03:07→07:03:29). 6 × ~11 s ≈ 66 s > the 60 s deadline, so the wait expires with five Bots made, the start is refused, the idle Bots are released and no run is ever written — which is why runs stayed 0 and freeing a tree slot changed nothing.
  • Aggravator, not cause: state.json is 8.1 MB and still carries 329 leaked ephemeral run-Bot records, so every create rewrites the whole team store and per-Bot creation has grown with the roster. Cleaning those records would give the lane back its margin; the durable fix on the owner's side is a shorter line (five nodes or fewer, or Triage/Escalate split out) — that is a live-definition change, left to the owner as the issue asks.

Main merged in (this pass)

cd1977ecde3ddd748eb34d65019f6d2b3098b1de — a real merge commit with both parents kept (0f2d6b378be87dcaad57ec07327f3db82c50238b and 59ee395726317d02c1de57134752ecba8d0d947d): no squash, no force-push, the two commits below it unchanged. #3283 touched core/src/host/capabilities.js, core/test/capabilities.test.js, docs/subsystems/core-host.md, patches/ds-bot/0001-core-fixes.patch, patches/ds-bot/0014-flybot-branding.patch and changelog.d/3281.fixed.md — none of this branch's six paths, so the merge tree is main's tree with this branch's six blobs. Every pushed blob was compared to the file the tests ran (git blob sha, SAME on all six), so the results below are results for the pushed bytes. node scripts/ds-bot-patches.mjs check after the merge: ok 10 patches apply cleanly in order and ok core/ equals ds-bot/ + patches/ds-bot/ — #3283's patch edits and the service.js change do not interfere.

Verification at the tip cd1977ecde3ddd748eb34d65019f6d2b3098b1de

This box: node 22.23.3, no git. Full tarball of main 59ee395 extracted, the branch's six files written over it, then npm ci --include=dev + npm run harness in core/ (655 packages) so ds-bot and the harness resolve.

Ran at this tip:

# tip cd1977ecde3ddd748eb34d65019f6d2b3098b1de; main 59ee395726317d02c1de57134752ecba8d0d947d merged in (merge commit, both parents kept)
# all six pushed blobs compared to the files these runs used: SAME

$ node --test plugins/workflow/test/spawn-failure-message.test.js
ok 1 - \#3282a: a node whose model does not exist names the node and the host reason
ok 2 - \#3282a: a node the host never answers names that node, not only the deadline
ok 3 - \#3282b: a refused start fails the envelope a caller checks first
1..3
# tests 3
# suites 0
# pass 3
# fail 0
# cancelled 0
# skipped 0
# todo 0

$ node --test plugins/workflow/test/late-spawn-release.test.js
ok 1 - \#2785: a late spawn failure still releases the Bots it made
ok 2 - \#2785 control: the same partial failure answered before the deadline is released
1..2
# tests 2
# suites 0
# pass 2
# fail 0
# cancelled 0
# skipped 0
# todo 0

$ npm --prefix plugins/workflow test        # node --import ./test/resolve.mjs --test "test/*.test.js"
ok 451 - \#3282a: a node whose model does not exist names the node and the host reason
ok 452 - \#3282a: a node the host never answers names that node, not only the deadline
ok 453 - \#3282b: a refused start fails the envelope a caller checks first
1..527
# tests 541
# suites 4
# pass 541
# fail 0
# cancelled 0
# skipped 0
# todo 0

$ node --test plugins/workflow/             # node 22.23.3 runs the directory as one unit and finds no test file in it
ok 1 - plugins/workflow
1..1
# tests 1
# suites 0
# pass 1
# fail 0
# cancelled 0
# skipped 0
# todo 0

# red first: plugins/workflow/src/service.js, client.js and index.js at main 59ee3957, new test unchanged
$ node --test plugins/workflow/test/spawn-failure-message.test.js
not ok 1 - \#3282a: a node whose model does not exist names the node and the host reason
  error: 'the node that failed is named: Unknown model opencsitool/Nope-1017'
  actual: 'Unknown model opencsitool/Nope-1017'
not ok 2 - \#3282a: a node the host never answers names that node, not only the deadline
  error: 'the node that never answered is named: the host did not create the run Bots within 60s'
  actual: 'the host did not create the run Bots within 60s'
not ok 3 - \#3282b: a refused start fails the envelope a caller checks first
  error: 'okSvc.assignEndpoint is not a function'
1..3
# tests 3
# suites 0
# pass 0
# fail 3
# cancelled 0
# skipped 0
# todo 0

# restored the branch files, same command again
$ node --test plugins/workflow/test/spawn-failure-message.test.js
ok 1 - \#3282a: a node whose model does not exist names the node and the host reason
ok 2 - \#3282a: a node the host never answers names that node, not only the deadline
ok 3 - \#3282b: a refused start fails the envelope a caller checks first
1..3
# tests 3
# suites 0
# pass 3
# fail 0
# cancelled 0
# skipped 0
# todo 0
  • Red first (implementation reverted to main's three files, the new test unchanged): node --test plugins/workflow/test/spawn-failure-message.test.js → # tests 3 / # pass 0 / # fail 3 — the node that failed is named: Unknown model opencsitool/Nope-1017 (the bare host text, no node), the node that never answered is named: the host did not create the run Bots within 60s (the bare deadline), okSvc.assignEndpoint is not a function (no seam to refuse through). Restored, same command → # tests 3 / # pass 3 / # fail 0.
  • node --test plugins/workflow/test/late-spawn-release.test.js → # tests 2 / # pass 2 / # fail 0.
  • npm --prefix plugins/workflow test (the package's own script: node --import ./test/resolve.mjs --test "test/*.test.js") → # tests 541 / # suites 4 / # pass 541 / # fail 0 — includes host.test.js, which needs the resolver to reach ds-bot.
  • node --test "plugins/workflow/test/*.test.js" (no resolver) → # tests 540 / # pass 539 / # fail 1; the one failure is host.test.js failing to load (Cannot find module 'ds-bot'), i.e. the same file the script above runs green.
  • node --test plugins/workflow/ on node 22.23.3 runs the directory as one unit and answers # tests 1 / # pass 1 (ok 1 - plugins/workflow): it finds no test file, so the two commands above are the real suite.
  • node core/scripts/build-client.mjs --check → exit 0 (no core/src/client/** change, so core/client.js needs no rebuild).
  • node scripts/ds-bot-patches.mjs check → ok 10 patches apply cleanly in order, ok core/ equals ds-bot/ + patches/ds-bot/ (skips the pin hash: no git).
  • node .github/scripts/changelog.mjs check → changelog: 5 fragment(s) in changelog.d/ are well formed.
  • node --test ".github/scripts/*.test.mjs" "policy/*.test.mjs" → # tests 717 / # pass 709 / # fail 6; all six are auto-queue-t1: spawnSync gh EACCES in auto-queue-t1.test.mjs (this box has no gh), unrelated to this change.
  • NODE_ENV=development npm --prefix core test → # tests 3417 / # pass 3413 / # fail 2, both in core/test/check-arch-docs.test.js because it shells out to git (cannot diff against HEAD); no core/** file is changed here.

Not run, and why:

  • TZ=UTC npm run verify — it stops on its first step, core/scripts/check-arch-docs.mjs: arch-docs: FAIL — cannot diff against origin/main, this box has no git. Equivalent check with the script's own loadSubsystemMap(): the only owned area touched is plugins/workflow/ (owned by docs/subsystems/workflow.md) and that page is edited in the same change.
  • node scripts/ds-bot-patches.mjs canonical — needs git to compare the committed patch series.
  • UI screenshots — the canvas refuses through the { ok: false, error } it already drew; client.test.js (180 tests) is part of the 541 green above.

Checkpoint

{
  "pr": 3284,
  "claim": "Dev",
  "issue": 3282,
  "summary": "A workflow start the host could not finish now names the node it waited on, how many node Bots were made, how long each took, and the seconds the lane needs past the 60s deadline; a refused node Bot carries the node and the host's reason. workflow/assign throws that refusal through assignEndpoint, so api/bot answers { ok: false, error }; assign() keeps its in-process answer and the canvas reads it through settledCall. wf-1cdf6ec5 is the only six-node lane: about 11s per Bot here runs past the 60s deadline before a run row exists; the live definition is untouched.",
  "files": ["plugins/workflow/src/service.js", "plugins/workflow/index.js", "plugins/workflow/client.js", "plugins/workflow/test/spawn-failure-message.test.js", "docs/subsystems/workflow.md", "changelog.d/3282.fixed.md"],
  "checks": { "ran": "red first at main 3/3 fail, green 3/3 after restore; workflow 541/541 via npm test; late-spawn 2/2; build-client --check 0; ds-bot-patches check ok; changelog ok; core 3413/3417 (2 need git)", "result": "green", "at": "cd1977ecde3ddd748eb34d65019f6d2b3098b1de" },
  "next": "none — ready for review"
}

… created

A start whose node Bots the host does not create in time answered only
"the host did not create the run Bots within 60s", so a lane that needs more
time than the deadline looked like a silent hang. The refusal now names the
node the host never answered, how many node Bots it did create and how long
each took, and the seconds the lane needs when that is past the deadline; a
node Bot the host refuses carries the node and the host's own reason.

The endpoint seam refuses by throwing, so api/bot answers the refusal as
{ ok: false, error } instead of putting { ok: false } inside the success
envelope; assign() keeps its own answer for in-process callers and the canvas
reads the thrown refusal through settledCall.

Fixes #3282
Written-by: Dev
@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Written-by: Dev

Note on the checkpoint: the enqueue-guard reading at this head (1698 characters, summary 821) was taken from the body as it was opened. The block is now 1203 characters with a 569-character summary — inside both limits, at still naming this tip — and no commit was added, so the rewrite is a body edit only. This comment is what re-judges it: the guard runs on issue_comment as well as on a push.

The remaining reason in enqueue-guard (no Verifier report names the tip c40d4b829e8f) is the T2 gate itself, for the PR 快线 to answer at this tip.

A start waits for its node Bots, and the host refuses it at its own 60 s
deadline, so the canvas must not cut the call at the shared 10 s: both assign
call sites pass ASSIGN_TIMEOUT_MS (70 s) and still show a refused answer as
the sentence the dialog already drew.

Fixes #3282
Written-by: Dev
#3283 changed core/src/host/capabilities.js, its test and the ds-bot patches; none of the
six paths this branch touches. The merge keeps both commits whole and adds no edit:
  plugins/workflow/src/service.js
  plugins/workflow/client.js
  plugins/workflow/index.js
  plugins/workflow/test/spawn-failure-message.test.js
  docs/subsystems/workflow.md
  changelog.d/3282.fixed.md
@Birfy Birfy self-assigned this Nov 21, 2026
@Birfy

Birfy commented Nov 21, 2026

Copy link
Copy Markdown
Collaborator Author

Verdict: PASS at cd1977e
Written-by: Reviewer
Reviewed-by: Reviewer
Attestation: cd1977e role=Reviewer
Reviewed-tip: cd1977e

Ran at this tip:

$ node --test "plugins/workflow/test/*.test.js"
# tests 540  # pass 539  # fail 1
$ node --test "test/spawn-failure-message.test.js"   # red tree: only src/service.js reverted to base blob 72043e24
# tests 3  # pass 0  # fail 3

The one local failure is plugins/workflow/test/host.test.js — ERR_MODULE_NOT_FOUND: Cannot find package 'ds-bot', because this checkout has no npm install; the job that does install it (plugins (workflow)) is success at this tip. The three #3282 tests pass here at the tip (ok 450–452) and all three fail against the base service.js, so the new tests are genuinely red-first.

Per acceptance point:

  1. Human sentence — plugins/workflow/src/service.js:69-120 (nodeLabel, SPAWN_REASONS, spawnReasonOf, spawnFailure) and :488-536 (spawnTimeoutText). A refused Bot carries node "Claim" (claim) could not create its Bot (model-missing): Unknown model …, with nodeId/nodeName/reason/details; the deadline sentence names the node still waiting, how many Bots were made with each latency, and the seconds a lane of N nodes needs at that host's observed rate. SPAWN_DEADLINE_MS is untouched (60 s), no error is swallowed — the host's own text is carried verbatim.

  2. Envelope — service.js:1946-1975 assignEndpoint rethrows a refused start as code: 'bot/refused' with details, which core/src/host/channel.js reports as { ok: false, error }; plugins/workflow/index.js:146 routes workflow/assign through it and leaves every other endpoint alone. Existing in-process callers are intact: assign() still answers { ok: false } (asserted in the third test), and plugins/workflow/client.js now reads the refusal through settledCall with a 70 s ASSIGN_TIMEOUT_MS so the 60 s host deadline is not cut off at the old 10 s.

  3. Red-first — plugins/workflow/test/spawn-failure-message.test.js: (a) unknown model names Claim + the host text, (a) a never-answering node names "取证" and 5 of 6 at the deadline, (b) the outer envelope is ok:false / bot/refused while assign() keeps its own answer. Reverting only service.js to the base blob turns all three red (3/3 fail), so the tests hold the fix, not the fixture.

  4. Docs + changelog — one sentence added to docs/subsystems/workflow.md and changelog.d/3282.fixed.md.

Scope: compare 59ee3957…cd1977ec touches exactly the 6 files above; no .github/**, no policy/, no change to the live wf-1cdf6ec5 definition. Checks at this tip are all terminal with only enqueue-guard red (its own "no Verifier report names the tip" line, which this verdict clears); tests-ok, policy-sign, merge-gate, plugins (workflow), package and judge are success, live/window check skipped.

@github-actions github-actions Bot added the pass label Nov 21, 2026
@Birfy
Birfy added this pull request to the merge queue Nov 21, 2026
Merged via the queue into main with commit 041342a Nov 21, 2026
21 checks passed
@Birfy
Birfy deleted the dev/3282-the-issue-fast-line-wf-1cdf6ec5-cannot-c branch November 21, 2026 09:24
@github-actions github-actions Bot removed the agent-pr Pull request opened by an agent label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The issue fast line (wf-1cdf6ec5) cannot create its run Bots: assign fails three times with a bare 60 s timeout

1 participant