diff --git a/changelog.d/3286.fixed.md b/changelog.d/3286.fixed.md new file mode 100644 index 000000000..e9ec89098 --- /dev/null +++ b/changelog.d/3286.fixed.md @@ -0,0 +1 @@ +A Workflow now creates its run Bots four at a time instead of one after another, so a six-node line starts in about 20 seconds instead of about 66 and no longer misses the 60-second batch deadline a slow host used to hit ([#3286](https://github.com/ThinkFlowLab/FlyBot/issues/3286)). If the host refuses one node, the line stops asking for new Bots, waits for the ones already in flight, and releases every Bot it did create; the run still maps the graph's nodes to their Bots in the graph's own order. diff --git a/docs/subsystems/workflow.md b/docs/subsystems/workflow.md index 6110f377d..72758387d 100644 --- a/docs/subsystems/workflow.md +++ b/docs/subsystems/workflow.md @@ -40,7 +40,7 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher, | --- | --- | | `plugins/workflow/index.js` | Plugin entry (`flybot-workflow`). Endpoints, `run_workflow`, `define_workflow`, the board listener, and the reply middleware that ingests a step result and denies delivery to persistent Bot chats. | | `plugins/workflow/src/graph.js` | Nodes, edges, positions, and named inputs and outputs. A node starts when its forward predecessors are settled and a taken edge reaches it. Same-rank nodes run together. A forward cycle is refused. A loop edge is the only cycle for success paths. A `when` edge chooses a branch; the other is skipped. A required input with no upstream provider is refused before a run starts. An `onFail` edge provides failure recovery: when a node fails with recoveries remaining, the workflow resets the body from the upstream target through the failing node and re-runs it in order (tracked by `failAttempts` per edge, with `max` counting recoveries, 1–8, default 3). After the recoveries are used, the node fails permanently. | -| `plugins/workflow/src/service.js` | The store, lanes (active runs, FIFO queue, terminal history), `maxConcurrent`, `rename`, `delete`, `saveGraph`, `define`, `runNamed`, `ingest`, `stepResult`. `rename` updates a workflow's name and optional description; `delete` removes the workflow, releases ephemeral Bots, and cleans logs/runs/routes and the task claims that named it — a claim is judged on its own record (`kind === 'workflow'` and that id), not by the route map, which its own cleanup has already emptied (#1632); a run Bot whose release failed is kept in the store's own `orphans` list — not in a variable — and the load-time `reap` retries it, once per id, capped, so deleting a workflow whose Bot will not go does not strand that Bot under a name nothing can reach (the #1616 contract). Factory template ids (e.g. `delivery`) are normal saved rows and may be deleted. A spawn hook makes one run Bot per node. Disposing the plugin ends every live run still marked `running` (nodes in flight are failed, the run is archived) and then releases every run Bot and drops those ids from the saved record only once the release has finished, handing the release back as a promise the host awaits, so a shutdown cannot persist a run as `running` with no ephemeral map and cannot leave hidden Bots that no state names, for every live run in a lane rather than only the one archived last (#1387, #1414, #1472). A released run Bot also gives its runner name back to the allocator instead of keeping it for the lifetime of the process. A refused start also says where it was: the deadline sentence names the node the host never answered, the Bots it did create and how long each took, and — when the lane's node count no longer fits the deadline at that host's speed — the seconds it needs, so a six-node line on a host that spends about eleven seconds per Bot reads `a lane of 6 nodes needs about 66s on this host at 11.0s per Bot, past the 60s deadline` instead of a bare timeout; a node Bot the host refuses carries the node and the host's own reason (an unavailable model, a taken Bot name, a full tree, or the host's error text) on the sentence and, for a machine reader, as `nodeId`/`reason` (#3282). `workflow/assign` reports a refused start the way the Bots API can: the endpoint throws, so `api/bot` answers `{ ok: false, error: { code: 'bot/refused', message, details: { workflowId, taskId, nodeId, nodeName, reason } } }` instead of `{ ok: true, value: { ok: false } }`, and the canvas reads that refusal through `settledCall`; `assign()` itself keeps its `{ ok: false }` answer for the in-process callers — the drain, `ingest`, `runNamed` and the tools (#3282). A dispatch that does not land is not marked handed: it is retried twice with a short backoff, and a step still unanswered after its workflow's `stepTimeoutMinutes` (45 minutes by default) fails the run and releases its Bots (#1530, #1580). A stored claim keeps a task only while the card names nothing new: when the card's own fields name another live workflow or Bot, the claim and its route are dropped and the card is matched again, and `rematch: true` on `ingest` drops that one claim first (#1647). `workflow/state` carries, per workflow, the ids it holds — `heldTasks`, a page of 20 sorted ids beside the whole `heldTaskCount` — and a Bot claim is not listed there (#1647); the read takes a `heldLimit` — 20 by default and 200 at most, read through `Number()` and kept only when it is a positive integer, so `'10'` reads 10 and `true` reads 1 while `0`, `-5`, `3.5`, `'many'` and `null` fall back to 20 — so a reader can ask for the rest of the page without turning the poll into a full read (#2256, #2260). Each public run also carries `finishedAt` (`null` while the run is live, else the moment it ended) and `currentStep` (`{ id, name }` of the node it walks now, else `null`) — pure additions for the Inbox, which reads them off `workflow/state` (#1706); a paused run keeps `status: 'running'` with `paused: true`, so it is not finished and still names its node. An assignment carries its instruction: `assign` reads `brief` — multi-line, up to 8000 characters, kept whole rather than flattened into the 300-character title — onto `task.brief`, and resolves it from the card first: a call that omits `brief` keeps the card's stored facts, a call that passes one stores it on a card that has none, and re-dispatching the same card to another workflow keeps the same facts, records `task.handoff` `{from, to, at, reason}` naming the lane it moved from and to, and hands the next node the stored facts (a brief input in `resolveInputs` falls back to the title only when the card has none; #2468). A call that passes `brief` is the latest word — `brief: ''` clears the card's facts instead of falling back to the cached ones — and deleting a lane drops the facts it cached, so a later dispatch records no handoff from a lane that no longer exists (#2475). It, a parked card keeps it in `workflow/state` and in `workflows.json`, and `stepBody` hands the node `task.brief`, falling back to the title only when there is none; before #1869 the node received the title twice and the brief never. `runNamed` gives a brief the same meaning and derives the title from its first line, and a field `assign` does not read comes back in `ignored` instead of disappearing (#1869). A call that omits `task` generates a random `task-` id (`idOf`), so two `run_workflow` calls in one millisecond start two runs instead of the second being answered `already: true` against the first one's id and losing its brief; an explicitly repeated `task` is still de-duplicated (#2662). A store whose `version` is higher than the `STORE_VERSION` (1) this build writes was written by a newer FlyBot, and is neither read as a store nor written over ([#2681](https://github.com/ThinkFlowLab/FlyBot/issues/2681)): `workflow/state` answers `ok: false` with `code: 'store-newer-version'`, the file and the version it carries, instead of the fixture list, and every write is refused before its first byte, so `workflows.json` stays byte-identical and no `.broken-` copy is made — that copy belongs to a store this build is about to replace ([#2603](https://github.com/ThinkFlowLab/FlyBot/issues/2603)). A store at this build's version, one below it, or one that does not parse keeps the #2603 answer (`store-unreadable`, the fixture for every other endpoint, the bytes kept aside on the first write). | +| `plugins/workflow/src/service.js` | The store, lanes (active runs, FIFO queue, terminal history), `maxConcurrent`, `rename`, `delete`, `saveGraph`, `define`, `runNamed`, `ingest`, `stepResult`. `rename` updates a workflow's name and optional description; `delete` removes the workflow, releases ephemeral Bots, and cleans logs/runs/routes and the task claims that named it — a claim is judged on its own record (`kind === 'workflow'` and that id), not by the route map, which its own cleanup has already emptied (#1632); a run Bot whose release failed is kept in the store's own `orphans` list — not in a variable — and the load-time `reap` retries it, once per id, capped, so deleting a workflow whose Bot will not go does not strand that Bot under a name nothing can reach (the #1616 contract). Factory template ids (e.g. `delivery`) are normal saved rows and may be deleted. A spawn hook makes one run Bot per node. Disposing the plugin ends every live run still marked `running` (nodes in flight are failed, the run is archived) and then releases every run Bot and drops those ids from the saved record only once the release has finished, handing the release back as a promise the host awaits, so a shutdown cannot persist a run as `running` with no ephemeral map and cannot leave hidden Bots that no state names, for every live run in a lane rather than only the one archived last (#1387, #1414, #1472). A released run Bot also gives its runner name back to the allocator instead of keeping it for the lifetime of the process. A refused start also says where it was: the deadline sentence names the node the host never answered, the Bots it did create and how long each took, and — when the lane's node count no longer fits the deadline at that host's speed — the seconds it needs, so a six-node line on a host that spends about forty seconds per Bot reads `a lane of 6 nodes needs about 80s on this host at 40.0s per Bot with 4 spawned at a time, past the 60s deadline` instead of a bare timeout; a node Bot the host refuses carries the node and the host's own reason (an unavailable model, a taken Bot name, a full tree, or the host's error text) on the sentence and, for a machine reader, as `nodeId`/`reason` (#3282). `workflow/assign` reports a refused start the way the Bots API can: the endpoint throws, so `api/bot` answers `{ ok: false, error: { code: 'bot/refused', message, details: { workflowId, taskId, nodeId, nodeName, reason } } }` instead of `{ ok: true, value: { ok: false } }`, and the canvas reads that refusal through `settledCall`; `assign()` itself keeps its `{ ok: false }` answer for the in-process callers — the drain, `ingest`, `runNamed` and the tools (#3282). A dispatch that does not land is not marked handed: it is retried twice with a short backoff, and a step still unanswered after its workflow's `stepTimeoutMinutes` (45 minutes by default) fails the run and releases its Bots (#1530, #1580). A stored claim keeps a task only while the card names nothing new: when the card's own fields name another live workflow or Bot, the claim and its route are dropped and the card is matched again, and `rematch: true` on `ingest` drops that one claim first (#1647). `workflow/state` carries, per workflow, the ids it holds — `heldTasks`, a page of 20 sorted ids beside the whole `heldTaskCount` — and a Bot claim is not listed there (#1647); the read takes a `heldLimit` — 20 by default and 200 at most, read through `Number()` and kept only when it is a positive integer, so `'10'` reads 10 and `true` reads 1 while `0`, `-5`, `3.5`, `'many'` and `null` fall back to 20 — so a reader can ask for the rest of the page without turning the poll into a full read (#2256, #2260). Each public run also carries `finishedAt` (`null` while the run is live, else the moment it ended) and `currentStep` (`{ id, name }` of the node it walks now, else `null`) — pure additions for the Inbox, which reads them off `workflow/state` (#1706); a paused run keeps `status: 'running'` with `paused: true`, so it is not finished and still names its node. An assignment carries its instruction: `assign` reads `brief` — multi-line, up to 8000 characters, kept whole rather than flattened into the 300-character title — onto `task.brief`, and resolves it from the card first: a call that omits `brief` keeps the card's stored facts, a call that passes one stores it on a card that has none, and re-dispatching the same card to another workflow keeps the same facts, records `task.handoff` `{from, to, at, reason}` naming the lane it moved from and to, and hands the next node the stored facts (a brief input in `resolveInputs` falls back to the title only when the card has none; #2468). A call that passes `brief` is the latest word — `brief: ''` clears the card's facts instead of falling back to the cached ones — and deleting a lane drops the facts it cached, so a later dispatch records no handoff from a lane that no longer exists (#2475). It, a parked card keeps it in `workflow/state` and in `workflows.json`, and `stepBody` hands the node `task.brief`, falling back to the title only when there is none; before #1869 the node received the title twice and the brief never. `runNamed` gives a brief the same meaning and derives the title from its first line, and a field `assign` does not read comes back in `ignored` instead of disappearing (#1869). A call that omits `task` generates a random `task-` id (`idOf`), so two `run_workflow` calls in one millisecond start two runs instead of the second being answered `already: true` against the first one's id and losing its brief; an explicitly repeated `task` is still de-duplicated (#2662). A store whose `version` is higher than the `STORE_VERSION` (1) this build writes was written by a newer FlyBot, and is neither read as a store nor written over ([#2681](https://github.com/ThinkFlowLab/FlyBot/issues/2681)): `workflow/state` answers `ok: false` with `code: 'store-newer-version'`, the file and the version it carries, instead of the fixture list, and every write is refused before its first byte, so `workflows.json` stays byte-identical and no `.broken-` copy is made — that copy belongs to a store this build is about to replace ([#2603](https://github.com/ThinkFlowLab/FlyBot/issues/2603)). A store at this build's version, one below it, or one that does not parse keeps the #2603 answer (`store-unreadable`, the fixture for every other endpoint, the bytes kept aside on the first write). | | `plugins/workflow/src/ephemeral.js` | Builds each node's run Bot from its prompt, role, model and tools, and releases the ids the host returns. `node.tools` names the connectors that node may call: `spawn` passes the cleaned list to `fb.bot.create`, the host records it as `stepTools`, and `connector-access.js` allows only the connectors it names (`github` covers `mcp__github__*`, `composio` covers `mcp__composio__*`) — a node that names none reaches none. File and shell rights do not come from that list: every node Bot gets the `danger-full-access` preset a normal sidebar Bot has, and `EPHEMERAL_DENIED_TOOLS` still refuses the team tools at the guard and approval layers. The list is also stored in `fields.tools` for the UI. | | `plugins/workflow/src/match.js` | Which open card matches a Workflow or a local Bot. | | `plugins/workflow/src/template.js` | Factory delivery/weekly graphs plus `EARLY_END_GUIDANCE` for Claim/Verify/Triage early-end markers; 发布交付流水线 (a parallel branch) and the weekly report, each node with the inputs and outputs that edge carries. | @@ -59,7 +59,7 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher, 1. Friends' Bots are not nodes. A `remote` Bot is refused by create, define and the matcher. 2. The canvas and the run share one graph: nodes, edges and positions. A drag commits `workflow/graph`, but only while **Edit** is on — a factory preset is copied first (#1378). A wire or a reconnect that would list a pair the graph already carries commits nothing new, because `normalizeGraph` refuses a graph that lists one pair twice and then saves none of it: unguarded, the plane would show no edge, no change and only a save error (#2163). Status highlighting reads that same graph. Replacing edges does not recompute who is already running. The Settings canvas paints that graph on the same 960px design plane, scaled to the width the settings page gives it, so the canvas is not a second scroll container inside the page's own scroll; its pointer deltas are divided by the scale, and its wires keep a stronger stroke than a plain divider so an edge can be followed at that size (#1483). While another dsh holds the Bot team the host is read-only (`roster.readOnly`): the page still reads through `workflow/state`, keeps its controls disabled, and offers no **Edit**, **Add link** or wire port, so it cannot take a change that core would refuse with `bot/read-only` and then drop (#2187). The same window draws the sidebar's workflow list without **New**, the row's **Rename**/**Delete** or **Hand a task to a Workflow**: the list is read-only too, and an `roster.readOnly` that is not a string, or holds only whitespace, counts as writable — trimmed the way `core/src/client/workflows.js:570` trims it (#2200). The Stage a row opens carries the same lock: it prints the host's sentence under `role="status"` and draws no **Hand a task to a Workflow**, no **Re-match**, no **Pause**/**Resume**/**Stop**, no **Edit**/drag palette and no **Reset**, while the row click, **Progress**, **Flow** and viewing a run still work; a writable roster draws every control exactly as before (#2213). Mounting the list in a read-only window still reads `projects` so the Project filter fills, but it sends no `workflow/ingest` for the open cards: ingest is a claim/match write the host refuses, so a reader opening the list writes nothing; a writable window still ingests every open card (#2222). The read-only sentence can also land after the list has mounted — a cold window starts on the empty roster, `ready: false` and no `readOnly` — so the ingest step asks the value the latest render saw rather than the one the mount saw, and it skips the board read and stops between two cards once the window has turned read-only. The Settings page draws no read-only sentence of its own: the host's Settings header already prints one for every page it shows (`core/src/client/settings.js` `bt-read-only`, `role="status"`, `hostText(roster.readOnly)`), and a second copy inside the page said the same thing twice (#2202). -3. **A live run advances only when its ephemeral Bot replies, following the graph's edges in order.** That reply advances the run and may appear on the workflow stage log; it is not delivered into a persistent sidebar Bot conversation (for example Chief). The `workflow-step` send middleware denies those replies after `stepResult` runs. Nodes start only when their forward predecessors finish and edges reach them. **Early termination (#1537, #1538, #1546, #1570, #1571):** When a node completes **or fails**, the run ends immediately with status `done`, sets `run.earlyEnd` to the condition name, and does not advance to successors, take a success-path `loop`, or burn `onFail` recovery attempts. The two markers that report the task itself end it only on a structured signal (#1546): `needs-human` and `already-claimed` must be a declared OUTPUT whose value **equals** the marker, or a whole reply line that is exactly `EARLY-END ` in any case. A step that quotes an issue title, negates the label or reports it absent keeps running. `tip-unchanged` (#1570) and `open-pr` (#1571) accept that same structured signal — in a declared OUTPUT the value counts as a whole word, so a brief that names `data-open-project` is not `open-pr` (#1866) — and also honor their prose fallbacks (`tip unchanged` / `unchanged tip`, `open PR already` / `already has an open PR` / `has an open PR` / `found an open PR`), so a Bot that forgets the structured marker cannot soft-lock Claim↔Verify on the same tip. See `checkEarlyEnd` in Key files. Factory prompt guidance lives in `EARLY_END_GUIDANCE` (`plugins/workflow/src/template.js`) for Settings graphs that wire Triage / Claim / Verify: Verify must escalate once on an unchanged tip, not re-run the full suite. There is no scripted or seeded run path: `workflow/state` exposes only `live === true` runs — the running ones, plus the last run when it is still `live` and ended `failed` — a failed sibling stays on the stage beside whatever is running now, with `run.failure` (`{ code, at, nodeName?, text?, attempts? }`) naming what ended it, so the room does not have to open Runs to learn that a run failed (#1493, #1494). The failed sibling is left out when a running run carries the same task id: that is a retry, and the live run keeps the row (#1507). History entries carry the run `id` and the same `failure`, so a Runs row shows when the run ended and why, and a row for a run already on the stage is not listed twice. A run does not create a schedule. `scheduleLabel` is display text. When the host can create Bots, starting a live run (`assign`, `run_workflow`, `ingest`) spawns one ephemeral Bot per node from that node's prompt, role, model and tools, sends it the step from the Main Bot, and removes every instance when the run finishes, fails for good, or is reset. A `failOnce` retry keeps them. They are hidden and left out of the sidebar. Loading the plugin does not spawn Bots for the default workflows. Plugin unload (`dispose`) and the load-time `reap` both end a live run still marked `running`: in-flight nodes are failed, the run is archived as `failed`, and any ephemeral Bot ids are released. A **read-only** host — a second window on the same `DSH_HOME`, the reader `home-lock` describes — does neither: it never ran those runs, so it cannot tell which of them a dead host left behind, and it must not end them, release their step Bots or write its older copy over the writer's store; both calls return without touching the record and the writer's own `reap` decides (#2673). **Taking the lock over is a state change (#2704):** the first call that finds the host writable after this window started read-only — a read, `reap`, `dispose` or any write — re-reads `workflows.json` once before it changes anything, because the copy it loaded predates what the dead window saved, and then runs the same load-time `reap` once on that adopted store. That reap ends the dead window's `running` run as `host-ended` (or heals a `live:true` run whose Bot map is already gone, #1414) and releases its step Bots; the reap itself leaves `lane.queue` alone, so a card that was merely `queued` keeps its place, and the drain that follows the reap starts it (#2787). Only the first such call re-reads — a writer that later loses the lock is still refused, exactly as before. A run already written as `running` with no ephemeral map (an older unload) is healed the same way on the next load (#1414). Loading is also the one moment a run the host left behind is noticed when its ephemeral map is still present: `reap` releases those instances and records the run as `failed`. Nothing resumes it, and that ending drains the lane's queue like any other terminal transition, so a card parked behind the dead run starts on load instead of waiting for the next assign (#2752) — a takeover reap starts the same reaper, so it drains with it too (#2787); that drain joins the serial tail the assigns use, because the load does not await the reap and a drain outside it would start a second card beside the assign at the lane's cap (#2757). A run that already ended keeps its own status (#1388). `workflow/reset` stopping one live run ends it the same way before its Bots are released: nodes still `running` become `failed`, so the strip cannot keep painting a run that is over (#1424). A graph edit that drops a node that is still running ends that run the way Stop does: the node is failed, the run is archived `failed`, its node Bots are released, and the queue drains (#1549); an id whose release failed stays named on that archived run, and the next load's `reap` retries it, instead of the edit forgetting it (the #1616 contract). Every wait is bounded (#1530). The deadline is not queued behind the wait it bounds (#2748): an expiry runs off the serial tail, so a host call that never answers — a dispatch, a release, a spawn — cannot park the one timer that ends the run, and the run still reaches `failed` with its Bots released. A start whose Bots the host does not create within `SPAWN_DEADLINE_MS` (60 s) is refused, and the card a drain had already taken out of `lane.queue` goes back to it instead of vanishing (a late spawn's Bots are released whether that answer is a success or a failure: a failed one carries the ids it already made on `error.spawned`, which the deadline branch now releases, #2785). A dispatch that does not land is not marked handed: the chat names the code and the attempt, and it is retried after a short backoff at most twice, never twice inside one hand-off, so a 429 is not hammered. When it still does not land, the step and the run fail with the reason written into the chat. Each running live step also holds a deadline — its workflow's `stepTimeoutMinutes`, `DEFAULT_STEP_TIMEOUT_MINUTES` (45 minutes) unless the record sets one, an integer from 1 to 120 given at create, through `define_workflow`, or through `workflow/step-timeout` (armed even when the host cannot spawn ephemeral Bots; #1530, #1580); when it expires the node and the run fail as `failed`, the run's ephemeral Bots are released, the chat and `run.failure` say the step's deadline was reached (`step deadline reached (45 min)`, the number its lane set), and `warn` names it once; the line never reads as a reply that did not arrive, because a step still working is killed by the same timer (#1868). The deadline is cleared as soon as the step moves, and by Pause, Reset, delete, `reap`, `dispose` and a Stop that released every Bot; a run the owner paused holds none, and Resume arms it again from that moment. A timer never outlives the run that started it. A reply that matches no running step — its run ended, or the node already moved — is reported through `warn` once per Bot and run instead of being dropped silently. A reply fails that node on a structured signal only: the host's failure notice carries `failure: true`, which the middleware hands `stepResult` as `ok: false` (a caller-set `ok: false` counts too), so an answer that quotes `could not answer your message` or `model call failed` while auditing it advances the step (#2677). An old host without the marker falls back to the notice shapes at the start of the text only (` could not answer your message: …`, ` was stopped before it could answer your message.`, and the bare `could not answer your message` / `The model call failed.` an old host wrote), never a free substring, so an ordinary answer that merely ends in the model-failure wording keeps advancing the step (#2741). A stopped turn is a failed step, with the stop notice as its reason, instead of finishing as done and handing that notice to the next node (#2741). A 429 or rate-limit mention in any other reply does not fail a step. Stop and Reset (`workflow/reset`) follow the same order as `dispose`: they release the run's Bots before the record forgets their ids, and an id whose release failed stays named on the still-live run, so the next Stop, Reset, dispose or load-time `reap` retries it (#1476). A run left that way is marked `paused` and `releasePending` and shows as Paused: it hands off no further node, `resume` is refused with an error, and Stop again is the retry. It also keeps the deadline of the node it was on — the one paused state that holds one — and a graph save re-syncs timers without taking it away (a save that cleared it left the run holding its slot with nothing left to free it); the deadline releases again there, ends the run as `paused-expired` and frees the slot once the Bot finally goes; while it still will not, that retry is bounded — `STOP_RELEASE_RETRIES` (two) more deadlines — and the last one ends the run `failed` as `release-failed` with the ids the host would not remove and the attempt count on the row, so the slot comes back and the queue drains (#2453, P-560, #2750). A whole-workflow Reset that could not release every Bot freezes every run it did not reset, because each of them has lost the ids whose release did go. A frozen run gets that deadline from the Reset itself: Reset clears every live run's timers before it releases, so `hold()` arms the deadline of each node those runs still have `running` when it writes the flags, and the retry that frees the slot no longer waits for the next graph save (P-716). A graph save still re-syncs every live run — `workflow/graph` (and `define_workflow`'s replace branch) — `paused` together with `releasePending` is the one paused state that keeps a deadline, and each frozen run is armed again for the node it was still on with its lane's `stepTimeoutMinutes` (45 minutes unless the lane set one, 1–120) (#2888). A save that changes that number arms the new one: the re-sync reads the value the same save writes, not the record's old one, and a deadline already armed at a different delay is re-armed (#2911, P-715) — saving five minutes used to leave the run on the 45-minute default, so nothing happened at five and the lane stayed held. When that deadline fires the run releases the Bots it still names: a clean release ends it `paused-expired`, a host that keeps refusing gets `STOP_RELEASE_RETRIES` (two) more deadlines, and the last one ends it `failed` as `release-failed` with the ids and the attempt count, so the lane slot comes back and the queue drains without the owner pressing Reset again. The markers go away when the run ends: Stop again, `dispose`, `reap`, or its own completion. +3. **A live run advances only when its ephemeral Bot replies, following the graph's edges in order.** That reply advances the run and may appear on the workflow stage log; it is not delivered into a persistent sidebar Bot conversation (for example Chief). The `workflow-step` send middleware denies those replies after `stepResult` runs. Nodes start only when their forward predecessors finish and edges reach them. **Early termination (#1537, #1538, #1546, #1570, #1571):** When a node completes **or fails**, the run ends immediately with status `done`, sets `run.earlyEnd` to the condition name, and does not advance to successors, take a success-path `loop`, or burn `onFail` recovery attempts. The two markers that report the task itself end it only on a structured signal (#1546): `needs-human` and `already-claimed` must be a declared OUTPUT whose value **equals** the marker, or a whole reply line that is exactly `EARLY-END ` in any case. A step that quotes an issue title, negates the label or reports it absent keeps running. `tip-unchanged` (#1570) and `open-pr` (#1571) accept that same structured signal — in a declared OUTPUT the value counts as a whole word, so a brief that names `data-open-project` is not `open-pr` (#1866) — and also honor their prose fallbacks (`tip unchanged` / `unchanged tip`, `open PR already` / `already has an open PR` / `has an open PR` / `found an open PR`), so a Bot that forgets the structured marker cannot soft-lock Claim↔Verify on the same tip. See `checkEarlyEnd` in Key files. Factory prompt guidance lives in `EARLY_END_GUIDANCE` (`plugins/workflow/src/template.js`) for Settings graphs that wire Triage / Claim / Verify: Verify must escalate once on an unchanged tip, not re-run the full suite. There is no scripted or seeded run path: `workflow/state` exposes only `live === true` runs — the running ones, plus the last run when it is still `live` and ended `failed` — a failed sibling stays on the stage beside whatever is running now, with `run.failure` (`{ code, at, nodeName?, text?, attempts? }`) naming what ended it, so the room does not have to open Runs to learn that a run failed (#1493, #1494). The failed sibling is left out when a running run carries the same task id: that is a retry, and the live run keeps the row (#1507). History entries carry the run `id` and the same `failure`, so a Runs row shows when the run ended and why, and a row for a run already on the stage is not listed twice. A run does not create a schedule. `scheduleLabel` is display text. When the host can create Bots, starting a live run (`assign`, `run_workflow`, `ingest`) spawns one ephemeral Bot per node from that node's prompt, role, model and tools, sends it the step from the Main Bot, and removes every instance when the run finishes, fails for good, or is reset. A `failOnce` retry keeps them. They are hidden and left out of the sidebar. Loading the plugin does not spawn Bots for the default workflows. Plugin unload (`dispose`) and the load-time `reap` both end a live run still marked `running`: in-flight nodes are failed, the run is archived as `failed`, and any ephemeral Bot ids are released. A **read-only** host — a second window on the same `DSH_HOME`, the reader `home-lock` describes — does neither: it never ran those runs, so it cannot tell which of them a dead host left behind, and it must not end them, release their step Bots or write its older copy over the writer's store; both calls return without touching the record and the writer's own `reap` decides (#2673). **Taking the lock over is a state change (#2704):** the first call that finds the host writable after this window started read-only — a read, `reap`, `dispose` or any write — re-reads `workflows.json` once before it changes anything, because the copy it loaded predates what the dead window saved, and then runs the same load-time `reap` once on that adopted store. That reap ends the dead window's `running` run as `host-ended` (or heals a `live:true` run whose Bot map is already gone, #1414) and releases its step Bots; the reap itself leaves `lane.queue` alone, so a card that was merely `queued` keeps its place, and the drain that follows the reap starts it (#2787). Only the first such call re-reads — a writer that later loses the lock is still refused, exactly as before. A run already written as `running` with no ephemeral map (an older unload) is healed the same way on the next load (#1414). Loading is also the one moment a run the host left behind is noticed when its ephemeral map is still present: `reap` releases those instances and records the run as `failed`. Nothing resumes it, and that ending drains the lane's queue like any other terminal transition, so a card parked behind the dead run starts on load instead of waiting for the next assign (#2752) — a takeover reap starts the same reaper, so it drains with it too (#2787); that drain joins the serial tail the assigns use, because the load does not await the reap and a drain outside it would start a second card beside the assign at the lane's cap (#2757). A run that already ended keeps its own status (#1388). `workflow/reset` stopping one live run ends it the same way before its Bots are released: nodes still `running` become `failed`, so the strip cannot keep painting a run that is over (#1424). A graph edit that drops a node that is still running ends that run the way Stop does: the node is failed, the run is archived `failed`, its node Bots are released, and the queue drains (#1549); an id whose release failed stays named on that archived run, and the next load's `reap` retries it, instead of the edit forgetting it (the #1616 contract). Every wait is bounded (#1530). The deadline is not queued behind the wait it bounds (#2748): an expiry runs off the serial tail, so a host call that never answers — a dispatch, a release, a spawn — cannot park the one timer that ends the run, and the run still reaches `failed` with its Bots released. A start whose Bots the host does not create within `SPAWN_DEADLINE_MS` (60 s) is refused, and the card a drain had already taken out of `lane.queue` goes back to it instead of vanishing (a late spawn's Bots are released whether that answer is a success or a failure: a failed one carries the ids it already made on `error.spawned`, which the deadline branch now releases, #2785). A start creates its node Bots four at a time (`SPAWN_CONCURRENCY`, four by default and overridable for a caller that needs another width): a six-node lane on the same host therefore needs two waves (~20s) instead of six sequential answers (~66s), the result still names the graph's nodes in their own order whatever order the host answers in, and a refused spawn starts no further Bot while the ones already in flight land, all of them released through `error.spawned` (#3286). A dispatch that does not land is not marked handed: the chat names the code and the attempt, and it is retried after a short backoff at most twice, never twice inside one hand-off, so a 429 is not hammered. When it still does not land, the step and the run fail with the reason written into the chat. Each running live step also holds a deadline — its workflow's `stepTimeoutMinutes`, `DEFAULT_STEP_TIMEOUT_MINUTES` (45 minutes) unless the record sets one, an integer from 1 to 120 given at create, through `define_workflow`, or through `workflow/step-timeout` (armed even when the host cannot spawn ephemeral Bots; #1530, #1580); when it expires the node and the run fail as `failed`, the run's ephemeral Bots are released, the chat and `run.failure` say the step's deadline was reached (`step deadline reached (45 min)`, the number its lane set), and `warn` names it once; the line never reads as a reply that did not arrive, because a step still working is killed by the same timer (#1868). The deadline is cleared as soon as the step moves, and by Pause, Reset, delete, `reap`, `dispose` and a Stop that released every Bot; a run the owner paused holds none, and Resume arms it again from that moment. A timer never outlives the run that started it. A reply that matches no running step — its run ended, or the node already moved — is reported through `warn` once per Bot and run instead of being dropped silently. A reply fails that node on a structured signal only: the host's failure notice carries `failure: true`, which the middleware hands `stepResult` as `ok: false` (a caller-set `ok: false` counts too), so an answer that quotes `could not answer your message` or `model call failed` while auditing it advances the step (#2677). An old host without the marker falls back to the notice shapes at the start of the text only (` could not answer your message: …`, ` was stopped before it could answer your message.`, and the bare `could not answer your message` / `The model call failed.` an old host wrote), never a free substring, so an ordinary answer that merely ends in the model-failure wording keeps advancing the step (#2741). A stopped turn is a failed step, with the stop notice as its reason, instead of finishing as done and handing that notice to the next node (#2741). A 429 or rate-limit mention in any other reply does not fail a step. Stop and Reset (`workflow/reset`) follow the same order as `dispose`: they release the run's Bots before the record forgets their ids, and an id whose release failed stays named on the still-live run, so the next Stop, Reset, dispose or load-time `reap` retries it (#1476). A run left that way is marked `paused` and `releasePending` and shows as Paused: it hands off no further node, `resume` is refused with an error, and Stop again is the retry. It also keeps the deadline of the node it was on — the one paused state that holds one — and a graph save re-syncs timers without taking it away (a save that cleared it left the run holding its slot with nothing left to free it); the deadline releases again there, ends the run as `paused-expired` and frees the slot once the Bot finally goes; while it still will not, that retry is bounded — `STOP_RELEASE_RETRIES` (two) more deadlines — and the last one ends the run `failed` as `release-failed` with the ids the host would not remove and the attempt count on the row, so the slot comes back and the queue drains (#2453, P-560, #2750). A whole-workflow Reset that could not release every Bot freezes every run it did not reset, because each of them has lost the ids whose release did go. A frozen run gets that deadline from the Reset itself: Reset clears every live run's timers before it releases, so `hold()` arms the deadline of each node those runs still have `running` when it writes the flags, and the retry that frees the slot no longer waits for the next graph save (P-716). A graph save still re-syncs every live run — `workflow/graph` (and `define_workflow`'s replace branch) — `paused` together with `releasePending` is the one paused state that keeps a deadline, and each frozen run is armed again for the node it was still on with its lane's `stepTimeoutMinutes` (45 minutes unless the lane set one, 1–120) (#2888). A save that changes that number arms the new one: the re-sync reads the value the same save writes, not the record's old one, and a deadline already armed at a different delay is re-armed (#2911, P-715) — saving five minutes used to leave the run on the 45-minute default, so nothing happened at five and the lane stayed held. When that deadline fires the run releases the Bots it still names: a clean release ends it `paused-expired`, a host that keeps refusing gets `STOP_RELEASE_RETRIES` (two) more deadlines, and the last one ends it `failed` as `release-failed` with the ids and the attempt count, so the lane slot comes back and the queue drains without the owner pressing Reset again. The markers go away when the run ends: Stop again, `dispose`, `reap`, or its own completion. Since #2742 every reply the host completes carries that field — an answer `failure: false`, a turn stopped before it answered `failure: true` with the stop notice as its reason — so the middleware classifies by the marker and the prose shapes are only the compatibility path for messages saved before it. 4. Each workflow keeps a FIFO queue and up to `maxConcurrent` live runs (default 2, an integer from 1 to 8 on the record, via `workflow/concurrency`, or in Settings → Workflow graphs). A start that is still waiting on the host counts against that cap: the card leaves the queue before its run exists, so two steps ending in the same turn would otherwise both read the free slot and start a card each, one past `maxConcurrent` (#2748). A task that does not fit is enqueued. When a run finishes, every workflow with a free slot is drained, oldest enqueue first. A new assign on a workflow that already has a queue waits behind it. A task does not replace a running run or release that run's Bots. Migrating an old store keeps only runs that were still `running` with ephemeral Bot ids; other legacy scripted rows are dropped from active state (finished tasks may remain in lane history). History records a finished or failed live run. Lowering the cap does not stop a run that has already started. Assigning `{ botId }` claims the card for that local Bot and does not enqueue. A remote Bot is refused. `activeCount` is `runningOf(lane)` — live runs with `live === true` and `status === 'running'`. The stage aggregate chip and the Runs tab (运行记录) must agree: every such run appears as its own running row and progress card with its own `status` (never a hardcoded Running), the chip count does not exceed that list, and a failed sibling appears with its time and reason. On the Runs list, the clock stays in a non-shrinking column with a gap before the status chip, and a Failed row paints the Failed chip once — a generic Failed with no host detail does not also draw `wf-hist-reason` (#1618). The line that names the other attempt (`superseded by ` / `retry of `) is drawn through `tr` like the rest of the tab, so a zh UI reads 已被 … 取代 / … 的重试 instead of English beside translated chips, and the raw id stays in the line's tooltip (#3040). 5. A node declares up to eight inputs and eight outputs. A required input is an output of the same name on an ancestor, or `task`, `title`, or `brief` on a root. A start with no provider is refused, and Bots spawned for that attempt are released. On a Bot-backed run a successor stays queued until its required inputs are present. If every forward predecessor is done, failed or skipped and an input is still missing, that node fails and the run stops. The dispatch text carries `Input : `. It also lists the OUTPUT lines the step owes — its declared ports plus every output its outgoing edges read, whether a condition's `when` or a loop's `until` (`edgeOutputs` in `src/graph.js`) — spelled `OUTPUT : ` when an edge compares that output against a value — so a node that declares no outputs is still told which line its own gate reads (#2669, #2685). The Bot reports `OUTPUT : `; a single declared output may be the reply body. A declared output a gate reads is required (§3): when a `when`/`else` or `until` edge reads it and the step's reply does not produce it, the step fails with `failure.text` `Missing OUTPUT ` instead of settling, so the gate never reads silence as "not this branch" and the run cannot close `done` on it; a delivered value that does not equal `eq` still completes the step and skips the untaken branch (#1773). One exception keeps the work a step already did (#2583, #2658): when `state` is the only gate-read output missing and the step's own reply shows its result — a pull request link, `pull/`, a line saying it opened PR #n, a `Claimed-by:` / `Claimed #n` line, or its own declared OUTPUT names the delivery (`pr`/`pull` whose value is only a pull number — a positive integer, optional leading `#`, optional trailing parenthetical such as `2657 (verified)` — or `merged`/`merge-commit` carrying a 40-character commit sha; any other output name counts for nothing) — the step stands as `done`, the run takes the node's first declared `state` branch (on the fast line `state: ready → claim`, so the pull request reaches gate/enqueue instead of the `else` route the step never chose), and the run record carries `outputMissing: true` and `producedPull: ` when a pull request was named; every other missing verdict still fails, and a step whose reply shows no such result still fails the same way. `stepArtifacts` reads that evidence and `firstBranchValue` names the branch, both in `src/graph.js`; the engine has no GitHub client, so the reply is the only evidence it can read. Nodes start in edge order. diff --git a/plugins/workflow/src/ephemeral.js b/plugins/workflow/src/ephemeral.js index c6a6b5737..c4124cc00 100644 --- a/plugins/workflow/src/ephemeral.js +++ b/plugins/workflow/src/ephemeral.js @@ -43,24 +43,58 @@ export function runCaps(callerBotId, authorBotId) { return [...new Set(ids)] } -export async function spawnAll(nodes, roster, spawn, workflowModel = '', caps = {}) { - const made = [] +// #3286: a host answers a spawn in about ten seconds, so a six-node lane created one +// Bot at a time needs ~66s and cannot fit the 60s batch deadline around a start. Four +// at a time finishes the same lane in two waves (~20s); a caller may pass a narrower +// or wider width as the last argument (tests use it to hold the waves still). +export const SPAWN_CONCURRENCY = 4 + +export async function spawnAll(nodes, roster, spawn, workflowModel = '', caps = {}, concurrency = SPAWN_CONCURRENCY) { + const list = Array.isArray(nodes) ? nodes : [] + const width = Number.isInteger(concurrency) && concurrency > 0 ? concurrency : SPAWN_CONCURRENCY const capping = runCaps(caps.callerBotId, caps.authorBotId) - try { - const ephemeral = {} - for (const node of nodes) { + // `made` is the order the host answered in: it is what `error.spawned` reports, so a + // failed start releases in the order the Bots came up, not the order they were asked for. + const made = [] + const byIndex = new Map() + let next = 0 + let failure = null + + const create = async () => { + for (;;) { + // Checked before taking the next index, so a refusal stops new spawns at once; the + // ones already in flight are still awaited below and still land in `made`. + if (failure !== null) return + const index = next + next += 1 + if (index >= list.length) return + const node = list[index] const spec = nodeConfig(node, roster, workflowModel) if (capping.length > 0) spec.cappedBy = capping - const id = await spawn(spec) - if (typeof id !== 'string' || id === '') throw new Error('The run could not start a Bot') + let id + try { + id = await spawn(spec) + if (typeof id !== 'string' || id === '') throw new Error('The run could not start a Bot') + } catch (error) { + if (failure === null) failure = error + return + } made.push(id) - ephemeral[node.id] = id + byIndex.set(index, id) } - return ephemeral - } catch (error) { - error.spawned = made - throw error } + + const workers = [] + for (let i = 0; i < Math.min(width, list.length); i += 1) workers.push(create()) + await Promise.all(workers) + if (failure !== null) { + failure.spawned = made.slice() + throw failure + } + // Keys follow the graph's nodes, whatever order the host answered in. + const ephemeral = {} + for (let i = 0; i < list.length; i += 1) ephemeral[list[i].id] = byIndex.get(i) + return ephemeral } // Returns the ids whose release failed: that Bot is still there, so the caller diff --git a/plugins/workflow/src/service.js b/plugins/workflow/src/service.js index f41e04e04..f1aae8279 100644 --- a/plugins/workflow/src/service.js +++ b/plugins/workflow/src/service.js @@ -5,7 +5,7 @@ import { randomBytes } from 'node:crypto' import { copyFileSync, lstatSync, mkdirSync, readFileSync, renameSync, writeFileSync } from 'node:fs' import { join } from 'node:path' -import { releaseAll, spawnAll } from './ephemeral.js' +import { SPAWN_CONCURRENCY, releaseAll, spawnAll } from './ephemeral.js' import { chainLabel, cleanLine, cleanPorts, cleanPrompt, cleanTools, completeNode, cutCodePoints, doneCount, edgeOutputs, failNode, freshStatus, layerMarks, firstBranchValue, missingProviders, missingVerdicts, nodeById, normalizeGraph, orderedIds, readStepResult, resolveInputs, retryNode, @@ -503,12 +503,13 @@ export function createWorkflowService({ const waited = current.startedAt > 0 ? now() - current.startedAt : 0 rest.push(`node ${current.label} still had no Bot${waited > 0 ? ` after ${formatMs(waited)}` : ''}`) } - // The average turns "slow" into a number the owner can act on: six nodes at eleven seconds - // each cannot fit a sixty-second deadline, whatever the settings say. + // The average turns "slow" into a number the owner can act on: the lane needs as many + // waves of SPAWN_CONCURRENCY as its node count requires (#3286). const average = created.length > 0 ? created.reduce((sum, one) => sum + (one.ms > 0 ? one.ms : 0), 0) / created.length : 0 if (average > 0 && total > 0) { - const needed = Math.round((average * total) / 1000) - if (needed > seconds) rest.push(`a lane of ${total} nodes needs about ${needed}s on this host at ${formatMs(average)} per Bot, past the ${seconds}s deadline`) + const waves = Math.ceil(total / SPAWN_CONCURRENCY) + const needed = Math.round((average * waves) / 1000) + if (needed > seconds) rest.push(`a lane of ${total} nodes needs about ${needed}s on this host at ${formatMs(average)} per Bot with ${SPAWN_CONCURRENCY} spawned at a time, past the ${seconds}s deadline`) } return rest.length === 0 ? `the host did not create the run Bots within ${seconds}s` @@ -1411,23 +1412,28 @@ export function createWorkflowService({ // this run may use; the host intersects them with each node's `tools`. // #3282: the loop reports the node it is on and the Bots already made, so the deadline // below can name the node that never answered instead of one bare sentence. - const progress = { total: item.nodes.length, created: [], current: null } + // #3286: with several spawns in flight, "the node the host is on" is a set, not one + // slot. The message names the node started last that the host has not answered yet. + const progress = { total: item.nodes.length, created: [], current: null, pending: new Map() } const spawnNode = async (spec) => { const startedAt = now() const nodeId = String(spec?.nodeId ?? '') const nodeName = String(spec?.name ?? '') - progress.current = { nodeId, nodeName, label: nodeLabel({ nodeId, nodeName }), startedAt } + const label = nodeLabel({ nodeId, nodeName }) + progress.pending.set(nodeId, { nodeId, nodeName, label, startedAt }) + progress.current = progress.pending.get(nodeId) try { const id = await spawn(spec) // A host that answers an empty id is a refusal too, and it has to name the node the // way a thrown one does: spawnAll() would otherwise refuse it without a node context. if (typeof id !== 'string' || id === '') throw new Error('the host returned no Bot id') - progress.created.push({ nodeId, nodeName, label: progress.current.label, ms: Math.max(0, now() - startedAt) }) - progress.current = null + progress.created.push({ nodeId, nodeName, label, ms: Math.max(0, now() - startedAt) }) return id } catch (error) { - progress.current = null throw spawnFailure(error, spec, item.id) + } finally { + progress.pending.delete(nodeId) + progress.current = [...progress.pending.values()].at(-1) ?? null } } ephemeral = await boundedSpawn(spawnAll(item.nodes, bots() ?? [], spawnNode, item.model ?? '', { diff --git a/plugins/workflow/test/parallel-spawn.test.js b/plugins/workflow/test/parallel-spawn.test.js new file mode 100644 index 000000000..512e6f925 --- /dev/null +++ b/plugins/workflow/test/parallel-spawn.test.js @@ -0,0 +1,125 @@ +// #3286: a run creates its node Bots with bounded concurrency. A host answers a +// spawn in about ten seconds, so a six-node lane built one Bot at a time needs +// ~66s and cannot fit the 60s batch deadline the service puts around a start. +// Four at a time finishes the same lane in two waves (~20s). +// +// The host is a fake clock: a spawn resolves only when the test moves time to its +// due moment, so the timing assertions are about how many waves `spawnAll` needs, +// not about how fast this machine is. +import assert from 'node:assert/strict' +import { test } from 'node:test' +import * as ephemeralModule from '../src/ephemeral.js' + +const NODES = Array.from({ length: 6 }, (_, index) => ({ + id: `node-${index + 1}`, + name: `Node ${index + 1}`, + role: 'dev', + prompt: `Do step ${index + 1}.`, +})) +const DELAY = 10_000 + +const tick = () => new Promise(done => setImmediate(done)) + +function fakeHost(delayFor = () => DELAY) { + let now = 0 + let active = 0 + let maxActive = 0 + const calls = [] + const queue = [] + const spawn = (spec) => { + active += 1 + maxActive = Math.max(maxActive, active) + const entry = { spec, id: `eph-${spec.nodeId}`, due: now + delayFor(spec) } + calls.push(entry.spec.nodeId) + return new Promise((resolve) => { + entry.finish = () => { + active -= 1 + resolve(entry.id) + } + queue.push(entry) + }) + } + async function advance() { + const due = Math.min(...queue.map(entry => entry.due)) + now = due + const ready = queue.filter(entry => entry.due <= due) + for (const entry of ready) queue.splice(queue.indexOf(entry), 1) + for (const entry of ready) entry.finish() + await tick() + } + return { + spawn, + calls, + advance, + get now() { return now }, + get maxActive() { return maxActive }, + get pending() { return queue.length }, + } +} + +// Drives the fake clock until `spawnAll` settles, failing loudly if it waits on +// a spawn that the test never released. +async function run(batch, host) { + let done = false + batch.then(() => { done = true }, () => { done = true }) + while (!done) { + await tick() + if (done) break + if (host.pending === 0) throw new Error('spawnAll waits with no spawn in flight') + await host.advance() + } + return batch +} + +test('#3286: spawns run four at a time and a six-node lane fits the 60s batch deadline', async () => { + const host = fakeHost() + const batch = ephemeralModule.spawnAll(NODES, [], host.spawn) + const ephemeral = await run(batch, host) + + assert.equal(host.maxActive, 4, `never more than four spawns in flight: ${host.maxActive}`) + assert.equal(Object.keys(ephemeral).length, 6, 'every node got a Bot') + // Six nodes at 10s each: serial would be 60s (the old failure) and four at a + // time is 20s. The bound keeps a slow machine from passing by luck. + assert.ok(host.now < 60_000, `the whole batch took ${host.now}ms of host time, past the 60s deadline`) + assert.ok(host.now <= 3 * DELAY, `two waves at ${DELAY}ms were expected, the batch took ${host.now}ms`) + assert.equal(ephemeralModule.SPAWN_CONCURRENCY, 4, 'the default width is four') +}) + +test('#3286: a failed spawn stops new starts, lets the ones in flight land, and reports what was made', async () => { + const calls = [] + const pending = [] + const spawn = (spec) => { + calls.push(spec.nodeId) + return new Promise((resolve, reject) => pending.push({ resolve, reject })) + } + const batch = ephemeralModule.spawnAll(NODES, [], spawn, '', {}, 2) + const failed = batch.then(() => null, error => error) + await tick() + assert.deepEqual(calls, ['node-1', 'node-2'], 'two spawns are in flight') + + pending[0].resolve('eph-node-1') + pending[1].resolve('eph-node-2') + await tick() + assert.deepEqual(calls, ['node-1', 'node-2', 'node-3', 'node-4'], 'both workers took the next nodes') + + // The third node the host answers is a refusal. The fourth is still in flight: + // it must land, and no fifth or sixth spawn may start. + pending[2].reject(new Error('the host could not create the coder')) + await tick() + assert.deepEqual(calls, ['node-1', 'node-2', 'node-3', 'node-4'], 'no new spawn starts after the refusal') + pending[3].resolve('eph-node-4') + + const error = await failed + assert.equal(error?.message, 'the host could not create the coder', 'the host reason is carried') + assert.deepEqual(error.spawned, ['eph-node-1', 'eph-node-2', 'eph-node-4'], 'exactly the Bots that were made, in the order they landed') + assert.deepEqual(calls, ['node-1', 'node-2', 'node-3', 'node-4'], 'the other two nodes were never asked for') +}) + +test('#3286: the result keeps the graph order even when the Bots land out of order', async () => { + const delays = [50_000, 10_000, 20_000, 30_000, 40_000, 5_000] + const host = fakeHost(spec => delays[Number(spec.nodeId.slice('node-'.length)) - 1]) + const ephemeral = await run(ephemeralModule.spawnAll(NODES, [], host.spawn), host) + + assert.deepEqual(Object.keys(ephemeral), NODES.map(node => node.id), 'the keys follow the nodes, not the answers') + for (const node of NODES) assert.equal(ephemeral[node.id], `eph-${node.id}`) +})