From c40d4b829e8fc2f08ba9dc715bebf41e6215561e Mon Sep 17 00:00:00 2001 From: Danyang Chen Date: Sat, 21 Nov 2026 16:14:22 +0800 Subject: [PATCH 1/2] Name the node and the reason when a workflow start's run Bots are not created A start whose node Bots the host does not create in time answered only "the host did not create the run Bots within 60s", so a lane that needs more time than the deadline looked like a silent hang. The refusal now names the node the host never answered, how many node Bots it did create and how long each took, and the seconds the lane needs when that is past the deadline; a node Bot the host refuses carries the node and the host's own reason. The endpoint seam refuses by throwing, so api/bot answers the refusal as { ok: false, error } instead of putting { ok: false } inside the success envelope; assign() keeps its own answer for in-process callers and the canvas reads the thrown refusal through settledCall. Fixes #3282 Written-by: Dev --- changelog.d/3282.fixed.md | 1 + docs/subsystems/workflow.md | 2 +- plugins/workflow/client.js | 7 +- plugins/workflow/index.js | 5 +- plugins/workflow/src/service.js | 158 ++++++++++++++++- .../test/spawn-failure-message.test.js | 163 ++++++++++++++++++ 6 files changed, 327 insertions(+), 9 deletions(-) create mode 100644 changelog.d/3282.fixed.md create mode 100644 plugins/workflow/test/spawn-failure-message.test.js diff --git a/changelog.d/3282.fixed.md b/changelog.d/3282.fixed.md new file mode 100644 index 000000000..38050c2b3 --- /dev/null +++ b/changelog.d/3282.fixed.md @@ -0,0 +1 @@ +A workflow start the host could not finish saying had one sentence — "the host did not create the run Bots within 60s" — with no node and no reason, so a lane whose node Bots were simply slower than the deadline looked like a silent hang and the same task was dispatched three times; the refusal now names the node the host never answered, the Bots it did create with how long each took, and, when the lane's node count no longer fits the deadline at that host's speed, the seconds it needs (a six-node line at about eleven seconds per Bot reads "needs about 66s … past the 60s deadline"), while a node Bot the host refuses carries the node and the host's own reason — an unavailable model, a taken Bot name, a full tree, or the host's error text — and `workflow/assign` reports a refused start as a failed answer (`ok: false` with `error.code` `bot/refused` and `error.details` naming the node) instead of a value inside the success envelope, so a caller cannot mistake a failed dispatch for a started run ([#3282](https://github.com/ThinkFlowLab/FlyBot/issues/3282)). diff --git a/docs/subsystems/workflow.md b/docs/subsystems/workflow.md index 77660e9b6..6110f377d 100644 --- a/docs/subsystems/workflow.md +++ b/docs/subsystems/workflow.md @@ -40,7 +40,7 @@ Everything in `plugins/workflow/`: the graph rules, the run store, the matcher, | --- | --- | | `plugins/workflow/index.js` | Plugin entry (`flybot-workflow`). Endpoints, `run_workflow`, `define_workflow`, the board listener, and the reply middleware that ingests a step result and denies delivery to persistent Bot chats. | | `plugins/workflow/src/graph.js` | Nodes, edges, positions, and named inputs and outputs. A node starts when its forward predecessors are settled and a taken edge reaches it. Same-rank nodes run together. A forward cycle is refused. A loop edge is the only cycle for success paths. A `when` edge chooses a branch; the other is skipped. A required input with no upstream provider is refused before a run starts. An `onFail` edge provides failure recovery: when a node fails with recoveries remaining, the workflow resets the body from the upstream target through the failing node and re-runs it in order (tracked by `failAttempts` per edge, with `max` counting recoveries, 1–8, default 3). After the recoveries are used, the node fails permanently. | -| `plugins/workflow/src/service.js` | The store, lanes (active runs, FIFO queue, terminal history), `maxConcurrent`, `rename`, `delete`, `saveGraph`, `define`, `runNamed`, `ingest`, `stepResult`. `rename` updates a workflow's name and optional description; `delete` removes the workflow, releases ephemeral Bots, and cleans logs/runs/routes and the task claims that named it — a claim is judged on its own record (`kind === 'workflow'` and that id), not by the route map, which its own cleanup has already emptied (#1632); a run Bot whose release failed is kept in the store's own `orphans` list — not in a variable — and the load-time `reap` retries it, once per id, capped, so deleting a workflow whose Bot will not go does not strand that Bot under a name nothing can reach (the #1616 contract). Factory template ids (e.g. `delivery`) are normal saved rows and may be deleted. A spawn hook makes one run Bot per node. Disposing the plugin ends every live run still marked `running` (nodes in flight are failed, the run is archived) and then releases every run Bot and drops those ids from the saved record only once the release has finished, handing the release back as a promise the host awaits, so a shutdown cannot persist a run as `running` with no ephemeral map and cannot leave hidden Bots that no state names, for every live run in a lane rather than only the one archived last (#1387, #1414, #1472). A released run Bot also gives its runner name back to the allocator instead of keeping it for the lifetime of the process. A dispatch that does not land is not marked handed: it is retried twice with a short backoff, and a step still unanswered after its workflow's `stepTimeoutMinutes` (45 minutes by default) fails the run and releases its Bots (#1530, #1580). A stored claim keeps a task only while the card names nothing new: when the card's own fields name another live workflow or Bot, the claim and its route are dropped and the card is matched again, and `rematch: true` on `ingest` drops that one claim first (#1647). `workflow/state` carries, per workflow, the ids it holds — `heldTasks`, a page of 20 sorted ids beside the whole `heldTaskCount` — and a Bot claim is not listed there (#1647); the read takes a `heldLimit` — 20 by default and 200 at most, read through `Number()` and kept only when it is a positive integer, so `'10'` reads 10 and `true` reads 1 while `0`, `-5`, `3.5`, `'many'` and `null` fall back to 20 — so a reader can ask for the rest of the page without turning the poll into a full read (#2256, #2260). Each public run also carries `finishedAt` (`null` while the run is live, else the moment it ended) and `currentStep` (`{ id, name }` of the node it walks now, else `null`) — pure additions for the Inbox, which reads them off `workflow/state` (#1706); a paused run keeps `status: 'running'` with `paused: true`, so it is not finished and still names its node. An assignment carries its instruction: `assign` reads `brief` — multi-line, up to 8000 characters, kept whole rather than flattened into the 300-character title — onto `task.brief`, and resolves it from the card first: a call that omits `brief` keeps the card's stored facts, a call that passes one stores it on a card that has none, and re-dispatching the same card to another workflow keeps the same facts, records `task.handoff` `{from, to, at, reason}` naming the lane it moved from and to, and hands the next node the stored facts (a brief input in `resolveInputs` falls back to the title only when the card has none; #2468). A call that passes `brief` is the latest word — `brief: ''` clears the card's facts instead of falling back to the cached ones — and deleting a lane drops the facts it cached, so a later dispatch records no handoff from a lane that no longer exists (#2475). It, a parked card keeps it in `workflow/state` and in `workflows.json`, and `stepBody` hands the node `task.brief`, falling back to the title only when there is none; before #1869 the node received the title twice and the brief never. `runNamed` gives a brief the same meaning and derives the title from its first line, and a field `assign` does not read comes back in `ignored` instead of disappearing (#1869). A call that omits `task` generates a random `task-` id (`idOf`), so two `run_workflow` calls in one millisecond start two runs instead of the second being answered `already: true` against the first one's id and losing its brief; an explicitly repeated `task` is still de-duplicated (#2662). A store whose `version` is higher than the `STORE_VERSION` (1) this build writes was written by a newer FlyBot, and is neither read as a store nor written over ([#2681](https://github.com/ThinkFlowLab/FlyBot/issues/2681)): `workflow/state` answers `ok: false` with `code: 'store-newer-version'`, the file and the version it carries, instead of the fixture list, and every write is refused before its first byte, so `workflows.json` stays byte-identical and no `.broken-` copy is made — that copy belongs to a store this build is about to replace ([#2603](https://github.com/ThinkFlowLab/FlyBot/issues/2603)). A store at this build's version, one below it, or one that does not parse keeps the #2603 answer (`store-unreadable`, the fixture for every other endpoint, the bytes kept aside on the first write). | +| `plugins/workflow/src/service.js` | The store, lanes (active runs, FIFO queue, terminal history), `maxConcurrent`, `rename`, `delete`, `saveGraph`, `define`, `runNamed`, `ingest`, `stepResult`. `rename` updates a workflow's name and optional description; `delete` removes the workflow, releases ephemeral Bots, and cleans logs/runs/routes and the task claims that named it — a claim is judged on its own record (`kind === 'workflow'` and that id), not by the route map, which its own cleanup has already emptied (#1632); a run Bot whose release failed is kept in the store's own `orphans` list — not in a variable — and the load-time `reap` retries it, once per id, capped, so deleting a workflow whose Bot will not go does not strand that Bot under a name nothing can reach (the #1616 contract). Factory template ids (e.g. `delivery`) are normal saved rows and may be deleted. A spawn hook makes one run Bot per node. Disposing the plugin ends every live run still marked `running` (nodes in flight are failed, the run is archived) and then releases every run Bot and drops those ids from the saved record only once the release has finished, handing the release back as a promise the host awaits, so a shutdown cannot persist a run as `running` with no ephemeral map and cannot leave hidden Bots that no state names, for every live run in a lane rather than only the one archived last (#1387, #1414, #1472). A released run Bot also gives its runner name back to the allocator instead of keeping it for the lifetime of the process. A refused start also says where it was: the deadline sentence names the node the host never answered, the Bots it did create and how long each took, and — when the lane's node count no longer fits the deadline at that host's speed — the seconds it needs, so a six-node line on a host that spends about eleven seconds per Bot reads `a lane of 6 nodes needs about 66s on this host at 11.0s per Bot, past the 60s deadline` instead of a bare timeout; a node Bot the host refuses carries the node and the host's own reason (an unavailable model, a taken Bot name, a full tree, or the host's error text) on the sentence and, for a machine reader, as `nodeId`/`reason` (#3282). `workflow/assign` reports a refused start the way the Bots API can: the endpoint throws, so `api/bot` answers `{ ok: false, error: { code: 'bot/refused', message, details: { workflowId, taskId, nodeId, nodeName, reason } } }` instead of `{ ok: true, value: { ok: false } }`, and the canvas reads that refusal through `settledCall`; `assign()` itself keeps its `{ ok: false }` answer for the in-process callers — the drain, `ingest`, `runNamed` and the tools (#3282). A dispatch that does not land is not marked handed: it is retried twice with a short backoff, and a step still unanswered after its workflow's `stepTimeoutMinutes` (45 minutes by default) fails the run and releases its Bots (#1530, #1580). A stored claim keeps a task only while the card names nothing new: when the card's own fields name another live workflow or Bot, the claim and its route are dropped and the card is matched again, and `rematch: true` on `ingest` drops that one claim first (#1647). `workflow/state` carries, per workflow, the ids it holds — `heldTasks`, a page of 20 sorted ids beside the whole `heldTaskCount` — and a Bot claim is not listed there (#1647); the read takes a `heldLimit` — 20 by default and 200 at most, read through `Number()` and kept only when it is a positive integer, so `'10'` reads 10 and `true` reads 1 while `0`, `-5`, `3.5`, `'many'` and `null` fall back to 20 — so a reader can ask for the rest of the page without turning the poll into a full read (#2256, #2260). Each public run also carries `finishedAt` (`null` while the run is live, else the moment it ended) and `currentStep` (`{ id, name }` of the node it walks now, else `null`) — pure additions for the Inbox, which reads them off `workflow/state` (#1706); a paused run keeps `status: 'running'` with `paused: true`, so it is not finished and still names its node. An assignment carries its instruction: `assign` reads `brief` — multi-line, up to 8000 characters, kept whole rather than flattened into the 300-character title — onto `task.brief`, and resolves it from the card first: a call that omits `brief` keeps the card's stored facts, a call that passes one stores it on a card that has none, and re-dispatching the same card to another workflow keeps the same facts, records `task.handoff` `{from, to, at, reason}` naming the lane it moved from and to, and hands the next node the stored facts (a brief input in `resolveInputs` falls back to the title only when the card has none; #2468). A call that passes `brief` is the latest word — `brief: ''` clears the card's facts instead of falling back to the cached ones — and deleting a lane drops the facts it cached, so a later dispatch records no handoff from a lane that no longer exists (#2475). It, a parked card keeps it in `workflow/state` and in `workflows.json`, and `stepBody` hands the node `task.brief`, falling back to the title only when there is none; before #1869 the node received the title twice and the brief never. `runNamed` gives a brief the same meaning and derives the title from its first line, and a field `assign` does not read comes back in `ignored` instead of disappearing (#1869). A call that omits `task` generates a random `task-` id (`idOf`), so two `run_workflow` calls in one millisecond start two runs instead of the second being answered `already: true` against the first one's id and losing its brief; an explicitly repeated `task` is still de-duplicated (#2662). A store whose `version` is higher than the `STORE_VERSION` (1) this build writes was written by a newer FlyBot, and is neither read as a store nor written over ([#2681](https://github.com/ThinkFlowLab/FlyBot/issues/2681)): `workflow/state` answers `ok: false` with `code: 'store-newer-version'`, the file and the version it carries, instead of the fixture list, and every write is refused before its first byte, so `workflows.json` stays byte-identical and no `.broken-` copy is made — that copy belongs to a store this build is about to replace ([#2603](https://github.com/ThinkFlowLab/FlyBot/issues/2603)). A store at this build's version, one below it, or one that does not parse keeps the #2603 answer (`store-unreadable`, the fixture for every other endpoint, the bytes kept aside on the first write). | | `plugins/workflow/src/ephemeral.js` | Builds each node's run Bot from its prompt, role, model and tools, and releases the ids the host returns. `node.tools` names the connectors that node may call: `spawn` passes the cleaned list to `fb.bot.create`, the host records it as `stepTools`, and `connector-access.js` allows only the connectors it names (`github` covers `mcp__github__*`, `composio` covers `mcp__composio__*`) — a node that names none reaches none. File and shell rights do not come from that list: every node Bot gets the `danger-full-access` preset a normal sidebar Bot has, and `EPHEMERAL_DENIED_TOOLS` still refuses the team tools at the guard and approval layers. The list is also stored in `fields.tools` for the UI. | | `plugins/workflow/src/match.js` | Which open card matches a Workflow or a local Bot. | | `plugins/workflow/src/template.js` | Factory delivery/weekly graphs plus `EARLY_END_GUIDANCE` for Claim/Verify/Triage early-end markers; 发布交付流水线 (a parallel branch) and the weekly report, each node with the inputs and outputs that edge carries. | diff --git a/plugins/workflow/client.js b/plugins/workflow/client.js index 62194f017..04e8c5a76 100644 --- a/plugins/workflow/client.js +++ b/plugins/workflow/client.js @@ -2284,7 +2284,10 @@ window.__ModuleLoader__.load({ setError('') if (String(target).startsWith('bot:')) { const botId = String(target).slice(4) - const value = await call('workflow/assign', { botId, taskId: task.id, title: task.title }) + // #3282: the host reports a refused start as a failed answer (`ok: false`), which + // `call()` throws, so this dialog reads it through `settledCall` — the same + // `{ ok: false, error }` it already knows how to show. + const value = await stageCall('workflow/assign', { botId, taskId: task.id, title: task.title }, 'Could not assign this task') if (value?.ok === false) { setError(value.error ?? ''); return } const host = globalThis.__flybot__?.host if (typeof host === 'function') { @@ -2299,7 +2302,7 @@ window.__ModuleLoader__.load({ // #2495: hand the card's own facts over, and say so explicitly even when it holds none — // an omitted `brief` keeps whatever an earlier call cached under this task id. const brief = typeof task.brief === 'string' ? task.brief : '' - const value = await call('workflow/assign', { workflowId: target, taskId: task.id, title: task.title, brief }) + const value = await stageCall('workflow/assign', { workflowId: target, taskId: task.id, title: task.title, brief }, 'Could not assign this task') if (value?.ok === false) { setError(value.error ?? ''); return } setPack(value) setOpenId(target) diff --git a/plugins/workflow/index.js b/plugins/workflow/index.js index cd2ee2a20..1dcc311c0 100644 --- a/plugins/workflow/index.js +++ b/plugins/workflow/index.js @@ -143,7 +143,10 @@ export function apply(ctx, config = {}) { fb.api.handle('workflow/create', body => service.create(body)), fb.api.handle('workflow/rename', body => service.rename(body)), fb.api.handle('workflow/delete', body => service.delete(body)), - fb.api.handle('workflow/assign', body => service.assign(body)), + // #3282: a refused start throws, so the Bots API reports it as `{ ok: false, error }` instead + // of putting `{ ok: false }` inside the success envelope a caller reads first (service.js + // `assignEndpoint`). Every other endpoint keeps answering its value. + fb.api.handle('workflow/assign', body => service.assignEndpoint(body)), fb.api.handle('workflow/concurrency', body => service.setConcurrency(body)), fb.api.handle('workflow/step-timeout', body => service.setStepTimeout(body)), fb.api.handle('workflow/set-model', body => service.setModel(body)), diff --git a/plugins/workflow/src/service.js b/plugins/workflow/src/service.js index 72043e24d..f41e04e04 100644 --- a/plugins/workflow/src/service.js +++ b/plugins/workflow/src/service.js @@ -66,6 +66,56 @@ export const STOP_RELEASE_RETRIES = 2 // deadline for an answer that may never come. export const SPAWN_DEADLINE_MS = 60 * 1000 +// #3282: which Bot a spawn was for, in the words a person uses for the node ("Claim" (claim)). +// A missing name or id still yields a label, so a message never reads "node undefined". +function nodeLabel(spec) { + const name = String(spec?.nodeName ?? spec?.name ?? '').trim() + const id = String(spec?.nodeId ?? '').trim() + if (name && id) return `"${name}" (${id})` + if (name) return `"${name}"` + if (id) return `"${id}"` + return 'a node' +} + +function formatMs(ms) { + return `${(Math.max(0, Number(ms) || 0) / 1000).toFixed(1)}s` +} + +// The host's own sentence is the reason a node Bot was refused; these are the shapes worth +// naming on top of it, so a reader does not have to recognise the host's wording (#3282). +const SPAWN_REASONS = [ + [/unknown model|no such model|model .{0,40}(not found|not available|missing|unknown)|cannot (find|resolve) (the )?model/i, 'model-missing'], + [/already exists|name .{0,20}not available|drawn clash|invisible characters/i, 'name-taken'], + [/max ?children|no (free )?(slot|room|space)|at capacity|too many (bot|child)|slot limit|children limit/i, 'no-slot'], + [/did not answer|timed? ?out|timeout/i, 'no-answer'], +] + +function spawnReasonOf(text) { + const said = String(text ?? '') + for (const [pattern, reason] of SPAWN_REASONS) if (pattern.test(said)) return reason + return 'refused' +} + +// A node Bot the host refused keeps the node and the reason beside the host's text, so the +// refusal names the Bot to look at instead of arriving as one unlabelled sentence (#3282). +function spawnFailure(error, spec, workflowId) { + const text = String(error?.message ?? error ?? '').trim() || 'the host refused to create this Bot' + const failure = error instanceof Error ? error : new Error(text) + const nodeName = String(spec?.name ?? '').trim() + const nodeId = String(spec?.nodeId ?? '').trim() + const reason = spawnReasonOf(text) + const named = nodeLabel({ nodeName, nodeId }) + failure.message = reason === 'refused' + ? `node ${named} could not create its Bot: ${text}` + : `node ${named} could not create its Bot (${reason}): ${text}` + failure.code = 'spawn-failed' + failure.nodeId = nodeId + failure.nodeName = nodeName + failure.reason = reason + failure.details = { reason, workflowId, nodeId, nodeName } + return failure +} + function stepTimeoutValue(raw) { const value = Number(raw) if (!Number.isInteger(value) || value < MIN_STEP_TIMEOUT_MINUTES || value > MAX_STEP_TIMEOUT_MINUTES) { @@ -435,16 +485,53 @@ export function createWorkflowService({ write() } + // #3282: a bare deadline said only that the host was slow. The spawn loop reports where it got + // to — the node it is waiting on, the Bots the host already answered, and how long each took — + // so the sentence can name the node and the reason, and a lane whose node count no longer fits + // this host's speed says so instead of reading as a silent hang. + function spawnTimeoutText(progress) { + const seconds = Math.round(SPAWN_DEADLINE_MS / 1000) + const total = progress?.total ?? 0 + const created = progress?.created ?? [] + const current = progress?.current ?? null + const rest = [] + if (total > 0) { + const made = created.map(one => (one.ms > 0 ? `${one.label} in ${formatMs(one.ms)}` : one.label)) + rest.push(`${created.length} of ${total} node Bots were created${created.length > 0 ? ` (${made.join(', ')})` : ''}`) + } + if (current) { + const waited = current.startedAt > 0 ? now() - current.startedAt : 0 + rest.push(`node ${current.label} still had no Bot${waited > 0 ? ` after ${formatMs(waited)}` : ''}`) + } + // The average turns "slow" into a number the owner can act on: six nodes at eleven seconds + // each cannot fit a sixty-second deadline, whatever the settings say. + const average = created.length > 0 ? created.reduce((sum, one) => sum + (one.ms > 0 ? one.ms : 0), 0) / created.length : 0 + if (average > 0 && total > 0) { + const needed = Math.round((average * total) / 1000) + if (needed > seconds) rest.push(`a lane of ${total} nodes needs about ${needed}s on this host at ${formatMs(average)} per Bot, past the ${seconds}s deadline`) + } + return rest.length === 0 + ? `the host did not create the run Bots within ${seconds}s` + : `the host did not create the run Bots within ${seconds}s: ${rest.join('; ')}` + } + // #2748 C4: a host that never answers a spawn must not keep the card out of the // queue with the call hanging. The wait is bounded; on the deadline the start is // refused, and the Bots a late answer creates are released rather than leaked. - function boundedSpawn(pending) { + function boundedSpawn(pending, progress) { return new Promise((resolve, reject) => { let settled = false const handle = setTimer(() => { settled = true - const error = new Error(`the host did not create the run Bots within ${Math.round(SPAWN_DEADLINE_MS / 1000)}s`) + const error = new Error(spawnTimeoutText(progress)) error.code = 'spawn-timeout' + // #3282: the node the host never answered, and the reason, travel with the refusal. + error.reason = 'no-answer' + if (progress?.current?.nodeId) { + error.nodeId = progress.current.nodeId + error.nodeName = progress.current.nodeName + } + error.details = { reason: 'no-answer', ...(error.nodeId ? { nodeId: error.nodeId, nodeName: error.nodeName } : {}) } Promise.resolve(pending) .then(ids => releaseAll(Object.values(ids ?? {}), release, warn)) // A late answer that failed carries the ids it had already made on the error @@ -730,7 +817,17 @@ export function createWorkflowService({ } function refuse(error) { - return { ok: false, error: error?.message ?? String(error) } + // #3282: a start the host refused says which node it was and why next to the sentence, so a + // caller that shows one line still names the Bot to look at. Every other refusal keeps the + // answer it always had. + const answer = { ok: false, error: error?.message ?? String(error) } + if (error?.code === 'spawn-failed' || error?.code === 'spawn-timeout') { + answer.code = 'spawn-failed' + if (error.nodeId) answer.nodeId = error.nodeId + if (error.nodeName) answer.nodeName = error.nodeName + if (error.reason) answer.reason = error.reason + } + return answer } function gateStarted(item, run, result) { @@ -1312,10 +1409,31 @@ export function createWorkflowService({ try { // #1544: the caller that started the run and the Bot that saved the graph cap the connectors // this run may use; the host intersects them with each node's `tools`. - ephemeral = await boundedSpawn(spawnAll(item.nodes, bots() ?? [], spawn, item.model ?? '', { + // #3282: the loop reports the node it is on and the Bots already made, so the deadline + // below can name the node that never answered instead of one bare sentence. + const progress = { total: item.nodes.length, created: [], current: null } + const spawnNode = async (spec) => { + const startedAt = now() + const nodeId = String(spec?.nodeId ?? '') + const nodeName = String(spec?.name ?? '') + progress.current = { nodeId, nodeName, label: nodeLabel({ nodeId, nodeName }), startedAt } + try { + const id = await spawn(spec) + // A host that answers an empty id is a refusal too, and it has to name the node the + // way a thrown one does: spawnAll() would otherwise refuse it without a node context. + if (typeof id !== 'string' || id === '') throw new Error('the host returned no Bot id') + progress.created.push({ nodeId, nodeName, label: progress.current.label, ms: Math.max(0, now() - startedAt) }) + progress.current = null + return id + } catch (error) { + progress.current = null + throw spawnFailure(error, spec, item.id) + } + } + ephemeral = await boundedSpawn(spawnAll(item.nodes, bots() ?? [], spawnNode, item.model ?? '', { callerBotId: task.callerBotId, authorBotId: item.authoredBy, - })) + }), progress) } catch (error) { await releaseAll(error.spawned ?? [], release, warn) return refuse(error) @@ -1825,6 +1943,36 @@ export function createWorkflowService({ if (typeof spawn !== 'function') return answer(enqueue(item, task)) return hostSerial(() => Promise.resolve(enqueue(item, task)).then(answer)) }, + // #3282: the Bots API answers every endpoint that returns as `{ ok: true, value }` + // (core/src/host/channel.js `ok()`), so a refused start reached the caller inside a success + // envelope and one of them dispatched the same task twice. A refusal has to cross that + // boundary the way the API can report it — thrown, which the host turns into + // `{ ok: false, error: { code, message, details } }` — so this is the one entry point that + // refuses by throwing. `assign()` keeps its own `{ ok: false }` answer for the in-process + // callers (the drain, `ingest`, `runNamed`, the tools and the plugin's own tests). + assignEndpoint(body = {}) { + const workflowId = cleanLine(body.workflowId ?? body.id, 80) + const taskId = cleanLine(body.taskId ?? body.id, 80) + const lift = (value) => { + if (!value || value.ok !== false) return value + const failure = new Error(value.error ?? 'The task could not be started') + const details = { + ...(workflowId ? { workflowId } : {}), + ...(taskId ? { taskId } : {}), + ...(value.reason ? { reason: value.reason } : {}), + ...(value.nodeId ? { nodeId: value.nodeId } : {}), + ...(value.nodeName ? { nodeName: value.nodeName } : {}), + ...(value.ignored ? { ignored: value.ignored } : {}), + } + // `bot/refused` is the code the host already reports for a refusal whose sentence is + // meant for the person who acted (core/src/host/channel.js `errorCode`). + failure.code = 'bot/refused' + failure.details = details + throw failure + } + const value = this.assign(body) + return value && typeof value.then === 'function' ? value.then(lift) : lift(value) + }, setConcurrency(body = {}) { const workflowId = String(body.workflowId ?? '') const value = Number(body.maxConcurrent) diff --git a/plugins/workflow/test/spawn-failure-message.test.js b/plugins/workflow/test/spawn-failure-message.test.js new file mode 100644 index 000000000..1527b6eee --- /dev/null +++ b/plugins/workflow/test/spawn-failure-message.test.js @@ -0,0 +1,163 @@ +// #3282: a start the host refused said only "the host did not create the run Bots within 60s". +// The issue fast line (a six-node graph) failed that way three times while three- and five-node +// lines started, and the sentence could not say which node it was waiting on or why — so the owner +// re-dispatched the same task twice. Two things are pinned here: +// a) a spawn the host refuses, and a spawn that never answers, name the node and the reason; +// b) the endpoint seam refuses the way the Bots API can report it, instead of returning +// `{ ok: false }` inside the `{ ok: true, value }` envelope every caller reads first. +import assert from 'node:assert/strict' +import { test } from 'node:test' +import { SPAWN_DEADLINE_MS, createWorkflowService } from '../src/service.js' + +const CLOCK = 1_700_000_000_000 +const ROSTER = [ + { id: 'main', name: 'Chief', role: 'lead', isMain: true, remote: false }, +] + +// Same fake timer wheel as late-spawn-release.test.js: the deadline is fired by the test. +function scheduler() { + let seq = 0 + const queue = new Map() + function setTimer(fn, ms) { + const id = ++seq + queue.set(id, { fn, due: Math.max(1, Number(ms) || 0) }) + return id + } + function clearTimer(id) { + queue.delete(id) + } + function fire(ms) { + let left = Math.max(0, Number(ms) || 0) + let fired = 0 + for (;;) { + let next + for (const [id, entry] of queue) { + if (entry.due > left) continue + if (!next || entry.due < next.entry.due) next = { id, entry } + } + if (!next) break + queue.delete(next.id) + left -= next.entry.due + for (const entry of queue.values()) entry.due -= next.entry.due + fired += 1 + Promise.resolve(next.entry.fn()).catch(() => {}) + } + for (const entry of queue.values()) entry.due -= left + return fired + } + return { setTimer, clearTimer, fire, pending: () => queue.size } +} + +async function settle(rounds = 25) { + for (let i = 0; i < rounds; i += 1) await new Promise(done => setImmediate(done)) +} + +// `spawnFor(nodeId)` answers one node: an id, a throw the host wrote, or a promise that never +// settles (the case that used to reach the caller as a bare deadline). +function harness(spawnFor) { + const released = [] + const warnings = [] + const timers = scheduler() + const svc = createWorkflowService({ + persist: false, + now: () => CLOCK, + bots: () => ROSTER, + spawn: spec => spawnFor(spec), + release: async (id) => { released.push(id) }, + dispatch: async () => ({ ok: true }), + setTimer: timers.setTimer, + clearTimer: timers.clearTimer, + warn: line => warnings.push(String(line)), + }) + return { svc, released, warnings, timers } +} + +function sixNodes(svc) { + const made = svc.define({ + name: 'Issue fast line', + nodes: [ + { id: 'triage', name: 'Triage', role: 'triage', model: 'opencsitool/DeepSeek-V4.1-Flash-1017', prompt: 'Triage it.' }, + { id: 'claim', name: 'Claim', role: 'dev', model: 'opencsitool/DeepSeek-V4.1-Flash-1017', prompt: 'Claim it.' }, + { id: 'gate', name: 'Gate', role: 'review', model: 'opencsitool/DeepSeek-V4.1-Flash-1017', prompt: 'Gate it.' }, + { id: 'enqueue', name: 'Enqueue', role: 'merge', model: 'opencsitool/DeepSeek-V4.1-Flash-1017', prompt: 'Queue it.' }, + { id: 'escalate', name: 'Escalate', role: 'stop', model: 'opencsitool/DeepSeek-V4.1-Flash-1017', prompt: 'Escalate it.' }, + { id: 'attest', name: '取证', role: 'attest', model: 'opencsitool/DeepSeek-V4.1-Flash-1017', prompt: 'Attest it.' }, + ], + edges: [{ from: 'triage', to: 'claim' }], + }) + assert.equal(made.ok, true) + return made.workflows.find(item => item.name === 'Issue fast line') +} + +test('#3282a: a node whose model does not exist names the node and the host reason', async () => { + const { svc } = harness((spec) => { + if (spec.nodeId === 'claim') throw new Error('Unknown model opencsitool/Nope-1017') + return `eph-${spec.nodeId}` + }) + const item = sixNodes(svc) + const refused = await svc.assign({ workflowId: item.id, taskId: '#3282-model', title: 'Bad model' }) + + assert.equal(refused.ok, false, 'the start is refused') + assert.match(refused.error, /Claim/, `the node that failed is named: ${refused.error}`) + assert.match(refused.error, /Unknown model opencsitool\/Nope-1017/, `the host reason is carried: ${refused.error}`) + assert.doesNotMatch(refused.error, /^the host did not create the run Bots within/, 'not the bare deadline sentence') + assert.equal(refused.nodeId, 'claim', 'the node id rides along for a machine reader') + assert.equal(refused.reason, 'model-missing', 'the reason is classified') +}) + +test('#3282a: a node the host never answers names that node, not only the deadline', async () => { + const hang = new Promise(() => {}) + const { svc, timers } = harness((spec) => (spec.nodeId === 'attest' ? hang : `eph-${spec.nodeId}`)) + const item = sixNodes(svc) + const assigning = svc.assign({ workflowId: item.id, taskId: '#3282-hang', title: 'Never answered' }) + assigning.catch(() => {}) + await settle() + + timers.fire(SPAWN_DEADLINE_MS + 1) + await settle() + const refused = await assigning + + assert.equal(refused.ok, false, 'the start is refused at the deadline') + assert.match(refused.error, /did not create the run Bots within 60s/, 'the deadline is still the frame') + assert.match(refused.error, /"取证"/, `the node that never answered is named: ${refused.error}`) + assert.match(refused.error, /5 of 6/, `how far the spawn got is said: ${refused.error}`) + assert.equal(refused.nodeId, 'attest', 'the node id rides along for a machine reader') + assert.equal(refused.reason, 'no-answer') +}) + +// core/src/host/channel.js answers every plugin endpoint as `ok(await rt.flybotEndpoint(...))` — +// `{ ok: true, value }` — and reports a thrown refusal as `fail(message, code, details)` — +// `{ ok: false, error }`. That is the envelope a caller sees, so the seam has to throw; `assign()` +// itself keeps its `{ ok: false }` answer for the in-process callers that read it. +async function channelEnvelope(run, details = {}) { + try { + return { ok: true, value: await run() } + } catch (error) { + return { ok: false, error: { code: error?.code ?? 'bot/error', message: error?.message ?? String(error), details: error?.details ?? details } } + } +} + +test('#3282b: a refused start fails the envelope a caller checks first', async () => { + const { svc: okSvc } = harness(spec => `eph-${spec.nodeId}`) + const startedLane = sixNodes(okSvc) + const started = await okSvc.assignEndpoint({ workflowId: startedLane.id, taskId: '#3282-envelope-ok', title: 'Envelope ok' }) + assert.equal(started.ok, true, 'a started answer passes through unchanged') + + const { svc } = harness((spec) => { + if (spec.nodeId === 'gate') throw new Error('A Bot named Gate 3f2a already exists') + return `eph-${spec.nodeId}` + }) + const item = sixNodes(svc) + const envelope = await channelEnvelope(() => svc.assignEndpoint({ workflowId: item.id, taskId: '#3282-envelope', title: 'Envelope' })) + + assert.equal(envelope.ok, false, 'the outer envelope says the dispatch failed') + assert.equal(envelope.error.code, 'bot/refused') + assert.match(envelope.error.message, /Gate/, `the sentence names the node: ${envelope.error.message}`) + assert.equal(envelope.error.details.nodeId, 'gate', 'the node is machine-readable on the envelope') + assert.equal(envelope.error.details.reason, 'name-taken') + + // The in-process contract is untouched: `assign()` still answers `{ ok: false }` and never throws. + const inProcess = await svc.assign({ workflowId: item.id, taskId: '#3282-envelope-2', title: 'In process' }) + assert.equal(inProcess.ok, false, 'assign() keeps its own answer') + assert.match(inProcess.error, /Gate/) +}) From 0f2d6b378be87dcaad57ec07327f3db82c50238b Mon Sep 17 00:00:00 2001 From: Danyang Chen Date: Sat, 21 Nov 2026 16:26:22 +0800 Subject: [PATCH 2/2] Read a refused assign through settledCall without cutting a slow start A start waits for its node Bots, and the host refuses it at its own 60 s deadline, so the canvas must not cut the call at the shared 10 s: both assign call sites pass ASSIGN_TIMEOUT_MS (70 s) and still show a refused answer as the sentence the dialog already drew. Fixes #3282 Written-by: Dev --- plugins/workflow/client.js | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/plugins/workflow/client.js b/plugins/workflow/client.js index 04e8c5a76..ee8105952 100644 --- a/plugins/workflow/client.js +++ b/plugins/workflow/client.js @@ -206,6 +206,10 @@ window.__ModuleLoader__.load({ // silently either. Same shape as readWorkflows()/workflowCallError() in // core/src/client/workflows.js, kept here because this page reads the plugin's own calls. const CALL_TIMEOUT_MS = 10000 + // A start waits for its node Bots before it answers, and the host refuses it at its own + // 60 s deadline (`SPAWN_DEADLINE_MS`), so cutting the call at 10 s would report a failure + // for a dispatch that then starts (#3282). + const ASSIGN_TIMEOUT_MS = 70000 function callError(failure, fallback) { if (typeof failure === 'string' && failure.trim()) return failure.trim() const message = typeof failure?.message === 'string' ? failure.message.trim() : '' @@ -2287,7 +2291,7 @@ window.__ModuleLoader__.load({ // #3282: the host reports a refused start as a failed answer (`ok: false`), which // `call()` throws, so this dialog reads it through `settledCall` — the same // `{ ok: false, error }` it already knows how to show. - const value = await stageCall('workflow/assign', { botId, taskId: task.id, title: task.title }, 'Could not assign this task') + const value = await settledCall(call, 'workflow/assign', { botId, taskId: task.id, title: task.title }, 'Could not assign this task', ASSIGN_TIMEOUT_MS) if (value?.ok === false) { setError(value.error ?? ''); return } const host = globalThis.__flybot__?.host if (typeof host === 'function') { @@ -2302,7 +2306,7 @@ window.__ModuleLoader__.load({ // #2495: hand the card's own facts over, and say so explicitly even when it holds none — // an omitted `brief` keeps whatever an earlier call cached under this task id. const brief = typeof task.brief === 'string' ? task.brief : '' - const value = await stageCall('workflow/assign', { workflowId: target, taskId: task.id, title: task.title, brief }, 'Could not assign this task') + const value = await settledCall(call, 'workflow/assign', { workflowId: target, taskId: task.id, title: task.title, brief }, 'Could not assign this task', ASSIGN_TIMEOUT_MS) if (value?.ok === false) { setError(value.error ?? ''); return } setPack(value) setOpenId(target)