fix(supervise): fail closed when a provider refuses an environment create - #1273
Merged
Merged
Conversation
…eate A provider SDK throws its own error classes, so a `create` the platform refused reached classifyDriverFailure as neither a BackendTransportError nor an AgentEvalError and took the foreign-accident default: retry. The status was on the error the whole time. classifyDriverFailure now reads a plain HTTP status off any thrown Error and applies the split the transport branch already promises: 408, 429 and 5xx are the upstream having a bad moment, any other 4xx will fail identically forever. Measured 2026-09-16 on discovery-lab: a Tangle Sandbox create refused HTTP 400 CONFIG_ERROR, for a key whose budget was fully reserved by an existing box, retried 14-22 times per node. Eleven of twelve roots showed a durable admission intent and no other event for 25 minutes, with healthy coordination servers and nothing for an operator to read. Only a thrown Error is read. `status` is an ordinary field name on settlements and run-state records, and reading one as an HTTP refusal stopped the retry that journals a cancelled bridge child's paid usage. Closes #1272.
The fixtures pin runtimeVersion against package.json, so a version bump without `pnpm run generate:testing-fixture` fails tests/testing-fixture.test.ts.
…ssage quotes error-message.ts already read an HTTP status off a thrown error, safely, to build the persisted failure text; the retry verdict had grown a second reader beside it. Export the original and delete the duplicate, so a failure is classified from exactly the status an operator sees quoted back to them, and a status behind a throwing getter cannot take the classifier down.
Contributor
Author
|
Two follow-ups pushed since the first review pass:
|
Three checked-in artifacts embed the runtime version, and a hand bump reaches only the first: the testing fixtures, docs/api/primitive-catalog.md, and docs/canonical-api.md. scripts/prepare-release.mjs exists to move all of them together, and this commit replaces the hand edits with its output.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1272.
Problem
A provider SDK throws its own error classes. So an environment
createthat the platform refused reachesclassifyDriverFailureas neither aBackendTransportErrornor anAgentEvalError, falls past every branch, and takes the foreign-accident default:transient, retry. The refusal is deterministic, so every retry reproduces it until the attempt ceiling or the deadline.Measured 2026-09-16 on discovery-lab, runtime 0.236.0, provider
tangle-sandbox. Twelve roots pressed ~20 s apart; one got a Sandbox and eleven produced exactly this and nothing more:root-stream.jsonlstayed 0 bytes. The coordination servers bound and answered 401 unauthenticated, so the runs looked healthy from every local angle. They sat for 25 minutes until an operator cancelled them. The platform's side of the same window: 200 rejections across about a dozen sandbox ids at 14-22 attempts each, each carryingHTTP 400 {"error":"Platform key delegation failed …","code":"CONFIG_ERROR"}. The underlying cause was budget reservation — a budgeted platform key funds one live sandbox, because the first box's router child reserves the whole remaining budget.The status was on the error the whole time. Nothing read it.
Change
classifyDriverFailurereads a plain HTTP status off any thrownErrorafter the existing branches, and applies the split the transport branch already promises consumers: 408, 429 and 5xx are the upstream having a bad moment; any other 4xx is a request that will fail identically forever.Only a thrown
Erroris read, and only a plain integer in 400-599 decides anything. That bound is not cosmetic:statusis an ordinary field name on settlements, run states and provider-model records. The first version of this patch testedtypeof error === 'object'and brokebridge-executor.test.ts— a bridge child that aborts mid-turn rejects with such a value, and classifying it as an HTTP refusal skipped the retry that journals its paid usage, turning{input: 17, output: 3}into{input: 0, output: 0}. That case is now a regression test.Checked
vitest run src/runtime/supervise tests/kernel— 1640 passed, 1 skipped, 0 failed.vitest run src/runtime/supervise/driver-retry.test.ts— 58 passed, including the new cases: a provider SDK error at 400/403/409 is terminal, at 408/429/502 transient, and a non-Errorvalue or a non-HTTPstatuskeeps the historical default.tsc --noEmitclean;biome checkclean on both changed files.What this does not fix
The refusal reaching the journal as a cause. A terminal create failure still settles
driver-failed; carrying it through asretainedPendingCause: 'request-rejected'(the vocabulary from #1204/#1247) is the follow-up, and #1272 stays open for it. This PR stops the silence and the 25-minute wait.Separately, hiding the real reason behind
CONFIG_ERRORis the sandbox platform's own bug and its team owns it.