Repository navigation
Conversation
…races The "Failed to place sandbox: not enough capacity" error (503, capacity_unavailable) is returned whenever placement finds no node with room, but it never says which capacity was short: the hugepage memory pool (MemoryAdmission), node-side ResourceExhausted refusals (running cap, starting cap, a guest still stopping, node not accepting), or an empty candidate set. Operators cannot tell a memory shortage from a running-count cap without guessing. Add a CapacityShortageReason and diagnostic fields (Attempts, Refusals, PoolsFull, LastRefusal) to NoNodesAvailableError, populated at every return site and recorded on the place-sandbox span plus a structured warn log. classifyRefusal labels the node ResourceExhausted message so the last refusal reason is carried too. The client-facing message is unchanged (still "no nodes available" / the 503 text), and the zero value keeps the original behaviour, so no wire or API change. Diagnostics are server-side only.
AdaAibaby
requested review from
ValentaTomas,
dobrac and
jakubno
as code owners
November 12, 2026 14:18
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3821. The
capacity_unavailable503 ("not enough capacity …") never says which capacity was short — hugepage memory pool, nodeResourceExhaustedrefusals, or an empty candidate set. The placement loop already knows (poolsFull,refusals, per-node status) but discards it at the return sites.This adds a
CapacityShortageReasonand diagnostic fields toNoNodesAvailableError, populated at every return site (the loop deadline, theanswered/allExcludedpaths, and theBestOfK.chooseNodefull-pool return), and records them on theplace-sandboxspan plus one structured warn log.Reasonhugepage_pool_full|node_refusals|no_candidate_nodeAttemptsRefusalsPoolsFullLastRefusalclassifyRefusalmaps the node'sResourceExhaustedwording (running_cap/starting_cap/guest_memory_stopping/node_not_accepting/lifecycle_busy) so a memory-pool shortage is distinguishable from a running-count cap at a glance.Not a behaviour change
Error()still returns"no nodes available"; the client-facing 503 message anderror_codeare unchanged.NoNodesAvailableErrorbehaves exactly as before, so every existingerrors.Asmatch keeps working.Test
TestPlaceSandbox_EveryPoolFull_ReasonIsHugepagePoolasserts the classified reason on the pool-full path.go build ./...,go vet, and the fullinternal/orchestrator/placementsuite pass. (Theevictorpackage's Redis-backed tests need a test Redis and are unrelated.)/cc @ryan-sana Looking forward to your code review.