Repository navigation
Residency failsafe (ADR 0010): fit before load, evict unlocked LRU, typed refusal with suggestions - #17
Merged
Conversation
… unlocked models, typed refusal with suggestions)
…LRU eviction of unlocked idle models, ModelDoesNotFitError with numbers and smaller suggestions; CPU fallback checked against RAM
…rned before measuring, in-use re-checked before each eviction, suggestions use the fit reserve, the CPU guard refuses without evicting and never on Metal
…at the end of [Unreleased])
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #9 (merge #9 first; this diff then shrinks to the failsafe).
ADR 0010 (docs/adr/0010-model-residency-fit-and-eviction.md): Core owns the failsafe (holders, claims and the estimate live here).
process_residency.ensure_fit / make_room: need = GGUF weights + KV at the smallest accepted context + reserve vs free device memory; evict resident HF models that are neither locked (a lock claim: the Models page Keep loaded) nor in use, least recently used first (InflightGenerations.last_used_at, registration time), one at a time, re-measuring; elseModelDoesNotFitError(needed/free bytes, kept in words, evicted, catalog suggestions of smaller artifacts of the same model or family) with the sentence.HuggingFaceProvider._load_gguf_model: the check runs before llama.cpp allocates (one full-offload CUDA card); the CPU fallback is checked against available RAM; the typed error is never wrapped.Tests:
tests/providers/test_residency_fit_and_evict_unit.py(5; mutation-checked: lock/in-use ignored, not LRU, no suggestions all go red; the load calls the check before allocating).Real weights on the OVH 8080 server (Quadro RTX 5000 16 GB, branch wheels installed with install.sh --with-wheel):
Re-run after the review fixes (one load at a time, CUDA cache returned before measuring, in-use re-check, reserve in suggestions, CPU guard without eviction): lock A, load B refused in 1.1 s with the same sentence, GPU 9,414 MiB unchanged; unlock A, load B: A evicted, B loaded in 4.8 s, GPU 11,424 MiB, the stale refusal cleared.