Skip to content

Residency failsafe (ADR 0010): fit before load, evict unlocked LRU, typed refusal with suggestions - #17

Merged
lpalbou merged 6 commits into
mainfrom
feat/residency-fit-and-evict
Oct 29, 2026
Merged

lpalbou merged 6 commits into
mainfrom
feat/residency-fit-and-evict

Conversation

@lpalbou

@lpalbou lpalbou commented Oct 29, 2026 •

Copy link
Copy Markdown
Owner

Stacked on #9 (merge #9 first; this diff then shrinks to the failsafe).

ADR 0010 (docs/adr/0010-model-residency-fit-and-eviction.md): Core owns the failsafe (holders, claims and the estimate live here).

  • process_residency.ensure_fit / make_room: need = GGUF weights + KV at the smallest accepted context + reserve vs free device memory; evict resident HF models that are neither locked (a lock claim: the Models page Keep loaded) nor in use, least recently used first (InflightGenerations.last_used_at, registration time), one at a time, re-measuring; else ModelDoesNotFitError (needed/free bytes, kept in words, evicted, catalog suggestions of smaller artifacts of the same model or family) with the sentence.
  • HuggingFaceProvider._load_gguf_model: the check runs before llama.cpp allocates (one full-offload CUDA card); the CPU fallback is checked against available RAM; the typed error is never wrapped.

Tests: tests/providers/test_residency_fit_and_evict_unit.py (5; mutation-checked: lock/in-use ignored, not LRU, no suggestions all go red; the load calls the check before allocating).

Real weights on the OVH 8080 server (Quadro RTX 5000 16 GB, branch wheels installed with install.sh --with-wheel):

  • A = mradermacher/Qwen3.5-9B-Kimi-k3-Distilled-GGUF resident (GPU 9,414 MiB), locked (Keep loaded).
  • Load B = mradermacher/Gemma-4-14B-A4B-Heretic-GGUF: refused in 1.1 s, GPU unchanged (9,414 MiB): "mradermacher/Gemma-4-14B-A4B-Heretic-GGUF needs 11.7 GB but only 5.8 GB is free on the GPU (kept: mradermacher/Qwen3.5-9B-Kimi-k3-Distilled-GGUF (locked by the runtime of account default/admin)). Choose a smaller model, or unlock a kept model on the Models page." (no catalog artifact of Gemma 4 fits 5.8 GB, so no suggestion).
  • Unlock A, load B: A evicted, B loaded in 27.6 s, GPU 11,424 MiB, B at 8,192-token context (10.4 GiB weights + KV).
  • Chat with B locked (default A): the run answers in 2.0 s with "... Error: mradermacher/Qwen3.5-9B-Kimi-k3-Distilled-GGUF needs 7.5 GB but only 3.7 GB is free on the GPU (kept: mradermacher/Gemma-4-14B-A4B-Heretic-GGUF (locked by ...)) ..."; GPU unchanged.

Re-run after the review fixes (one load at a time, CUDA cache returned before measuring, in-use re-check, reserve in suggestions, CPU guard without eviction): lock A, load B refused in 1.1 s with the same sentence, GPU 9,414 MiB unchanged; unlock A, load B: A evicted, B loaded in 4.8 s, GPU 11,424 MiB, the stale refusal cleared.

… unlocked models, typed refusal with suggestions)
…LRU eviction of unlocked idle models, ModelDoesNotFitError with numbers and smaller suggestions; CPU fallback checked against RAM
…rned before measuring, in-use re-checked before each eviction, suggestions use the fit reserve, the CPU guard refuses without evicting and never on Metal
@lpalbou
lpalbou merged commit 2cae18b into main Oct 29, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant