GGUF: honest KV and context in residency, device-fit context rule, clear context-full failure

CHANGELOG.md

@@ -20,6 +20,27 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **GGUF residency rows report the KV cache llama.cpp really allocates, and the context it was built
+  with.** The estimate counted every block as an attention layer: `Qwen3.5-9B-Kimi-k3-Distilled-GGUF`
+  (qwen35, 33 blocks, a KV cache on one block in four) at 65,536 tokens read "+ 8.3 GiB cache" on a
+  16 GB card whose GPU held 9.1 GiB in total; it is 2.0 GiB. The layer walk (`utils.kv_geometry`)
+  reads the layout from the GGUF header (`full_attention_interval`, per-layer `head_count_kv`,
+  sliding-window layers and widths, shared/MTP layers) and is used by the residency rows, the
+  context-fit estimate and the load. Rows and claims carry `context_length`, `kv_cache_bytes`
+  (`kv_cache_estimated: true`).
+- **The GGUF context is sized to the card before allocating** (one CUDA/ROCm card, full offload):
+  the KV cache may take at most half of the free device memory after the weights and a reserve of
+  max(1 GiB, 10%); rungs above are skipped and the rule's pick is not stored as a calibration
+  (docs/memory-management.md). An explicit `max_tokens` is never resized.
+- **A skipped eject names its holders in words** (`owner_label` of the claim, "the runtime of account
+  default/bob"), never "runtime client 70a5b844f350" (ADR 0043).
+- **A prompt longer than the loaded GGUF context fails clearly, before any eval**
+  (`ContextWindowExceededError`, a subclass of `InvalidRequestError`): it used to fill the KV cache
+  ("decode: failed to find a memory slot for batch of size 512"), retry cold, rebuild the engine and
+  answer "Error: failed to prefill GGUF prompt cache ..." as text; llama-cpp-python's own "exceed
+  context window" refusal is raised the same way. Decoding that reaches the end of the context
+  stops with `finish_reason="length"` instead of appending "Error: llama_decode returned 1".
+
 - **`/v1/chat/completions` (and `/v1/responses`) reuse a model already resident in the process** instead of loading
   another copy per request (backlog 0859). With no model loaded in the server's own registry, the chat route called
   `create_llm` for every request; a host that keeps the model in another holder (AbstractGateway's runtime pool, which

abstractcore/exceptions/__init__.py

@@ -97,6 +97,22 @@ class InvalidRequestError(ProviderError):
     pass
 
 
+class ContextWindowExceededError(InvalidRequestError):
+    """The request needs more tokens than the context the model is loaded with.
+
+    Raised by local engines (llama.cpp) BEFORE evaluating a prompt that cannot
+    fit, instead of letting the engine fail mid-prefill ("failed to find a
+    memory slot") and answering an "Error: ..." text. `prompt_tokens` and
+    `context_length` say by how much."""
+
+    def __init__(self, message: str, *, prompt_tokens: "int | None" = None,
+                 context_length: "int | None" = None, model: "str | None" = None):
+        super().__init__(message)
+        self.prompt_tokens = prompt_tokens
+        self.context_length = context_length
+        self.model = model
+
+
 class UnsupportedFeatureError(AbstractCoreError):
     """Feature not supported by provider"""
     pass
@@ -237,6 +253,7 @@ def format_provider_error(provider: str, reason: str) -> str:
 
 # Export all exceptions for easy importing
 __all__ = [
+    'ContextWindowExceededError',
     'AbstractCoreError',
     'ProviderError',
     'ProviderAPIError',

abstractcore/providers/hf_residency.py

@@ -34,12 +34,14 @@
 """
 from __future__ import annotations
 
+import functools
 import gc
 import logging
 import sys
 import threading
 import time
 import weakref
+from pathlib import Path
 from typing import Any, Dict, Iterable, List, Optional, Tuple
 
 logger = logging.getLogger("abstractcore.providers.huggingface")
@@ -254,46 +256,69 @@ def llama_state_bytes(root: Any, *, max_depth: int = 10, max_nodes: int = 50_000
     return int(total)
 
 
-def llama_kv_alloc_bytes(metadata: Any, n_ctx: Optional[int], *, bytes_per_elem: int = 2) -> Optional[int]:
-    """The KV cache llama.cpp ALLOCATES for a context of `n_ctx` tokens, from
-    the GGUF metadata (`<arch>.block_count`, `.attention.head_count_kv`,
-    `.attention.key_length` / `embedding_length` / `head_count`), f16 K and V.
-    This is what the process holds from the moment the engine is built --
-    `llama_state_get_size` only measures the tokens in use (measured:
-    0.05 GB of state beside a 4.7 GB allocation). None when the
-    metadata does not describe the geometry. An estimate: it is labelled so."""
+def llama_kv_alloc_bytes(metadata: Any, n_ctx: Optional[int], *, bytes_per_elem: int = 2,
+                         geometry: Optional[Dict[str, Any]] = None, swa_full: bool = True,
+                         pad_v: bool = False) -> Optional[int]:
+    """The KV cache llama.cpp ALLOCATES for a context of `n_ctx` tokens (plus
+    the fixed recurrent state of hybrid models), from the GGUF geometry:
+    `geometry` (the file header, `utils.model_cache.read_gguf_geometry`, which
+    carries the per-layer arrays) when given, else llama-cpp-python's
+    `metadata`. Only the layers llama.cpp gives a cache are counted, at their
+    own widths (`utils.kv_geometry`): a Qwen3.5 hybrid holds a cache on one
+    block in four, not on all 33. This is what the process holds from the
+    moment the engine is built -- `llama_state_get_size` only measures the
+    tokens in use (measured: 0.05 GB of state beside a 4.7 GB allocation).
+    None when the geometry is unknown. An estimate: it is labelled so."""
     try:
-        if not isinstance(metadata, dict) or not n_ctx:
-            return None
-        arch = str(metadata.get("general.architecture") or "").strip()
-        if not arch:
-            return None
-
-        def _int_key(*keys: str) -> Optional[int]:
-            for key in keys:
-                value = metadata.get(key)
-                if value is None:
-                    continue
-                try:
-                    return int(str(value).strip())
-                except Exception:
-                    continue
-            return None
-
-        n_layer = _int_key(f"{arch}.block_count")
-        n_head = _int_key(f"{arch}.attention.head_count")
-        n_head_kv = _int_key(f"{arch}.attention.head_count_kv") or n_head
-        head_dim = _int_key(f"{arch}.attention.key_length")
-        if head_dim is None:
-            n_embd = _int_key(f"{arch}.embedding_length")
-            head_dim = (n_embd // n_head) if (n_embd and n_head) else None
-        if not (n_layer and n_head_kv and head_dim):
-            return None
-        return int(n_ctx) * int(n_layer) * 2 * int(n_head_kv) * int(head_dim) * int(bytes_per_elem)
+        from ..utils.kv_geometry import geometry_from_metadata, kv_cache_bytes
+
+        geom = geometry if isinstance(geometry, dict) and geometry else geometry_from_metadata(metadata)
+        return kv_cache_bytes(geom, n_ctx, swa_full=swa_full, pad_v=pad_v, bytes_per_element=bytes_per_elem)
+    except Exception:
+        return None
+
+
+@functools.lru_cache(maxsize=32)
+def _gguf_file_geometry(path: str, mtime_ns: int, size: int) -> Optional[Dict[str, Any]]:
+    from ..utils.model_cache import read_gguf_geometry
+
+    return read_gguf_geometry(Path(path))
+
+
+def gguf_file_geometry(model_path: Any) -> Optional[Dict[str, Any]]:
+    """The header geometry of a GGUF file (cached by path, mtime and size), or None."""
+    try:
+        path = Path(str(model_path))
+        st = path.stat()
+        geom = _gguf_file_geometry(str(path), int(st.st_mtime_ns), int(st.st_size))
+        return dict(geom) if isinstance(geom, dict) else None
     except Exception:
         return None
 
 
+def _llama_cache_layout(llm: Any) -> Dict[str, bool]:
+    """(swa_full, pad_v) of a built engine, from its context params: llama.cpp
+    pads V to the widest layer unless flash attention is ON."""
+    out = {"swa_full": True, "pad_v": True}
+    params = getattr(llm, "context_params", None)
+    if params is None:
+        return out
+    try:
+        if hasattr(params, "swa_full"):
+            out["swa_full"] = bool(params.swa_full)
+    except Exception:
+        pass
+    try:
+        fa_type = getattr(params, "flash_attn_type", None)
+        if fa_type is not None:
+            out["pad_v"] = int(fa_type) != 1  # 1 = enabled; -1 auto / 0 disabled pad
+        elif hasattr(params, "flash_attn"):
+            out["pad_v"] = not bool(params.flash_attn)
+    except Exception:
+        pass
+    return out
+
+
 def llama_bytes(llm: Any) -> Dict[str, Optional[int]]:
     """A `llama_cpp.Llama`'s weights (`llama_model_size`: bytes of the loaded
     tensors -- file-backed when memory-mapped), its KV ALLOCATION for `n_ctx`
@@ -316,7 +341,12 @@ def llama_bytes(llm: Any) -> Dict[str, Optional[int]]:
     except Exception:
         pass
     try:
-        out["kv_alloc_bytes"] = llama_kv_alloc_bytes(getattr(llm, "metadata", None), out["n_ctx"])
+        layout = _llama_cache_layout(llm)
+        out["kv_alloc_bytes"] = llama_kv_alloc_bytes(
+            getattr(llm, "metadata", None), out["n_ctx"],
+            geometry=gguf_file_geometry(getattr(llm, "model_path", None)),
+            swa_full=layout["swa_full"], pad_v=layout["pad_v"],
+        )
     except Exception:
         pass
     if llama_cpp is None:
@@ -438,6 +468,12 @@ def resident_models() -> List[Dict[str, Any]]:
         row["holders"] += 1
         row["copies"] += 1
         row["holder_rows"].append(hrow)
+        # The context each copy was built for and its KV allocation (an
+        # estimate from the GGUF geometry): what the "cache" figure is made of.
+        if isinstance(hrow.get("n_ctx"), int):
+            row["context_length"] = max(int(row.get("context_length") or 0), int(hrow["n_ctx"]))
+        if isinstance(hrow.get("kv_alloc_bytes"), int):
+            row["kv_cache_bytes"] = int(row.get("kv_cache_bytes") or 0) + int(hrow["kv_alloc_bytes"])
         if hrow.get("kv_bytes_estimated"):
             # `cache_bytes` / `held_bytes` include an f16 KV estimate.
             row["kv_bytes_estimated"] = True