Repository navigation
0862: a GGUF context that fills up mid-request is freed and fails in words (stacked on #9) - #11
Merged
Merged
Conversation
…ry_clear) and fails with a clear ContextWindowExceededError; tests
…frees-and-says-so # Conflicts: # CHANGELOG.md
…LOG union, merge desk)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
0862: A GGUF context that fills up mid-request is freed and the request fails in words
Status: completed 2026-10-29 (unreleased; stacked on
fix/gguf-kv-context-honesty, PR #9).Found by: the OVH release-test server (Core 2.31.0,
unsloth/gemma-4-E4B-it-GGUFon a 16 GB GPU): gateway.logends with
decode: failed to find a memory slot for batch of size 512repeated on every later call.Cause
On the delegated llama-cpp-python lanes (
create_chat_completion, plain and streamed), a decode that cannot place abatch in the context memory raises
RuntimeError("llama_decode returned 1"). The GGUF lane turned it into an"Error: llama_decode returned 1" answer and left the engine's memory as it was, so the next request met the same
full memory. (PR #9 handles the prompt-cache lane and prompts that do not fit before eval.)
Fix
HuggingFaceProvider._gguf_raise_if_decode_failed: on a decode failure,Llama.reset()and the context'smemory_clear(True)(kv_cache_clear()on an older llama-cpp-python), thenContextWindowExceededErrorwith thesentence " ran out of context memory while answering (its N-token context is full; llama.cpp could not
place the next batch, code 1). The request was stopped and that memory freed. Start a new conversation or shorten
this one, or load the model with a larger context if the device has room." Applied in the plain lane's handler and
in
_gguf_context_checked_stream(a streamed failure).Evidence
tests/providers/test_gguf_memory_full_unit.py(memory freed + the sentence; a streamed failure; the older API);red when the freeing is removed.
tests/providers: 1727 passed.