Feature Request: --live-context-workspace

[Almazick]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Please implement the feature from https://github.com/GenerelSchwerz/llama.cpp

--live-context-workspace, --no-live-context-workspace
for supported attention caches, grow the compute workspace reservation
with the padded live physical KV extent instead of reserving the full
context up front (default: disabled)
(env: LLAMA_ARG_LIVE_CONTEXT_WORKSPACE)

--phase-aware-workspace, --no-phase-aware-workspace
resize compute workspaces between prompt processing and token
generation; later prompt turns regrow the prompt reservation (default:
disabled)
(env: LLAMA_ARG_PHASE_AWARE_WORKSPACE)

Motivation

It speeds up the token generation at lower context size

Possible Implementation

No response

[Anbeeld]

Please clarify the source for "It speeds up the token generation at lower context size"? I don't see any evidence for that, these features only affect VRAM usage.

[Almazick]

Please clarify the source for "It speeds up the token generation at lower context size"? I don't see any evidence for that, these features only affect VRAM usage.

Let's assume you are running a 256k context size. Prefill and token generation will be slow due to the big context size, but when you run a 20k context size, your prefill and token generation will be faster. What it does is shift the context size from smaller to bigger on the fly and keeping the max context size to the max limit -c N. For the first prompt people don't need full 256k context size. First prompt will use context size enough just for the prompt and adjust context size to a smaller number. It causes at least 10% faster token generation and prefill in the beginning, and then it'll stabilize back to the speed it should be. This is just an example and I'm talking from my own personal experience. I tested yesterday. My initial t/s was 46 but after enabling --live-context-workspace My speed went up to 56 t/s. It stayed much higher for the first 70k context size compared to my regular speed. My total context size is only 110k. Tested with Qwen 3.8 27B Q4 with RTX5080

[Anbeeld]

This explanation does not match how llama.cpp actually handles context.

Setting -c 256k does not make a request with 20k live tokens perform attention over the full 256k. llama.cpp already calculates n_kv from the actually occupied KV extent (padded for graph reuse), and the runtime K/V views and attention mask use that live n_kv. The full context is used for the worst-case compute workspace reservation.

--live-context-workspace changes that reservation: instead of reserving compute buffers for the full configured context up front, it grows them with the live KV extent. It does not turn a "256k attention workload" into a "20k attention workload", because the latter was already the case.

So the 46 -> 56 t/s result may be real, but this mechanism does not explain it. A ~22% gain from changing workspace reservation alone would be interesting enough to benchmark separately. I would want to see a controlled A/B with the same commit, model, prompt, starting KV state, GPU clocks and settings, multiple runs, and only --live-context-workspace toggled before treating it as a performance feature.

I also think both flags are somewhat at odds with llama.cpp's design. llama.cpp reserves worst-case memory up front so allocation stays predictable and stable. If 256k fits, reserving for it is fine; if it does not, growing allocations later just postpones the OOM.

--phase-aware-workspace has the same issue: real workloads repeatedly switch between prefill and decode, so it turns one stable allocation into repeated resizing, synchronization and graph invalidation for mostly temporary VRAM savings.

[Almazick]

If you want me to provide any logs, I'll be happy to do that or what ever you need. I can provide a complete command that I'm using to run it, for you to test it.

I tested yesterday. My initial t/s was 46 but after enabling --live-context-workspace My speed went up to 56 t/s. It stayed much higher for the first 70k context size compared to my regular speed. My total context size is only 110k. Tested with Qwen 3.8 27B Q4 with RTX5080

[Anbeeld]

To verify this properly, please do a full A/B where the only changed parameter is --live-context-workspace.

Keep the same commit/build, model, quant, -c, batch/ubatch, GPU settings and prompt. Start from a fresh process for every run so the KV cache, allocator and CUDA state are reset.

Please test a few fixed live-context sizes, for example 10k, 30k, 50k, 70k and ~100k, while keeping the configured -c identical. For each size:

1. Run without --live-context-workspace.
2. Run with --live-context-workspace.
3. Prefill exactly the same prompt.
4. Generate a fixed number of tokens, ideally 512-1024, with deterministic sampling (temp 0) and EOS ignored so both runs do the same amount of work.
5. Record prompt t/s, generation t/s and VRAM usage.

Do at least 3-5 runs per configuration and alternate the order (off/on, then on/off) to reduce clock/temperature effects. Discard a warm-up run if necessary.

Please also include the exact command line, commit SHA and complete logs for both sides.

The important part is that -c stays fixed while only the actual live context changes. That directly tests the claim that this flag improves performance at low live context despite the same maximum configured context.

[Almazick]

ok, I'll do that.

[Almazick]

Just a quick log files for you to see. It's not great but it shows around 10% improvement.

with live-context-workspace.txt
without live-context-workspace.txt

[Almazick]

Here is another one with TEMP 0, and with and without --live-context-workspace. Same prompt. It shows a good 10% difference. With --live-context-workspace It doesn't drop under 50 t/s and without --live-context-workspace it only runs high 40's t/s. 4+ t/s difference without dropping the quality and hardware

TEMP 0 with live-context-workspace.txt
TEMP 0 withOUT live-context-workspace.txt

[Almazick]

If you need anything else, just let me know but I'd love to see that feature with beellama to have dynamic context size. I hope these logs shows enough proof and information for you.

[Anbeeld]

Thanks, this is useful and the ~10% difference is interesting, but I don't think these logs are enough to call it confirmed yet.

What I need now is a very small controlled test: pick one fixed prompt/context depth, keep the exact same command and build, and do 5 fresh-process runs with "--live-context-workspace" off and 5 with it on. Alternate the order between runs to reduce temperature/clock effects, and record pp/tg timings for each run.

Please also include the exact command line and commit SHA.

If the ~10% difference remains consistently across those runs, then there is clearly a real performance effect worth investigating. It still would not be explained by a smaller attention context, since llama.cpp already uses the live "n_kv", but it would show that the workspace change itself has some secondary performance effect on this setup.

[Almazick]

Sure, I'll do that later today. I compiled it yesterday from https://github.com/GenerelSchwerz/llama.cpp

[Almazick]

Another log, found out TEMP 0 makes Qwen 3.8 27B dumber.

TEMP 0 with live-context-workspace + Compaction.txt
TEMP 0 withOUT live-context-workspace + Compaction.txt

[Almazick]

git -C I:\llamaSchwerz\llama.cpp rev-parse HEAD

50df2c7cf761a4647bd2555cb4e0f0fabdaeef11

[Anbeeld]

Proper controlled A/B benchmarks showed no difference performance-wise. Which means there's no sense in porting these features, as they range from anti-pattern for llama.cpp, to not having any real impact at all, except adding complexity and requiring a lot of work.

live-context-workspace-ab-benchmark-20260929.md