diff --git a/docs/README.md b/docs/README.md index 47adbd1d..d39dae96 100644 --- a/docs/README.md +++ b/docs/README.md @@ -47,8 +47,8 @@ and coda, including the LM head. The [Dory baseline notes](dory.md) list the narrower scope qualified for Dory: synchronous fixed-loop execution with boundary KV in both KV modes, sliding-window masking on the Torch and Triton backends, and BF16 agreement with the pinned -checkpoint at 1/2/4 loops; Graphs, WCPB, async, speculation, prefix reuse and PD -remain out of scope. +checkpoint at 1/2/4 loops. The notes also list the qualified scope of Graphs, +async, speculation, prefix reuse and PD; WCPB remains out of scope. CPU execution provides a Torch reference backend. FlashAttention hardware validation is currently documented for FA4 on B300; FA2/FA3 require validation on their target devices. Disaggregated serving currently targets multiple GPUs on a single host. diff --git a/docs/dory.md b/docs/dory.md index 46cfd8d6..35e0f6a6 100644 --- a/docs/dory.md +++ b/docs/dory.md @@ -50,10 +50,12 @@ to one cache layout: | `last_loop` | `shared` | One plane per layer, overwritten by each loop | Boundary stages always write depth 0, so input and output KV are never -replicated per loop. In `last_loop` mode a token's current-step KV is overwritten -by the next loop, so results depend on the chunk schedule; the qualified -schedule is one token per step. `per_loop` gives the same result for -token-at-a-time and all-positions-at-once execution. +replicated per loop. In `last_loop` mode a token's KV is overwritten by each of +its loops, so the runtime completes every loop of a token before the next one, +also inside a prefill chunk: later tokens read final-loop KV whatever the chunk +size. `per_loop` preserves the same mathematical +computation for token-at-a-time and all-positions-at-once execution; floating-point +rounding can still vary with chunk and batch shapes. Only the 36 attention sublayers own KV. The cache pool is sized by that compact count (`DoryConfig.num_kv_layers`), not by all 72 physical sublayers, and @@ -61,13 +63,94 @@ count (`DoryConfig.num_kv_layers`), not by all 72 physical sublayers, and declares one. Attention layers therefore reach the cache through a packed layer id rather than their physical sublayer index. +## Grouped storage and SWA reclamation + +Dory's ordinary synchronous eager execution uses one pool of attention-layer +pages, grouped by window policy and boundary/recurrent plane domain. The engine +selects this grouped storage automatically and returns expired sliding-window +pages to the pool as it runs; no cache flag turns reclamation on. Reclamation +needs fixed-loop `ouro` exits; other exit modes keep grouped pages resident. +The engine keeps all-layer blocks for the execution modes listed below. Packed +prefill also stores all layers together and is preferred where a backend +supports it; pass `packed_prefill=False` to keep grouped storage. + +```python +from vllm_rlt import CacheConfig +from vllm_rlt.config import ExecutionConfig +from vllm_rlt.engine.llm_engine import LLMEngine + +engine = LLMEngine( + model, + attention_backend="torch", + execution_config=ExecutionConfig(use_uva=False, packed_prefill=False), + cache_config=CacheConfig( + num_blocks=256, + layout=model.kv_cache_layout, + incremental_allocation=True, + ), +) +``` + +`num_blocks` preserves the existing all-layer KV byte budget: the native pool +converts it to `num_blocks * model.config.num_kv_layers` single-layer pages. +Each `per_loop` plane advances independently; shared recurrent history in +`last_loop` advances only after the token's final loop. Pages retire only after +all layers in their group have consumed the range and the submission completes. +Full-attention history remains resident. Incremental allocation bounds the +resident pages to the window plus one submission; preallocation reserves each +request's full peak and still returns expired pages to the pool. + +CUDA Graphs and asynchronous scheduling borrow device-owned KV metadata; +self-speculation needs suffix truncation after a rejection; packed prefill does +not record which SWA positions it read; FP8 KV keeps per-layer scales; +preemption may offload KV to a CPU snapshot. Grouped pages do not support these +yet. In all six cases the engine uses all-layer blocks, including for prefix +caching, and reclamation is off. + +PD stages keep grouped storage. A transfer addresses one table per group, +plane and layer, because each grouped page holds one layer. P holds a prompt +only until its transfer completes, so P does not reclaim: reclamation could +return a page that a transfer still reads. D first runs the output stage of the +last prompt token. D reserves no SWA page that this query cannot read, and P +sends no KV to those pages. After the import, D reclaims from the positions +that P has read. + +Native SWA attention stages only the needed page-aligned read span. Logical token +positions, write addresses and public layer maps remain absolute; the backend's +read-table context lengths are local to that span. Native attention calls must +match the layer's declared window, including `None` for full-attention layers. + +For read-only group policy, `cache.layer_group(kv_layer_id)` returns +`(group_id, policy)`: a manager-local group ID and the immutable `KVGroup` with +compact layer IDs, window, recurrent domain and plane count. The layer ID is a +compact KV index, not a physical sublayer index. + +For read-only cache inspection, `cache.retained_range(request_id, kv_layer_id, +depth)` returns `(resident_start, written_end)` in absolute token positions. +`cache.layer_block_table(request_id, kv_layer_id, depth)` returns the absolute +logical-page map, with `-1` for reclaimed pages. Here `cache` is +`engine.cache_manager`, and `kv_layer_id` is a compact KV index. Reading a +reclaimed full prefix is rejected; these queries do not retain page references +or expose mutable allocator state. + +Grouped prefix caching publishes one entry per token page holding every +owned (group, plane, layer) page, ordered as the allocator splits a claim. +The entry takes its own reference, so the publishing request can finish or +release its pages without freeing the cached ones. Publication stops at the +first token page with any owned page missing; each step publishes its prompt +pages before retiring expired ones. When reclamation is active, a hit takes +request references only to SWA pages still within the next token's window. +The prefix cache can hold older pages for reuse by shorter hits. Each request +is budgeted for its own peak KV use so sharing does not reserve less memory +than the requests may need as they advance. + ## Supported today - Native strict safetensors loading with the artifact's per-projection key names (`q/k/v/g/o_proj`, `gate_proj`/`up_proj`) and the learned scale tensors. There is no fused-key remapping: a checkpoint that packs `qkv_proj` or `gate_up_proj` fails loudly instead of loading approximately. -- Synchronous eager execution: full-prompt prefill followed by ordinary token +- Synchronous eager execution: ordinary bounded-chunk prefill followed by token decoding, with the configured fixed loop count. - `loops=` at load time to fix 1, 2 or 4 loops. - Greedy and sampled decoding, request seeds, EOS (`2`, `11`) and length @@ -86,9 +169,31 @@ layer id rather than their physical sublayer index. the next input stage and only its readback is deferred. The two switches are independent: sync eager, sync Graph, async eager and async Graph are tested separately. +- Same-host PD (`PDEngine`). P runs the input stage, every recurrent loop and + the output stage of each prompt token except the last, so its output-stage + KV covers the prompt history. D receives the KV of every plane and the + carried state of the last prompt token. D then runs that token's output stage + and samples, so no prompt stage runs twice or is skipped. PD runs in the four + execution modes above, with prefix caching on either side. In sync eager, + both stages keep grouped storage and D reclaims SWA pages; the other three + modes use all-layer blocks. With SWA reclamation on D, an imported prompt + publishes no prefix on D, because D never holds its first SWA pages. See + [PD](#pd). - The official tokenizer, including the Hub's `tokenizer.json` and chat template, resolved without requiring transformers to accept the checkpoint config (its generic validator rejects Dory's `mlp` sublayer entries). +- Prefix caching (`enable_prefix_caching=True`). In `per_loop` mode a + published page carries boundary KV from plane 0 and recurrent KV from every + plane; boundary layers never write deeper planes, so they are not required + there. In `last_loop` mode the single shared plane holds each cached token's + final-loop KV, which is what a cold prefill leaves; this mode is qualified + for synchronous eager execution only. A hit restores exactly that coverage + and the scheduler resumes prefill from the cached offset. + Prefix keys hash tokens only. That is sound because the loop count is fixed + when the model loads and the layout is fixed by `CacheConfig`, and each + engine owns its own KV manager, so pages from different loop counts or + layouts never share a cache. A cache shared across loop counts would have to + add both to the key. Evicted prefixes fall back to a full cold prefill. ## Packed Triton prefill @@ -134,26 +239,29 @@ even with per-query attention. These fail fast rather than degrade silently: -- WCPB, mixed or staged prefill, and PD. -- Prefix caching. +- WCPB, and mixed or staged prefill. +- Speculative decoding with PD. +- Prefix caching in `last_loop` mode with CUDA Graphs or asynchronous + scheduling. - Self-speculation combined with CUDA Graphs, asynchronous scheduling, - preemption or `last_loop` KV, and sampled token-pipeline speculation. The - synchronous eager form is supported; see below. + preemption or `last_loop` KV, and sampled token-pipeline + speculation. The synchronous eager form is supported; see below. - Per-request loop counts other than the configured full depth: Dory has no trained exit gate, so the runtime refuses to invent one, including via `min_loops` with `exit_threshold=0`. -Preemption and snapshots are **not qualified** rather than enforced: the -generic KV snapshot mechanism copies every plane, so it may well work, but no -restoration test exists for Dory's boundary state. Keep `enable_preemption` -off until 2E covers it. +`enable_preemption` selects the all-layer fallback, so suspend, CPU offload and +restore work, with or without prefix caching, in execution modes that accept +preemption (self-speculation does not). Dory preemption has not been qualified +on GPU. Sliding-window masking exists in the Torch reference backend and in the Triton decode kernel (`window=` on `KVCacheManager.attend`). Backend selection requires the `sliding_window` capability, so a Flash-backed Dory engine fails at construction instead of attending full history. Masking accepts host-planned, -graph-private and device-owned metadata; storage reclamation of out-of-window -pages stays with 2B. +graph-private and device-owned metadata. Physical reclamation of out-of-window +pages happens on grouped storage, which the engine selects automatically as +described above. ## Synchronous self-speculation @@ -227,19 +335,82 @@ views; all views are bound before execution. `tests/test_dory.py` and `tests/test_dory_gpu.py` hold the regression suites; `tests/test_dory_real_checkpoint.py` holds the pinned-weight run. Measured results are below, including what they do not cover. + ### CPU suite -`tests/test_dory.py` (44 cases) covers configuration validation, strict loading +`tests/test_dory.py` covers configuration validation, strict loading with single-file and sharded checkpoints, stage/KV contracts, sliding-window masking, KV layer compaction, engine scheduling, EOS stopping, and numerical agreement with an independent dense oracle (`tests/reference/dory.py`) for both -KV modes — including greedy generation against that oracle. The full CPU run is -945 passed with only the pre-existing failures of the reviewed baseline -(`test_ouro` packed prefill, one `test_pd`, eleven `test_serving`). +KV modes — including greedy generation against that oracle. The full CPU run on +2026-11-16 passed 1109 tests with no failures; GPU-marked tests were skipped. + +### Prefix reuse + +Prefix reuse is checked at loops 1, 2 and 4: cold and cache-hit prefill +logits match the oracle, cached prefixes longer than the sliding window, +block-aligned prompts, diverging suffixes, concurrent sharers and a +re-request after its prefix was evicted all reproduce the cold greedy tokens, +and publication refuses pages missing any owned plane. Prefix pages stay +pinned for the life of the cache entry. Prefix tests run on both the legacy +all-layer blocks and grouped storage. On a small FP32 model, prefix hits and +diverging branches match the oracle in sync eager, sync Graph, async eager and +async Graph modes. The checks above are `per_loop`. `last_loop` checks the +cache-hit logits against the oracle on both storages, and block-aligned, +diverging and concurrent requests after a hit against the cold greedy tokens, +with prefill chunks of 3 and 4 tokens; on CUDA with Triton, a hit and a +diverging branch match the oracle with chunks of 3 and 16. With grouped +reclamation, in both modes, hits of 2 and 8 tokens match the oracle logits and +reclamation advances past the hit, including submissions longer than the window. + +Real checkpoint, one L20, two loops, 16 greedy tokens, `prefill_chunk_size=256`, +prompts cut from the model card. Each cell compares prefix caching on and off in +a fresh engine; time to first token in seconds: + +| Prompt tokens | Request | TTFT off | TTFT on | Tokens equal (BF16) | +| --- | --- | --- | --- | --- | +| 1100 | repeat | 15.3 | 0.30 | yes | +| 1100 | branch | 15.2 | 0.60 | yes | +| 4096 | repeat | 60.4 | 0.41 | yes | +| 4096 | branch | 60.3 | 0.78 | yes | + +An evicted prefix falls back to a cold prefill with equal tokens, and a control +without eviction hits 1088 tokens. In BF16, one of four concurrent sharers and a +seeded sampled (temperature 0.8) repeat differed from the uncached run; in FP32 +all of them match. A hit resumes prefill at the cached offset, which changes +the chunk split, and with caching off, changing `prefill_chunk_size` alone +already changes BF16 greedy tokens. FP32 tokens matched in these checks; BF16 +agreement is not promised across chunk splits. These are single runs on a +shared host. + +The same checks were repeated on one RTX PRO 5000 Blackwell (torch 2.13+cu130, +Torch backend), BF16, at loops 1, 2 and 4: + +| Loops | Prompt tokens | TTFT off | TTFT on (repeat) | +| --- | --- | --- | --- | +| 1 | 1100 | 6.6 | 0.13 | +| 1 | 4096 | 27.9 | 0.20 | +| 2 | 1100 | 9.3 | 0.18 | +| 2 | 4096 | 40.9 | 0.29 | +| 4 | 1100 | 16.4 | 0.31 | +| 4 | 4096 | 69.9 | 0.49 | + +Repeat tokens were equal everywhere; eviction and its 1088-token control +behaved as on the L20. At two loops and 1100 tokens in FP32, all eight +concurrent requests, the seeded sampled repeat, every `prefill_chunk_size` +from 16 to 256 and the cache hit produced identical tokens. + +With the Triton backend at two loops and 1100 tokens, sync eager, sync Graph, +async eager and async Graph each produced the uncached tokens for the cold, +repeat and branch requests, with the hit cutting time to first token from +0.54-0.79 s to 0.06-0.10 s. Through the OpenAI-compatible HTTP server (1516 +prompt tokens), every request succeeded and hits cut time to first token from +1.5-7 s to 0.09-0.35 s; the branch and one concurrent response differed in +BF16 text, which is the chunk-split effect above. ### CUDA -`tests/test_dory_gpu.py` (66 cases) checks the Triton backend against the Torch +`tests/test_dory_gpu.py` checks the Triton backend against the Torch reference: windowed decode for `head_dim` 8/128/256, windows 1/3/16/64 and lengths 1/7/63/64/65; unwindowed decode unchanged; both KV modes against the dense oracle; a greedy engine run whose tokens match the oracle; and @@ -272,7 +443,7 @@ to be complete. ### Execution modes -`tests/test_dory_gpu.py` adds 36 cases on a small FP32 model. For each of sync +`tests/test_dory_gpu.py` checks a small FP32 model. For each of sync eager, sync Graph, async eager and async Graph, with 1, 2 and 3 concurrent requests of different prompt lengths, the generated tokens equal the dense oracle's, and the Graph modes must replay the input-stage, recurrent and @@ -312,12 +483,12 @@ BF16 the four modes are identical up to 16 requests and at every prompt length measured there; from 32 requests on, sync eager, sync Graph and async eager still agree while async Graph differs, because it merges the two prefill cohorts into one decode wave and BF16 greedy tokens depend on batch -composition. FP32 is therefore the exact check; BF16 agreement is not promised -across batch shapes. +composition. FP32 tokens matched in these measured workloads; BF16 agreement is +not promised across batch shapes. ### Self-speculation -`tests/test_dory_speculative.py` (54 CPU cases) checks block and token-pipeline +`tests/test_dory_speculative.py` checks block and token-pipeline speculation for the depth pairs 1→2, 1→4 and 2→4 against the dense oracle: greedy tokens for ragged batches, a forced rejection at every draft position, all-accepted rounds with the bonus token, EOS inside an accepted prefix, output @@ -346,9 +517,9 @@ same eight BF16 requests without speculation in waves of four or one instead of eight also leaves only 2–4 of 8 requests token-identical, while the FP32 baseline is identical across wave sizes (32/32). Speculation changes the batch shapes of the same per-row arithmetic, BF16 rounding then differs, and greedy -near-ties flip; every later token of that request diverges. FP32 is therefore -the exact check, and BF16 output is a valid greedy continuation up to rounding, -not a bit-reproduction of the baseline. +near-ties flip; later tokens of that request can then diverge. FP32 tokens matched +in these measured workloads; BF16 does not reproduce the baseline token for token +across these batch shapes. Acceptance (accepted / drafted, BF16): 1→2 0.78–0.93, 2→4 0.90–0.98, 1→4 0.72–0.93. @@ -390,6 +561,34 @@ Reading the numbers: memory was 74–82 GiB with the automatically sized KV pool; every arm released its KV blocks. +### PD + +`tests/test_dory_pd.py` runs P on one GPU and D on another, with NIXL and FP32 +Triton: + +- CUDA, 19 cases. Greedy output in both KV modes, in sync eager (grouped + storage), sync Graph, async eager and async Graph (all-layer blocks), equals + the dense oracle and a single-stage engine. A seeded sampled request equals + the single-stage engine, so sampling stays on D. Prefix caching on P, on D or + on both skips prompt work and keeps the outputs. For a 40-token prompt, + grouped PD sends fewer bytes than all-layer PD. Cancellation during prefill, + during decode and while a reservation is in flight releases the KV of both + stages, with grouped and with all-layer storage. +- CPU, 8 cases. P and D keep grouped storage, and only D reclaims. An import + reserves only the SWA pages that D reads, sets the read positions for + reclamation, and drops the attached prefix references when the pool is + short. A CPU handoff that copies only the pages D holds matches the oracle, + and D reclamation then continues. An import after a prefix hit on D marks + only the planes each layer owns. PD rejects speculative decoding for Dory + before it starts the workers. + +The CPU tests fail when an import reserves one SWA page too few, or when the +read positions stop reclamation. On two H100 GPUs (torch 2.15.0+cu132, NIXL +1.4.1), the file passes together with `tests/test_pd.py`, the stage engine, +transfer connector, Dory, hybrid KV and SWA reclamation tests (321 passed). +Two `tests/test_pd.py` cases skip: one needs FlashAttention-4 and one needs +four GPUs. + ### Not yet performed - A runtime-to-runtime comparison against vLLM once a Dory-capable build is @@ -406,3 +605,4 @@ Reading the numbers: sampled token-pipeline, adaptive exit, or a draft width above the fixed K. - Sampled speculation on the real checkpoint (the rejection-sampling contract is qualified on the tiny CPU model only). +- PD on the real checkpoint, and PD with speculative decoding. diff --git a/tests/test_backend_selection.py b/tests/test_backend_selection.py index cc90cf90..446300f4 100644 --- a/tests/test_backend_selection.py +++ b/tests/test_backend_selection.py @@ -378,6 +378,8 @@ def plan_cache( speculative=None, staged_prefill=None, fp8_scale=None, + layer_windows=None, + reclaim_swa=False, ): assert selected == "triton" # FA2 cannot use the configured 16-token pages. assert staged_prefill is False diff --git a/tests/test_cdb_runtime.py b/tests/test_cdb_runtime.py index 740572f0..311f3266 100644 --- a/tests/test_cdb_runtime.py +++ b/tests/test_cdb_runtime.py @@ -686,8 +686,18 @@ def test_cuda_graph_reserve_reduces_budget_with_fixed_memory_inputs(monkeypatch, scheduler = SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=3) execution = ExecutionConfig(cuda_graph_memory_reserve_bytes=32 * mib) - _, eager = plan_cache(base, cache, scheduler, execution, "triton") - _, graphed = plan_cache(base, cache, scheduler, replace(execution, cuda_graphs=True), "triton") + _, eager = plan_cache( + base, cache, scheduler, execution, "triton", layer_windows=None, reclaim_swa=False + ) + _, graphed = plan_cache( + base, + cache, + scheduler, + replace(execution, cuda_graphs=True), + "triton", + layer_windows=None, + reclaim_swa=False, + ) assert eager["cuda_graph_reserve_bytes"] == 0 assert graphed["cuda_graph_reserve_bytes"] == 32 * mib diff --git a/tests/test_device.py b/tests/test_device.py index c33f8282..4ea035c2 100644 --- a/tests/test_device.py +++ b/tests/test_device.py @@ -60,9 +60,23 @@ def test_npu_requires_bf16_kv_before_allocation(backend): def test_npu_does_not_use_cpu_default_memory_budget(): model = npu_model_stub() with pytest.raises(ValueError, match="explicit num_blocks"): - plan_cache(model, CacheConfig(), SchedulerConfig(), ExecutionConfig(), "torch") + plan_cache( + model, + CacheConfig(), + SchedulerConfig(), + ExecutionConfig(), + "torch", + layer_windows=None, + reclaim_swa=False, + ) blocks, info = plan_cache( - model, CacheConfig(num_blocks=17), SchedulerConfig(), ExecutionConfig(), "torch" + model, + CacheConfig(num_blocks=17), + SchedulerConfig(), + ExecutionConfig(), + "torch", + layer_windows=None, + reclaim_swa=False, ) assert blocks == 17 and info["source"] == "blocks" diff --git a/tests/test_dory.py b/tests/test_dory.py index 45260837..7b51a95e 100644 --- a/tests/test_dory.py +++ b/tests/test_dory.py @@ -7,6 +7,7 @@ import pytest import torch +from tests.helpers import tiny_ouro_config from tests.reference.dory import dense_dory_reference from vllm_rlt import CacheConfig, ExecutionConfig, SamplingParams, SchedulerConfig from vllm_rlt.attention import BackendCapabilities @@ -14,7 +15,7 @@ from vllm_rlt.core.kv_cache_manager import KVCacheManager from vllm_rlt.engine.llm_engine import LLMEngine from vllm_rlt.kernels.paged_attention import torch_paged_attention -from vllm_rlt.models import AutoModelForCausalLM, DoryConfig, DoryForCausalLM +from vllm_rlt.models import AutoModelForCausalLM, DoryConfig, DoryForCausalLM, OuroForCausalLM ARTIFACT_PAIRS = 36 @@ -111,9 +112,11 @@ def model(): return DoryForCausalLM(tiny_dory_config()).eval() -def make_cache(model, layout=None, block_size=2): +def make_cache(model, layout=None, block_size=2, enable_prefix_caching=False, grouped=False): config = model.config return KVCacheManager( + layer_windows=model.kv_layer_windows if grouped else None, + enable_prefix_caching=enable_prefix_caching, num_layers=config.num_kv_layers, num_kv_heads=config.num_key_value_heads, head_dim=config.head_dim, @@ -128,10 +131,10 @@ def make_cache(model, layout=None, block_size=2): ) -def run_one_token_per_step(model, tokens): +def run_one_token_per_step(model, tokens, cache=None): """Execute the declared reference schedule: one token row at a time.""" config = model.config - cache = make_cache(model) + cache = make_cache(model) if cache is None else cache assert cache.allocate("request", len(tokens)) loops = [[] for _ in range(config.total_ut_steps)] logits = [] @@ -148,10 +151,10 @@ def run_one_token_per_step(model, tokens): return {"logits": torch.stack(logits), "loops": [torch.stack(item) for item in loops]}, cache -def run_all_positions_at_once(model, tokens): +def run_all_positions_at_once(model, tokens, cache=None): """Same schedule, but every stage call carries all causal positions.""" config = model.config - cache = make_cache(model) + cache = make_cache(model) if cache is None else cache assert cache.allocate("request", len(tokens)) positions = list(range(len(tokens))) ids = ["request"] * len(tokens) @@ -690,3 +693,232 @@ def test_dory_packed_prefill_rejects_graphs(model): model, execution_config=ExecutionConfig(packed_prefill=True, cuda_graphs=True), ) + + +def prefill_rows(model, cache, request_id, tokens, start): + """Run positions ``start:`` one row at a time on an allocated request.""" + config = model.config + logits = [] + for position in range(start, len(tokens)): + boundary = cache._prepare_batch([request_id], [0], [position]) + hidden = model.prelude_prepared(tokens[position : position + 1], boundary, cache) + for depth in range(config.total_ut_steps): + core = cache._prepare_batch([request_id], [depth], [position]) + hidden, _ = model.recurrent_prepared(hidden, core, cache) + logits.append(model.coda_prepared(hidden, boundary, cache)[0]) + cache.finalize_token(request_id, position, config.total_ut_steps - 1) + return torch.stack(logits) + + +@pytest.mark.parametrize("kv_mode", ["per_loop", "last_loop"]) +@pytest.mark.parametrize("grouped", [False, True], ids=["legacy", "grouped"]) +@pytest.mark.parametrize("loops", [1, 2, 4]) +def test_prefix_hit_resumes_boundary_and_recurrent_kv(loops, grouped, kv_mode): + torch.manual_seed(42) + model = DoryForCausalLM( + tiny_dory_config(n_recurrent_loops=loops, recurrent_kv_cache_mode=kv_mode) + ).eval() + cache = make_cache(model, enable_prefix_caching=True, grouped=grouped) + # The cached prefix exceeds the four-token sliding attention window. + tokens = torch.tensor([4, 7, 3, 9, 5, 2, 8, 6, 3]) + expected = dense_dory_reference(model, tokens)["logits"] + + assert cache.allocate("cold", len(tokens)) + torch.testing.assert_close( + prefill_rows(model, cache, "cold", tokens, 0), expected, atol=3e-6, rtol=3e-5 + ) + cache.publish_prefix("cold", tokens.tolist(), len(tokens)) + # The last prompt token is always recomputed, so four whole pages publish. + assert len(cache._prefixes) == 4 + cache.free("cold") + + prefix = cache.lookup_prefix(tokens.tolist()) + assert len(prefix) == 4 + assert cache.allocate("hit", len(tokens), prefix=prefix) + written = cache._get_allocation("hit").written + for plane, layers in enumerate(written): + for layer, positions in enumerate(layers): + owns = plane == 0 or layer in model.recurrent_kv_layers + # Boundary KV is restored only where the cold run wrote it. + assert positions.prefix == (8 if owns else 0) + torch.testing.assert_close( + prefill_rows(model, cache, "hit", tokens, 8), expected[8:], atol=3e-6, rtol=3e-5 + ) + cache.free("hit") + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize("grouped", [False, True], ids=["legacy", "grouped"]) +@pytest.mark.parametrize("missing", ["boundary_plane0", "recurrent_plane0", "recurrent_deep_plane"]) +def test_prefix_publication_requires_every_owned_plane(model, missing, grouped): + cache = make_cache(model, enable_prefix_caching=True, grouped=grouped) + tokens = torch.tensor([4, 7, 3, 9, 5]) + assert cache.allocate("request", len(tokens)) + prefill_rows(model, cache, "request", tokens, 0) + layers = cache._get_allocation("request").written + boundary = model.prelude_kv_layers[0] + recurrent = model.recurrent_kv_layers[0] + plane, layer = { + "boundary_plane0": (0, boundary), + "recurrent_plane0": (0, recurrent), + "recurrent_deep_plane": (cache.storage_depths - 1, recurrent), + }[missing] + # Mark page 1 (positions 2-3) incomplete in one plane: only page 0 may publish. + layers[plane][layer].prefix = 2 + cache.publish_prefix("request", tokens.tolist(), len(tokens)) + assert len(cache._prefixes) == 1 + + +def test_grouped_prefix_pages_outlive_the_publishing_request(model): + cache = make_cache(model, enable_prefix_caching=True, grouped=True) + tokens = torch.tensor([4, 7, 3, 9, 5]) + expected = dense_dory_reference(model, tokens)["logits"] + assert cache.allocate("cold", len(tokens)) + prefill_rows(model, cache, "cold", tokens, 0) + cache.publish_prefix("cold", tokens.tolist(), len(tokens)) + published = {b for blocks in cache._prefixes.values() for b in blocks} + # Two token pages, each one physical page per (group, plane, layer). + assert len(published) == 2 * cache.pages_per_token_block + cache.free("cold") + # Allocate another request; the published pages must not be handed out. + assert cache.allocate("filler", len(tokens)) + assert not published & set( + b + for layer in cache.layer_groups + for depth in range(cache.max_loops if cache.layer_group(layer)[1].recurrent else 1) + for b in cache.layer_block_table("filler", layer, depth) + ) + cache.free("filler") + prefix = cache.lookup_prefix(tokens.tolist()) + assert cache.allocate("hit", len(tokens), prefix=prefix) + for layer in cache.layer_groups: + assert cache.retained_range("hit", layer, 0) == (0, 4) + torch.testing.assert_close( + prefill_rows(model, cache, "hit", tokens, 4), expected[4:], atol=3e-6, rtol=3e-5 + ) + cache.free("hit") + assert cache.num_used_blocks == 0 + + +def run_engine(engine, requests, params): + for request_id, prompt in requests: + engine.add_request(request_id, prompt, params) + finished = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + finished[output.request_id] = output.token_ids + return finished + + +def greedy_reference(model, prompt, steps): + tokens = list(prompt) + for _ in range(steps): + logits = dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + return tokens[len(prompt) :] + + +@pytest.mark.parametrize("kv_mode,chunk", [("per_loop", 4), ("last_loop", 3), ("last_loop", 4)]) +@pytest.mark.parametrize("loops", [1, 2, 4]) +def test_engine_prefix_hits_match_cold_greedy_reference(loops, kv_mode, chunk): + torch.manual_seed(42) + model = DoryForCausalLM( + tiny_dory_config(n_recurrent_loops=loops, recurrent_kv_cache_mode=kv_mode) + ).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=512, + block_size=2, + enable_prefix_caching=True, + layout=model.kv_cache_layout, + ), + scheduler_config=SchedulerConfig( + max_num_seqs=2, max_num_batched_tokens=4, prefill_chunk_size=chunk + ), + ) + steps = model.config.total_ut_steps + params = SamplingParams(max_tokens=3, min_loops=steps, max_loops=steps, ignore_eos=True) + shared = [4, 7, 3, 9, 5, 2, 8, 6] + prompts = { + "cold": shared + [3], + # Block-aligned: the last full page is still recomputed. + "aligned": shared + [3, 5], + "branch": shared + [10, 11, 12], + "same": shared + [3], + } + cache = engine.cache_manager + assert run_engine(engine, [("cold", prompts["cold"])], params) == { + "cold": greedy_reference(model, prompts["cold"], 3) + } + assert cache.prefix_hits == 0 + # Run the remaining requests concurrently over the shared published prefix. + finished = run_engine( + engine, [(rid, prompts[rid]) for rid in ("aligned", "branch", "same")], params + ) + for rid in ("aligned", "branch", "same"): + assert finished[rid] == greedy_reference(model, prompts[rid], 3), rid + assert cache.prefix_hits >= 3 * len(shared) + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize( + "execution", [ExecutionConfig(async_scheduling=True), ExecutionConfig(cuda_graphs=True)] +) +def test_last_loop_prefix_caching_requires_sync_eager(execution): + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode="last_loop")).eval() + with pytest.raises(ValueError, match="synchronous eager"): + LLMEngine( + model, + execution_config=execution, + cache_config=CacheConfig( + num_blocks=64, block_size=2, enable_prefix_caching=True, layout="shared" + ), + ) + + +def test_shared_prefix_caching_requires_an_opted_in_model(): + with pytest.raises(ValueError, match="shared-KV prefix caching"): + LLMEngine( + OuroForCausalLM(tiny_ouro_config()), + cache_config=CacheConfig(enable_prefix_caching=True, layout="shared"), + ) + + +@pytest.mark.parametrize("loops", [1, 2, 4]) +def test_evicted_prefix_falls_back_to_cold_prefill(loops): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(n_recurrent_loops=loops)).eval() + steps = model.config.total_ut_steps + # Grouped KV stores one layer per physical page: boundary layers own one + # plane, recurrent layers own ``steps``. A configured block is + # ``num_kv_layers`` physical pages. Each request reserves eleven tokens, + # spanning six token pages; those plus four cached pages exceed the pool's eight. + recurrent = set(model.recurrent_kv_layers) + pages_per_token_page = sum( + steps if layer in recurrent else 1 for layer in model.kv_layer_windows + ) + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=8 * pages_per_token_page // model.config.num_kv_layers, + block_size=2, + enable_prefix_caching=True, + ), + scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=4), + ) + cache = engine.cache_manager + params = SamplingParams(max_tokens=3, min_loops=steps, max_loops=steps, ignore_eos=True) + prompt = [4, 7, 3, 9, 5, 2, 8, 6, 3] + other = [10, 11, 12, 13, 14, 15, 16, 17, 18] + expected = greedy_reference(model, prompt, 3) + + assert run_engine(engine, [("cold", prompt)], params) == {"cold": expected} + assert len(cache.lookup_prefix(prompt)) == 4 + run_engine(engine, [("other", other)], params) + assert cache.lookup_prefix(prompt) == () + hits = cache.prefix_hits + assert run_engine(engine, [("again", prompt)], params) == {"again": expected} + assert cache.prefix_hits == hits + assert cache.num_used_blocks == 0 diff --git a/tests/test_dory_gpu.py b/tests/test_dory_gpu.py index 3a04240f..51ec93be 100644 --- a/tests/test_dory_gpu.py +++ b/tests/test_dory_gpu.py @@ -10,7 +10,13 @@ from tests.reference.dory import dense_dory_reference from tests.test_dory import tiny_dory_config -from vllm_rlt import CacheConfig, ExecutionConfig, SamplingParams, SchedulerConfig +from vllm_rlt import ( + CacheConfig, + ExecutionConfig, + SamplingParams, + SchedulerConfig, + SpeculativeConfig, +) from vllm_rlt.attention import create_backend from vllm_rlt.core.kv_cache_manager import KVCacheManager from vllm_rlt.core.memory import plan_cache @@ -164,18 +170,23 @@ def test_plan_cache_profiles_dory_on_cuda(): model = make_gpu_model() cache = CacheConfig(block_size=2, memory_reserve_bytes=0) scheduler = SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=8) - blocks, plan = plan_cache(model, cache, scheduler, ExecutionConfig(), "triton") + blocks, plan = plan_cache( + model, + cache, + scheduler, + ExecutionConfig(), + "triton", + layer_windows=model.kv_layer_windows, + reclaim_swa=False, + ) assert blocks > 0 # The probe ran the real staged path, including windowed boundary attention. assert plan["source"] == "cuda_profile" config = model.config + # Grouped storage budgets one physical attention-layer page per block. + assert plan["block_unit"] == "physical_layer_page" assert plan["bytes_per_block"] == ( - 2 - * config.num_kv_layers - * cache.block_size - * config.num_key_value_heads - * config.head_dim - * 4 + 2 * cache.block_size * config.num_key_value_heads * config.head_dim * 4 ) @@ -189,10 +200,10 @@ def test_plan_cache_profiles_dory_on_cuda(): PROMPTS = {"a": [4, 7, 3], "b": [9, 2, 6, 5], "c": [8, 3, 3, 11, 5, 2, 7, 13, 6]} -def reference_pair(): - reference = DoryForCausalLM(tiny_dory_config()).eval() +def reference_pair(**overrides): + reference = DoryForCausalLM(tiny_dory_config(**overrides)).eval() torch.manual_seed(42) - model = make_gpu_model() + model = make_gpu_model(**overrides) reference.load_state_dict({name: value.cpu() for name, value in model.state_dict().items()}) return reference, model @@ -510,3 +521,312 @@ def test_dory_packed_prefill_keeps_small_value_residual(): actual = paged_prefill_attention(q, k, v, table, length, offset, 1, compensate=True) assert expected[0, 0, 0] > 0.0006 torch.testing.assert_close(actual, expected, atol=5e-6, rtol=0) + + +@pytest.mark.parametrize("mode", EXECUTION_MODES) +def test_dory_prefix_hits_match_cpu_oracle_in_every_execution_mode(mode): + """Prefix reuse composes with Graph replay and async submission: a cache hit + and a diverging branch serve the same greedy tokens as the dense oracle.""" + reference, model = reference_pair() + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig(num_blocks=512, block_size=2, enable_prefix_caching=True), + scheduler_config=SchedulerConfig(max_num_seqs=3, max_num_batched_tokens=16), + execution_config=ExecutionConfig(**EXECUTION_MODES[mode]), + ) + params = SamplingParams(max_tokens=6, min_loops=2, max_loops=2, ignore_eos=True) + prompt = PROMPTS["c"] + branch = prompt[:6] + [4, 9, 1] + cache = engine.cache_manager + try: + assert cache.grouped is (mode == "sync_eager") + expected = greedy_reference(reference, prompt, 6) + engine.add_request("cold", prompt, params) + assert drain(engine)["cold"].token_ids == expected + assert cache.prefix_hits == 0 + graphs = ( + ( + engine.model_runner.prelude_graphs, + engine.model_runner.graphs, + engine.model_runner.coda_graphs, + ) + if "graph" in mode + else () + ) + replays = [graph.replays for graph in graphs] + + engine.add_request("hit", prompt, params) + engine.add_request("branch", branch, params) + actual = drain(engine) + assert actual["hit"].token_ids == expected + assert actual["branch"].token_ids == greedy_reference(reference, branch, 6) + assert cache.prefix_hits == 14 # Eight shared tokens for hit, six for branch. + assert all(graph.replays > before for graph, before in zip(graphs, replays)) + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +@pytest.mark.parametrize("chunk", [3, 16]) +def test_dory_last_loop_prefix_hits_match_cpu_oracle(chunk): + reference, model = reference_pair(recurrent_kv_cache_mode="last_loop") + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig( + num_blocks=512, block_size=2, enable_prefix_caching=True, layout="shared" + ), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=16, prefill_chunk_size=chunk + ), + ) + params = SamplingParams(max_tokens=6, min_loops=2, max_loops=2, ignore_eos=True) + prompt = PROMPTS["c"] + branch = prompt[:6] + [4, 9, 1] + cache = engine.cache_manager + try: + expected = greedy_reference(reference, prompt, 6) + engine.add_request("cold", prompt, params) + assert drain(engine)["cold"].token_ids == expected + engine.add_request("hit", prompt, params) + engine.add_request("branch", branch, params) + actual = drain(engine) + assert actual["hit"].token_ids == expected + assert actual["branch"].token_ids == greedy_reference(reference, branch, 6) + assert cache.prefix_hits == 14 + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +@pytest.mark.parametrize("speculative_mode", ["block", "pipeline"]) +def test_dory_speculative_prefix_hits_match_cpu_oracle(speculative_mode): + reference, model = reference_pair() + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig(num_blocks=512, block_size=2, enable_prefix_caching=True), + scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=16), + speculative_config=SpeculativeConfig( + 2, draft_loops=1, target_loops=2, mode=speculative_mode + ), + ) + prompt = PROMPTS["c"] + branch = prompt[:6] + [4, 9, 1] + params = SamplingParams(max_tokens=4, min_loops=2, max_loops=2, ignore_eos=True) + try: + cache = engine.cache_manager + assert not cache.grouped + expected = greedy_reference(reference, prompt, 4) + engine.add_request("cold", prompt, params) + assert drain(engine)["cold"].token_ids == expected + assert cache.prefix_hits == 0 + rounds = engine.speculative_runner.stats.rounds + + engine.add_request("hit", prompt, params) + engine.add_request("branch", branch, params) + actual = drain(engine) + assert actual["hit"].token_ids == expected + assert actual["branch"].token_ids == greedy_reference(reference, branch, 4) + assert cache.prefix_hits == 14 + assert engine.speculative_runner.stats.rounds > rounds + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +def test_grouped_static_buffers_keep_three_decode_rows_aligned(): + reference, model = reference_pair() + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig(num_blocks=64, block_size=2), + scheduler_config=SchedulerConfig(max_num_seqs=3, max_num_batched_tokens=3), + execution_config=ExecutionConfig(static_buffers=True, pad_to_power_of_two=True), + ) + prompts = {"a": [4], "b": [7], "c": [3]} + params = SamplingParams(max_tokens=3, min_loops=2, max_loops=2, ignore_eos=True) + try: + assert engine.cache_manager.grouped + assert engine.model_runner.requestspaces + for request_id, prompt in prompts.items(): + engine.add_request(request_id, prompt, params) + actual = drain(engine) + assert set(actual) == set(prompts) + for request_id, prompt in prompts.items(): + assert actual[request_id].token_ids == greedy_reference(reference, prompt, 3) + assert engine.cache_manager.num_used_blocks == 0 + finally: + engine.close() + + +@pytest.mark.parametrize("kv_mode", ["per_loop", "last_loop"]) +def test_grouped_swa_reclamation_uses_cuda_completion_and_reuses_pages(kv_mode, monkeypatch): + model = make_gpu_model(recurrent_kv_cache_mode=kv_mode) + reference = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=kv_mode)).eval() + reference.load_state_dict({name: value.cpu() for name, value in model.state_dict().items()}) + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig( + num_blocks=8, + block_size=2, + layout=model.kv_cache_layout, + incremental_allocation=True, + ), + # One token per chunk preserves last_loop's token-then-loop ordering. + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=1, prefill_chunk_size=1 + ), + ) + prompt = [4, 7, 9, 3, 12, 5, 6, 8, 11, 13, 14, 15] + cache = engine.cache_manager + retired, reused = set(), set() + finish = cache.finish_submission + + def resident_pages(): + return { + page + for layer in model.kv_layer_windows + for depth in ( + range(cache.storage_depths) if layer in model.recurrent_kv_layers else (0,) + ) + for page in cache.layer_block_table("request", layer, depth) + if page >= 0 + } + + def observe_completion(event=None): + assert isinstance(event, torch.cuda.Event) + if not cache.has_allocation("request"): + return finish(event) + before = resident_pages() + reused.update(before & retired) + finish(event) + retired.update(before - resident_pages()) + + monkeypatch.setattr(cache, "finish_submission", observe_completion) + try: + assert cache.grouped and cache.reclaim_swa + engine.add_request( + "request", + prompt, + SamplingParams(max_tokens=4, min_loops=2, max_loops=2, ignore_eos=True), + ) + actual = drain(engine)["request"] + assert actual.token_ids == greedy_reference(reference, prompt, 4) + assert retired and reused + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +class HeldEvent: + """A CUDA event that reports pending until the test releases it.""" + + def __init__(self, event): + self.event, self.released = event, False + + def query(self): + return self.released and self.event.query() + + +@pytest.mark.parametrize("kv_mode", ["per_loop", "last_loop"]) +def test_deferred_prefix_publication_survives_swa_reclamation(kv_mode, monkeypatch): + reference, model = reference_pair(recurrent_kv_cache_mode=kv_mode) + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig( + num_blocks=16, + block_size=2, + layout=model.kv_cache_layout, + incremental_allocation=True, + enable_prefix_caching=True, + ), + # Six-token submissions expire pages of the window-4 layers before they end. + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=6, prefill_chunk_size=6 + ), + ) + cache = engine.cache_manager + held, deferred = [], [] + publish, finish = cache.publish_prefix, cache.finish_submission + + def publish_later(request_id, tokens, length, event=None): + if event is not None: + event = HeldEvent(event) + held.append(event) + return publish(request_id, tokens, length, event) + + def release_then_finish(event=None): + deferred.append(len(cache._pending_prefixes)) + # The submission's own wait completes every earlier publication event. + for pending in held: + pending.released = True + return finish(event) + + monkeypatch.setattr(cache, "publish_prefix", publish_later) + monkeypatch.setattr(cache, "finish_submission", release_then_finish) + params = SamplingParams(max_tokens=4, min_loops=2, max_loops=2, ignore_eos=True) + publisher = [4, 7, 3, 9, 5, 2, 8, 6, 10] + prompt = publisher[:8] + [11, 12, 13] + try: + assert cache.grouped and cache.reclaim_swa + engine.add_request("publisher", publisher, params) + assert drain(engine)["publisher"].token_ids == greedy_reference(reference, publisher, 4) + assert any(deferred) + assert len(cache._prefixes) == 4 + engine.add_request("hit", prompt, params) + assert drain(engine)["hit"].token_ids == greedy_reference(reference, prompt, 4) + assert cache.prefix_hits == 8 + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +def test_packed_prefill_prefix_hits_use_all_layer_blocks(): + """Packed prefill falls back to all-layer pages; a prefix hit matches cold tokens.""" + model = make_gpu_model(head_dim=256, max_position_embeddings=256, sliding_window=32).to( + dtype=torch.bfloat16 + ) + config = model.config + _packed_tile( + config.num_attention_heads, + config.num_key_value_heads, + config.num_kv_layers, + window=config.sliding_window, + ) + engine = LLMEngine( + model, + attention_backend="triton", + cache_config=CacheConfig(num_blocks=64, enable_prefix_caching=True), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=128, prefill_chunk_size=64 + ), + execution_config=ExecutionConfig(packed_prefill=True), + ) + packed_layers = set() + attend = engine.cache_manager._attend_prepared + + def record_packed(layer, metadata, q, *, window=None): + if metadata.cu_seqlens_q is not None: + packed_layers.add(layer) + return attend(layer, metadata, q, window=window) + + engine.cache_manager._attend_prepared = record_packed + params = SamplingParams(max_tokens=4, min_loops=2, max_loops=2, ignore_eos=True) + # Packed prefill applies only to prompts longer than 64 tokens. + prompt = ([4, 7, 9, 3, 12, 5, 23] * 12)[:80] + cache = engine.cache_manager + try: + assert engine.execution_config.packed_prefill and not cache.grouped + engine.add_request("cold", prompt, params) + cold = drain(engine)["cold"].token_ids + assert packed_layers and cache.prefix_hits == 0 + engine.add_request("hit", prompt, params) + assert drain(engine)["hit"].token_ids == cold + assert cache.prefix_hits > 0 + assert cache.num_used_blocks == 0 + finally: + engine.close() diff --git a/tests/test_dory_pd.py b/tests/test_dory_pd.py new file mode 100644 index 00000000..bda2f784 --- /dev/null +++ b/tests/test_dory_pd.py @@ -0,0 +1,385 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Dory same-host PD against the dense oracle and a single-stage engine. + +P runs the input stage, every recurrent loop and the output stage of every +prompt token except the last. D receives that KV and the last token's carried +state, runs the last output stage and samples. Neither side repeats or skips a +stage of the prompt. +""" + +import time + +import pytest +import torch + +from tests.helpers import RecordingConnector +from tests.reference.dory import dense_dory_reference +from tests.test_dory import make_cache, tiny_dory_config +from vllm_rlt import ( + CacheConfig, + ExecutionConfig, + ExitConfig, + SamplingParams, + SchedulerConfig, + SpeculativeConfig, +) +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.engine.llm_engine import LLMEngine +from vllm_rlt.engine.stage_config import StageConfig +from vllm_rlt.engine.stage_engine import StageEngine +from vllm_rlt.models import DoryForCausalLM +from vllm_rlt.pd.config import PDConfig +from vllm_rlt.request import RequestPhase + +LOOPS = 2 +SEED = 123 +KV_MODES = ["per_loop", "last_loop"] +# Prompts longer than the four-token window cross reclaimable SWA pages. +PROMPTS = {"0": [4, 7, 3, 9, 5, 6, 2], "1": [8, 1, 6], "2": [3] * 9} +EXECUTION_MODES = { + "sync_eager": dict(), + "sync_graph": dict(cuda_graphs=True), + "async_eager": dict(async_scheduling=True), + "async_graph": dict(async_scheduling=True, cuda_graphs=True), +} + + +def config(kv_mode): + return tiny_dory_config(n_recurrent_loops=LOOPS, recurrent_kv_cache_mode=kv_mode) + + +def params(max_tokens, **kwargs): + kwargs.setdefault("ignore_eos", True) + return SamplingParams(max_tokens=max_tokens, min_loops=LOOPS, max_loops=LOOPS, **kwargs) + + +def oracle(kv_mode): + """The CPU model that each PD worker builds from the same seed.""" + torch.manual_seed(SEED) + return DoryForCausalLM(config(kv_mode)) + + +def greedy_reference(model, prompt, count): + tokens = list(prompt) + for _ in range(count): + logits = dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + return tokens[len(prompt) :] + + +def drain(engine): + outputs = {} + deadline = time.monotonic() + 90 + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + outputs[output.request_id] = output + assert time.monotonic() < deadline + return outputs + + +def tokens(outputs): + return {rid: output.token_ids for rid, output in outputs.items()} + + +def cache(kv_mode, prefix_caching=False): + return CacheConfig( + num_blocks=128, + block_size=2, + layout=config(kv_mode).kv_cache_layout, + enable_prefix_caching=prefix_caching, + incremental_allocation=True, + ) + + +def create_pd(kv_mode, mode, *, p_cache=False, d_cache=False): + pytest.importorskip("nixl") + from vllm_rlt.pd.engine import PDEngine + + if torch.cuda.device_count() < 2: + pytest.skip("requires 2 visible GPUs") + return PDEngine( + config(kv_mode), + pd_config=PDConfig( + prefill_devices=(0,), + decode_devices=(1,), + max_receiving_requests=3, + max_draining_requests=2, + transfer_chunk_bytes=4096, + max_inflight_bytes=16384, + request_timeout=90, + startup_timeout=90, + ), + prefill_cache_config=cache(kv_mode, p_cache), + decode_cache_config=cache(kv_mode, d_cache), + prefill_scheduler_config=SchedulerConfig( + max_num_seqs=2, max_num_batched_tokens=5, prefill_chunk_size=3 + ), + decode_scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=2), + exit_config=ExitConfig("ouro"), + execution_config=ExecutionConfig(**EXECUTION_MODES[mode]), + attention_backend="triton", + dtype="float32", + seed=SEED, + ) + + +def local_engine(kv_mode): + model = oracle(kv_mode).to("cuda:0") + return LLMEngine( + model, + attention_backend="triton", + cache_config=cache(kv_mode), + scheduler_config=SchedulerConfig( + max_num_seqs=2, max_num_batched_tokens=5, prefill_chunk_size=3 + ), + exit_config=ExitConfig("ouro"), + ) + + +def test_pd_rejects_dory_speculation_before_spawning(): + from vllm_rlt.pd.engine import PDEngine + + with pytest.raises(ValueError, match="boundary-KV speculative decoding"): + PDEngine(config("per_loop"), speculative_config=SpeculativeConfig(3)) + + +def test_import_after_a_prefix_hit_marks_only_the_owned_planes(): + model = oracle("per_loop") + cache = make_cache(model, enable_prefix_caching=True) + prompt = list(range(9)) + assert cache.allocate("source", 12) + cache.mark_imported_prefix("source", len(prompt)) + cache.publish_prefix("source", prompt, len(prompt)) + prefix = cache.lookup_prefix(prompt) + assert len(prefix) * cache.block_size == 8 + # D's own prefix hit leaves boundary layers unwritten in deeper planes. + assert cache.allocate("hit", 12, prefix=prefix) + cache.mark_imported_prefix("hit", len(prompt), start=8) + for plane, layers in enumerate(cache._get_allocation("hit").written): + for layer, written in enumerate(layers): + owned = plane == 0 or layer in model.recurrent_kv_layers + assert written.prefix == (len(prompt) if owned else 0) + + +def planes(model): + boundary = (*model.prelude_kv_layers, *model.coda_kv_layers) + for layer in range(model.config.num_kv_layers): + for depth in (0,) if layer in boundary else range(model.config.total_ut_steps): + yield layer, depth + + +def receiver_cache(model, num_blocks=64): + config = model.config + return KVCacheManager( + num_layers=config.num_kv_layers, + num_kv_heads=config.num_key_value_heads, + head_dim=config.head_dim, + num_blocks=num_blocks, + block_size=2, + max_loops=config.total_ut_steps, + layout=config.kv_cache_layout, + recurrent_layers=model.recurrent_kv_layers, + layer_windows=model.kv_layer_windows, + enable_prefix_caching=True, + incremental_allocation=True, + reclaim_swa=True, + max_submission_tokens=3, + ) + + +def test_import_reserves_only_the_swa_pages_that_decode_reads(): + model = oracle("per_loop") + cache = receiver_cache(model) + prompt = 13 + # A local allocation would compute these tokens, so the bound rejects it. + with pytest.raises(ValueError, match="submission bound"): + cache.allocate("local", 20, initial_tokens=prompt) + assert cache.allocate_import("r", 20, prompt, initial_tokens=prompt) + tables = cache.transfer_tables("r") + assert len(tables) == cache.pages_per_token_block + held = sum(block >= 0 for table in tables for block in table) + assert held == cache.allocated_blocks("r") == cache.num_used_blocks + # The output stage of token 12 reads positions 9 to 12, so pages 0 to 3 stay empty. + for layer, depth in planes(model): + window = model.kv_layer_windows[layer] + assert cache.retained_range("r", layer, depth)[0] == (8 if window else 0) + cache.mark_imported_prefix("r", prompt, output_layers=model.coda_kv_layers) + for plane in cache._get_allocation("r").consumed: + for layer, consumed in enumerate(plane): + assert consumed.prefix == (prompt - 1 if layer in model.coda_kv_layers else prompt) + cache.free("r") + assert cache.num_used_blocks == 0 + + +def test_short_pool_import_drops_the_attached_prefix_references(): + model = oracle("per_loop") + cache = receiver_cache(model, num_blocks=48) + tokens = [5, 6, 7, 8] + assert cache.allocate("source", 4, initial_tokens=3) + cache.mark_imported_prefix("source", 3) + cache.publish_prefix("source", tokens, 3) + cache.free("source") + [page] = cache.lookup_prefix(tokens) + free = cache.num_free_blocks + # Thirty token pages need more fresh pages than the pool holds. + assert not cache.allocate_import("r", 60, 4, initial_tokens=60, prefix=(page,)) + assert not cache.has_allocation("r") and cache.num_free_blocks == free + assert cache.lookup_prefix(tokens) == (page,) + assert cache.allocate_import("r", 8, 4, initial_tokens=4, prefix=(page,)) + assert [table[0] for table in cache.transfer_tables("r")] == list(page) + cache.free("r") + assert cache.num_free_blocks == cache.num_blocks + + +def stage(kv_mode, role): + return StageEngine( + oracle(kv_mode), + cache_config=cache(kv_mode), + scheduler_config=SchedulerConfig( + max_num_seqs=2, max_num_batched_tokens=5, prefill_chunk_size=3 + ), + stage_config=StageConfig(stage_id=role, role=role), + exit_config=ExitConfig("ouro"), + ) + + +@pytest.mark.parametrize("kv_mode", KV_MODES) +def test_cpu_handoff_copies_only_the_retained_swa_pages(kv_mode): + """Copy the pages that both stages hold, as the connector sends them.""" + p, d = stage(kv_mode, "prefill"), stage(kv_mode, "decode") + p.connector = RecordingConnector() + prompt = list(range(1, 14)) + source = p.reserve_transfer("r", prompt, params(6), receiving=False) + target = d.reserve_transfer("r", prompt, params(6), receiving=True) + for _ in range(100): + p.step() + if source.phase == RequestPhase.TRANSFERRING: + break + else: + pytest.fail("source prompt did not complete") + assert d.cache_manager.allocated_blocks("r") < p.cache_manager.allocated_blocks("r") + sent = 0 + for left, right in zip( + p.cache_manager.transfer_tables("r"), d.cache_manager.transfer_tables("r") + ): + for src, dst in zip(left, right): + if dst >= 0: + d.cache_manager.key_cache[dst].copy_(p.cache_manager.key_cache[src]) + d.cache_manager.value_cache[dst].copy_(p.cache_manager.value_cache[src]) + sent += 1 + assert sent < p.cache_manager.allocated_blocks("r") + assert d.accept_import(target, source.hidden_state.clone()) + assert p.release_transfer(source) + windowed = [layer for layer, window in d.model.kv_layer_windows.items() if window] + starts = {layer: d.cache_manager.retained_range("r", layer, 0)[0] for layer in windowed} + final = {} + while len(target.generated_token_ids) < 4: + final.update((o.request_id, o) for o in d.step() if o.finished) + # Reclamation continues from the imported read positions. + for layer in windowed: + assert d.cache_manager.retained_range("r", layer, 0)[0] > starts[layer] + final.update(drain(d)) + assert final["r"].token_ids == greedy_reference(oracle(kv_mode), prompt, 6) + assert p.cache_manager.num_used_blocks == d.cache_manager.num_used_blocks == 0 + + +@pytest.mark.parametrize("role", ["prefill", "decode"]) +def test_pd_stages_keep_grouped_kv_and_only_decode_reclaims(role): + """P holds a prompt only until its transfer completes, so it reclaims nothing.""" + stage = StageEngine( + oracle("per_loop"), + cache_config=cache("per_loop"), + stage_config=StageConfig(stage_id=role, role=role), + exit_config=ExitConfig("ouro"), + ) + assert stage.cache_manager.grouped + assert stage.cache_manager.reclaim_swa is (role == "decode") + + +@pytest.mark.gpu +@pytest.mark.parametrize("kv_mode", KV_MODES) +@pytest.mark.parametrize("mode", EXECUTION_MODES) +def test_dory_pd_matches_the_oracle_and_a_local_engine(kv_mode, mode): + model = oracle(kv_mode) + expected = {rid: greedy_reference(model, prompt, 5) for rid, prompt in PROMPTS.items()} + stochastic = params(5, temperature=0.8, top_k=16, seed=947) + local = local_engine(kv_mode) + for rid, prompt in PROMPTS.items(): + local.add_request(rid, prompt, params(5)) + local.add_request("seeded", PROMPTS["0"], stochastic) + reference = tokens(drain(local)) + with create_pd(kv_mode, mode) as e: + for rid, prompt in PROMPTS.items(): + e.add_request(rid, prompt, params(5)) + e.add_request("seeded", PROMPTS["0"], stochastic) + final = drain(e) + assert tokens(final) == reference + for rid in PROMPTS: + assert final[rid].token_ids == expected[rid] + assert final[rid].exit_depths == [LOOPS] * 5 + # Cancel while a reservation can still be in flight, then reuse the ID. + e.add_request("reuse", PROMPTS["0"], params(5)) + e.step() + e.abort_request("reuse") + e.add_request("reuse", PROMPTS["1"], params(5)) + assert drain(e)["reuse"].token_ids == expected["1"] + assert e.stats()["reserved_blocks"] == 0 + assert all(p.slots == 0 and p.reserved_blocks == 0 for p in e.peers.values()) + assert all(m["used_blocks"] == 0 for m in e.worker_metrics.values()) + assert sum(m["bytes_sent"] for m in e.worker_metrics.values()) > 0 + + +@pytest.mark.gpu +@pytest.mark.parametrize("p_cache,d_cache", [(True, True), (True, False), (False, True)]) +@pytest.mark.parametrize("mode", ["sync_eager", "async_graph"]) +def test_dory_pd_prefix_reuse_skips_work_and_keeps_outputs(p_cache, d_cache, mode): + prompt = [3, 1, 4, 1, 5, 9, 2, 6, 5] + expected = greedy_reference(oracle("per_loop"), prompt, 6) + with create_pd("per_loop", mode, p_cache=p_cache, d_cache=d_cache) as e: + for round_index in range(2): + for i in range(2): + e.add_request(f"{round_index}-{i}", prompt, params(6)) + assert all(t == expected for t in tokens(drain(e)).values()) + assert all(p.slots == 0 and p.reserved_blocks == 0 for p in e.peers.values()) + stats = list(e.worker_metrics.values()) + p = next(x for x in stats if x["prefill_tokens"]) + assert (p["prefill_tokens"] < 4 * len(prompt)) == p_cache + assert all(x["used_blocks"] == 0 for x in stats) + + +@pytest.mark.gpu +def test_grouped_pd_sends_less_kv_than_all_layer_pd(): + """Grouped PD skips boundary layers in deeper planes and expired SWA pages.""" + prompt = list(range(1, 41)) + expected = greedy_reference(oracle("per_loop"), prompt, 4) + sent = {} + for mode in ("sync_eager", "async_eager"): + with create_pd("per_loop", mode) as e: + e.add_request("long", prompt, params(4)) + assert drain(e)["long"].token_ids == expected + sent[mode] = sum(m["bytes_sent"] for m in e.worker_metrics.values()) + assert 0 < sent["sync_eager"] < sent["async_eager"] + + +@pytest.mark.gpu +@pytest.mark.parametrize("mode", ["sync_eager", "async_eager"]) +@pytest.mark.parametrize("phase", ["prefill", "decode"]) +def test_dory_pd_cancel_releases_both_stages(phase, mode): + with create_pd("per_loop", mode) as e: + e.add_request("cancel", [1] * 40, params(20)) + deadline = time.monotonic() + 30 + while True: + e.step() + w = next(iter(e.transfers.values())) + if (phase == "prefill" and "prefill_started" in w.timings) or ( + phase == "decode" and w.phase == "decode" + ): + break + assert time.monotonic() < deadline + e.abort_request("cancel") + assert drain(e) == {} + assert all(p.slots == 0 and p.reserved_blocks == 0 for p in e.peers.values()) + e.add_request("after", PROMPTS["0"], params(5)) + assert drain(e)["after"].token_ids == greedy_reference(oracle("per_loop"), PROMPTS["0"], 5) + assert all(m["used_blocks"] == 0 for m in e.worker_metrics.values()) diff --git a/tests/test_dory_speculative.py b/tests/test_dory_speculative.py index e52b2832..a3265f3d 100644 --- a/tests/test_dory_speculative.py +++ b/tests/test_dory_speculative.py @@ -64,6 +64,57 @@ def greedy_reference(model, prompt, count): return tokens +@pytest.mark.parametrize("mode", ["block", "pipeline"]) +def test_prefix_hits_keep_speculative_outputs_on_all_layer_kv(mode): + model, count = dory(2), 5 + prompt = [4, 7, 3, 9, 5, 2, 8, 6, 3] + branch = prompt[:6] + [10, 11, 12] + sampling = params(model, count) + expected = greedy_reference(model, prompt, count)[len(prompt) :] + e = engine( + model, + 2, + 1, + mode, + cache_config=CacheConfig(num_blocks=256, block_size=2, enable_prefix_caching=True), + scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=4), + ) + try: + cache = e.cache_manager + assert not cache.grouped + assert e.memory_plan["block_unit"] == "all_layer_block" + e.add_request("cold", prompt, sampling) + assert drain(e)["cold"].token_ids == expected + assert cache.prefix_hits == 0 + + rounds = e.speculative_runner.stats.rounds + e.add_request("hit", prompt, sampling) + e.add_request("branch", branch, sampling) + actual = drain(e) + assert actual["hit"].token_ids == expected + assert actual["branch"].token_ids == greedy_reference(model, branch, count)[len(branch) :] + assert cache.prefix_hits == 14 # Eight shared tokens for hit, six for branch. + assert e.speculative_runner.stats.rounds > rounds + assert cache.num_used_blocks == 0 + finally: + e.close() + + +@pytest.mark.parametrize("mode", ["block", "pipeline"]) +def test_speculation_uses_all_layer_kv_instead_of_grouped_reclamation(mode): + e = engine( + dory(2), + 2, + 1, + mode, + cache_config=CacheConfig(num_blocks=256, block_size=2, incremental_allocation=True), + ) + try: + assert not e.cache_manager.grouped and not e.cache_manager.reclaim_swa + finally: + e.close() + + def test_round_layouts_align_partial_coda_rows_and_upload_once(monkeypatch): model = dory(4) e = engine(model, 3, 2, scheduler_config=SchedulerConfig(max_num_seqs=2)) diff --git a/tests/test_engine.py b/tests/test_engine.py index e738d8f4..98c26ebb 100644 --- a/tests/test_engine.py +++ b/tests/test_engine.py @@ -312,6 +312,13 @@ def test_invalid_request_id_does_not_poison_admission(request_id): assert drain(engine)[0]["valid"].finished +def test_cache_config_preserves_existing_positional_arguments(): + config = CacheConfig(64, 2, "last_exited", 0.9, None, 256 * 1024 * 1024, False, True, 0.25) + assert config == CacheConfig( + num_blocks=64, block_size=2, incremental_allocation=True, watermark_ratio=0.25 + ) + + @pytest.mark.parametrize("invalid", [1.5, float("nan"), float("inf"), True, 0]) def test_integer_limits_rejected_before_scheduling(invalid): for constructor, name in [ diff --git a/tests/test_hybrid_kv.py b/tests/test_hybrid_kv.py new file mode 100644 index 00000000..12361962 --- /dev/null +++ b/tests/test_hybrid_kv.py @@ -0,0 +1,402 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Native Dory storage: numerical dataflow, ownership and physical-page budgets.""" + +import pytest +import torch + +from tests.reference import dory as reference +from tests.test_dory import ( + assert_matches_reference, + run_all_positions_at_once, + run_one_token_per_step, + tiny_dory_config, +) +from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig +from vllm_rlt.config import ExecutionConfig +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.core.memory import plan_cache +from vllm_rlt.engine.llm_engine import LLMEngine +from vllm_rlt.models import DoryForCausalLM +from vllm_rlt.worker.metadata import GroupedKVMetadata, MetadataBuilder + + +def native_cache(model, blocks=48): + config = model.config + return KVCacheManager( + num_layers=config.num_kv_layers, + num_kv_heads=config.num_key_value_heads, + head_dim=config.head_dim, + num_blocks=blocks, + block_size=2, + max_loops=config.total_ut_steps, + layout=config.kv_cache_layout, + recurrent_layers=model.recurrent_kv_layers, + layer_windows=model.kv_layer_windows, + ) + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +def test_native_dory_matches_unchanged_dense_reference_and_kv(monkeypatch, mode): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + tokens = torch.tensor([4, 7, 9, 3, 12, 5]) + histories = [] + original = reference._History + + class ObservedHistory(original): + def __init__(self): + super().__init__() + histories.append(self) + + # Observe the existing oracle's dense K/V without changing its operations. + monkeypatch.setattr(reference, "_History", ObservedHistory) + expected = reference.dense_dory_reference(model, tokens) + cache = native_cache(model) + actual, cache = run_one_token_per_step(model, tokens, cache=cache) + assert_matches_reference(actual, expected) + for (physical_layer, plane), entries in histories[0].entries.items(): + layer = model.config.kv_layer_index(physical_layer) + keys, values = cache.read(layer, "request", plane) + torch.testing.assert_close( + keys, torch.stack([entries[p][0] for p in range(6)]), atol=3e-6, rtol=3e-5 + ) + torch.testing.assert_close( + values, torch.stack([entries[p][1] for p in range(6)]), atol=3e-6, rtol=3e-5 + ) + slots = 6 if mode == "per_loop" else 4 + assert cache.allocated_blocks("request") == cache.num_used_blocks == 3 * slots + tables = [] + for layer in model.kv_layer_windows: + depths = range(2) if layer in model.recurrent_kv_layers and mode == "per_loop" else (0,) + for depth in depths: + tables.extend(cache.layer_block_table("request", layer, depth)) + assert len(set(tables)) == 3 * slots + assert set(cache.layer_groups) == {0, 1, 2, 3} + assert model.config.attention_layer_ids == (0, 2, 4, 6) + assert len(cache.groups) == 3 # SWA boundary, FA recurrent, SWA recurrent. + with pytest.raises(ValueError, match="no attention KV"): + model.config.kv_layer_index(1) + with pytest.raises(ValueError, match="layer must be"): + cache.read(4, "request", 0) + with pytest.raises(ValueError, match="depth zero"): + cache.read(0, "request", 1) + if mode == "last_loop": + assert cache.layer_block_table("request", 2, 0) == cache.layer_block_table("request", 2, 1) + assert cache.token_written("request", 5, 1) + cache.free("request") + assert cache.num_used_blocks == 0 + + +def test_native_per_loop_batched_queries_and_sliced_metadata(): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config()).eval() + tokens = torch.tensor([4, 7, 9, 3, 12, 5]) + cache = native_cache(model) + boundary_id, boundary = cache.layer_group(0) + assert cache.layer_group(3) == (boundary_id, boundary) + assert boundary.layers == (0, 3) + assert (boundary.window, boundary.recurrent, boundary.planes) == (4, False, 1) + swa_id, swa = cache.layer_group(2) + assert swa_id != boundary_id + assert (swa.window, swa.recurrent, swa.planes) == (4, True, 2) + assert model.config.kv_layer_index(4) == 2 + assert cache.layer_group(1)[1].window is None + actual, cache = run_all_positions_at_once(model, tokens, cache=cache) + assert_matches_reference(actual, reference.dense_dory_reference(model, tokens)) + batch = cache._prepare_batch(["request"] * 2, [0, 0], [4, 5], padded_rows=3) + assert isinstance(batch, GroupedKVMetadata) + assert not hasattr(batch, "write_blocks") + view = cache.layer_view(batch, 1) + assert cache.layer_view(batch, 1) is view + assert view.write_blocks.tolist() == [cache.layer_block_table("request", 1, 0)[2]] * 2 + assert view.write_offsets.tolist() == [0, 1] + assert view.context_lengths.tolist() == [5, 6, 0] + assert view.position_ids.tolist() == [4, 5, 0] + windowed = cache.layer_view(batch, 0) + table = cache.layer_block_table("request", 0, 0) + assert windowed.context_lengths.tolist() == [5, 4, 0] + assert windowed.position_ids.tolist() == [4, 5, 0] + assert windowed.block_tables.tolist() == [list(table[:3]), [*table[1:3], -1], [0, 0, 0]] + assert windowed.write_blocks.tolist() == [table[2], table[2]] + assert windowed.write_offsets.tolist() == [0, 1] + sliced = MetadataBuilder.slice_rows(batch, 1, 2) + assert not sliced.layer_views and len(sliced.rows) == 1 + assert cache.layer_view(sliced, 2).write_blocks.tolist() == [ + cache.layer_block_table("request", 2, 0)[2] + ] + assert len(MetadataBuilder.without_padding(batch).position_ids) == 2 + with pytest.raises(NotImplementedError, match="ordinary layer views"): + cache.get_block_table("request", 0) + cache.free("request") + assert cache.allocate("request", 6) + with pytest.raises(RuntimeError, match="stale prepared"): + cache.layer_view(batch, 1) + + +@pytest.mark.parametrize("loops", [1, 2, 4]) +@pytest.mark.parametrize("tail_pages", [0, 1, 3]) +def test_grouped_prefix_import_preserves_page_order_and_shared_ownership(loops, tail_pages): + cached_pages, block_size = 64, 2 + recurrent = (2, 3, 4, 5) + # Every group has multiple layers; recurrent groups also have multiple + # planes. Distinct data per layer/plane detects any flattening mismatch. + windows = {layer: (4 if layer % 2 == 0 else None) for layer in range(8)} + pages_per_block = 4 + 4 * loops + pool_pages = max(cached_pages + 1, cached_pages + 2 * tail_pages) + cache = KVCacheManager( + num_layers=8, + num_kv_heads=1, + head_dim=2, + num_blocks=pool_pages * pages_per_block, + block_size=block_size, + max_loops=loops, + recurrent_layers=recurrent, + layer_windows=windows, + enable_prefix_caching=True, + ) + tokens = list(range(cached_pages * block_size + 1)) + assert cache.allocate("source", len(tokens)) + expected_tables = {} + for layer in windows: + for depth in range(loops) if layer in recurrent else (0,): + values = torch.full((len(tokens), 1, 2), float(layer * loops + depth)) + cache.write( + layer, ["source"] * len(tokens), [depth] * len(tokens), tokens, values, values + 100 + ) + expected_tables[layer, depth] = cache.layer_block_table("source", layer, depth)[ + :cached_pages + ] + cache.publish_prefix("source", tokens, len(tokens)) + prefix = cache.lookup_prefix(tokens) + assert len(prefix) == cached_pages + cache.free("source") + + capacity = (cached_pages + tail_pages) * block_size + fresh_pages = set() + for request_id in ("hit", "sharer"): + assert cache.allocate(request_id, capacity, prefix=prefix) + for (layer, depth), expected in expected_tables.items(): + table = cache.layer_block_table(request_id, layer, depth) + assert table[:cached_pages] == expected + assert len(table) == cached_pages + tail_pages + suffix = set(table[cached_pages:]) + assert not suffix & fresh_pages + fresh_pages.update(suffix) + keys, values = cache.read(layer, request_id, depth) + expected_keys = torch.full_like(keys, float(layer * loops + depth)) + torch.testing.assert_close(keys, expected_keys) + torch.testing.assert_close(values, expected_keys + 100) + assert cache.num_used_blocks == (cached_pages + 2 * tail_pages) * pages_per_block + if tail_pages: + # A failed third import must roll back its shared-page references. + assert not cache.allocate("blocked", capacity, prefix=prefix) + assert not cache.has_allocation("blocked") + cache.free("hit") + cache.free("sharer") + assert cache.num_used_blocks == 0 + # Evict every cached page to prove no import left a reference behind. + assert cache.allocate("evict", pool_pages * block_size) + cache.free("evict") + assert cache.num_free_blocks == cache.num_blocks + + +def test_native_atomic_growth_failure_free_and_early_exit_copy(): + model = DoryForCausalLM(tiny_dory_config()).eval() + cache = native_cache(model, blocks=18) + assert cache.required_blocks(6) == 18 + assert cache.allocate("a", 6, initial_tokens=2) + assert cache.allocate("b", 6, initial_tokens=2) + assert cache.num_used_blocks == 12 + before = tuple(cache.layer_block_table("a", layer, 0) for layer in model.kv_layer_windows) + assert not cache.ensure_capacity("a", 6) + assert cache.num_used_blocks == 12 + assert before == tuple( + cache.layer_block_table("a", layer, 0) for layer in model.kv_layer_windows + ) + assert cache.ensure_capacity("a", 4) + assert cache.num_used_blocks == 18 + assert not cache.allocate("c", 6, initial_tokens=2) + assert not cache.has_allocation("c") + cache.free("b") + assert cache.ensure_capacity("a", 6) + for layer in model.kv_layer_windows: + values = torch.full((6, 2, 8), float(layer + 1)) + cache.write(layer, ["a"] * 6, [0] * 6, list(range(6)), values, values + 10) + for position in range(6): + cache.finalize_token("a", position, 0) + for layer in model.recurrent_kv_layers: + k0, v0 = cache.read(layer, "a", 0) + k1, v1 = cache.read(layer, "a", 1) + torch.testing.assert_close(k0, k1, atol=0, rtol=0) + torch.testing.assert_close(v0, v1, atol=0, rtol=0) + assert cache.token_written("a", 5, 1) + cache.free("a") + assert cache.num_used_blocks == 0 and cache.num_free_blocks == 18 + assert cache.allocate("c", 6) + assert not cache.token_written("c", 0, 0) + cache.free("c") + assert cache.num_free_blocks == 18 + + +def test_native_planner_preserves_configured_all_layer_byte_budget(): + model = DoryForCausalLM(tiny_dory_config()).eval() + scheduler, execution = SchedulerConfig(), ExecutionConfig() + native = native_cache(model) + page_bytes = 2 * 2 * 2 * 8 * 4 + assert native.bytes_per_block == page_bytes + windows = model.kv_layer_windows + blocks, plan = plan_cache( + model, + CacheConfig(num_blocks=3, block_size=2), + scheduler, + execution, + "torch", + layer_windows=windows, + reclaim_swa=False, + ) + assert model.config.num_kv_layers == 4 < model.config.num_hidden_layers + assert blocks == 3 * model.config.num_kv_layers + assert blocks * plan["bytes_per_block"] == 3 * 4 * page_bytes + assert plan["block_unit"] == "physical_layer_page" and plan["configured_num_blocks"] == 3 + blocks, plan = plan_cache( + model, + CacheConfig(kv_cache_memory_bytes=24 * page_bytes, block_size=2), + scheduler, + execution, + "torch", + layer_windows=windows, + reclaim_swa=False, + ) + assert blocks == 24 and plan["bytes_per_block"] == page_bytes + blocks, _ = plan_cache( + model, + CacheConfig(block_size=2), + scheduler, + execution, + "torch", + layer_windows=windows, + reclaim_swa=False, + ) + assert blocks == 256 * 4 + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +def test_native_engine_generation_uses_grouped_storage(mode): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=8, block_size=2, layout=model.config.kv_cache_layout), + scheduler_config=SchedulerConfig(max_num_seqs=2, max_num_batched_tokens=4), + ) + cache = engine.cache_manager + assert cache.grouped and cache.num_blocks == 8 * 4 + assert cache.key_cache.shape[1] == 1 + prompt = [4, 7, 3] + params = SamplingParams(max_tokens=3, min_loops=2, max_loops=2, ignore_eos=True) + engine.add_request("request", prompt, params) + result = None + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + result = output + assert result is not None + tokens = list(prompt) + for _ in range(3): + logits = reference.dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + assert result.token_ids == tokens[len(prompt) :] + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +@pytest.mark.parametrize("loops", [1, 2, 4]) +def test_grouped_static_buffers_with_three_padded_rows_match_dense_reference(mode, loops): + torch.manual_seed(42) + model = DoryForCausalLM( + tiny_dory_config(recurrent_kv_cache_mode=mode, n_recurrent_loops=loops) + ).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=64, block_size=2, layout=model.kv_cache_layout), + scheduler_config=SchedulerConfig(max_num_seqs=3, max_num_batched_tokens=3), + execution_config=ExecutionConfig(static_buffers=True, pad_to_power_of_two=True), + ) + prompts = {"a": [4], "b": [7], "c": [3]} + try: + cache = engine.cache_manager + assert cache.grouped + assert engine.model_runner.requestspaces + for request_id, prompt in prompts.items(): + engine.add_request( + request_id, + prompt, + SamplingParams(max_tokens=3, min_loops=loops, max_loops=loops, ignore_eos=True), + ) + completed = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + if output.finished: + completed[output.request_id] = output.token_ids + assert set(completed) == set(prompts) + for request_id, prompt in prompts.items(): + tokens = list(prompt) + for _ in range(3): + logits = reference.dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + assert completed[request_id] == tokens[len(prompt) :] + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +def test_preemption_selects_all_layer_kv_and_restores_a_prefix_hit(): + model = DoryForCausalLM(tiny_dory_config()).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=64, block_size=2, enable_prefix_caching=True), + scheduler_config=SchedulerConfig(enable_preemption=True), + ) + cache = engine.cache_manager + greedy = SamplingParams(max_tokens=6, temperature=0, ignore_eos=True) + prompt = [3, 1, 4, 1, 5, 9, 2, 6, 5] + + def finish(): + outputs = {} + while engine.has_unfinished_requests(): + for output in engine.step(): + outputs[output.request_id] = output.token_ids + return outputs + + try: + assert not cache.grouped + engine.add_request("cold", prompt, greedy) + cold = finish()["cold"] + engine.add_request("hit", prompt, greedy) + hit = engine.scheduler.requests["hit"] + while len(hit.generated_token_ids) < 2: + engine.step() + assert cache.prefix_hits > 0 + engine.add_request("other", [7], SamplingParams(max_tokens=1)) + assert engine.scheduler.preempt(engine.scheduler.requests["other"]) + assert engine.preemption.snapshots["hit"]["kv"] is not None + assert finish()["hit"] == cold + assert engine.preemption.resumptions == 1 + assert cache.num_used_blocks == 0 + finally: + engine.close() + + +def test_non_ouro_exit_keeps_grouped_storage_without_reclamation(): + from vllm_rlt.config import ExitConfig + + model = DoryForCausalLM(tiny_dory_config()).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=64, block_size=2), + exit_config=ExitConfig(mode="ouro_delayed"), + ) + try: + assert engine.cache_manager.grouped and not engine.cache_manager.reclaim_swa + finally: + engine.close() diff --git a/tests/test_prefix_growth.py b/tests/test_prefix_growth.py index 9fe717c6..4e9062de 100644 --- a/tests/test_prefix_growth.py +++ b/tests/test_prefix_growth.py @@ -1,3 +1,6 @@ +import hashlib +import weakref + import pytest import torch @@ -66,6 +69,228 @@ def test_prefix_refcounts_and_eviction(): assert c.num_free_blocks == 12 +def test_completed_chunks_hash_each_prefix_page_once(monkeypatch): + count, block_size = 8192, 16 + cache = KVCacheManager(1, 1, 4, count // block_size, block_size, 1, enable_prefix_caching=True) + tokens = list(range(count)) + assert cache.allocate("request", count) + cache.mark_imported_prefix("request", count) + hash_calls = 0 + sha256 = hashlib.sha256 + + def counted(data): + nonlocal hash_calls + hash_calls += 1 + return sha256(data) + + monkeypatch.setattr(hashlib, "sha256", counted) + for length in range(128, count + 1, 128): + cache.publish_prefix("request", tokens, length, PrefixEvent(ready=True)) + cache.publish_prefix("request", tokens, count) + assert hash_calls == (count - 1) // block_size + assert len(cache.lookup_prefix(tokens)) == (count - 1) // block_size + + +@pytest.mark.parametrize("missing", ["coverage", "physical_page"]) +def test_prefix_publication_retries_the_first_incomplete_page(missing): + cache = KVCacheManager( + 2, + 1, + 4, + 12, + 2, + 2, + enable_prefix_caching=True, + recurrent_layers=(1,), + layer_windows={0: 4, 1: None}, + ) + tokens = list(range(7)) + assert cache.allocate( + "request", len(tokens), initial_tokens=2 if missing == "physical_page" else 7 + ) + allocation = cache._get_allocation("request") + for plane, layers in enumerate(allocation.written): + for layer, written in enumerate(layers): + if plane == 0 or layer == 1: + written.prefix = len(tokens) + if missing == "coverage": + allocation.written[1][1].prefix = 2 + + cache.publish_prefix("request", tokens, len(tokens)) + assert len(cache.lookup_prefix(tokens)) == 1 + if missing == "physical_page": + assert cache.ensure_capacity("request", len(tokens)) + else: + allocation.written[1][1].prefix = len(tokens) + cache.publish_prefix("request", tokens, len(tokens)) + assert len(cache.lookup_prefix(tokens)) == 3 + + +def test_active_request_republishes_its_prefix_after_lru_eviction(): + cache = KVCacheManager(1, 1, 4, 10, 2, 1, enable_prefix_caching=True) + tokens, other = list(range(7)), list(range(100, 107)) + assert cache.allocate("publisher", len(tokens)) + cache.mark_imported_prefix("publisher", len(tokens)) + cache.publish_prefix("publisher", tokens, len(tokens)) + cache.free("publisher") + assert cache.allocate("source", len(tokens), prefix=cache.lookup_prefix(tokens)) + cache.publish_prefix("source", tokens, len(tokens)) + assert cache.allocate("other", len(other)) + cache.mark_imported_prefix("other", len(other)) + cache.publish_prefix("other", other, len(other)) + cache.free("other") + + # Reaching the evictable pages also removes the older source entries, + # whose physical pages remain pinned by the live request. + assert cache.allocate("pressure", 10) + assert not cache.lookup_prefix(tokens) + cache.publish_prefix("source", tokens, len(tokens)) + assert len(cache.lookup_prefix(tokens)) == 3 + cache.free("source") + cache.free("pressure") + assert cache.num_free_blocks == cache.num_blocks + + +class PrefixEvent: + def __init__(self, ready=False): + self.ready = ready + self.queries = 0 + + def query(self): + self.queries += 1 + return self.ready + + def synchronize(self): + raise AssertionError("prefix publication must not wait for an event") + + +@pytest.mark.parametrize("leased", [False, True]) +def test_free_drops_pending_prefix_without_polling_or_waiting(leased): + cache = KVCacheManager(1, 1, 4, 6, 2, 1, enable_prefix_caching=True) + tokens = [1, 2, 3, 4, 5] + events = [] + for request_id in ("request", "other"): + assert cache.allocate(request_id, len(tokens)) + event = PrefixEvent() + events.append(event) + cache.publish_prefix(request_id, tokens, len(tokens), event) + owner = weakref.ref(cache._get_allocation("request")) + if leased: + cache.pin_transfer("request", "transfer") + cache.free("request") + if leased: + assert cache.has_allocation("request") + assert len(cache._pending_prefixes) == 2 + cache.unpin_transfer("request", "transfer") + assert owner() is None + assert not cache.has_allocation("request") + assert len(cache._pending_prefixes) == 1 + assert cache._pending_prefixes[0][0] == "other" + + assert cache.allocate("request", len(tokens)) + events.append(PrefixEvent()) + cache.publish_prefix("request", tokens, len(tokens), events[-1]) + cache.free("unknown") + assert len(cache._pending_prefixes) == 2 + cache.free("request") + cache.free("request") + assert len(cache._pending_prefixes) == 1 + assert cache._pending_prefixes[0][0] == "other" + cache.free("other") + assert not cache._pending_prefixes + assert all(event.queries == 1 for event in events) + assert cache.num_free_blocks == cache.num_blocks + + +def test_completed_prefix_event_publishes_without_pending_storage(): + cache = KVCacheManager(1, 1, 4, 6, 2, 1, enable_prefix_caching=True) + tokens = [1, 2, 3, 4, 5] + assert cache.allocate("request", len(tokens)) + cache.mark_imported_prefix("request", len(tokens)) + + cache.publish_prefix("request", tokens, len(tokens), PrefixEvent(ready=True)) + + assert not cache._pending_prefixes + assert len(cache.lookup_prefix(tokens)) == 2 + + +def test_pending_prefix_chunks_share_one_prompt_and_do_not_rescan_events(): + count, chunk_size = 8192, 128 + cache = KVCacheManager(1, 1, 4, count // 16, 16, 1, enable_prefix_caching=True) + tokens = list(range(count)) + assert cache.allocate("request", count) + cache.mark_imported_prefix("request", count) + events = [] + for length in range(chunk_size, count + 1, chunk_size): + event = PrefixEvent() + events.append(event) + cache.publish_prefix("request", tokens, length, event) + + assert len(cache._pending_prefixes) == 1 + _, _, snapshot, frontier, pending_events = cache._pending_prefixes[0] + assert snapshot == tuple(tokens) + assert frontier == count + assert len(pending_events) == len(events) + assert all(event.queries == 1 for event in events) + assert not cache._prefixes + + cache.poll_prefixes() + assert cache._pending_prefixes[0][2] is snapshot + assert all(event.queries == 2 for event in events) + for event in events: + event.ready = True + cache.poll_prefixes() + assert not cache._pending_prefixes + assert len(cache.lookup_prefix(tokens)) == (count - 1) // cache.block_size + + +def test_pending_prefix_waits_for_all_events_and_keeps_the_largest_frontier(): + cache = KVCacheManager(1, 1, 4, 8, 2, 1, enable_prefix_caching=True) + tokens = list(range(9)) + assert cache.allocate("request", len(tokens)) + cache.mark_imported_prefix("request", len(tokens)) + earlier, later = PrefixEvent(), PrefixEvent() + cache.publish_prefix("request", tokens, 4, earlier) + cache.publish_prefix("request", tokens, 8, later) + # A completed submission with a smaller frontier must not discard either + # outstanding writer or the larger prefix awaiting publication. + cache.publish_prefix("request", tokens, 2, PrefixEvent(ready=True)) + later.ready = True + cache.poll_prefixes() + assert not cache._prefixes + assert cache._pending_prefixes[0][3] == 8 + assert cache._pending_prefixes[0][4] == [earlier] + + earlier.ready = True + cache.poll_prefixes() + assert not cache._pending_prefixes + assert len(cache.lookup_prefix(tokens)) == 4 + + +def test_pending_prefix_does_not_publish_a_replaced_allocation(): + cache = KVCacheManager(1, 1, 4, 6, 2, 1, enable_prefix_caching=True) + old_tokens, new_tokens = [1, 2, 3, 4, 5], [6, 7, 8, 9, 10] + assert cache.allocate("request", len(old_tokens)) + cache.mark_imported_prefix("request", len(old_tokens)) + old_event = PrefixEvent() + cache.publish_prefix("request", old_tokens, len(old_tokens), old_event) + cache.free("request") + assert cache.allocate("request", len(new_tokens)) + cache.mark_imported_prefix("request", len(new_tokens)) + new_event = PrefixEvent() + cache.publish_prefix("request", new_tokens, len(new_tokens), new_event) + + old_event.ready = True + cache.poll_prefixes() + assert not cache._prefixes + assert len(cache._pending_prefixes) == 1 + assert cache._pending_prefixes[0][2] == tuple(new_tokens) + new_event.ready = True + cache.poll_prefixes() + assert len(cache.lookup_prefix(new_tokens)) == 2 + assert not cache.lookup_prefix(old_tokens) + + def test_incremental_growth_matches_reserved_generation(): m = model() p = SamplingParams(max_tokens=9, exit_threshold=0, ignore_eos=True) diff --git a/tests/test_swa_reclaim.py b/tests/test_swa_reclaim.py new file mode 100644 index 00000000..203ae318 --- /dev/null +++ b/tests/test_swa_reclaim.py @@ -0,0 +1,701 @@ +# SPDX-License-Identifier: Apache-2.0 +"""Ordinary fixed-loop SWA reclamation, including shared final-depth readers.""" + +import math + +import pytest +import torch + +from tests.reference.dory import dense_dory_reference +from tests.test_dory import run_engine, tiny_dory_config +from vllm_rlt import CacheConfig, SamplingParams, SchedulerConfig +from vllm_rlt.core.kv_cache_manager import KVCacheManager +from vllm_rlt.engine.llm_engine import LLMEngine +from vllm_rlt.models import DoryForCausalLM +from vllm_rlt.request import RequestPhase + + +def live_pages(cache, model, request_id, *, sliding): + pages = set() + for layer, window in model.kv_layer_windows.items(): + if (window is not None) != sliding: + continue + depths = range(cache.storage_depths) if layer in model.recurrent_kv_layers else (0,) + for depth in depths: + pages.update( + page for page in cache.layer_block_table(request_id, layer, depth) if page >= 0 + ) + return pages + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +@pytest.mark.parametrize("hit_tokens", [2, 8]) +# A 6-token submission retires pages of the window-4 layers before it ends. +@pytest.mark.parametrize("submission", [4, 6]) +def test_reclaim_continues_after_prefix_hit(monkeypatch, mode, hit_tokens, submission): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=8 if submission == 4 else 16, + block_size=2, + layout=model.kv_cache_layout, + incremental_allocation=True, + enable_prefix_caching=True, + ), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=submission, prefill_chunk_size=6 + ), + ) + cache = engine.cache_manager + publisher = [4, 7, 3, 9, 5, 2, 8, 6, 10] + prompt = publisher[:hit_tokens] + [11, 12, 13, 14, 15, 16] + params = SamplingParams(max_tokens=2, min_loops=2, max_loops=2, ignore_eos=True) + run_engine(engine, [("publisher", publisher)], params) + assert cache.prefix_hits == 0 + assert len(cache._prefixes) == 4 + assert cache.num_used_blocks == 0 + + window_layer = next(layer for layer, window in model.kv_layer_windows.items() if window) + full_layer = next(layer for layer, window in model.kv_layer_windows.items() if window is None) + frontiers = [] + finish_submission = cache.finish_submission + + def record_reclamation(event=None): + if not cache.has_allocation("hit"): + return finish_submission(event) + if not frontiers: + # The hit starts retaining only what its next token can read. + frontiers.append(cache.retained_range("hit", window_layer, 0)[0]) + assert frontiers[0] == (4 if hit_tokens == 8 else 0) + finish_submission(event) + frontiers.append(cache.retained_range("hit", window_layer, 0)[0]) + assert cache.retained_range("hit", full_layer, 0)[0] == 0 + + observed = [] + logits = model.logits + + def record_logits(state): + output = logits(state) + observed.append(output.detach().clone()) + return output + + monkeypatch.setattr(cache, "finish_submission", record_reclamation) + monkeypatch.setattr(model, "logits", record_logits) + actual = run_engine(engine, [("hit", prompt)], params)["hit"] + assert cache.prefix_hits == hit_tokens + assert frontiers[-1] > frontiers[0] + assert cache.num_used_blocks == 0 + assert len(observed) == len(actual) == 2 + tokens = list(prompt) + for output, token in zip(observed, actual): + expected = dense_dory_reference(model, torch.tensor(tokens))["logits"][-1:] + torch.testing.assert_close(output, expected, atol=3e-6, rtol=3e-5) + tokens.append(token) + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +def test_reclaim_engine_crosses_chunks_and_matches_dense_logits(monkeypatch, mode): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + observed = [] + logits = model.logits + + def observe(state): + result = logits(state) + observed.append(result.detach().clone()) + return result + + monkeypatch.setattr(model, "logits", observe) + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=8, + block_size=2, + layout=model.config.kv_cache_layout, + incremental_allocation=True, + ), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=3, prefill_chunk_size=5 + ), + ) + cache = engine.cache_manager + assert cache.max_submission_tokens == 3 + prompt = [4, 7, 9, 3, 12, 5, 6, 8, 11, 13, 14, 15] + engine.add_request( + "request", prompt, SamplingParams(max_tokens=4, min_loops=2, max_loops=2, ignore_eos=True) + ) + request = engine.scheduler.requests["request"] + assert engine.scheduler._plan_admission(request, 0).initial_tokens == 3 + occupancy = [] + finish = cache.finish_submission + + def record_finish(event=None): + finish(event) + if cache.has_allocation("request"): + occupancy.append( + ( + len(live_pages(cache, model, "request", sliding=True)), + len(live_pages(cache, model, "request", sliding=False)), + ) + ) + + monkeypatch.setattr(cache, "finish_submission", record_finish) + output = None + while engine.has_unfinished_requests(): + for item in engine.step(): + if item.finished: + output = item + assert output is not None and len(observed) == 4 + tokens = list(prompt) + for actual in observed: + expected = dense_dory_reference(model, torch.tensor(tokens))["logits"][-1:] + torch.testing.assert_close(actual, expected, atol=3e-6, rtol=3e-5) + tokens.append(int(expected[-1].argmax())) + assert output.token_ids == tokens[len(prompt) :] + peak_swa_pages = (2 + cache.storage_depths) * math.ceil((4 - 1 + 3 + 2 - 1) / 2) + assert max(swa for swa, _ in occupancy) <= peak_swa_pages + assert max(full for _, full in occupancy) >= 6 * cache.storage_depths + assert any(swa < full for swa, full in occupancy) + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +def test_reclaim_waits_for_group_reader_reuses_pages_and_rebuilds_or_rejects_views(mode): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + pool_size = 20 if mode == "per_loop" else 13 + cache = KVCacheManager( + num_layers=4, + num_kv_heads=2, + head_dim=8, + num_blocks=pool_size, + layout=model.config.kv_cache_layout, + block_size=2, + max_loops=2, + recurrent_layers=model.recurrent_kv_layers, + layer_windows=model.kv_layer_windows, + incremental_allocation=True, + reclaim_swa=True, + max_submission_tokens=1, + ) + assert cache.allocate("request", 8) + original_swa_pages = live_pages(cache, model, "request", sliding=True) + old = cache._prepare_batch(["request"], [0], [0]) + cache.layer_view(old, 0) + recent = recent_view = None + waits = [] + + class Completion: + def __init__(self): + self.used = cache.num_used_blocks + + def synchronize(self): + # No references or physical mapping may change before completion. + assert cache.num_used_blocks == self.used + assert cache.layer_block_table("request", 0, 0)[0] >= 0 + waits.append(True) + + for position in range(7): + assert cache.ensure_capacity("request", position + 1) + if position == 4: + rebuilt = cache.layer_view(recent, 0) + assert rebuilt is not recent_view # Growth epoch: live rows rebuild safely. + torch.testing.assert_close(rebuilt.write_blocks, recent_view.write_blocks) + if position == 6: + # The tight pool must actually reuse at least one physically retired page. + assert original_swa_pages & live_pages(cache, model, "request", sliding=True) + boundary = cache._prepare_batch(["request"], [0], [position]) + hidden = model.prelude_prepared(torch.tensor([position + 2]), boundary, cache) + for depth in range(2): + core = cache._prepare_batch(["request"], [depth], [position]) + hidden, _ = model.recurrent_prepared(hidden, core, cache) + if mode == "last_loop" and position == 4 and depth == 0: + # q4's next loop still reads key1 from page0: first-loop completion + # cannot advance the shared recurrent consumption frontier. + cache.finish_submission(Completion()) + assert cache.layer_block_table("request", 2, 0)[0] >= 0 + assert cache.retained_range("request", 2, 0)[0] == 0 + assert not waits + if position == 4: + # Input consumed position 4, but output still needs position 1 in page 0. + cache.finish_submission(Completion()) + assert cache.layer_block_table("request", 0, 0)[0] >= 0 + assert cache.layer_block_table("request", 2, 0)[0] == -1 + model.coda_prepared(hidden, boundary, cache) + cache.finish_submission(Completion() if position == 4 else None) + cache.finalize_token("request", position, 1) + if position == 3: + recent = cache._prepare_batch(["request"], [0], [3]) + recent_view = cache.layer_view(recent, 0) + if position == 4: + assert len(waits) == 2 + assert cache.retained_range("request", 0, 0) == (2, 5) + assert cache.retained_range("request", 1, 0) == (0, 5) + for layer in (*model.prelude_kv_layers, *model.coda_kv_layers): + with pytest.raises( + ValueError, match="boundary attention layers only own depth zero" + ): + cache.retained_range("request", layer, 1) + assert cache.token_written("request", position, 1) + with pytest.raises(RuntimeError, match="resident"): + cache.layer_view(old, 0) + with pytest.raises(ValueError, match="window must match the layer policy"): + cache.attend(0, ["request"], [0], [4], torch.zeros(1, 4, 8)) + with pytest.raises(RuntimeError, match="prefix has been reclaimed"): + cache.read(0, "request", 0) + cache.free("request") + assert cache.num_used_blocks == 0 and cache.num_free_blocks == pool_size + + +def test_window_one_page_one_finalization_and_swa_occupancy_plateau(monkeypatch): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(sliding_window=1)).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=9, block_size=1, incremental_allocation=True), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=3, prefill_chunk_size=5 + ), + ) + cache = engine.cache_manager + occupancy = [] + finish = cache.finish_submission + + def record_finish(event=None): + finish(event) + if cache.has_allocation("request"): + occupancy.append( + ( + len(live_pages(cache, model, "request", sliding=True)), + len(live_pages(cache, model, "request", sliding=False)), + ) + ) + + monkeypatch.setattr(cache, "finish_submission", record_finish) + prompt = [4, 7, 9, 3, 12, 5, 6, 8, 11] + engine.add_request( + "request", prompt, SamplingParams(max_tokens=3, min_loops=2, max_loops=2, ignore_eos=True) + ) + output = None + while engine.has_unfinished_requests(): + for item in engine.step(): + if item.finished: + output = item + assert output is not None + tokens = list(prompt) + for _ in range(3): + expected = dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(expected[-1].argmax())) + assert output.token_ids == tokens[len(prompt) :] + # SWA stays within its append bound while the full-attention history grows. + assert max(swa for swa, _ in occupancy) <= 4 * 3 + assert max(full for _, full in occupancy) >= 2 * len(prompt) + assert any(swa == 0 and full > 0 for swa, full in occupancy) + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize("layout", ["last_exited", "shared"]) +def test_long_reclaimed_history_is_not_copied_on_growth_or_retirement(monkeypatch, layout): + history_tokens, extra_tokens, loops = 4096, 8, 2 + planes = loops if layout == "last_exited" else 1 + cache = KVCacheManager( + num_layers=1, + num_kv_heads=1, + head_dim=2, + num_blocks=history_tokens * planes, + block_size=1, + max_loops=loops, + layout=layout, + layer_windows={0: 1}, + incremental_allocation=True, + reclaim_swa=True, + max_submission_tokens=history_tokens, + ) + assert cache.allocate("request", history_tokens + extra_tokens, initial_tokens=history_tokens) + snapshots = tuple(cache.layer_block_table("request", 0, depth) for depth in range(planes)) + # Exercise real coverage and consumption bookkeeping without benchmarking attention. + monkeypatch.setattr(cache.attention, "windowed", lambda query, *args: query) + values = torch.zeros(history_tokens, 1, 2) + for depth in range(loops): + batch = cache._prepare_batch( + ["request"] * history_tokens, [depth] * history_tokens, list(range(history_tokens)) + ) + cache._write_prepared(0, batch, values, values) + cache._attend_prepared(0, batch, values, window=1) + cache.finish_submission() + assert cache.num_used_blocks == 0 + assert all(isinstance(snapshot, tuple) and min(snapshot) >= 0 for snapshot in snapshots) + + class NoHistoryCopy(list): + # A one-page append or retirement must not traverse thousands of dead slots. + def __iter__(self): + raise AssertionError("allocator scanned the full historical page table") + + def __add__(self, other): + raise AssertionError("allocator copied the full historical page table") + + allocation = cache._get_allocation("request") + tables = tuple(NoHistoryCopy(table) for table in allocation.group_tables[0]) + allocation.group_tables = (tables,) + assert cache.allocated_blocks("request") == 0 + epochs = tuple(allocation.group_epochs[0]) + for position in range(history_tokens, history_tokens + extra_tokens): + assert cache.ensure_capacity("request", position + 1) + assert cache.num_used_blocks == planes + assert cache.allocated_blocks("request") == planes + for depth in range(loops): + batch = cache._prepare_batch(["request"], [depth], [position]) + cache._write_prepared(0, batch, values[:1], values[:1]) + cache._attend_prepared(0, batch, values[:1], window=1) + + class Completion: + def synchronize(self): + # Residency and references remain intact until the final reader completes. + assert cache.num_used_blocks == planes + assert all(table[position] is not None for table in tables) + + cache.finish_submission(Completion()) + assert cache.num_used_blocks == 0 + assert cache.allocated_blocks("request") == 0 + assert all(table[position] is None for table in tables) + assert all(current is saved for current, saved in zip(allocation.group_tables[0], tables)) + assert allocation.group_epochs[0] == [epoch + 2 * extra_tokens for epoch in epochs] + # Public inspection still returns immutable snapshots independent of internal mutation. + allocation.group_tables = (tuple(table[:] for table in tables),) + for depth in range(planes): + assert cache.layer_block_table("request", 0, depth) == (-1,) * ( + history_tokens + extra_tokens + ) + assert cache.retained_range("request", 0, depth) == ( + history_tokens + extra_tokens, + history_tokens + extra_tokens, + ) + cache.free("request") + assert cache.num_free_blocks == cache.num_blocks + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +def test_reclaim_multi_request_abort_and_reuse(monkeypatch, mode): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + reference = type(model)(model.config).eval() + reference.load_state_dict(model.state_dict()) + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=16 if mode == "per_loop" else 10, + block_size=2, + layout=model.config.kv_cache_layout, + incremental_allocation=True, + ), + scheduler_config=SchedulerConfig( + max_num_seqs=2, + max_num_batched_tokens=3, + prefill_chunk_size=3, + ), + ) + cache = engine.cache_manager + prompts = { + "long": [4, 7, 9, 3, 12, 5, 6, 8, 11, 13, 14, 15], + "reused": [8, 2, 5, 9, 3, 7, 12], + } + replacement = [21, 6, 17, 5, 14, 2, 19, 4] + for rid, count in (("long", 7), ("reused", 4)): + engine.add_request( + rid, + prompts[rid], + SamplingParams( + max_tokens=count, + min_loops=2, + max_loops=2, + ignore_eos=True, + ), + ) + original_b = engine.scheduler.requests["reused"] + observed, completed = [], {} + original_logits = model.logits + + def observe(state): + values = original_logits(state) + scheduled = engine.last_schedule.requests + assert len(values) == len(scheduled) + for value, item in zip(values, scheduled): + request = item.request + context = tuple(request.prompt_token_ids + request.generated_token_ids) + observed.append( + (request, len(request.generated_token_ids), context, value.detach().clone()) + ) + return values + + monkeypatch.setattr(model, "logits", observe) + first_b_pages, reused_while_b_active = set(), set() + stale = None + aborted = stale_checked = False + for _ in range(128): + if not engine.has_unfinished_requests(): + break + for output in engine.step(): + if output.finished: + completed[output.request_id] = output + owned = { + rid: live_pages(cache, model, rid, sliding=True) + | live_pages(cache, model, rid, sliding=False) + for rid in ("long", "reused") + if cache.has_allocation(rid) + } + assert sum(map(len, owned.values())) == cache.num_used_blocks + if len(owned) == 2: + assert owned["long"].isdisjoint(owned["reused"]) + current_b = engine.scheduler.requests.get("reused") + if current_b is original_b and "reused" in owned: + if not first_b_pages: + first_b_pages = { + cache.layer_block_table("reused", layer, depth)[0] + for layer, window in model.kv_layer_windows.items() + if window is not None + for depth in ( + range(cache.storage_depths) if layer in model.recurrent_kv_layers else (0,) + ) + } + assert min(first_b_pages) >= 0 + reused_while_b_active |= first_b_pages & owned.get("long", set()) + if stale is not None and "reused" in owned and not stale_checked: + assert current_b is not original_b + with pytest.raises(RuntimeError, match="stale prepared KV batch"): + cache.layer_view(stale, 0) + stale_checked = True + a = engine.scheduler.requests.get("long") + if ( + not aborted + and reused_while_b_active + and a is not None + and current_b is original_b + and a.num_prefilled_tokens >= 9 + and current_b.num_prefilled_tokens >= 6 + and len(current_b.generated_token_ids) == 1 + and current_b.phase == RequestPhase.RECURRENT + and current_b.loops_done == 0 + ): + position = cache.retained_range("reused", 0, 0)[1] - 1 + stale = cache._prepare_batch(["reused"], [0], [position], for_write=False) + cache.layer_view(stale, 0) # A valid, resident descriptor before cancellation. + engine.abort_request("reused") + engine.add_request( + "reused", + replacement, + SamplingParams( + max_tokens=4, + min_loops=2, + max_loops=2, + ignore_eos=True, + ), + ) + assert engine.scheduler.requests["reused"].num_prefilled_tokens == 0 + prompts["reused"] = replacement + aborted = True + assert not engine.has_unfinished_requests(), "small-pool admission/growth did not complete" + assert aborted and stale_checked and reused_while_b_active + assert set(completed) == {"long", "reused"} + for request, index, context, actual in observed: + assert index == len(context) - len(request.prompt_token_ids) + expected = dense_dory_reference(reference, torch.tensor(context))["logits"][-1] + torch.testing.assert_close(actual.cpu(), expected, atol=3e-6, rtol=3e-5) + assert any(request is original_b for request, _, _, _ in observed) + assert any( + request is not original_b and request.request_id == "reused" + for request, _, _, _ in observed + ) + for rid, count in (("long", 7), ("reused", 4)): + tokens = list(prompts[rid]) + for _ in range(count): + logits = dense_dory_reference(reference, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + assert completed[rid].token_ids == tokens[len(prompts[rid]) :] + assert cache.num_used_blocks == 0 and cache.num_free_blocks == cache.num_blocks + + +def test_fp8_kv_falls_back_to_all_layer_blocks(): + model = DoryForCausalLM(tiny_dory_config()).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=64, block_size=2, kv_cache_dtype="fp8_e4m3", enable_prefix_caching=True + ), + ) + assert not engine.cache_manager.grouped + + +def test_grouped_prefill_ignores_advertised_packing(monkeypatch): + """A natively packing backend must not get packed metadata for grouped pages.""" + from dataclasses import replace + + from vllm_rlt import ExecutionConfig + from vllm_rlt.attention import AttentionBackend + from vllm_rlt.engine import stage_engine + + create_backend = stage_engine.create_backend + + def packing_backend(*args, **kwargs): + backend = create_backend(*args, **kwargs) + capabilities = replace( + backend.capabilities, packed_prefill=True, packed_sliding_window=True + ) + return AttentionBackend(backend.kernel, capabilities, backend.info) + + monkeypatch.setattr(stage_engine, "create_backend", packing_backend) + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(max_position_embeddings=128)).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=64, + block_size=2, + incremental_allocation=True, + ), + scheduler_config=SchedulerConfig(max_num_seqs=1, max_num_batched_tokens=16), + execution_config=ExecutionConfig(packed_prefill=False), + ) + assert engine.cache_manager.grouped + assert engine.cache_manager.attention_capabilities.packed_prefill + assert not engine.execution_config.packed_prefill + # Runner packing applies only to prompts longer than 64 tokens. + prompt = [(index * 7) % 50 + 2 for index in range(65)] + params = SamplingParams(max_tokens=2, min_loops=2, max_loops=2, ignore_eos=True) + tokens = list(prompt) + for _ in range(2): + logits = dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + assert run_engine(engine, [("long", prompt)], params)["long"] == tokens[len(prompt) :] + assert engine.cache_manager.num_used_blocks == 0 + + +def test_shared_fp8_prefix_hits_ignore_advertised_packing(monkeypatch): + """last_loop + FP8 keeps shared all-layer pages, which have no packed metadata.""" + from dataclasses import replace + + from vllm_rlt.attention import AttentionBackend + from vllm_rlt.engine import stage_engine + + create_backend = stage_engine.create_backend + + def packing_backend(*args, **kwargs): + backend = create_backend(*args, **kwargs) + capabilities = replace(backend.capabilities, packed_prefill=True) + return AttentionBackend(backend.kernel, capabilities, backend.info) + + monkeypatch.setattr(stage_engine, "create_backend", packing_backend) + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode="last_loop")).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=256, + block_size=2, + layout="shared", + kv_cache_dtype="fp8_e4m3", + enable_prefix_caching=True, + ), + scheduler_config=SchedulerConfig( + max_num_seqs=3, max_num_batched_tokens=12, prefill_chunk_size=4 + ), + ) + cache = engine.cache_manager + assert cache.layout == "shared" and not cache.grouped + assert cache.attention_capabilities.packed_prefill + params = SamplingParams(max_tokens=3, min_loops=2, max_loops=2, ignore_eos=True) + long = [4, 7, 3, 9, 5, 2, 8, 6, 3, 10] + # "short" ends its prompt inside the first chunk while the 10-token prompts + # on either side continue, so CODA rows 0 and 2 are rebuilt non-contiguously. + requests = [("first", long), ("short", long[:4]), ("last", long[:9] + [11])] + cold = run_engine(engine, requests, params) + assert cache.prefix_hits == 0 + hits = run_engine(engine, [(f"hit-{rid}", prompt) for rid, prompt in requests], params) + assert {rid: hits[f"hit-{rid}"] for rid, _ in requests} == cold + assert cache.prefix_hits > 0 + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize("mode", ["per_loop", "last_loop"]) +def test_preallocation_returns_expired_swa_pages(monkeypatch, mode): + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig(num_blocks=16, block_size=2, layout=model.kv_cache_layout), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=3, prefill_chunk_size=3 + ), + ) + cache = engine.cache_manager + assert cache.grouped and cache.reclaim_swa and not cache.incremental_allocation + prompt = [4, 7, 9, 3, 12, 5, 6, 8, 11, 13, 14, 15] + params = SamplingParams(max_tokens=4, min_loops=2, max_loops=2, ignore_eos=True) + frontiers = [] + finish = cache.finish_submission + + def record(event=None): + finish(event) + if cache.has_allocation("request"): + allocation = cache._allocations["request"] + frontiers.append(max(v for planes in allocation.retired_pages for v in planes)) + + monkeypatch.setattr(cache, "finish_submission", record) + reserved = [] + step = engine.step + + def record_reserved(): + outputs = step() + reserved.append(engine.scheduler._reserved_growth_blocks(reserve_outputs=False)) + return outputs + + monkeypatch.setattr(engine, "step", record_reserved) + tokens = list(prompt) + for _ in range(4): + logits = dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + assert run_engine(engine, [("request", prompt)], params)["request"] == tokens[len(prompt) :] + assert frontiers[0] == 0 and frontiers[-1] > 0 + # Retired pages are not budgeted again as growth. + assert set(reserved) == {0} + assert cache.num_used_blocks == 0 + + +@pytest.mark.parametrize(("mode", "blocks"), [("per_loop", 17), ("last_loop", 11)]) +def test_prefix_hit_admission_skips_expired_cached_pages(mode, blocks): + # A hit attaches only in-window SWA pages; expired cached pages stay + # evictable, so admission must not charge them. + torch.manual_seed(42) + model = DoryForCausalLM(tiny_dory_config(recurrent_kv_cache_mode=mode)).eval() + engine = LLMEngine( + model, + cache_config=CacheConfig( + num_blocks=blocks, + block_size=2, + layout=model.kv_cache_layout, + enable_prefix_caching=True, + ), + scheduler_config=SchedulerConfig( + max_num_seqs=1, max_num_batched_tokens=4, prefill_chunk_size=4 + ), + ) + cache = engine.cache_manager + params = SamplingParams(max_tokens=2, min_loops=2, max_loops=2, ignore_eos=True) + publisher = list(range(3, 23)) + run_engine(engine, [("publisher", publisher)], params) + # Leave only prefix-only pages free: enough for the hit, not for the + # hit plus its expired cached pages. + entries = len(cache._prefixes) + occupied = cache._claim(len(cache._free_blocks)) + assert len(cache._prefixes) == entries + prompt = publisher[:16] + [30, 31] + actual = run_engine(engine, [("hit", prompt)], params)["hit"] + tokens = list(prompt) + for _ in range(2): + logits = dense_dory_reference(model, torch.tensor(tokens))["logits"] + tokens.append(int(logits[-1].argmax())) + assert actual == tokens[len(prompt) :] + assert cache.prefix_hits == 16 + cache._drop_refs(occupied) + assert cache.num_used_blocks == 0 diff --git a/vllm_rlt/config.py b/vllm_rlt/config.py index 7213adb0..732e5608 100644 --- a/vllm_rlt/config.py +++ b/vllm_rlt/config.py @@ -30,8 +30,6 @@ def __post_init__(self): raise ValueError(f"{name} must be a boolean") if not 0 <= self.watermark_ratio < 1: raise ValueError("watermark_ratio must be in [0, 1)") - if self.enable_prefix_caching and self.layout != "last_exited": - raise ValueError("prefix caching requires last_exited KV") if self.num_blocks is not None: _positive("num_blocks", self.num_blocks) _positive("block_size", self.block_size) diff --git a/vllm_rlt/core/kv_cache_manager.py b/vllm_rlt/core/kv_cache_manager.py index c6187d77..db196002 100644 --- a/vllm_rlt/core/kv_cache_manager.py +++ b/vllm_rlt/core/kv_cache_manager.py @@ -14,7 +14,7 @@ import struct import weakref from collections import OrderedDict -from collections.abc import Callable, Sequence +from collections.abc import Callable, Mapping, Sequence from copy import deepcopy from dataclasses import dataclass, field from numbers import Integral @@ -23,15 +23,23 @@ from vllm_rlt.attention import ( AttentionBackend, + AttentionRows, BackendCapabilities, backend_capabilities, create_backend, + plan_attention_metadata, validate_backend_name, ) from vllm_rlt.device import resolve_device from vllm_rlt.request import ExecutionBatch from vllm_rlt.worker.buffers import stage_metadata -from vllm_rlt.worker.metadata import DeviceKVMetadata, KVMetadata, MetadataBuilder +from vllm_rlt.worker.metadata import ( + DeviceKVMetadata, + GroupedKVMetadata, + KVLayout, + KVMetadata, + MetadataBuilder, +) KV_CACHE_DTYPES = ("auto", "fp8_e4m3") FP8_E4M3_MAX = 448.0 @@ -106,8 +114,24 @@ class _Allocation: max_tokens: int block_tables: tuple[tuple[int, ...], ...] written: list[list[_WrittenPositions]] + # Native group -> plane -> mutable logical-page list -> one page per group layer. + group_tables: tuple = () + group_epochs: list[list[int]] = field(default_factory=list) + retired_pages: list[list[int]] = field(default_factory=list) + consumed: list[list[_WrittenPositions]] = field(default_factory=list) + reclaim_pending: bool = False transfer_leases: set[str] = field(default_factory=set) release_requested: bool = False + prefix_keys: list[bytes] = field(default_factory=list) + prefix_epoch: int = 0 + + +@dataclass(frozen=True) +class KVGroup: + layers: tuple[int, ...] + window: int | None + recurrent: bool + planes: int @dataclass(frozen=True) @@ -170,8 +194,9 @@ class KVCacheManager: """Own a fixed physical page pool shared by request/depth allocations. ``num_blocks`` counts total physical blocks across all recurrence depths. - Every physical block contains K and V for all decoder layers. Depth indices - are zero-based throughout the interface. + Legacy blocks contain all decoder layers. With ``layer_windows``, a block + is one physical attention-layer page; groups own independent page bundles. + Depth indices are zero-based throughout the interface. """ def __init__( @@ -194,6 +219,9 @@ def __init__( attention: AttentionBackend | None = None, kv_cache_dtype: str = "auto", fp8_scale: float | None = None, + layer_windows: Mapping[int, int | None] | None = None, + reclaim_swa: bool = False, + max_submission_tokens: int = 1, ): for name, value in ( ("num_layers", num_layers), @@ -237,6 +265,46 @@ def __init__( ): raise ValueError("recurrent_layers must be valid cache layer indices") self.storage_depths = max_loops if layout == "last_exited" else 1 + self.grouped = layer_windows is not None + if type(reclaim_swa) is not bool: + raise ValueError("reclaim_swa must be a boolean") + if type(max_submission_tokens) is not int or max_submission_tokens <= 0: + raise ValueError("max_submission_tokens must be a positive integer") + if reclaim_swa and not self.grouped: + raise ValueError("SWA reclamation requires grouped storage") + self.reclaim_swa = reclaim_swa + self.max_submission_tokens = max_submission_tokens + self.groups = () + self.layer_groups = {} + if self.grouped: + policies = dict(layer_windows) + if not policies or any( + type(layer) is not int or not 0 <= layer < num_layers for layer in policies + ): + raise ValueError("layer_windows must name actual attention layer indices") + if not set(self.recurrent_layers) <= policies.keys(): + raise ValueError("recurrent layers must be actual attention layers") + groups = {} + for layer, window in sorted(policies.items()): + if window is not None and (type(window) is not int or window <= 0): + raise ValueError("layer windows must be positive integers or None") + recurrent = layer in self.recurrent_layers + groups.setdefault((window, recurrent), []).append(layer) + self.groups = tuple( + KVGroup(tuple(layers), window, recurrent, self.storage_depths if recurrent else 1) + for (window, recurrent), layers in groups.items() + ) + self.layer_groups = { + layer: (group_id, local) + for group_id, group in enumerate(self.groups) + for local, layer in enumerate(group.layers) + } + # Physical pool pages needed for one token block across all layers and planes. + self.pages_per_token_block = ( + sum(len(group.layers) * group.planes for group in self.groups) + if self.grouped + else self.storage_depths + ) self.device = resolve_device(device) self.dtype = dtype self.attention = attention or create_backend( @@ -258,7 +326,9 @@ def __init__( f"the {self.attention_info['backend']} backend does not read FP8 KV " f"with {'per-vector' if fp8_scale is None else 'fixed'} scales" ) - shape = (num_blocks, num_layers, block_size, num_kv_heads, head_dim) + if self.grouped and kv_cache_dtype != "auto": + raise NotImplementedError("grouped KV does not support FP8 storage yet") + shape = (num_blocks, 1 if self.grouped else num_layers, block_size, num_kv_heads, head_dim) storage = torch.float8_e4m3fn if kv_cache_dtype == "fp8_e4m3" else dtype self.key_cache = torch.empty(shape, device=self.device, dtype=storage) self.value_cache = torch.empty_like(self.key_cache) @@ -294,10 +364,9 @@ def __init__( self.watermark_blocks = int(watermark_ratio * num_blocks) if incremental_allocation else 0 self._refs = [0] * num_blocks self._prefixes = OrderedDict() + self._prefix_epoch = 0 self._pending_prefixes = [] self.prefix_hits = self.prefix_queries = 0 - if enable_prefix_caching and layout != "last_exited": - raise ValueError("prefix caching requires last_exited KV") @property def attention_capabilities(self) -> BackendCapabilities: @@ -309,6 +378,11 @@ def num_free_blocks(self) -> int: self._refs[b] == 1 for blocks in self._prefixes.values() for b in blocks ) + @property + def submission_bounded(self) -> bool: + """Whether SWA pages stay within the window plus one submission.""" + return self.reclaim_swa and self.incremental_allocation + @property def num_used_blocks(self) -> int: return self.num_blocks - self.num_free_blocks @@ -316,7 +390,7 @@ def num_used_blocks(self) -> int: @property def bytes_per_block(self) -> int: return kv_bytes_per_block( - self.num_layers, + 1 if self.grouped else self.num_layers, self.block_size, self.num_kv_heads, self.head_dim, @@ -327,7 +401,22 @@ def bytes_per_block(self) -> int: def required_blocks(self, max_tokens: int) -> int: if not isinstance(max_tokens, Integral) or isinstance(max_tokens, bool) or max_tokens <= 0: raise ValueError("max_tokens must be a positive integer") - return ((max_tokens + self.block_size - 1) // self.block_size) * self.storage_depths + pages = (max_tokens + self.block_size - 1) // self.block_size + if self.submission_bounded: + # W-1 retained keys plus one append S, including worst page alignment. + return sum( + min( + pages, + (group.window - 1 + self.max_submission_tokens + 2 * self.block_size - 2) + // self.block_size, + ) + * len(group.layers) + * group.planes + if group.window is not None + else pages * len(group.layers) * group.planes + for group in self.groups + ) + return pages * self.pages_per_token_block def _drop_refs(self, blocks): for b in blocks: @@ -349,6 +438,7 @@ def _claim(self, count): return None while len(self._free_blocks) < count and self._prefixes: _, blocks = self._prefixes.popitem(last=False) + self._prefix_epoch += 1 self._drop_refs(blocks) if len(self._free_blocks) < count: return None @@ -358,8 +448,7 @@ def _claim(self, count): self._refs[b] = 1 return blocks - def _prefix_keys(self, tokens): - digest = b"" + def _prefix_keys(self, tokens, digest=b""): for start in range(0, len(tokens) // self.block_size * self.block_size, self.block_size): values = tokens[start : start + self.block_size] digest = hashlib.sha256(digest + struct.pack(f"<{len(values)}q", *values)).digest() @@ -367,11 +456,12 @@ def _prefix_keys(self, tokens): def poll_prefixes(self): pending, self._pending_prefixes = self._pending_prefixes, [] - for request_id, allocation, tokens, length, event in pending: + for request_id, allocation, tokens, length, events in pending: if self._allocations.get(request_id) is not allocation: continue - if event is not None and not event.query(): - self._pending_prefixes.append((request_id, allocation, tokens, length, event)) + events[:] = [event for event in events if not event.query()] + if events: + self._pending_prefixes.append((request_id, allocation, tokens, length, events)) continue self.publish_prefix(request_id, tokens, length) @@ -380,29 +470,99 @@ def publish_prefix(self, request_id, tokens, length, event=None): length is the prefilled extent; len(tokens) is the full prompt length. A supplied event defers publication until the GPU writes complete. + The full prompt stays immutable for the allocation's lifetime. """ if not self.enable_prefix_caching: return allocation = self._get_allocation(request_id) if event is not None: - self._pending_prefixes.append((request_id, allocation, tuple(tokens), length, event)) - return + ready = event.query() + for index, (_, owner, snapshot, frontier, events) in enumerate(self._pending_prefixes): + if owner is allocation: + if not ready: + events.append(event) + # Keep one prompt snapshot per allocation. Retain every + # unfinished event: later submissions may complete first. + self._pending_prefixes[index] = ( + request_id, + allocation, + snapshot, + max(frontier, length), + events, + ) + return + if not ready: + self._pending_prefixes.append( + (request_id, allocation, tuple(tokens), length, [event]) + ) + return # Prefix entries retain KV, not the final hidden state needed by CODA. # Leave at least the last prompt token to recompute on a cache hit; # full-page alignment may leave additional tokens to recompute. publishable_tokens = min(length, len(tokens) - 1) - for index, key in enumerate(self._prefix_keys(tokens[:publishable_tokens])): + if allocation.prefix_epoch != self._prefix_epoch: + allocation.prefix_keys.clear() + allocation.prefix_epoch = self._prefix_epoch + keys = allocation.prefix_keys + start = len(keys) + digest = keys[-1] if keys else b"" + # Eviction invalidates the saved chain's publication frontier. Without + # eviction, hash only new token pages instead of revisiting every chunk. + for index, key in enumerate( + self._prefix_keys(tokens[start * self.block_size : publishable_tokens], digest), + start=start, + ): if key in self._prefixes: + keys.append(key) continue end = (index + 1) * self.block_size - # Every depth and layer must have a contiguous written prefix. - # GPU completion alone does not establish this coverage. - if any(w.prefix < end for plane in allocation.written for w in plane): + # Every layer that attention reads must have a contiguous written + # prefix. GPU completion alone does not establish this coverage. + if any( + written.prefix < end + for plane, layers in enumerate(allocation.written) + for layer, written in enumerate(layers) + if self._plane_owns(plane, layer) + ): + break + blocks = self._prefix_page(allocation, index) + if blocks is None: break - blocks = tuple(t[index] for t in allocation.block_tables) for b in blocks: self._refs[b] += 1 self._prefixes[key] = blocks + keys.append(key) + + def _prefix_page(self, allocation, index): + """Every physical page holding token page ``index``, or None if any is gone. + + Grouped entries are flattened group -> plane -> layer for + ``_group_prefix_tables`` to split back into bundles. + """ + if not self.grouped: + return tuple(t[index] for t in allocation.block_tables) + bundles = [ + table[index] if index < len(table) else None + for planes in allocation.group_tables + for table in planes + ] + if any(bundle is None for bundle in bundles): + return None + return tuple(b for bundle in bundles for b in bundle) + + def _plane_owns(self, plane: int, layer: int) -> bool: + """Whether ``layer`` keeps KV history in ``plane``. + + Boundary (non-recurrent) layers run once per token, at depth 0, so + they only ever write plane 0. Recurrent layers own every plane. + Grouped storage takes ownership from the layer's group policy. + """ + if self.grouped: + if layer not in self.layer_groups: + return False + group_id, _ = self.layer_groups[layer] + return plane == 0 or self.groups[group_id].recurrent + return plane == 0 or layer in self.recurrent_layers def lookup_prefix(self, tokens): if not self.enable_prefix_caching: @@ -428,43 +588,106 @@ def allocate(self, request_id: str, max_tokens: int, *, initial_tokens=None, pre raise ValueError( f"request needs {required} physical KV blocks, but the pool has {self.num_blocks}" ) - initial_tokens = max_tokens if initial_tokens is None else initial_tokens + cached_tokens = len(prefix) * self.block_size + if initial_tokens is None: + initial_tokens = ( + min(max_tokens, cached_tokens + self.max_submission_tokens) + if self.submission_bounded + else max_tokens + ) + if self.submission_bounded and initial_tokens - cached_tokens > self.max_submission_tokens: + raise ValueError("uncached initial KV exceeds the configured submission bound") if not 0 < initial_tokens <= max_tokens: raise ValueError("invalid initial KV frontier") - pages = (initial_tokens + self.block_size - 1) // self.block_size - if len(prefix) > pages: + if cached_tokens > initial_tokens: raise ValueError("prefix exceeds initial KV frontier") - claimed = [b for group in prefix for b in group] + pages = (initial_tokens + self.block_size - 1) // self.block_size + retired_pages, cached_tables, claimed = self._prefix_claim(prefix) for b in claimed: self._refs[b] += 1 - fresh = self._claim((pages - len(prefix)) * self.storage_depths) - if fresh is None: + tail = pages - len(prefix) + if self.grouped: + tables = self._claim_group_tables(tail, tables=cached_tables) + else: + blocks = self._claim(tail * self.storage_depths) + tables = ( + None + if blocks is None + else tuple( + tuple(g[d] for g in prefix) + tuple(blocks[d * tail : (d + 1) * tail]) + for d in range(self.storage_depths) + ) + ) + if tables is None: self._drop_refs(claimed) return False - tail = pages - len(prefix) - tables = tuple( - tuple(g[d] for g in prefix) + tuple(fresh[d * tail : (d + 1) * tail]) - for d in range(self.storage_depths) - ) - length = len(prefix) * self.block_size + # A hit restores exactly the coverage publish_prefix required. + written = [ + [ + _WrittenPositions(prefix=cached_tokens if self._plane_owns(plane, layer) else 0) + for layer in range(self.num_layers) + ] + for plane in range(self.storage_depths) + ] self._allocations[request_id] = _Allocation( int(max_tokens), - tables, - [ - [_WrittenPositions(prefix=length) for _ in range(self.num_layers)] - for _ in range(self.storage_depths) - ], + () if self.grouped else tables, + written, + group_tables=tables if self.grouped else (), + group_epochs=[[0] * group.planes for group in self.groups], + retired_pages=retired_pages, + consumed=( + [ + [_WrittenPositions(prefix=positions.prefix) for positions in plane] + for plane in written + ] + if self.reclaim_swa + else [] + ), ) self._allocation_epoch += 1 self.prefix_queries += 1 - self.prefix_hits += length + self.prefix_hits += cached_tokens return True + def _claim_group_tables(self, new_pages, *, tables=()): + """Append ``new_pages`` fresh bundles to every plane, or None if the pool is short.""" + fresh = self._claim(new_pages * self.pages_per_token_block) + if fresh is None: + return None + if not tables: + tables = tuple(tuple([] for _ in range(group.planes)) for group in self.groups) + offset = 0 + for group, planes in zip(self.groups, tables): + width = len(group.layers) + for table in planes: + for _ in range(new_pages): + table.append(tuple(fresh[offset : offset + width])) + offset += width + assert offset == len(fresh) + return tables + def ensure_capacity(self, request_id, tokens): allocation = self._get_allocation(request_id) if not 0 < tokens <= allocation.max_tokens: raise ValueError("KV growth exceeds logical capacity") pages = (tokens + self.block_size - 1) // self.block_size + if self.grouped: + current = len(allocation.group_tables[0][0]) + extra = pages - current + if extra <= 0: + return True + added = self._claim_group_tables(extra) + if added is None: + return False + for planes, new_planes in zip(allocation.group_tables, added): + for table, tail in zip(planes, new_planes): + table.extend(tail) + for epochs in allocation.group_epochs: + for plane in range(len(epochs)): + epochs[plane] += 1 + self._allocation_epoch += 1 + return True extra = pages - len(allocation.block_tables[0]) if extra <= 0: return True @@ -485,6 +708,7 @@ def truncate_suffix(self, request_id: str, frontier: int) -> None: admission reservation; stale bytes are hidden by written/context lengths. Shared prompt pages must never be truncated or subsequently overwritten. """ + self._require_legacy() allocation = self._get_allocation(request_id) if type(frontier) is not int or not 0 <= frontier <= allocation.max_tokens: raise ValueError("invalid KV truncation frontier") @@ -506,7 +730,106 @@ def free(self, request_id: str) -> None: allocation = self._allocations.pop(request_id, None) if allocation is not None: self._allocation_epoch += 1 - self._drop_refs(b for table in allocation.block_tables for b in reversed(table)) + if self.grouped: + self._drop_refs( + b + for planes in allocation.group_tables + for table in planes + for bundle in reversed(table) + if bundle is not None + for b in reversed(bundle) + ) + else: + self._drop_refs(b for table in allocation.block_tables for b in reversed(table)) + self._pending_prefixes = [ + pending for pending in self._pending_prefixes if pending[1] is not allocation + ] + + def allocate_import( + self, request_id: str, max_tokens: int, prompt_tokens: int, *, initial_tokens, prefix=() + ) -> bool: + """Reserve the pages that a received prompt needs, or return False if the pool is short. + + The receiver first runs the output stage of the last prompt token. With + SWA reclamation, the receiver holds no SWA page that this query cannot + read, so the sender writes no KV there. ``allocate`` would claim those + pages and apply the submission bound, which counts computed tokens only. + """ + if not self.reclaim_swa: + return self.allocate( + request_id, max_tokens, initial_tokens=initial_tokens, prefix=prefix + ) + pages = (initial_tokens + self.block_size - 1) // self.block_size + retired = [ + [self._expired_swa_pages(group, prompt_tokens - 1)] * group.planes + for group in self.groups + ] + # Prefix entries are flattened group -> plane -> layer, like a claim. + spans, start = [], 0 + for group, planes in zip(self.groups, retired): + for first in planes: + spans.append((first, start, start + len(group.layers))) + start += len(group.layers) + attached = [b for first, lo, hi in spans for page in prefix[first:] for b in page[lo:hi]] + # References first: the claim below can evict the entries being attached. + for b in attached: + self._refs[b] += 1 + fresh = self._claim( + sum((pages - max(first, len(prefix))) * (hi - lo) for first, lo, hi in spans) + ) + if fresh is None: + self._drop_refs(attached) + return False + fresh = iter(fresh) + rows = iter( + [None] * first + + [ + tuple(prefix[page][lo:hi]) + if page < len(prefix) + else tuple(next(fresh) for _ in range(hi - lo)) + for page in range(first, pages) + ] + for first, lo, hi in spans + ) + cached_tokens = len(prefix) * self.block_size + written = [ + [ + _WrittenPositions(prefix=cached_tokens if self._plane_owns(plane, layer) else 0) + for layer in range(self.num_layers) + ] + for plane in range(self.storage_depths) + ] + self._allocations[request_id] = _Allocation( + int(max_tokens), + (), + written, + group_tables=tuple( + tuple(next(rows) for _ in range(group.planes)) for group in self.groups + ), + group_epochs=[[0] * group.planes for group in self.groups], + retired_pages=retired, + consumed=deepcopy(written), + ) + self._allocation_epoch += 1 + self.prefix_queries += 1 + self.prefix_hits += cached_tokens + return True + + def transfer_tables(self, request_id: str) -> tuple[tuple[int, ...], ...]: + """Block tables that a KV transfer addresses, with -1 for a page not held. + + All-layer storage has one table per plane. Grouped storage has one table + per group, plane and layer, in that order, because each page holds one layer. + """ + allocation = self._get_allocation(request_id) + if not self.grouped: + return allocation.block_tables + return tuple( + tuple(-1 if bundle is None else bundle[local] for bundle in table) + for group, planes in zip(self.groups, allocation.group_tables) + for table in planes + for local in range(len(group.layers)) + ) def pin_transfer(self, request_id: str, transfer_id: str): allocation = self._get_allocation(request_id) @@ -520,18 +843,33 @@ def unpin_transfer(self, request_id: str, transfer_id: str): if allocation.release_requested and not allocation.transfer_leases: self.free(request_id) - def mark_imported_prefix(self, request_id: str, length: int, *, start=0): - """Publish received KV only after the connector confirms remote writes are complete.""" + def mark_imported_prefix(self, request_id: str, length: int, *, start=0, output_layers=()): + """Publish received KV only after the connector confirms remote writes are complete. + + Like a prefix hit, the import covers only the planes each layer owns: + boundary layers never write deeper planes, so the sender has no KV there. + ``output_layers`` still have to read the last imported position here, + because the receiver runs the output stage of that token. + """ allocation = self._get_allocation(request_id) if type(length) is not int or not 0 < length <= allocation.max_tokens: raise ValueError("imported prefix exceeds reserved capacity") if allocation.release_requested: raise RuntimeError("cannot activate cancelled KV") - if any(w.prefix != start or w.pending for plane in allocation.written for w in plane): + owned = [ + written + for plane, layers in enumerate(allocation.written) + for layer, written in enumerate(layers) + if self._plane_owns(plane, layer) + ] + if any(w.prefix != start or w.pending for w in owned): raise RuntimeError("KV prefix was already initialized") - for plane in allocation.written: - for written in plane: - written.prefix = length + for written in owned: + written.prefix = length + # The sender read every other position; reclamation continues from here. + for plane in allocation.consumed: + for layer, consumed in enumerate(plane): + consumed.prefix = length - 1 if layer in output_layers else length # Other modules learn about allocations only through the methods below. # They return plain facts, never _Allocation objects or reference counts. @@ -541,9 +879,33 @@ def has_allocation(self, request_id: str) -> bool: def allocated_blocks(self, request_id: str) -> int: """How many physical blocks this request holds, counting every plane.""" - return len(self._get_allocation(request_id).block_tables[0]) * self.storage_depths + allocation = self._get_allocation(request_id) + if self.grouped: + return sum( + (len(table) - allocation.retired_pages[group_id][plane]) * len(group.layers) + for group_id, group in enumerate(self.groups) + for plane, table in enumerate(allocation.group_tables[group_id]) + ) + return len(allocation.block_tables[0]) * self.storage_depths + + def charged_blocks(self, request_id: str) -> int: + """Blocks already spent against this request's ``required_blocks`` peak. + + A window-bounded peak covers resident pages only. A full peak also + covers retired SWA pages: they went back to the pool and are never + needed again, so they are not future growth. + """ + if not self.grouped or self.submission_bounded: + return self.allocated_blocks(request_id) + allocation = self._get_allocation(request_id) + return sum( + len(table) * len(group.layers) + for group_id, group in enumerate(self.groups) + for table in allocation.group_tables[group_id] + ) def get_block_table(self, request_id: str, depth: int) -> tuple[int, ...]: + self._require_legacy() self._validate_depth(depth) return self._get_allocation(request_id).block_tables[self._plane(depth)] @@ -554,19 +916,22 @@ def plane_block_tables(self, request_id: str) -> tuple[tuple[int, ...], ...]: ``max_loops`` planes and SHARED has one, so ``tables[2]`` fails under SHARED while ``get_block_table(rid, 2)`` works under both layouts. """ + self._require_legacy() return self._get_allocation(request_id).block_tables def has_transfer_lease(self, request_id: str) -> bool: return bool(self._get_allocation(request_id).transfer_leases) def exclusive_prefix_blocks(self, prefix: Sequence[Sequence[int]]) -> int: - """How many blocks of a prefix hit are held only by the prefix cache. + """How many blocks a prefix hit claims that only the prefix cache holds. ``num_free_blocks`` counts those blocks as free because they could be evicted. Claiming them makes them non-evictable, so admission has to - budget them on top of the fresh blocks it needs. + budget them on top of the fresh blocks it needs. Expired SWA pages the + hit does not attach stay evictable. """ - return sum(self._refs[b] == 1 for group in prefix for b in group) + _, _, claimed = self._prefix_claim(prefix) + return sum(self._refs[b] == 1 for b in claimed) def token_written(self, request_id: str, position: int, depth: int) -> bool: """Whether every layer has written KV for ``position`` at ``depth``.""" @@ -576,6 +941,12 @@ def token_written(self, request_id: str, position: int, depth: int) -> bool: return self._token_written(allocation, position, depth) def _token_written(self, allocation: _Allocation, position: int, depth: int) -> bool: + if self.grouped: + return all( + position in allocation.written[self._group_plane(group, depth)][layer] + for group in self.groups + for layer in group.layers + ) plane = allocation.written[self._plane(depth)] return all(position in plane[layer] for layer in range(self.num_layers)) @@ -629,6 +1000,7 @@ def snapshot(self, request_id: str) -> KVSnapshot: Copying page by page avoids a request-sized temporary on a device that is already full, which is exactly when preemption happens. """ + self._require_legacy() allocation = self._get_allocation(request_id) if allocation.transfer_leases: raise RuntimeError("cannot snapshot KV during a transfer") @@ -660,6 +1032,7 @@ def restore(self, request_id: str, snapshot: KVSnapshot) -> None: decision belongs to the scheduler. The copies go on the current stream; the caller decides when other streams may read them. """ + self._require_legacy() allocation = self._get_allocation(request_id) if allocation.max_tokens != snapshot.max_tokens: raise ValueError("snapshot capacity does not match the allocation") @@ -680,9 +1053,238 @@ def restore(self, request_id: str, snapshot: KVSnapshot) -> None: self.value_scale[block].copy_(snapshot.value_scales[row]) allocation.written = deepcopy(snapshot.written) + def _prefix_claim(self, prefix): + """Expired pages per group plane, the tables a hit attaches, and its blocks. + + A hit skips the leading SWA pages its next token cannot read, so it + takes no references to them. + """ + cached_tokens = len(prefix) * self.block_size + retired_pages = [ + [self._expired_swa_pages(group, cached_tokens) if self.reclaim_swa else 0] + * group.planes + for group in self.groups + ] + if not (self.grouped and prefix): + return retired_pages, (), [b for bundle in prefix for b in bundle] + tables = self._group_prefix_tables(prefix, retired_pages) + claimed = [ + b + for planes in tables + for table in planes + for bundle in table + if bundle is not None + for b in bundle + ] + return retired_pages, tables, claimed + + def _group_prefix_tables(self, prefix, retired_pages): + """Attach only the cached pages each group can still read.""" + offset, result = 0, [] + for group, retired_planes in zip(self.groups, retired_pages): + planes = [] + width = len(group.layers) + for retired in retired_planes: + end = offset + width + planes.append( + [None] * retired + [tuple(page[offset:end]) for page in prefix[retired:]] + ) + offset = end + result.append(tuple(planes)) + assert all(len(page) == offset for page in prefix) + return tuple(result) + + def _group_plane(self, group: KVGroup, depth: int) -> int: + """Storage plane owning ``depth`` for ``group``; boundary layers keep depth 0.""" + return self._plane(depth) if group.recurrent else 0 + + def _group_table(self, allocation, layer, depth): + group_id, local = self.layer_groups[layer] + group = self.groups[group_id] + if not group.recurrent and depth != 0: + raise ValueError("boundary attention layers only own depth zero") + return allocation.group_tables[group_id][self._group_plane(group, depth)], local + + def _layer_table(self, allocation, layer, depth): + bundles, local = self._group_table(allocation, layer, depth) + return tuple(-1 if bundle is None else bundle[local] for bundle in bundles) + + def layer_group(self, layer: int) -> tuple[int, KVGroup]: + """Read-only group policy for a compact KV layer; this does not pin pages. + + Group IDs are local to this manager. The returned policy is immutable. + """ + self._validate_layer(layer) + if not self.grouped: + raise NotImplementedError("layer groups require grouped KV storage") + group_id, _ = self.layer_groups[layer] + return group_id, self.groups[group_id] + + def layer_block_table(self, request_id, layer, depth): + """Read-only absolute logical-page map for one actual attention layer.""" + self._validate_layer(layer) + self._validate_depth(depth) + allocation = self._get_allocation(request_id) + if not self.grouped: + return allocation.block_tables[self._plane(depth)] + return self._layer_table(allocation, layer, depth) + + def layer_view(self, batch, layer): + """Resolve native ownership; SWA read tables use a page-aligned local span. + + Rows, RoPE positions and write addresses retain their absolute meaning. + Only the attention planner sees positions relative to each row's span. + """ + self._validate_layer(layer) + self._require_live_batch(batch) + if not isinstance(batch, GroupedKVMetadata): + raise TypeError("grouped KV requires a logical grouped descriptor") + group_id, _ = self.layer_groups[layer] + group = self.groups[group_id] + epochs = tuple( + a.group_epochs[group_id][self._group_plane(group, d)] for a, d, _ in batch.rows + ) + cached = batch.layer_views.get(layer) + if cached is not None and cached[0] == epochs: + return cached[1] + tables, blocks, read_positions = [], [], [] + positions = [p for _, _, p in batch.rows] + for allocation, depth, position in batch.rows: + bundles, local = self._group_table(allocation, layer, depth) + page = position // self.block_size + if page >= len(bundles): + raise ValueError("KV row exceeds the acquired physical capacity") + bundle = bundles[page] + blocks.append(-1 if bundle is None else bundle[local]) + first = ( + 0 + if group.window is None + else max(0, position - group.window + 1) // self.block_size + ) + tables.append(tuple(-1 if b is None else b[local] for b in bundles[first : page + 1])) + read_positions.append(position - first * self.block_size) + offsets = [p % self.block_size for p in positions] + if any(block < 0 for block in blocks): + raise RuntimeError("current KV write/query page is no longer resident") + if self.reclaim_swa: + self._require_resident_rows(batch, layer, group.window) + if batch.writable and len(set(zip(blocks, offsets))) != len(blocks): + raise ValueError("duplicate physical KV write addresses") + attention = plan_attention_metadata( + AttentionRows(block_tables=tables, positions=read_positions), + block_size=self.block_size, + ) + layout = KVLayout( + self, + batch.rows, + batch.allocations, + positions, + blocks, + offsets, + attention, + batch.writable, + ) + view = stage_metadata((layout,), self._stage, padded_rows=len(batch.position_ids))[0] + batch.layer_views[layer] = (epochs, view) + return view + + def retained_range(self, request_id, layer, depth): + """Absolute resident start and logical written end for a layer/plane.""" + self._validate_layer(layer) + self._validate_depth(depth) + allocation = self._get_allocation(request_id) + start = 0 + if self.grouped: + self._group_table(allocation, layer, depth) + group_id, _ = self.layer_groups[layer] + plane = self._group_plane(self.groups[group_id], depth) + start = allocation.retired_pages[group_id][plane] * self.block_size + return start, allocation.written[self._plane(depth)][layer].prefix + + def _require_resident_rows(self, batch, layer, window): + for allocation, depth, position in batch.rows: + bundles, _ = self._group_table(allocation, layer, depth) + start = 0 if window is None else max(0, position - window + 1) + if position // self.block_size >= len(bundles): + raise ValueError("KV row exceeds the acquired physical capacity") + if any( + bundles[page] is None + for page in range(start // self.block_size, position // self.block_size + 1) + ): + raise RuntimeError("required KV history is no longer resident") + + def _record_consumed(self, layer, batch): + group_id, _ = self.layer_groups[layer] + if self.groups[group_id].window is None: + return + group = self.groups[group_id] + for allocation, depth, position in batch.rows: + # Shared recurrent history is still read by later loops of this token. + if self.layout == "shared" and group.recurrent and depth != self.max_loops - 1: + continue + allocation.consumed[self._plane(depth)][layer].add(position) + allocation.reclaim_pending = True + + def _expired_swa_pages(self, group, frontier): + """Count whole pages the next token at ``frontier`` cannot read.""" + if group.window is None: + return 0 + return max(0, frontier - group.window + 1) // self.block_size + + def finish_submission(self, event=None): + """Return whole expired bundles only after this submission's final reader. + + Consumption candidates are host progress, not device completion. Reuse + the ordinary runner's event and wait only when physical pages retire. + """ + if not self.reclaim_swa: + return + plans = [] + pending = [a for a in self._allocations.values() if a.reclaim_pending] + for allocation in pending: + for group_id, group in enumerate(self.groups): + if group.window is None: + continue + for plane, table in enumerate(allocation.group_tables[group_id]): + frontier = min( + allocation.consumed[plane][layer].prefix for layer in group.layers + ) + stop = self._expired_swa_pages(group, frontier) + start = allocation.retired_pages[group_id][plane] + if stop > start: + plans.append((allocation, group_id, plane, start, stop)) + if plans: + if event is None and self.device.type == "cuda": + raise ValueError("CUDA reclamation requires the submission completion event") + if event is not None: + event.synchronize() + # Deferred publications are now complete; index them before retiring. + self.poll_prefixes() + for allocation, group_id, plane, start, stop in plans: + table = allocation.group_tables[group_id][plane] + released = [] + for page in range(start, stop): + if table[page] is not None: + released.extend(table[page]) + table[page] = None + allocation.retired_pages[group_id][plane] = stop + allocation.group_epochs[group_id][plane] += 1 + self._drop_refs(released) + for allocation in pending: + allocation.reclaim_pending = False + def _plane(self, depth: int) -> int: return depth if self.layout == "last_exited" else 0 + def _storage_layer(self, layer: int) -> int: + """Physical page slot holding ``layer``; grouped pages hold exactly one.""" + return 0 if self.grouped else layer + + def _require_legacy(self) -> None: + """Require legacy all-layer KV storage.""" + if self.grouped: + raise NotImplementedError("grouped KV supports ordinary layer views only") + def _get_allocation(self, request_id: str) -> _Allocation: try: return self._allocations[request_id] @@ -696,6 +1298,8 @@ def _validate_layer(self, layer: int) -> None: or not 0 <= layer < self.num_layers ): raise ValueError(f"layer must be in [0, {self.num_layers})") + if self.grouped and layer not in self.layer_groups: + raise ValueError("layer has no attention KV storage") def _validate_depth(self, depth: int) -> None: if ( @@ -732,7 +1336,9 @@ def _validate_rows(self, request_ids, depths, positions): def metadata_builder(self): return MetadataBuilder(self) - def prepare_recurrent(self, batch: ExecutionBatch, *, packed=False) -> KVMetadata: + def prepare_recurrent( + self, batch: ExecutionBatch, *, packed=False + ) -> KVMetadata | GroupedKVMetadata: """Compatibility entry for standalone cache consumers.""" ids, depths, positions = batch.addresses() return self._prepare_batch( @@ -752,7 +1358,7 @@ def _prepare_batch( packed_prefill: bool = False, stage: Callable[[Sequence[int], torch.dtype], torch.Tensor] | None = None, padded_rows: int | None = None, - ) -> KVMetadata: + ) -> KVMetadata | GroupedKVMetadata: """Build layer-independent addresses once; do not initialize any KV slot.""" return self._prepare_batches( request_ids, @@ -774,7 +1380,7 @@ def _prepare_batches( packed_prefill: bool = False, stage: Callable[[Sequence[int], torch.dtype], torch.Tensor] | None = None, padded_rows: int | None = None, - ) -> tuple[KVMetadata, ...]: + ) -> tuple[KVMetadata | GroupedKVMetadata, ...]: """Convenience API; layout construction belongs to MetadataBuilder.""" layouts = self.metadata_builder.build( request_ids, @@ -791,8 +1397,12 @@ def _stage(self, values, dtype): host = torch.tensor(values, dtype=dtype, pin_memory=self.device.type == "cuda") return host.to(self.device, non_blocking=True) - def _require_live_batch(self, batch: KVMetadata | DeviceKVMetadata): - if isinstance(batch, KVMetadata) and batch.owner is not self: + def _require_live_batch( + self, batch: KVMetadata | GroupedKVMetadata | DeviceKVMetadata + ) -> _PreparedCoverage | None: + if self.grouped and isinstance(batch, DeviceKVMetadata): + raise NotImplementedError("grouped KV does not support device-owned metadata") + if isinstance(batch, (KVMetadata, GroupedKVMetadata)) and batch.owner is not self: raise ValueError("prepared KV batch belongs to a different cache manager") plan = None reusable = self._prepared_coverage is not None and isinstance(batch, KVMetadata) @@ -848,7 +1458,7 @@ def write( def _write_prepared( self, layer: int, - batch: KVMetadata | DeviceKVMetadata, + batch: KVMetadata | GroupedKVMetadata | DeviceKVMetadata, k: torch.Tensor, v: torch.Tensor, ) -> None: @@ -869,6 +1479,8 @@ def _write_prepared( self.key_cache[batch.write_blocks, layer, batch.write_offsets] = k self.value_cache[batch.write_blocks, layer, batch.write_offsets] = v return + if self.grouped: + batch = self.layer_view(batch, layer) if not batch.writable: raise ValueError("a read-only prepared KV batch cannot be written") self._validate_tensor(k, len(batch.position_ids), "k") @@ -876,7 +1488,9 @@ def _write_prepared( rows = len(batch.rows) if not rows: return - self._store_layer(layer, batch.write_blocks, batch.write_offsets, k[:rows], v[:rows]) + self._store_layer( + self._storage_layer(layer), batch.write_blocks, batch.write_offsets, k[:rows], v[:rows] + ) if coverage is None: for allocation, depth, position in batch.rows: allocation.written[self._plane(depth)][layer].add(position) @@ -970,7 +1584,7 @@ def attend( def _attend_prepared( self, layer: int, - batch: KVMetadata | DeviceKVMetadata, + batch: KVMetadata | GroupedKVMetadata | DeviceKVMetadata, q: torch.Tensor, *, window: int | None = None, @@ -978,9 +1592,20 @@ def _attend_prepared( self._validate_layer(layer) coverage = self._require_live_batch(batch) self._validate_tensor(q, len(batch.position_ids), "q", query=True) + logical = batch + if self.grouped: + group_id, _ = self.layer_groups[layer] + if window != self.groups[group_id].window: + raise ValueError("native attention window must match the layer policy") + batch = self.layer_view(batch, layer) if window is not None: if batch.cu_seqlens_q is None: - return self._attend_windowed(layer, batch, q, window) + output = self._attend_windowed(layer, batch, q, window) + if self.reclaim_swa: + self._record_consumed(layer, logical) + return output + if self.reclaim_swa: + raise NotImplementedError("SWA reclamation requires unpacked prefill") if not self.attention_capabilities.packed_sliding_window: raise NotImplementedError("sliding-window attention needs one query row per token") if type(window) is not int or window <= 0: @@ -1010,8 +1635,8 @@ def _attend_prepared( ) return self.attention.attend_metadata( q, - self.key_cache[:, layer], - self.value_cache[:, layer], + self.key_cache[:, self._storage_layer(layer)], + self.value_cache[:, self._storage_layer(layer)], batch, **self._layer_scales(layer), **({} if window is None else {"window": window}), @@ -1041,8 +1666,8 @@ def _attend_windowed( self._require_prefix(allocation, layer, depth, position + 1) return self.attention.windowed( q, - self.key_cache[:, layer], - self.value_cache[:, layer], + self.key_cache[:, self._storage_layer(layer)], + self.value_cache[:, self._storage_layer(layer)], batch.block_tables, batch.context_lengths, window, @@ -1052,10 +1677,29 @@ def _attend_windowed( @torch.no_grad() def finalize_token(self, request_id: str, position: int, exit_depth: int) -> None: """Propagate the final executed depth's K/V into every unexecuted depth.""" + if self.reclaim_swa and exit_depth != self.max_loops - 1: + raise NotImplementedError( + "SWA reclamation currently requires full fixed-loop execution" + ) allocation = self._require_exit_written(request_id, position, exit_depth) - if self.layout == "shared": + if self.layout == "shared" or exit_depth == self.max_loops - 1: return logical_page, offset = divmod(position, self.block_size) + if self.grouped: + for layer in self.recurrent_layers: + source = self.layer_block_table(request_id, layer, exit_depth)[logical_page] + if source < 0: + raise RuntimeError("cannot finalize reclaimed KV") + for depth in range(exit_depth + 1, self.max_loops): + destination = self.layer_block_table(request_id, layer, depth)[logical_page] + if destination < 0: + raise RuntimeError("cannot finalize into reclaimed KV") + self.key_cache[destination, 0, offset].copy_(self.key_cache[source, 0, offset]) + self.value_cache[destination, 0, offset].copy_( + self.value_cache[source, 0, offset] + ) + self._record_finalized(allocation, position, exit_depth) + return source = allocation.block_tables[self._plane(exit_depth)][logical_page] destinations = [ allocation.block_tables[self._plane(depth)][logical_page] @@ -1081,6 +1725,8 @@ def read(self, layer: int, request_id: str, depth: int, length: int | None = Non self._validate_layer(layer) self._validate_depth(depth) allocation = self._get_allocation(request_id) + if self.grouped: + self._group_table(allocation, layer, depth) if length is None: length = allocation.written[self._plane(depth)][layer].prefix if ( @@ -1090,17 +1736,28 @@ def read(self, layer: int, request_id: str, depth: int, length: int | None = Non ): raise ValueError("read length exceeds the request's reserved token budget") self._require_prefix(allocation, layer, depth, length) + if self.grouped and length and self.retained_range(request_id, layer, depth)[0]: + raise RuntimeError( + "KV prefix has been reclaimed; use retained_range and layer_block_table" + ) positions = torch.arange(length, device=self.device) table = torch.tensor( - allocation.block_tables[self._plane(depth)], device=self.device, dtype=torch.long + ( + self._layer_table(allocation, layer, depth) + if self.grouped + else allocation.block_tables[self._plane(depth)] + ), + device=self.device, + dtype=torch.long, ) blocks = table[positions // self.block_size] offsets = positions % self.block_size - keys = self.key_cache[blocks, layer, offsets] - values = self.value_cache[blocks, layer, offsets] + storage_layer = self._storage_layer(layer) + keys = self.key_cache[blocks, storage_layer, offsets] + values = self.value_cache[blocks, storage_layer, offsets] if self.key_scale is None: return keys, values return ( - dequantize_fp8(keys, self.key_scale[blocks, layer, offsets], self.dtype), - dequantize_fp8(values, self.value_scale[blocks, layer, offsets], self.dtype), + dequantize_fp8(keys, self.key_scale[blocks, storage_layer, offsets], self.dtype), + dequantize_fp8(values, self.value_scale[blocks, storage_layer, offsets], self.dtype), ) diff --git a/vllm_rlt/core/memory.py b/vllm_rlt/core/memory.py index 1c52c16c..1cfdff5a 100644 --- a/vllm_rlt/core/memory.py +++ b/vllm_rlt/core/memory.py @@ -135,14 +135,24 @@ def plan_cache( speculative=None, staged_prefill=None, fp8_scale=None, + layer_windows, + reclaim_swa, ): parameter = next(model.parameters()) config, device = model.config, parameter.device # Models whose physical stack mixes KV-owning and KV-free sublayers (Dory) # expose the compact count; everything else keeps the layer count. kv_layers = getattr(config, "num_kv_layers", config.num_hidden_layers) + # The engine picks the layout: grouped windows, or None for all-layer blocks. + native = layer_windows is not None + unit = "physical_layer_page" if native else "all_layer_block" + recurrent = set(getattr(model, "recurrent_kv_layers", range(kv_layers))) + depths = config.total_ut_steps if cache.layout == "last_exited" else 1 + page_factor = ( + sum(depths if layer in recurrent else 1 for layer in layer_windows) if native else depths + ) per_block = kv_bytes_per_block( - kv_layers, + 1 if native else kv_layers, cache.block_size, config.num_key_value_heads, config.head_dim, @@ -150,14 +160,25 @@ def plan_cache( cache.kv_cache_dtype, ) if cache.num_blocks is not None: - return cache.num_blocks, {"source": "blocks", "bytes_per_block": per_block} + blocks = cache.num_blocks * (kv_layers if native else 1) + return blocks, { + "source": "blocks", + "bytes_per_block": per_block, + "block_unit": unit, + "configured_num_blocks": cache.num_blocks, + } if cache.kv_cache_memory_bytes is not None: blocks = budget_blocks(cache.kv_cache_memory_bytes, per_block) - return blocks, {"source": "bytes", "bytes_per_block": per_block} + return blocks, {"source": "bytes", "bytes_per_block": per_block, "block_unit": unit} if device.type == "npu": raise ValueError("NPU requires explicit num_blocks or kv_cache_memory_bytes") if device.type != "cuda": - return 256, {"source": "cpu_default", "bytes_per_block": per_block} + return 256 * (kv_layers if native else 1), { + "source": "cpu_default", + "bytes_per_block": per_block, + "block_unit": unit, + "configured_num_blocks": 256, + } torch.cuda.synchronize(device) # Return unused allocator segments before measuring driver-free memory. # In particular, a previous engine may have left its KV pool cached here. @@ -182,8 +203,7 @@ def plan_cache( num_kv_heads=config.num_key_value_heads, head_dim=config.head_dim, max_loops=config.total_ut_steps, - num_blocks=sum(math.ceil(length / cache.block_size) for length in lengths) - * (config.total_ut_steps if cache.layout == "last_exited" else 1), + num_blocks=sum(math.ceil(length / cache.block_size) for length in lengths) * page_factor, block_size=cache.block_size, layout=cache.layout, device=device, @@ -192,6 +212,7 @@ def plan_cache( recurrent_layers=getattr(model, "recurrent_kv_layers", None), kv_cache_dtype=cache.kv_cache_dtype, fp8_scale=fp8_scale, + layer_windows=layer_windows, ) ids, positions = [], [] for index, length in enumerate(lengths): @@ -243,15 +264,37 @@ def plan_cache( ) # More blocks than all concurrent full-context requests can use only waste # memory, particularly for tiny-model GPU tests with very small pages. - useful_blocks = ( - math.ceil(config.max_position_embeddings / cache.block_size) - * (config.total_ut_steps if cache.layout == "last_exited" else 1) - * scheduler.max_num_seqs - ) + if reclaim_swa and cache.incremental_allocation: + submission = min(scheduler.prefill_chunk_size, scheduler.max_num_batched_tokens) + pages = math.ceil(config.max_position_embeddings / cache.block_size) + useful_blocks = ( + sum( + ( + min( + pages, + math.ceil( + (window - 1 + submission + cache.block_size - 1) / cache.block_size + ), + ) + if window is not None + else pages + ) + * (depths if layer in recurrent else 1) + for layer, window in layer_windows.items() + ) + * scheduler.max_num_seqs + ) + else: + useful_blocks = ( + math.ceil(config.max_position_embeddings / cache.block_size) + * page_factor + * scheduler.max_num_seqs + ) blocks = min(budget_blocks(budget, per_block), useful_blocks) return blocks, dict( source="cuda_profile", bytes_per_block=per_block, + block_unit=unit, profile_peak_bytes=peak, concurrent_peak_bytes=concurrent_peak, max_useful_blocks=useful_blocks, diff --git a/vllm_rlt/core/scheduler.py b/vllm_rlt/core/scheduler.py index 14d95950..a3f10f4e 100644 --- a/vllm_rlt/core/scheduler.py +++ b/vllm_rlt/core/scheduler.py @@ -346,15 +346,23 @@ def _try_resume(self, request: Request) -> ResumeResult: self.enqueue(request, restored) return ResumeResult.RESTORED - def _reserved_growth_blocks(self, *, reserve_outputs: bool) -> int: + def _reserved_growth_blocks( + self, *, reserve_outputs: bool, independent_budget: bool = False + ) -> int: """Budget future growth of existing requests without allocating it. Example: a request needs 12 blocks eventually but owns 4. Without preemption, reserve the remaining 8 even though they still look free. With preemption, protect in-progress prompts; decode growth may instead reclaim capacity by suspending another request later. + + ``independent_budget`` counts every request's full peak and subtracts + current use once, so shared pages cannot hide growth. That subtraction + assumes every allocated request is counted, which needs reserve_outputs. """ cache = self.cache_manager + if independent_budget and not reserve_outputs: + raise ValueError("independent budgeting requires reserved outputs") reserved = 0 for request in self.requests.values(): # Imported/reserved requests already own pages before compute admission. @@ -369,9 +377,9 @@ def _reserved_growth_blocks(self, *, reserve_outputs: bool) -> int: tokens = len(request.prompt_token_ids) if reserve_outputs: tokens += request.sampling_params.max_tokens - 1 - allocated = cache.allocated_blocks(request.request_id) + allocated = 0 if independent_budget else cache.charged_blocks(request.request_id) reserved += max(0, cache.required_blocks(tokens) - allocated) - return reserved + return max(0, reserved - cache.num_used_blocks) if independent_budget else reserved def _plan_admission(self, request: Request, active_count: int) -> AdmissionPlan: """Describe prefix reuse, physical allocation and logical admission cost. @@ -388,26 +396,37 @@ def _plan_admission(self, request: Request, active_count: int) -> AdmissionPlan: capacity = len(request.prompt_token_ids) + request.sampling_params.max_tokens - 1 prefix = cache.lookup_prefix(request.prompt_token_ids) cached_tokens = len(prefix) * cache.block_size - initial_tokens = min( - len(request.prompt_token_ids), cached_tokens + self.config.prefill_chunk_size + chunk_size = ( + cache.max_submission_tokens + if cache.submission_bounded + else self.config.prefill_chunk_size + ) + initial_tokens = ( + min(len(request.prompt_token_ids), cached_tokens + chunk_size) + if cache.incremental_allocation + else capacity ) - if not cache.incremental_allocation: - initial_tokens = capacity reserve_outputs = cache.incremental_allocation and self.suspend_callback is None budget_tokens = ( len(request.prompt_token_ids) if cache.incremental_allocation and not reserve_outputs else capacity ) - required = cache.required_blocks(budget_tokens) - len(prefix) * cache.storage_depths - reserved = self._reserved_growth_blocks(reserve_outputs=reserve_outputs) + # SWA sharers can retire the same page at different times. Budget each + # request's peak independently so sharing cannot leave growth underfunded. + # Preallocation already reserves each whole peak. + independent_budget = cache.submission_bounded and cache.enable_prefix_caching + required = cache.required_blocks(budget_tokens) + cached_claim = 0 + if not independent_budget: + required -= len(prefix) * cache.pages_per_token_block + # Prefix-only blocks count as free until a request claims them. + cached_claim = cache.exclusive_prefix_blocks(prefix) + reserved = self._reserved_growth_blocks( + reserve_outputs=reserve_outputs, independent_budget=independent_budget + ) # Allow an otherwise idle engine to start a large request. admission_headroom_blocks = cache.watermark_blocks if active_count else 0 - # num_free_blocks includes evictable prefix-only blocks. Reusing those - # blocks makes them non-evictable: do not count the same capacity twice. - # Example: free=8 fresh + 4 cached, demand=12 with a 4-block prefix. - # New demand is 8, but the cached claim is 4: admission costs 12, not 8. - cached_claim = cache.exclusive_prefix_blocks(prefix) return AdmissionPlan( capacity_tokens=capacity, initial_tokens=initial_tokens, @@ -419,7 +438,7 @@ def _plan_admission(self, request: Request, active_count: int) -> AdmissionPlan: cached_claim_blocks=cached_claim, ) - def reserve_transfer(self, request: Request) -> AdmissionPlan | None: + def reserve_transfer(self, request: Request, *, receiving: bool) -> AdmissionPlan | None: """Reserve transport KV under the same growth budget as local admission. A transfer needs its whole prompt mapped before compute starts. Reserving @@ -430,15 +449,23 @@ def reserve_transfer(self, request: Request) -> AdmissionPlan | None: plan = self._plan_admission(request, int(occupied)) if self.cache_manager.incremental_allocation: plan = replace(plan, initial_tokens=len(request.prompt_token_ids)) - if not self._allocate(request, plan): + if not self._allocate(request, plan, receiving=receiving): return None self.cache_manager.pin_transfer(request.request_id, request.request_id) return plan - def _allocate(self, request: Request, plan: AdmissionPlan) -> bool: + def _allocate(self, request: Request, plan: AdmissionPlan, *, receiving=False) -> bool: cache = self.cache_manager if plan.total_budget_blocks > cache.num_free_blocks: return False + if receiving: + return cache.allocate_import( + request.request_id, + plan.capacity_tokens, + len(request.prompt_token_ids), + initial_tokens=plan.initial_tokens, + prefix=plan.prefix_blocks, + ) return cache.allocate( request.request_id, plan.capacity_tokens, diff --git a/vllm_rlt/engine/stage_engine.py b/vllm_rlt/engine/stage_engine.py index 3e3866af..c6f1685c 100644 --- a/vllm_rlt/engine/stage_engine.py +++ b/vllm_rlt/engine/stage_engine.py @@ -69,7 +69,11 @@ def __init__( required_layout = getattr(model, "kv_cache_layout", "last_exited") if cache_config.layout != required_layout: raise ValueError(f"boundary-KV models require {required_layout} KV") - if cache_config.enable_prefix_caching: + # Only models that verified hit/miss parity opt in: the cache must + # cover boundary KV at depth 0 and recurrent KV at every depth. + if cache_config.enable_prefix_caching and not getattr( + model, "supports_boundary_prefix_caching", False + ): raise ValueError("boundary-KV prefix caching is not yet supported") self.exit_config = exit_config or ExitConfig() self.execution_config = resolve_cuda_graph_rows( @@ -100,6 +104,8 @@ def __init__( scheduler_config, produces_tokens=self.stage_config.produces_tokens ) sliding_window = getattr(model, "requires_sliding_window", False) + # Automatic packed prefill wins where supported; it needs all-layer + # blocks, so packed_prefill=False keeps grouped storage and reclamation. prefer_packed = packed_prefill_eligible( self.execution_config, device_type=parameter.device.type, @@ -133,6 +139,13 @@ def __init__( "boundary-KV speculative decoding requires synchronous eager " "execution without CUDA graphs or preemption" ) + if cache_config.enable_prefix_caching and cache_config.layout == "shared": + # The shared runner finishes every loop of a token before the next, + # so cached final-loop KV is the cold prefill state. + if not getattr(model, "supports_boundary_prefix_caching", False): + raise ValueError("this model does not support shared-KV prefix caching") + if self.execution_config.async_scheduling or self.execution_config.cuda_graphs: + raise ValueError("shared-KV prefix caching requires synchronous eager execution") if not getattr(model, "supports_cuda_graphs", True) and self.execution_config.cuda_graphs: raise ValueError("this model does not support CUDA graphs yet") self.speculative_config = speculative_config @@ -258,6 +271,28 @@ def __init__( self.execution_config = replace( self.execution_config, packed_prefill=prefer_packed and supported ) + layer_windows = getattr(model, "kv_layer_windows", None) + # Graph, async, packed, speculative, FP8 and preemption paths keep + # all-layer blocks: their metadata, rollback, scales or CPU offload have + # no grouped support yet. + if layer_windows is not None and ( + self.execution_config.cuda_graphs + or self.execution_config.async_scheduling + or self.execution_config.packed_prefill + or speculative_config is not None + or cache_config.kv_cache_dtype != "auto" + or scheduler_config.enable_preemption + ): + layer_windows = None + # Grouped pages expire only when every token runs all loops in ouro order. + # A prefill stage holds a prompt only until its transfer completes, and + # reclamation there could return a page that a transfer still reads. + reclaim_swa = ( + layer_windows is not None + and getattr(model, "fixed_loop_count", False) + and self.exit_config.mode == "ouro" + and self.stage_config.produces_tokens + ) num_blocks, self.memory_plan = plan_cache( model, cache_config, @@ -267,6 +302,8 @@ def __init__( speculative=speculative_config, staged_prefill=staged_prefill, fp8_scale=fp8_scale, + layer_windows=layer_windows, + reclaim_swa=reclaim_swa, ) self.cache_manager = KVCacheManager( num_layers=kv_layers, @@ -286,6 +323,11 @@ def __init__( recurrent_layers=getattr(model, "recurrent_kv_layers", None), kv_cache_dtype=cache_config.kv_cache_dtype, fp8_scale=fp8_scale, + layer_windows=layer_windows, + reclaim_swa=reclaim_swa, + max_submission_tokens=min( + scheduler_config.prefill_chunk_size, scheduler_config.max_num_batched_tokens + ), ) self.scheduler = Scheduler( scheduler_config, @@ -589,7 +631,10 @@ def _step(self) -> list[RequestOutput]: # Only host-dependent decisions collect now. Prompt/PRELUDE submissions # advance logically under their event without a readback wait. result = ticket.collect() if ticket.has_result else None - return self._update(batch, result) + outputs = self._update(batch, result) + # Publish this submission's prompt pages before SWA retires them. + self.cache_manager.finish_submission(ticket.event) + return outputs except Exception: if self.connector is not None: raise # Transfer leases remain quarantined until peer shutdown. @@ -656,7 +701,7 @@ def reserve_transfer( else: self.add_request(request_id, prompt_token_ids, sampling_params, trace_id=trace_id) request = self.scheduler.requests[request_id] - plan = self.scheduler.reserve_transfer(request) + plan = self.scheduler.reserve_transfer(request, receiving=receiving) if plan is None: self.scheduler.abort(request_id) return None @@ -728,7 +773,12 @@ def accept_import(self, request, hidden, *, cached_tokens=0): ): raise ValueError("request is not awaiting an import") count = len(request.prompt_token_ids) - self.cache_manager.mark_imported_prefix(request.request_id, count, start=cached_tokens) + self.cache_manager.mark_imported_prefix( + request.request_id, + count, + start=cached_tokens, + output_layers=getattr(self.model, "coda_kv_layers", ()), + ) self.cache_manager.publish_prefix(request.request_id, request.prompt_token_ids, count) request.num_prefilled_tokens = request.num_preluded_tokens = count request.prefill_window_origin = count diff --git a/vllm_rlt/models/dory.py b/vllm_rlt/models/dory.py index 8e6a388e..1e5b92bc 100644 --- a/vllm_rlt/models/dory.py +++ b/vllm_rlt/models/dory.py @@ -491,6 +491,9 @@ class DoryForCausalLM(nn.Module): # Synchronous self-speculation with per-loop KV: a draft's loops are the # target's first loops, and boundary KV is committed by the target pass. supports_self_speculation = True + # Complete-state prefix reuse: boundary KV is published from depth 0 and + # recurrent KV from every depth (per_loop) or the final loop (last_loop). + supports_boundary_prefix_caching = True def __init__(self, config: DoryConfig) -> None: super().__init__() @@ -512,6 +515,12 @@ def __init__(self, config: DoryConfig) -> None: self.coda_kv_layers = tuple( config.kv_layer_index(index) for index in config.attention_layers("output") ) + self.kv_layer_windows = { + config.kv_layer_index(index): ( + config.sliding_window if config.layer_types[index] == "sliding_attention" else None + ) + for index in config.attention_layer_ids + } self.apply(self._initialize) self.requires_grad_(False) self.eval() diff --git a/vllm_rlt/pd/connector.py b/vllm_rlt/pd/connector.py index 9b5ed676..b3a7c875 100644 --- a/vllm_rlt/pd/connector.py +++ b/vllm_rlt/pd/connector.py @@ -219,7 +219,7 @@ def prefill_submitted(self, request, *, phase, start, progress, event, stream): def queue_chunk(self, transfer, start, end, event, final): peer = self.transport.peers[transfer.peer] - tables = self.cache.plane_block_tables(transfer.tid) + tables = self.cache.transfer_tables(transfer.tid) start = max(start, transfer.target_cached_tokens) if start < end: boundary = (start, end) @@ -227,8 +227,14 @@ def queue_chunk(self, transfer, start, end, event, final): transfer.transfer_boundary_first = boundary transfer.transfer_boundary_last = boundary transfer.transfer_boundary_count += 1 - source = list(kv_segments(self.transport.info, tables, start, end)) if start < end else [] - target = list(kv_segments(peer, transfer.target_tables, start, end)) if start < end else [] + source, target = [], [] + page = self.transport.info["block_size"] + for local, remote in zip(tables, transfer.target_tables): + # The receiver holds no SWA page that its first query cannot read. + lo = max(start, next(i for i, block in enumerate(remote) if block >= 0) * page) + if lo < end: + source.extend(kv_segments(self.transport.info, (local,), lo, end)) + target.extend(kv_segments(peer, (remote,), lo, end)) if final: size = self.transport.info["hidden_bytes"] source.append( @@ -291,7 +297,7 @@ def reserve( self.notify( "reserved", tid=tid, - tables=self.cache.plane_block_tables(tid), + tables=self.cache.transfer_tables(tid), slot=transfer.slot, cached_tokens=hit, ) diff --git a/vllm_rlt/pd/engine.py b/vllm_rlt/pd/engine.py index 26f7e51d..0394f8fd 100644 --- a/vllm_rlt/pd/engine.py +++ b/vllm_rlt/pd/engine.py @@ -14,7 +14,7 @@ from vllm_rlt.config import CacheConfig, ExecutionConfig, ExitConfig, SchedulerConfig from vllm_rlt.engine.llm_engine import StageClient from vllm_rlt.engine.stage_config import StageConfig -from vllm_rlt.models import prepare_model +from vllm_rlt.models import MODEL_MAPPING, prepare_model from vllm_rlt.profiling import ProfileConfig from vllm_rlt.profiling_artifacts import _write_json_atomic, profile_timestamp from vllm_rlt.request import FinishReason, Request, RequestOutput, RequestPhase @@ -76,10 +76,11 @@ def __init__( ) if self.model_config.model_type == "huginn_raven": raise ValueError("PD does not support Huginn") - if getattr(model, "requires_boundary_kv", False): - raise ValueError( - f"PD does not support {self.model_config.model_type} boundary-KV models" - ) + # A path or a config source has no model instance yet; the class + # declares the capability for every source. + architecture = MODEL_MAPPING[self.model_config.model_type] + if speculative_config is not None and getattr(architecture, "requires_boundary_kv", False): + raise ValueError("PD does not support boundary-KV speculative decoding yet") self.config = pd_config or PDConfig() self.exit_config = exit_config or ExitConfig( "ouro" if speculative_config else "ouro_delayed" diff --git a/vllm_rlt/pd/transport.py b/vllm_rlt/pd/transport.py index f1b2a3d5..332f18af 100644 --- a/vllm_rlt/pd/transport.py +++ b/vllm_rlt/pd/transport.py @@ -29,8 +29,9 @@ def _kv_region(key, value): def kv_segments(info, tables, start, end): """Map valid token ranges to contiguous byte segments, preserving every depth. - A full physical block holds all layers. Partial blocks need one range per - layer; unused tail bytes are never sent or advertised as initialized. + A full physical block holds every layer that it stores: all layers, or one + layer for a grouped page. Partial blocks need one range per layer; unused + tail bytes are never sent or advertised as initialized. """ page = info["block_size"] if not 0 <= start < end: @@ -146,7 +147,8 @@ def __init__(self, name, cache, hidden, config): device=k.device.index, num_blocks=cache.num_blocks, block_size=cache.block_size, - num_layers=cache.num_layers, + # Layers per physical block: a grouped page holds one layer. + num_layers=k.shape[1], depths=cache.storage_depths, kv_dtype=str(k.dtype), fp8_scale=cache.fp8_scale, diff --git a/vllm_rlt/worker/buffers.py b/vllm_rlt/worker/buffers.py index 621e1715..8a7836f0 100644 --- a/vllm_rlt/worker/buffers.py +++ b/vllm_rlt/worker/buffers.py @@ -5,7 +5,12 @@ import torch -from vllm_rlt.worker.metadata import KVLayout, KVMetadata +from vllm_rlt.worker.metadata import ( + GroupedKVLayout, + GroupedKVMetadata, + KVLayout, + KVMetadata, +) from vllm_rlt.worker.submission_queue import MAX_CONCURRENT_SUBMISSIONS, SubmissionQueue from vllm_rlt.worker.tensors import UvaBackedTensor, copy_tensor, use_uva_on @@ -95,6 +100,8 @@ def record_done(self): def store_metadata(self, layout: KVLayout, size): """Fill fixed host buffers from a storage-independent layout.""" + if isinstance(layout, GroupedKVLayout): + raise NotImplementedError("grouped KV does not support UVA metadata workspaces") n = len(layout.rows) attention = layout.attention if attention.cu_seqlens_q is not None: @@ -155,6 +162,18 @@ def tokens(self, ids, size): def stage_metadata(layouts, stage, *, padded_rows=None): """Store computed layouts in two typed buffers using caller-owned staging.""" first = layouts[0] + if isinstance(first, GroupedKVLayout): + n = len(first.rows) + size = n if padded_rows is None else padded_rows + if size < n: + raise ValueError("padding requires at least the real row count") + positions = stage(first.positions + [0] * (size - n), torch.int64) + return tuple( + GroupedKVMetadata( + layout.owner, layout.rows, layout.allocations, positions, layout.writable + ) + for layout in layouts + ) n = len(first.rows) size = n if padded_rows is None else padded_rows if size < n or (size != n and first.attention.cu_seqlens_q is not None): diff --git a/vllm_rlt/worker/metadata.py b/vllm_rlt/worker/metadata.py index 95fde7bb..5617fd31 100644 --- a/vllm_rlt/worker/metadata.py +++ b/vllm_rlt/worker/metadata.py @@ -7,7 +7,7 @@ from __future__ import annotations from collections.abc import Sequence -from dataclasses import dataclass, replace +from dataclasses import dataclass, field, replace from typing import TYPE_CHECKING import torch @@ -26,6 +26,8 @@ class KVMetadata: no write address. Packed query boundaries describe layout, not request stage. Host rows and allocation identities are captured independently of caller lists. Device tensors are private, read-only inputs to the cache operations. + Read tables and context lengths are backend-local spans: native SWA views + may start at a window page boundary. Rows and position_ids remain absolute. """ owner: KVCacheManager @@ -41,6 +43,31 @@ class KVMetadata: max_seqlen_q: int = 1 +@dataclass(frozen=True) +class GroupedKVLayout: + """Logical rows only; physical addresses depend on the attention layer.""" + + owner: KVCacheManager + rows: tuple[tuple[_Allocation, int, int], ...] + allocations: tuple[tuple[str, _Allocation], ...] + positions: list[int] + writable: bool + + +@dataclass(frozen=True, eq=False) +class GroupedKVMetadata: + """Borrowed logical traversal with descriptor-local resolved layer views.""" + + owner: KVCacheManager + rows: tuple[tuple[_Allocation, int, int], ...] + allocations: tuple[tuple[str, _Allocation], ...] + position_ids: torch.Tensor + writable: bool + layer_views: dict[int, tuple[tuple[int, ...], KVMetadata]] = field(default_factory=dict) + cu_seqlens_q: None = None + max_seqlen_q: int = 1 + + @dataclass(frozen=True, eq=False) class DeviceKVMetadata: """Stream-ordered speculative metadata with device-owned positions. @@ -107,6 +134,8 @@ def prepare_device( until GPU completion. This does not advance host written-position state. Allocation identity also rejects descriptors from another cache manager. """ + if self.cache.grouped: + raise NotImplementedError("grouped KV does not support device-owned metadata") batch = DeviceKVMetadata( allocations=tuple(allocations), position_ids=position_ids, @@ -165,6 +194,8 @@ def _write_addresses(self, rows, *, offsets=None, for_write=True): def resolve_rows(self, ids, depths, positions): """Validate host-known rows for device routing without building host tables.""" + if self.cache.grouped: + raise NotImplementedError("grouped KV does not support device routing") rows = tuple(self.cache._validate_rows(ids, depths, positions)) self._write_addresses(rows) return rows @@ -177,9 +208,11 @@ def build( *, packed=False, for_write=True, - ) -> tuple[KVLayout, ...]: + ) -> tuple[KVLayout | GroupedKVLayout, ...]: """Calculate all requested depth layouts without selecting a storage path.""" cache = self.cache + if cache.grouped and packed: + raise NotImplementedError("grouped KV supports ordinary host-planned rows only") if not depth_sets: raise ValueError("at least one depth assignment is required") if packed and ( @@ -204,6 +237,12 @@ def build( allocations = tuple(dict(zip(request_ids, (a for a, _, _ in first))).items()) layouts = [] for rows in row_sets: + if cache.grouped: + identities = [(id(a), cache._plane(d), p) for a, d, p in rows] + if for_write and len(set(identities)) != len(rows): + raise ValueError("duplicate logical KV write rows") + layouts.append(GroupedKVLayout(cache, rows, allocations, positions, for_write)) + continue blocks, _ = self._write_addresses(rows, offsets=offsets, for_write=for_write) attention = plan_attention_metadata( AttentionRows( @@ -283,6 +322,8 @@ def view(tensor): tensor.shape, tensor.stride(), tensor.storage_offset() ) + if isinstance(metadata, GroupedKVMetadata): + return replace(metadata, position_ids=view(metadata.position_ids), layer_views={}) return replace( metadata, position_ids=view(metadata.position_ids), @@ -296,6 +337,13 @@ def view(tensor): @staticmethod def slice_rows(metadata, start, end): """Select contiguous single-query rows, e.g. a dependent coda subset.""" + if isinstance(metadata, GroupedKVMetadata): + return replace( + metadata, + rows=metadata.rows[start:end], + position_ids=metadata.position_ids[start:end], + layer_views={}, + ) if metadata.cu_seqlens_q is not None: raise ValueError("packed query boundaries must be rebuilt when selecting rows") return replace( @@ -310,6 +358,10 @@ def slice_rows(metadata, start, end): @staticmethod def without_padding(metadata): + if isinstance(metadata, GroupedKVMetadata): + return replace( + metadata, position_ids=metadata.position_ids[: len(metadata.rows)], layer_views={} + ) if metadata.cu_seqlens_q is not None: return metadata count = len(metadata.rows) @@ -333,6 +385,8 @@ def prepare_shared_rows( loop_count, ): """Reuse one shared-plane table across token groups and loop assignments.""" + if isinstance(base, GroupedKVMetadata): + raise NotImplementedError("grouped KV does not support shared device row routing") count = len(spans) rows = tuple( (base.rows[row][0], span.depth, positions[row]) for row, span in enumerate(spans) diff --git a/vllm_rlt/worker/model_runner.py b/vllm_rlt/worker/model_runner.py index d7312d92..4e4fda1c 100644 --- a/vllm_rlt/worker/model_runner.py +++ b/vllm_rlt/worker/model_runner.py @@ -646,9 +646,7 @@ def _prepare_prefill_coda(self, ids, positions, requests, kv=None, *, packed=Non [ids[row] for row in rows], [[0] * len(rows)], [positions[row] for row in rows], - packed=self.cache_manager.attention_capabilities.packed_prefill - if packed is None - else packed, + packed=self._packs_prefill() if packed is None else packed, )[0] return ForwardBatch( Operation.CODA, prelude_coda_kv_metadata=kv_metadata, compute_logits=False @@ -658,12 +656,16 @@ def _prepare_shared_loop_inputs(self, batch): cache = self.cache_manager spans = sorted(batch.spans, key=lambda span: span.token_count, reverse=True) ids = [span.request.request_id for span in spans] - base = self._stage_metadata( - ids, - [[span.depth for span in spans]], - [span.token_start + span.token_count - 1 for span in spans], - padded_rows=self._size(len(spans)), - )[0] + base = ( + None + if cache.grouped + else self._stage_metadata( + ids, + [[span.depth for span in spans]], + [span.token_start + span.token_count - 1 for span in spans], + padded_rows=self._size(len(spans)), + )[0] + ) groups, positions, blocks, tokens, contexts = [], [], [], [], [] for offset in range(spans[0].token_count): active = [span for span in spans if offset < span.token_count] @@ -671,13 +673,15 @@ def _prepare_shared_loop_inputs(self, batch): for row, span in enumerate(active): position = span.token_start + offset positions.append(position) - blocks.append(base.rows[row][0].block_tables[0][position // cache.block_size]) + if not cache.grouped: + blocks.append(base.rows[row][0].block_tables[0][position // cache.block_size]) tokens.append(span.input_tokens[offset]) contexts.append(position + 1) size = self._size(len(active)) padding = size - len(active) positions.extend([0] * padding) - blocks.extend([0] * padding) + if not cache.grouped: + blocks.extend([0] * padding) tokens.extend([0] * padding) contexts.extend([0] * padding) outputs = tuple( @@ -687,24 +691,33 @@ def _prepare_shared_loop_inputs(self, batch): ) groups.append((start, size, active, outputs)) n = len(positions) - wide = self.input_buffers[torch.int64].append( - positions + blocks + [p % cache.block_size for p in positions] - ) - lengths = self.input_buffers[torch.int32].append(contexts) + if not cache.grouped: + wide = self.input_buffers[torch.int64].append( + positions + blocks + [p % cache.block_size for p in positions] + ) + lengths = self.input_buffers[torch.int32].append(contexts) token_ids = self.input_buffers[torch.int64].append(tokens) inputs = [] for start, size, active, outputs in groups: count, end = len(active), start + size - metadata = self.metadata.prepare_shared_rows( - base, - active, - positions[start : start + count], - position_ids=wide[start:end], - write_blocks=wide[n + start : n + start + count], - write_offsets=wide[2 * n + start : 2 * n + start + count], - context_lengths=lengths[start:end], - loop_count=batch.loop_count, - ) + if cache.grouped: + metadata = self._stage_metadata( + ids[:count], + [[span.depth + loop for span in active] for loop in range(batch.loop_count)], + positions[start : start + count], + padded_rows=size, + ) + else: + metadata = self.metadata.prepare_shared_rows( + base, + active, + positions[start : start + count], + position_ids=wide[start:end], + write_blocks=wide[n + start : n + start + count], + write_offsets=wide[2 * n + start : 2 * n + start + count], + context_lengths=lengths[start:end], + loop_count=batch.loop_count, + ) kv = metadata[0] requests = tuple(span.request for span in active) coda, coda_rows = self._prepare_prefill_coda( @@ -724,6 +737,16 @@ def _prepare_shared_loop_inputs(self, batch): inputs.append(PreparedForward(coda, outputs=(), hidden_rows=coda_rows)) return tuple(inputs) + def _packs_prefill(self): + # Packed metadata needs last_exited all-layer pages. Native-packing + # backends advertise it regardless of the cache layout. + cache = self.cache_manager + return ( + cache.attention_capabilities.packed_prefill + and cache.layout == "last_exited" + and not cache.grouped + ) + def _uses_packed_loop_inputs(self, batch): capabilities = self.cache_manager.attention_capabilities return ( @@ -751,7 +774,7 @@ def _prepare_loop_inputs(self, batch, packed_metadata=None): tokens.extend(span.input_tokens) requests.extend([request] * count) outputs.append((len(tokens) - 1, request)) - packed = cache.attention_capabilities.packed_prefill and ( + packed = self._packs_prefill() and ( not cache.attention_capabilities.packed_sliding_window or all(len(span.request.prompt_token_ids) > 64 for span in batch.spans) ) @@ -867,8 +890,7 @@ def prepare_on_cpu(self, batch: ExecutionBatch): for offset in range(start, batch.loop_count) ], positions, - packed=batch.has_chunks - and self.cache_manager.attention_capabilities.packed_prefill, + packed=batch.has_chunks and self._packs_prefill(), ) elif getattr(self.model, "requires_boundary_kv", False): ids, depths, positions = batch.addresses() @@ -959,6 +981,7 @@ def _prepare_cpu_inputs(self, batch: ExecutionBatch): prepared = replace(prepared, workspace=workspace) if ( workspace is not None + and not self.cache_manager.grouped and batch.operation == Operation.RECURRENT and not (batch.emit_gate and self.graphs is not None) and batch.loop_count == 1 @@ -1052,10 +1075,11 @@ def _prepare_forward(self, prepared): kv = prepared.kv if kv is not None: metadata = (kv,) + metadata + # Grouped views omit padding even when the hidden workspace includes it. forward = ForwardBatch( Operation.RECURRENT, hidden=hidden[: batch.num_rows] - if graphs is not None or batch.loop_count > 1 + if graphs is not None or batch.loop_count > 1 or self.cache_manager.grouped else hidden, metadata=metadata, graphs=graphs,