Skip to content

feat(dory): preempt and restore requests on grouped KV storage - #202

Draft
XinhaoTheo wants to merge 8 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-preemption
Draft

XinhaoTheo wants to merge 8 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-preemption

Conversation

@XinhaoTheo

@XinhaoTheo XinhaoTheo commented Nov 21, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?
Lossless Dory preemption that keeps the memory savings of grouped KV storage. This is item 2E of RFC #176.

2. What is the highest-leverage variable for that outcome?
The KV pages that a request holds when preemption is enabled. Before this change, enable_preemption=True selected all-layer blocks and disabled SWA reclamation.

3. How will this work move that variable rather than create busywork?
Snapshot and restore now work on grouped pages, so the engine keeps grouped storage and SWA reclamation with preemption. A CPU snapshot copies only the pages that the request holds. A resumed request continues the reclamation from where it stopped.

4. Who is the customer or user, and what do they actually need?
People who serve Dory with priority scheduling or under KV pressure. They need preemption without the loss of the SWA memory savings. Also the owner of 3C, which needs 2E for speculative preemption.

PR necessity
A PR is necessary. On the 2B code, snapshot() and restore() reject grouped storage, so the engine selects all-layer blocks when preemption is enabled. This is a feature, not a bugfix.

Modification

  • KVCacheManager.snapshot() and restore() accept grouped storage. A snapshot copies only the pages that the allocation holds. It also keeps the reclaimed page count per group plane and the positions that each layer has read.
  • The new KVCacheManager.allocate_snapshot() claims the same number of pages for a resumed request, and PreemptionManager.resume() uses it. allocate() cannot do this: it claims every logical page again, and a resumed frontier is larger than the submission bound.
  • The engine keeps grouped storage with preemption in sync eager. CUDA Graphs, async scheduling, packed prefill, speculation and FP8 KV still use all-layer blocks.
  • With preemption, admission does not budget the peak of each prefix sharer separately. Growth that sharing leaves short suspends a victim, as decode growth does. Without this change, prefix caching with SWA reclamation and preemption raised an error at admission.
  • A resumed request holds private copies of the pages that it shared through a prefix hit, as all-layer resumption on main does. Resumption does not depend on the prefix entry, because the cache can evict that entry while the request is suspended. The allocate_snapshot() docstring says this.
  • All-layer snapshots do not change.

Out of scope: preemption with WCPB or staged prefill (2A), speculative preemption (3C), grouped storage for Graph and async modes.

Related PR

Accuracy Test

Tests passed on one H100 (torch 2.15.0+cu132, Triton) and on the CPU. The GPU tests ran at 98858b4. ac34a21 adds a docstring, two comments, an assert and a CPU test, and the full CPU suite passed at ac34a21.

  • tests/test_dory_preemption.py (67 CPU and 48 CUDA cases): suspension at six stage boundaries in both KV modes, with resident and offloaded KV. Greedy output matches the dense oracle, sampled output matches an uninterrupted run, and restored KV matches the serial history.
  • Grouped cases: a request with reclaimed SWA pages resumes with the same pages, ranges and KV, and reclamation continues. A prefix hit with SWA reclamation resumes. A snapshot waits for pool capacity. A resumed prefix sharer frees only its own pages at offload and holds private copies of the shared pages after resume, in grouped and all-layer storage.
  • CUDA capacity and priority preemption in sync eager (grouped), sync Graph, async eager and async Graph (all-layer).
  • The tests fail when restoration drops output-stage KV, reseeds the RNG, moves the carried state, resets the read positions or claims the reclaimed pages again.
  • tests/test_dory_gpu.py passed together with the file (245 passed). The full CPU suite also passed. 12 CPU tests need Triton and 6 GPU tests need flash-attn-4. These packages are not in the test environment.

Benchmark and Profiling

Not run yet. The benchmark runs after 2B merges.

With preemption enabled, every sync eager step now uses grouped storage, also when no request is preempted. The plan compares this PR with all-layer blocks (the #190 head) on one H100 with the real checkpoint:

  • No KV pressure: 1024-token prompts at 1, 4, 16 and 64 requests. Wall time, TTFT and TPOT.
  • KV pressure: a small KV budget that forces preemption. Preemption count, offloaded bytes and wall time.

Conclusion

Dory preemption now keeps grouped KV storage and SWA reclamation in sync eager. Resumed requests match uninterrupted runs. The performance effect is not measured yet.

prettygirlisnotme and others added 7 commits November 20, 2026 15:31
Publish complete prefix pages using boundary and recurrent plane ownership,
retain cached pages after request release, and restore coverage on hits.
Use incremental prefix hashing and completion-aware publication, releasing
pending records with their allocation.

Keep compatible all-layer storage for graph, async and speculative execution,
and harden grouped allocation, padding, reclamation and initialization paths.
Preserve CacheConfig positional compatibility and document CPU/GPU validation.

This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B
storage/reclamation change remains a separate commit by its original author.
- Remove CacheConfig.reclaim_swa. The engine enables reclamation whenever
  grouped storage runs with fixed-loop ouro exits; other exit modes keep
  grouped pages resident.
- Decouple reclamation from incremental allocation. Preallocation reserves
  the full peak, still returns expired pages, and no longer budgets retired
  pages as growth.
- Charge a prefix hit only for the cached pages it attaches, so expired SWA
  pages stay evictable and do not delay admission.
- Select all-layer KV for preemption instead of raising, so suspend, CPU
  offload and restore work with prefix caching.
- Make plan_cache layer_windows and reclaim_swa required keywords chosen by
  the engine.
Suspend Dory requests at every stage boundary and resume them, with resident
and CPU-offloaded KV, in both KV modes. Greedy output must equal the dense
oracle, seeded sampled output must equal an uninterrupted run, and every
restored plane must equal the serial KV. CUDA cases cover capacity-pressure and
priority preemption in sync and async, eager and graphed execution.

The existing suspension path needed no Dory-specific change. Describe the
supported preemption scope and remove the note that called it unqualified.
A grouped allocation holds one page per attention layer, and SWA reclamation
sets the expired leading pages to None. Snapshot and restore rejected grouped
storage, so preemption had to use all-layer blocks.

A snapshot now copies only the pages that the allocation holds. It also keeps
the count of reclaimed pages per group plane and the positions that each layer
has read. The new allocate_snapshot claims the same number of pages and keeps
the reclaimed pages empty. The resume path cannot use allocate, because
allocate claims every logical page again and a resumed frontier is larger than
the submission bound. Restore then loads the pages and the read positions, so
reclamation continues from where it stopped. All-layer snapshots do not change.
Preemption no longer selects all-layer blocks. Sync eager Dory keeps grouped
storage and SWA reclamation with preemption; CUDA Graphs, async scheduling,
packed prefill, speculation and FP8 KV still use all-layer blocks.

Admission budgets each sharer's peak separately only without preemption. That
budget subtracts the pages of every allocation, so it must count every peak,
and with preemption it raised an error for prefix caching with SWA
reclamation. With preemption, growth that prefix sharing leaves short now
suspends a victim, as decode growth does.

The Dory preemption tests now run on grouped storage and compare only the
retained KV. New cases check that an offloaded request with reclaimed SWA
pages resumes with the same pages, ranges and KV and that reclamation then
continues; that a prefix hit with SWA reclamation resumes; and that a grouped
snapshot waits until the pool has room. The capacity-pressure pool is derived
from the per-request peak, because grouped pages are counted per layer.

@0z5a 0z5a left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, the test matrix is thorough (every stage boundary × KV mode × resident/offload, sampled RNG, SWA retained pages, prefix hit, CUDA). Tokens and cleanup look right to me. One accounting question and a few small things.

1. Shared prefix pages in the grouped snapshot. held_blocks() returns every physical page the allocation maps, including prefix pages it shares with other requests. snapshot() copies them, and allocate_snapshot() then claims len(snapshot.keys) fresh private pages and does not restore prefix_keys. So offloading a request that shares a prefix frees fewer pages than resuming it needs (the shared pages stay with the other sharers), and after resumption it no longer shares them. With several sharers under pressure, can this make suspend/resume take turns without progress? Since 98858b4 admission no longer budgets each sharer's peak when preemption is on, so this path is the one that absorbs the shortfall. If this is intended (simple, and correct in tokens), a sentence in the docstring and a test that pins the page count after a shared-prefix resume would make it explicit. Otherwise, the snapshot could skip pages that other allocations still reference and re-link them at resume when the prefix entry still exists.

2. Small things.

  • allocate_snapshot: an assert offset == len(fresh) after the table loop, as _claim_group_tables has, would catch a retired-page mismatch early.
  • group_epochs restart at 0 for the new allocation, so a layer_views entry keyed by epochs could in principle match an old view. It cannot be reached: _require_live_batch raises for a batch whose allocation object is no longer the live one, so a batch built before the suspension is never used after the resume. Fine as is.
  • reclaim_pending and _pending_prefixes entries are dropped on free and not carried in the snapshot. Both look benign (reclamation reruns on the next step; prefixes republish). Worth one comment line each.

3. Before merge. The benchmark is still open, and this needs a rebase after #190.

A resumed request holds private copies of the pages that it shared
through a prefix hit, in grouped and all-layer storage. The
allocate_snapshot docstring now says this, and a new test checks the
free page count after the offload and after the resume.

Also check that allocate_snapshot uses every claimed page, and say why
a restore does not need reclaim_pending or the deferred prefix
publications.
@XinhaoTheo

Copy link
Copy Markdown
Author

@0z5a Thanks for the review. Here is my answer for your questions

  1. Shared prefix pages in the grouped snapshot. This is intended. Resumption claims a private page for every snapshot page, as the all-layer path on main does. It does not depend on the prefix entry, because the cache can evict that entry while the request is suspended. A blocked resume offloads only requests that already wait, never a running request, so running requests can finish and return their pages. The allocate_snapshot docstring now says this, and test_a_resumed_prefix_sharer_holds_private_copies_of_the_shared_pages checks the free page count after the offload and after the resume, in grouped and all-layer storage.
  2. Added the assert offset == len(fresh), and one comment each for reclaim_pending and the deferred publications.
  3. Yes. I will rebase after [Dory] Enable prefix caching for per_loop and last_loop KV with SWA reclamation #190 merges and then run the benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants