Repository navigation
feat(dory): preempt and restore requests on grouped KV storage - #202
XinhaoTheo wants to merge 8 commits into
Conversation
Publish complete prefix pages using boundary and recurrent plane ownership, retain cached pages after request release, and restore coverage on hits. Use incremental prefix hashing and completion-aware publication, releasing pending records with their allocation. Keep compatible all-layer storage for graph, async and speculative execution, and harden grouped allocation, padding, reclamation and initialization paths. Preserve CacheConfig positional compatibility and document CPU/GPU validation. This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B storage/reclamation change remains a separate commit by its original author.
- Remove CacheConfig.reclaim_swa. The engine enables reclamation whenever grouped storage runs with fixed-loop ouro exits; other exit modes keep grouped pages resident. - Decouple reclamation from incremental allocation. Preallocation reserves the full peak, still returns expired pages, and no longer budgets retired pages as growth. - Charge a prefix hit only for the cached pages it attaches, so expired SWA pages stay evictable and do not delay admission. - Select all-layer KV for preemption instead of raising, so suspend, CPU offload and restore work with prefix caching. - Make plan_cache layer_windows and reclaim_swa required keywords chosen by the engine.
Suspend Dory requests at every stage boundary and resume them, with resident and CPU-offloaded KV, in both KV modes. Greedy output must equal the dense oracle, seeded sampled output must equal an uninterrupted run, and every restored plane must equal the serial KV. CUDA cases cover capacity-pressure and priority preemption in sync and async, eager and graphed execution. The existing suspension path needed no Dory-specific change. Describe the supported preemption scope and remove the note that called it unqualified.
A grouped allocation holds one page per attention layer, and SWA reclamation sets the expired leading pages to None. Snapshot and restore rejected grouped storage, so preemption had to use all-layer blocks. A snapshot now copies only the pages that the allocation holds. It also keeps the count of reclaimed pages per group plane and the positions that each layer has read. The new allocate_snapshot claims the same number of pages and keeps the reclaimed pages empty. The resume path cannot use allocate, because allocate claims every logical page again and a resumed frontier is larger than the submission bound. Restore then loads the pages and the read positions, so reclamation continues from where it stopped. All-layer snapshots do not change.
Preemption no longer selects all-layer blocks. Sync eager Dory keeps grouped storage and SWA reclamation with preemption; CUDA Graphs, async scheduling, packed prefill, speculation and FP8 KV still use all-layer blocks. Admission budgets each sharer's peak separately only without preemption. That budget subtracts the pages of every allocation, so it must count every peak, and with preemption it raised an error for prefix caching with SWA reclamation. With preemption, growth that prefix sharing leaves short now suspends a victim, as decode growth does. The Dory preemption tests now run on grouped storage and compare only the retained KV. New cases check that an offloaded request with reclaimed SWA pages resumes with the same pages, ranges and KV and that reclamation then continues; that a prefix hit with SWA reclamation resumes; and that a grouped snapshot waits until the pool has room. The capacity-pressure pool is derived from the per-request peak, because grouped pages are counted per layer.
0z5a
left a comment
There was a problem hiding this comment.
Thanks, the test matrix is thorough (every stage boundary × KV mode × resident/offload, sampled RNG, SWA retained pages, prefix hit, CUDA). Tokens and cleanup look right to me. One accounting question and a few small things.
1. Shared prefix pages in the grouped snapshot. held_blocks() returns every physical page the allocation maps, including prefix pages it shares with other requests. snapshot() copies them, and allocate_snapshot() then claims len(snapshot.keys) fresh private pages and does not restore prefix_keys. So offloading a request that shares a prefix frees fewer pages than resuming it needs (the shared pages stay with the other sharers), and after resumption it no longer shares them. With several sharers under pressure, can this make suspend/resume take turns without progress? Since 98858b4 admission no longer budgets each sharer's peak when preemption is on, so this path is the one that absorbs the shortfall. If this is intended (simple, and correct in tokens), a sentence in the docstring and a test that pins the page count after a shared-prefix resume would make it explicit. Otherwise, the snapshot could skip pages that other allocations still reference and re-link them at resume when the prefix entry still exists.
2. Small things.
allocate_snapshot: anassert offset == len(fresh)after the table loop, as_claim_group_tableshas, would catch a retired-page mismatch early.group_epochsrestart at 0 for the new allocation, so alayer_viewsentry keyed by epochs could in principle match an old view. It cannot be reached:_require_live_batchraises for a batch whose allocation object is no longer the live one, so a batch built before the suspension is never used after the resume. Fine as is.reclaim_pendingand_pending_prefixesentries are dropped on free and not carried in the snapshot. Both look benign (reclamation reruns on the next step; prefixes republish). Worth one comment line each.
3. Before merge. The benchmark is still open, and this needs a rebase after #190.
A resumed request holds private copies of the pages that it shared through a prefix hit, in grouped and all-layer storage. The allocate_snapshot docstring now says this, and a new test checks the free page count after the offload and after the resume. Also check that allocate_snapshot uses every claimed page, and say why a restore does not need reclaim_pending or the deferred prefix publications.
|
@0z5a Thanks for the review. Here is my answer for your questions
|
Required human contributor answers
1. What outcome are you trying to achieve?
Lossless Dory preemption that keeps the memory savings of grouped KV storage. This is item 2E of RFC #176.
2. What is the highest-leverage variable for that outcome?
The KV pages that a request holds when preemption is enabled. Before this change,
enable_preemption=Trueselected all-layer blocks and disabled SWA reclamation.3. How will this work move that variable rather than create busywork?
Snapshot and restore now work on grouped pages, so the engine keeps grouped storage and SWA reclamation with preemption. A CPU snapshot copies only the pages that the request holds. A resumed request continues the reclamation from where it stopped.
4. Who is the customer or user, and what do they actually need?
People who serve Dory with priority scheduling or under KV pressure. They need preemption without the loss of the SWA memory savings. Also the owner of 3C, which needs 2E for speculative preemption.
PR necessity
A PR is necessary. On the 2B code,
snapshot()andrestore()reject grouped storage, so the engine selects all-layer blocks when preemption is enabled. This is a feature, not a bugfix.Modification
KVCacheManager.snapshot()andrestore()accept grouped storage. A snapshot copies only the pages that the allocation holds. It also keeps the reclaimed page count per group plane and the positions that each layer has read.KVCacheManager.allocate_snapshot()claims the same number of pages for a resumed request, andPreemptionManager.resume()uses it.allocate()cannot do this: it claims every logical page again, and a resumed frontier is larger than the submission bound.allocate_snapshot()docstring says this.Out of scope: preemption with WCPB or staged prefill (2A), speculative preemption (3C), grouped storage for Graph and async modes.
Related PR
Accuracy Test
Tests passed on one H100 (torch 2.15.0+cu132, Triton) and on the CPU. The GPU tests ran at 98858b4. ac34a21 adds a docstring, two comments, an assert and a CPU test, and the full CPU suite passed at ac34a21.
tests/test_dory_preemption.py(67 CPU and 48 CUDA cases): suspension at six stage boundaries in both KV modes, with resident and offloaded KV. Greedy output matches the dense oracle, sampled output matches an uninterrupted run, and restored KV matches the serial history.tests/test_dory_gpu.pypassed together with the file (245 passed). The full CPU suite also passed. 12 CPU tests need Triton and 6 GPU tests needflash-attn-4. These packages are not in the test environment.Benchmark and Profiling
Not run yet. The benchmark runs after 2B merges.
With preemption enabled, every sync eager step now uses grouped storage, also when no request is preempted. The plan compares this PR with all-layer blocks (the #190 head) on one H100 with the real checkpoint:
Conclusion
Dory preemption now keeps grouped KV storage and SWA reclamation in sync eager. Resumed requests match uninterrupted runs. The performance effect is not measured yet.
docs/dory.md)