Repository navigation
feat(dory): hand off prompts between prefill and decode workers - #203
Draft
XinhaoTheo wants to merge 7 commits into
Draft
XinhaoTheo wants to merge 7 commits into
XinhaoTheo wants to merge 7 commits into
Conversation
Publish complete prefix pages using boundary and recurrent plane ownership, retain cached pages after request release, and restore coverage on hits. Use incremental prefix hashing and completion-aware publication, releasing pending records with their allocation. Keep compatible all-layer storage for graph, async and speculative execution, and harden grouped allocation, padding, reclamation and initialization paths. Preserve CacheConfig positional compatibility and document CPU/GPU validation. This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B storage/reclamation change remains a separate commit by its original author.
- Remove CacheConfig.reclaim_swa. The engine enables reclamation whenever grouped storage runs with fixed-loop ouro exits; other exit modes keep grouped pages resident. - Decouple reclamation from incremental allocation. Preallocation reserves the full peak, still returns expired pages, and no longer budgets retired pages as growth. - Charge a prefix hit only for the cached pages it attaches, so expired SWA pages stay evictable and do not delay admission. - Select all-layer KV for preemption instead of raising, so suspend, CPU offload and restore work with prefix caching. - Make plan_cache layer_windows and reclaim_swa required keywords chosen by the engine.
A prefix hit marks the cached tokens as written only in the planes that each layer owns. Boundary layers own plane 0 only. The import check required every layer in every plane to start at the cached length, so a decode-side prefix hit for a boundary-KV model failed with "KV prefix was already initialized". The import now checks and marks the owned planes only, as a prefix hit does. The sender never writes boundary layers in deeper planes, so the import no longer marks them as written. Models without boundary layers own every plane and do not change.
PD needed all-layer blocks. Grouped storage had no transfer tables, and the lease and import calls rejected it. - transfer_tables gives one table per group, plane and layer for grouped storage, with -1 for a page that is not held. The transport counts one layer per grouped page. - The receiver first runs the output stage of the last prompt token. With SWA reclamation, allocate_import does not reserve the SWA pages that this query cannot read. The submission bound does not apply, because the receiver computes no prompt token. - The sender sends each table from the first page that the receiver holds. - The import sets the read positions for reclamation: the output layers have still to read the last prompt token, and every other layer has read it. - A prefill stage does not reclaim. It holds a prompt only until the transfer completes, and reclamation could return a page that a transfer still reads. All-layer transfers send the same segments as before.
PD rejected boundary-KV models, but only when the source was a model instance. A config or a path source passed the check. The check now reads the model class, so it applies to every source. Dory needs no new handoff state. P runs the input stage, every loop and the output stage of each prompt token except the last, so P sends output-stage KV for the prompt history, as for the other planes. D receives the carried state of the last prompt token, runs its output stage and samples. No prompt stage runs twice or is skipped. PD with speculative decoding stays rejected for boundary-KV models until it is qualified. The new tests compare Dory PD with the dense oracle and a single-stage engine in both KV modes and four execution modes; sync eager uses grouped storage. They also cover seeded sampling on D, prefix caching on either side, cancellation in each phase, and the smaller transfer of grouped PD. CPU tests cover the grouped import and a handoff that copies only the pages D holds.
Open
1 of 2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Required human contributor answers
1. What outcome are you trying to achieve?
Dory requests that prefill on one GPU and decode on another GPU of the same host, with the same output as one engine. This is item 2F of RFC #176.
2. What is the highest-leverage variable for that outcome?
The prompt state that goes from P to D: which stage runs the output stage of the last prompt token, and which KV pages D must receive.
3. How will this work move that variable rather than create busywork?
P keeps the existing PD contract: it runs every prompt stage except the output stage of the last token, and D runs that stage and samples. On grouped storage, P sends one table per layer, and D receives only the SWA pages that it reads. Sync PD therefore keeps the memory savings of SWA reclamation on D.
4. Who is the customer or user, and what do they actually need?
People who serve Dory with separate prefill and decode GPUs. They need PD output that is equal to one engine, and the SWA memory savings on the decode side.
PR necessity
A PR is necessary. PD rejected Dory model instances, and grouped storage had no transfer path. The PD check also let config and path sources through without qualification. This PR also fixes a bug: a decode-side prefix hit for a boundary-KV model failed the import.
Modification
mark_imported_prefix()marks only the planes that each layer owns, as a prefix hit does. Before this change, a decode-side prefix hit raised "KV prefix was already initialized".transfer_tables()gives one table per group, plane and layer, with -1 for a page that is not held.allocate_import()does not reserve the SWA pages that the first D query cannot read.Out of scope: PD with speculative decoding (2F with 3A/3B), WCPB (2A), PD on the real checkpoint.
Related PR
stage_engine.pyandkv_cache_manager.py. Without feat(dory): preempt and restore requests on grouped KV storage #202, a sync eager D with preemption uses all-layer blocks while P uses grouped pages, and the PD handshake rejects the mismatch. With feat(dory): preempt and restore requests on grouped KV storage #202, both stages keep grouped pages.Accuracy Test
Tests passed on two H100 GPUs (torch 2.15.0+cu132, Triton, NIXL 1.4.1) and on the CPU. The code at 985b12f is the code that was tested.
tests/test_dory_pd.py(19 CUDA and 8 CPU cases): Dory PD output matches the dense oracle and a single-stage engine in both KV modes, in sync eager (grouped), sync Graph, async eager and async Graph (all-layer). It also checks seeded sampling on D, prefix caching on P, on D or on both, and cancellation in each phase with grouped and all-layer storage.tests/test_pd.py, the stage engine, transfer connector, Dory GPU, hybrid KV and SWA reclamation tests also passed (321 passed in total). The full CPU suite passed. 12 CPU tests need Triton, and 2 PD cases need FlashAttention-4 or four GPUs. These are not in the test environment.Benchmark and Profiling
Not run yet. The benchmark runs after 2B merges, together with the 2E benchmark.
The plan uses the real checkpoint on two H100 GPUs. It compares PD with a single-stage engine, and grouped PD with all-layer PD: TTFT, TPOT, transferred bytes and D KV memory.
Conclusion
Dory PD now matches a single-stage engine. Sync PD keeps grouped storage, and D reclaims SWA pages. The performance effect is not measured yet.
docs/dory.md)