Skip to content

feat(dory): hand off prompts between prefill and decode workers - #203

Draft
XinhaoTheo wants to merge 7 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-pd-handoff
Draft

XinhaoTheo wants to merge 7 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-pd-handoff

Conversation

@XinhaoTheo

@XinhaoTheo XinhaoTheo commented Nov 21, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?
Dory requests that prefill on one GPU and decode on another GPU of the same host, with the same output as one engine. This is item 2F of RFC #176.

2. What is the highest-leverage variable for that outcome?
The prompt state that goes from P to D: which stage runs the output stage of the last prompt token, and which KV pages D must receive.

3. How will this work move that variable rather than create busywork?
P keeps the existing PD contract: it runs every prompt stage except the output stage of the last token, and D runs that stage and samples. On grouped storage, P sends one table per layer, and D receives only the SWA pages that it reads. Sync PD therefore keeps the memory savings of SWA reclamation on D.

4. Who is the customer or user, and what do they actually need?
People who serve Dory with separate prefill and decode GPUs. They need PD output that is equal to one engine, and the SWA memory savings on the decode side.

PR necessity
A PR is necessary. PD rejected Dory model instances, and grouped storage had no transfer path. The PD check also let config and path sources through without qualification. This PR also fixes a bug: a decode-side prefix hit for a boundary-KV model failed the import.

Modification

  • PD reads the model class, so the boundary-KV check applies to every source. PD with speculative decoding stays rejected for boundary-KV models.
  • Handoff: P runs the input stage, every loop and the output stage of each prompt token except the last. D receives the KV and the carried state of the last token, runs its output stage and samples. Dory needs no new handoff state.
  • mark_imported_prefix() marks only the planes that each layer owns, as a prefix hit does. Before this change, a decode-side prefix hit raised "KV prefix was already initialized".
  • Grouped transfers:
    • transfer_tables() gives one table per group, plane and layer, with -1 for a page that is not held.
    • allocate_import() does not reserve the SWA pages that the first D query cannot read.
    • P sends each table from the first page that D holds.
    • The import sets the read positions, so D reclamation continues.
    • P does not reclaim, because reclamation could return a page that a transfer still reads.
  • All-layer transfers send the same segments as before.

Out of scope: PD with speculative decoding (2F with 3A/3B), WCPB (2A), PD on the real checkpoint.

Related PR

Accuracy Test

Tests passed on two H100 GPUs (torch 2.15.0+cu132, Triton, NIXL 1.4.1) and on the CPU. The code at 985b12f is the code that was tested.

  • tests/test_dory_pd.py (19 CUDA and 8 CPU cases): Dory PD output matches the dense oracle and a single-stage engine in both KV modes, in sync eager (grouped), sync Graph, async eager and async Graph (all-layer). It also checks seeded sampling on D, prefix caching on P, on D or on both, and cancellation in each phase with grouped and all-layer storage.
  • Grouped PD sends fewer bytes than all-layer PD for a 40-token prompt.
  • CPU cases: an import reserves only the SWA pages that D reads and drops attached prefix references when the pool is short. A CPU handoff that copies only the pages D holds matches the oracle, and D reclamation continues.
  • The CPU tests fail when an import reserves one SWA page too few, or when the read positions stop reclamation.
  • tests/test_pd.py, the stage engine, transfer connector, Dory GPU, hybrid KV and SWA reclamation tests also passed (321 passed in total). The full CPU suite passed. 12 CPU tests need Triton, and 2 PD cases need FlashAttention-4 or four GPUs. These are not in the test environment.

Benchmark and Profiling

Not run yet. The benchmark runs after 2B merges, together with the 2E benchmark.

The plan uses the real checkpoint on two H100 GPUs. It compares PD with a single-stage engine, and grouped PD with all-layer PD: TTFT, TPOT, transferred bytes and D KV memory.

Conclusion

Dory PD now matches a single-stage engine. Sync PD keeps grouped storage, and D reclaims SWA pages. The performance effect is not measured yet.

prettygirlisnotme and others added 7 commits November 20, 2026 15:31
Publish complete prefix pages using boundary and recurrent plane ownership,
retain cached pages after request release, and restore coverage on hits.
Use incremental prefix hashing and completion-aware publication, releasing
pending records with their allocation.

Keep compatible all-layer storage for graph, async and speculative execution,
and harden grouped allocation, padding, reclamation and initialization paths.
Preserve CacheConfig positional compatibility and document CPU/GPU validation.

This commit squashes only the 12 HowardYangAI commits. The prerequisite 2B
storage/reclamation change remains a separate commit by its original author.
- Remove CacheConfig.reclaim_swa. The engine enables reclamation whenever
  grouped storage runs with fixed-loop ouro exits; other exit modes keep
  grouped pages resident.
- Decouple reclamation from incremental allocation. Preallocation reserves
  the full peak, still returns expired pages, and no longer budgets retired
  pages as growth.
- Charge a prefix hit only for the cached pages it attaches, so expired SWA
  pages stay evictable and do not delay admission.
- Select all-layer KV for preemption instead of raising, so suspend, CPU
  offload and restore work with prefix caching.
- Make plan_cache layer_windows and reclaim_swa required keywords chosen by
  the engine.
A prefix hit marks the cached tokens as written only in the planes that each
layer owns. Boundary layers own plane 0 only. The import check required every
layer in every plane to start at the cached length, so a decode-side prefix
hit for a boundary-KV model failed with "KV prefix was already initialized".

The import now checks and marks the owned planes only, as a prefix hit does.
The sender never writes boundary layers in deeper planes, so the import no
longer marks them as written. Models without boundary layers own every plane
and do not change.
PD needed all-layer blocks. Grouped storage had no transfer tables, and the
lease and import calls rejected it.

- transfer_tables gives one table per group, plane and layer for grouped
  storage, with -1 for a page that is not held. The transport counts one layer
  per grouped page.
- The receiver first runs the output stage of the last prompt token. With SWA
  reclamation, allocate_import does not reserve the SWA pages that this query
  cannot read. The submission bound does not apply, because the receiver
  computes no prompt token.
- The sender sends each table from the first page that the receiver holds.
- The import sets the read positions for reclamation: the output layers have
  still to read the last prompt token, and every other layer has read it.
- A prefill stage does not reclaim. It holds a prompt only until the transfer
  completes, and reclamation could return a page that a transfer still reads.

All-layer transfers send the same segments as before.
PD rejected boundary-KV models, but only when the source was a model
instance. A config or a path source passed the check. The check now reads the
model class, so it applies to every source.

Dory needs no new handoff state. P runs the input stage, every loop and the
output stage of each prompt token except the last, so P sends output-stage KV
for the prompt history, as for the other planes. D receives the carried state
of the last prompt token, runs its output stage and samples. No prompt stage
runs twice or is skipped. PD with speculative decoding stays rejected for
boundary-KV models until it is qualified.

The new tests compare Dory PD with the dense oracle and a single-stage engine
in both KV modes and four execution modes; sync eager uses grouped storage.
They also cover seeded sampling on D, prefix caching on either side,
cancellation in each phase, and the smaller transfer of grouped PD. CPU tests
cover the grouped import and a handoff that copies only the pages D holds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants