Skip to content

[Dory] PR 3C: preemption for synchronous self-speculation - #204

Open
0z5a wants to merge 1 commit into
InferMatrixLab:mainfrom
0z5a:dory-3c-sync-preemption
Open

0z5a wants to merge 1 commit into
InferMatrixLab:mainfrom
0z5a:dory-3c-sync-preemption

Conversation

@0z5a

@0z5a 0z5a commented Nov 21, 2026

Copy link
Copy Markdown
Contributor

Summary

RFC #176 3C, synchronous part: preemption for Dory self-speculation with synchronous eager rounds, in block and token-pipeline mode. Async rounds and CUDA Graphs stay rejected together with preemption; they belong to 3B (#197).

This does not depend on 2E (#202). Speculation already stores every plane at every depth, so the generic KV snapshot covers it. Preemption for non-speculative grouped Dory is still 2E's.

Required human contributor answers

1. What outcome are you trying to achieve?

Dory self-speculation that can run with enable_preemption=True, the 3C item of RFC #176 that I claimed in #176 (comment).

2. What is the highest-leverage variable for that outcome?

Whether a suspended request resumes in exactly the state it left: committed KV, the KV that in-flight pipeline tokens wrote above the committed frontier, and the pipeline itself.

3. How will this work move that variable rather than create busywork?

It reuses the existing preemption manager and the runner's snapshot_pipeline / restore_pipeline; the code change is the engine gate. The work is in the tests, which check the restored KV and schedule directly instead of only the output tokens (a lost pipeline is silently recomputed and keeps tokens correct).

4. Who is the customer or user, and what do they actually need?

People serving Dory with priority scheduling or under KV pressure who want speculation on, and the 3B owner, whose async path can reuse these checks.

PR necessity and bugfix evidence

A PR is needed: main rejects preemption for every boundary-KV speculative configuration at construction. This is a feature, not a bugfix.

What changes

  • stage_engine.py: the boundary-KV speculation gate rejects only async scheduling and CUDA Graphs.
  • Suspension happens only between collected submissions. The KV snapshot holds every written plane, and the pipeline snapshot holds each retained token's position, depth, carried hidden state and draft candidate. Block mode keeps no state between rounds. Resident suspension keeps the pages; offload copies KV and hidden state to the CPU and frees them. The request keeps its sampling generator.
  • docs/dory.md: preemption moves from "rejected" to a paragraph under synchronous self-speculation.

Test Plan

tests/test_dory_speculative_preemption.py, 52 CPU and 12 CUDA cases:

  • Suspension in block and pipeline mode, depth pairs 1→2, 1→4, 2→4, resident and offloaded, after a pipeline has tokens in flight. At resumption the KV of every written position (in-flight positions above the frontier included) equals the KV before suspension, the restored pipeline holds the saved tokens, tokens match the dense oracle, and the pipeline schedule (positions and depths per submission) matches an uninterrupted run.
  • Sampled block rounds keep the RNG stream; a resident snapshot later offloaded; abort while suspended with ID reuse; suspension between chunked-prefill steps before the first token; growth preemption in a pool that holds two requests or only one.
  • Two mutation tests: dropping the restored pipeline keeps tokens correct but changes the schedule, and zeroing the snapshot above the frontier fails the KV comparison.
  • CUDA cases compare against the CPU oracle (Triton backend).

vllm-rlt Version / Commit: 42b869d on main ce200f0.

Test Result

  • CPU suite: 1348 passed, 925 skipped.
  • One GH200, tests/test_dory_speculative_preemption.py, tests/test_dory_speculative.py and tests/test_speculative.py together: 244 passed, 1 skipped, 12 failed. All 12 failures are flash_attn_4 cases in tests/test_speculative.py; the optional package is not installed there.
  • ruff check and ruff format --check pass on the changed files.

Merge order

#197 edits the same gate block and the self-speculation part of docs/dory.md. Whichever lands second needs a small rebase.

Allow preemption with synchronous eager boundary-KV speculation. Suspension
happens between collected submissions, so the KV snapshot holds every plane,
in-flight pipeline KV above the committed frontier included, and the pipeline
snapshot holds each retained token's depth, hidden state and candidate.
Async rounds and CUDA Graphs stay rejected (RFC InferMatrixLab#176 3B).

Tests check, at resumption, the KV of every written position and the restored
pipeline; tokens against the dense oracle; and the pipeline schedule against
an uninterrupted run. Two mutation tests show the checks fail when the
pipeline or the KV above the frontier is lost.

Signed-off-by: 0z5a <dezhen.lu@student.uni-tuebingen.de>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant