Skip to content

feat(dory): add asynchronous self-speculation and speculative CUDA Graphs - #197

Open
XinhaoTheo wants to merge 7 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-async-speculation
Open

XinhaoTheo wants to merge 7 commits into
InferMatrixLab:mainfrom
XinhaoTheo:feat/rfc176-dory-async-speculation

Conversation

@XinhaoTheo

@XinhaoTheo XinhaoTheo commented Nov 18, 2026 •

Copy link
Copy Markdown

Required human contributor answers

1. What outcome are you trying to achieve?
Dory self-speculation that runs with asynchronous scheduling and CUDA Graphs. This is item 3B of RFC #176.

2. What is the highest-leverage variable for that outcome?
Host-side cost per committed token. Eager Dory is bound by kernel launches, so the draft and verify steps must run as graph replays without a host read.

3. How will this work move that variable rather than create busywork?
PRELUDE and CODA now replay bucketed graphs with device-owned positions, and the async pipeline updates all requests in one batch. At 1 to 16 concurrent requests, the async Graph pipeline is 10–29 % faster than ordinary async Graph decode. At 64 requests the gain is 3 %. Tokens and GSM8K accuracy do not change.

4. Who is the customer or user, and what do they actually need?
People who serve Dory with this runtime at low or medium concurrency and want fast greedy decode. Also the owner of 3C, which needs async rounds for its async qualification.

PR necessity
A PR is necessary. Before this change, the engine rejected Dory self-speculation with async scheduling or CUDA Graphs. This is a feature, not a bugfix.

Modification

  • Async rounds give PRELUDE and CODA depth-0 device KV metadata. A pipeline step runs both stages on all rows, with one shared view, as synchronous steps and async block rounds already do. Only the write masks are different, and they keep the KV of rows that do not use a stage.
  • DeviceBoundaryGraphs replays PRELUDE and CODA in the same buckets as the recurrent graphs. Positions, write masks and context lengths are graph inputs.
  • Sync Graph rounds use the KV-aware PreludeGraphs and CodaGraphs from 2D.
  • The async pipeline keeps the state of all requests in one slot pool. A step does one gather, one update and one scatter. Before this change, a step used about 35 operations per request, and async eager was up to 27 % slower than sync eager at 64 requests. This change also applies to Ouro.
  • The engine accepts async and Graph modes for Dory. Sampled async requests and preemption stay rejected.

Out of scope: sampled async rounds, preemption (3C), last_loop KV, HTTP serving and open-loop arrivals.

Related PR

Accuracy Test

GPU tests passed on one H100 (torch 2.15.0+cu132, Triton) at d74adfa: 610 passed. The real-checkpoint and GSM8K runs below used a6e2fe2, before the shared boundary view.

  • tests/test_dory_speculative_gpu.py (97 cases): async eager, async Graph and sync Graph against the dense oracle; 1→2, 1→4 and 2→4; rejection at each draft position; the KV of rows in flight; cancellation and failure cleanup.
  • The existing async/sync speculation, Dory and CUDA Graph tests.
  • The pipeline tests fail (18 cases) when CODA does not write the rows that reach the target depth.
  • Real checkpoint, FP32, 32 requests × 512 prompt tokens + 128 generated tokens, at 1→2, 1→4 and 2→4: all 7 speculative modes give tokens identical to sync decode without speculation.
  • GSM8K, BF16, first 200 test questions, 8-shot: baseline 161/200, async Graph pipeline 164/200, async Graph block 163/200.

Benchmark and Profiling

Real checkpoint, BF16, 2 loops, a one-loop draft, K = 3, greedy, WikiText-103 prompts. All requests are sent at the same time. Each arm runs one warmup, then five timed repetitions, all in one H100 container. Wall times are the mean whole-run time in seconds, prefill included. Every Graph arm uses cuda_graph_max_graphs=128, so no arm runs eager because its graph cache is full; every Graph arm had zero graph fallbacks. BF16 acceptance is 0.85–0.88. These tables were measured before the shared boundary view.

Table 1: With CUDA Graphs, concurrency sweep. Prompt 1024 tokens, 128 generated tokens for each request. Percentages are against the baseline.

Requests Baseline (async Graph, no spec) Baseline, default graph cache Sync Graph pipeline Async Graph pipeline Async Graph block
1 2.50 2.49 (−0.1 %) 1.83 (−27 %) 1.78 (−29 %) 1.91 (−23 %)
4 3.17 3.18 (+0.4 %) 2.66 (−16 %) 2.63 (−17 %) 2.95 (−7 %)
16 5.89 5.90 (+0.1 %) 5.51 (−6 %) 5.30 (−10 %) 5.63 (−5 %)
64 15.60 25.72 (+65 %) 16.00 (+3 %) 15.10 (−3 %) 16.45 (+5 %)

Table 2: Latency with CUDA Graphs. Same workload. TTFT is the time to the first token; TPOT is the time per output token after the first token. Percentages are against the baseline at the same concurrency.

Requests Arm TTFT p50 (s) TTFT p90 (s) TPOT p50 (ms) TPOT p90 (ms)
1 Baseline (async Graph, no spec) 0.18 0.18 18.3 18.3
1 Async Graph pipeline 0.17 (−2 %) 0.20 (+13 %) 12.6 (−31 %) 12.6 (−31 %)
1 Async Graph block 0.17 (−1 %) 0.19 (+7 %) 13.7 (−25 %) 13.7 (−25 %)
4 Baseline (async Graph, no spec) 0.67 0.67 20.0 20.0
4 Async Graph pipeline 0.66 (−1 %) 0.67 (−0.8 %) 16.6 (−17 %) 17.4 (−13 %)
4 Async Graph block 0.69 (+3 %) 0.69 (+3 %) 18.1 (−10 %) 19.0 (−5 %)
16 Baseline (async Graph, no spec) 2.01 2.75 35.2 40.9
16 Async Graph pipeline 1.68 (−17 %) 2.70 (−2 %) 28.8 (−18 %) 36.1 (−12 %)
16 Async Graph block 1.79 (−11 %) 2.93 (+6 %) 30.1 (−15 %) 36.1 (−12 %)
64 Baseline (async Graph, no spec) 6.21 10.58 76.6 108.5
64 Async Graph pipeline 5.78 (−7 %) 10.02 (−5 %) 70.9 (−7 %) 102.5 (−5 %)
64 Async Graph block 6.43 (+3 %) 11.54 (+9 %) 68.1 (−11 %) 96.4 (−11 %)

Table 3: Without CUDA Graphs (eager), sync against async. Same workload. There is no eager baseline without speculation, so each async percentage is against the sync value of the same mode.

Requests Pipeline sync Pipeline async Block sync Block async
1 6.80 6.42 (−6 %) 6.83 7.16 (+5 %)
4 7.80 8.24 (+6 %) 9.64 9.71 (+0.8 %)
16 10.74 10.57 (−2 %) 13.24 13.25 (+0.1 %)
64 21.22 20.49 (−3 %) 28.02 27.29 (−3 %)

Table 4: Long context. 8 requests, prompt 8192 tokens, 128 generated tokens for each request.

Arm Wall (s) TTFT p50 (s) TTFT p90 (s) TPOT p50 (ms) TPOT p90 (ms)
Baseline (async Graph, no spec) 23.73 18.60 20.71 53.2 68.9
Sync Graph pipeline 23.52 (−0.9 %) 17.85 (−4 %) 20.73 (+0.1 %) 51.2 (−4 %) 72.5 (+5 %)
Async Graph pipeline 23.18 (−2 %) 17.69 (−5 %) 20.58 (−0.6 %) 49.4 (−7 %) 70.3 (+2 %)
Async Graph block 23.13 (−3 %) 17.60 (−5 %) 20.42 (−1 %) 49.9 (−6 %) 67.9 (−1 %)
Sync eager pipeline 28.83 18.04 21.08 86.1 111.4
Async eager pipeline 28.52 (−1 % vs sync) 18.10 (+0.3 % vs sync) 21.15 (+0.3 % vs sync) 87.2 (+1 % vs sync) 108.0 (−3 % vs sync)

Reading the numbers:

  • The gain of speculation falls as concurrency rises. At one request the GPU has spare capacity, so draft and verify work is almost free. At 64 requests the GPU is busy, and the extra verify rows cancel most of the saved steps.
  • In pipeline mode, every row runs both boundary stages, so that each call keeps one shape. PRELUDE also runs on rows that already hold their input-stage KV, and CODA also runs on rows that reach neither the draft nor the target depth. The runtime discards these outputs. At 64 requests, this extra work can explain a part of the small gain.
  • At 8 × 8192, prefill takes about 18 of 24 seconds. Speculation shortens only decode, so the whole run is 2–3 % faster.
  • The baseline with the default graph cache (16 shapes) runs eager in most calls at 64 requests and takes 25.72 s instead of 15.60 s. This is a property of ordinary async Graph decode, not of this PR. Comparisons against that baseline overstate the gain of speculation.

Known issues

1. Async Graph block is slower than the baseline at 64 requests (+5 %). Its time per output token is 11 % lower, but its time to the first token is 3–9 % higher, so the whole run is longer. The async Graph pipeline is still 3 % faster at 64 requests. At high concurrency, the pipeline mode is the better choice.

2. Eager async rounds at low concurrency. Without CUDA Graphs, async against sync varies from −6 % to +6 % at 1 and 4 requests (Table 3). A dedicated run kept a sync and an async engine alive and alternated them in ABBA order, 10 timed repetitions each, at 1 request in pipeline mode:

Container Sync Async Difference Rounds with async slower
1 5.34 s (5.26–5.44) 5.49 s (5.38–5.68) +2.7 % 9 of 10
2 6.27 s (5.80–7.18) 6.51 s (6.05–7.01) +3.9 % 6 of 10

The profiler counts 5503 kernel launches per generated token for sync and 5626 for async (+124, +2.2 %), the same in both containers. Eager Dory is bound by host-side launch cost, so the extra launches give the extra time. Async rounds keep positions on the GPU. Thus the GPU computes the KV metadata, the PRELUDE and CODA write masks, and the slot-pool gather, update and scatter. Sync rounds do this work on the CPU, without kernel launches. At 16 and 64 requests, async eager is 0–3 % faster than sync eager.

Decision from the review: accept the gap (option 1), share one boundary view in this PR (option 2), and fuse the per-step update in a separate PR with an Ouro run (option 3).

Option 2 is in d74adfa. CODA reuses the PRELUDE view with its own write mask, so the pipeline step now builds its boundary view once, as synchronous steps and async block rounds do. Async now makes 5615 launches per generated token (+113 against sync, before +124). The same ABBA run in two new containers:

Container Sync Async Difference Rounds with async slower
3 5.50 s (5.36–5.71) 5.49 s (5.36–5.63) −0.2 % 5 of 10
4 6.72 s (6.39–7.53) 6.75 s (6.42–7.07) +0.4 % 5 of 10

The time between containers varies by a few percent, so these runs do not show that the gap is closed.

Conclusion

Dory self-speculation now runs async and with CUDA Graphs, and tokens and GSM8K accuracy do not change. With CUDA Graphs, the async pipeline is 10–29 % faster at 1 to 16 requests and 3 % faster at 64 requests; block mode is 5 % slower at 64 requests. The Known issues section gives the data and the decisions from the review.

  • Async eager rounds with boundary KV
  • Async and sync speculative CUDA Graphs
  • Batched async pipeline step
  • GPU tests on H100
  • Real-checkpoint FP32 token agreement at 1→2, 1→4 and 2→4
  • GSM8K accuracy, BF16
  • Real-checkpoint BF16 benchmark: concurrency sweep, long context, TTFT and TPOT
  • Decision on eager async rounds at low concurrency: options 1 and 2 here, option 3 later
  • Shared boundary view (option 2)

Asynchronous block and pipeline rounds now give PRELUDE and CODA their
depth-0 KV. The positions of these rows exist only on the device. Thus the
runner builds device metadata for the boundary stages, as it does for the
recurrent loops.

A pipeline step runs both boundary stages on all rows. Write masks keep the
input-stage KV of retained rows and the output-stage KV of rows between the
draft and target depths. Each step stores the same KV as a synchronous step.

The engine accepts asynchronous eager rounds for boundary-KV models. CUDA
Graphs, preemption and sampled requests stay rejected.
Asynchronous rounds capture bucketed PRELUDE and CODA graphs
(DeviceBoundaryGraphs) next to the recurrent graphs. Positions, write
addresses, write masks and context lengths are graph inputs. Thus the graphs
keep the device-owned progress of the rounds. The engine captures all buckets
at construction.

Synchronous rounds of a boundary-KV model use the host-planned boundary
graphs of ordinary decode (PreludeGraphs, and CodaGraphs with the cache). The
previous synchronous CODA graph had no KV input, and Dory cannot run it.

The engine accepts CUDA Graphs for boundary-KV self-speculation. Preemption
stays rejected.
The asynchronous pipeline step updated the device state of each request with
its own small operations, about 35 kernel launches for each request in each
step. At 64 requests, these launches made asynchronous eager rounds 27 %
slower than synchronous eager rounds. They also ran outside the CUDA Graphs.

The pipeline state of all live requests is now one pool, indexed by the slot
table that block rounds already use. A step gathers the rows of its requests,
updates them in one batch and scatters them back. Thus the number of
operations does not change with the request count. The rows, their order and
the results do not change. Preemption snapshots and restoration use the pool
slot of the request.
Describe how asynchronous rounds give PRELUDE and CODA their depth-0 KV, the
pipeline write masks, the sampling limit, the shared pipeline slot pool and
the graph families of each mode.

Add the validation status: the CUDA test file, the real-checkpoint FP32 token
agreement at 1 to 2, 1 to 4 and 2 to 4 loops, GSM8K accuracy, and BF16 wall
time, first-token time and time per output token on one H100.
@XinhaoTheo
XinhaoTheo force-pushed the feat/rfc176-dory-async-speculation branch from 8fbe505 to a6e2fe2 Compare November 18, 2026 04:55
@XinhaoTheo
XinhaoTheo marked this pull request as ready for review November 18, 2026 06:44
The previous wall times compared speculation with ordinary async Graph decode
that used the default graph cache of 16 shapes. At 64 requests that baseline
ran eager in most calls, so the reported gain was too large.

Replace the results with a concurrency sweep at a fixed prompt length (1, 4,
16 and 64 requests) and a separate long-context run. Every Graph arm now
captures up to 128 shapes and has no graph fallback. Add the time to the first
token and the time per output token, the GSM8K accuracy, and the known issues:
async Graph block at 64 requests and eager async rounds at low concurrency.

@0z5a 0z5a left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I read the async pipeline step, the boundary metadata and DeviceBoundaryGraphs. The logic matches the sync contract as far as I can see:

  • PRELUDE stores only for depth-0 rows (row_active & fresh); a retained row already holds its input-stage KV, and a restarted root is depth 0 again, so it rewrites the rejected candidate's slot.
  • CODA stores only for rows that reach the draft or the target depth, so the draft CODA KV of rows in between survives until the target CODA of that position overwrites it.
  • Padding rows get write mask false and context length 0 in both the eager and the graph path (_copy_graph_inputs zeroes the tail).
  • The batched update reproduces the per-request one (shift on target completion, restart at a rejection, limit cap), and slots return to free_slots on release.

Comments:

  1. Eager async at low concurrency (your open decision). I would take option 1 now, and option 2 in this PR if it is small: device_metadata runs twice per step with the same positions and tables and differs only in the write mask, so one metadata with two masks removes the duplicate gather and prepare_device. Option 3 (fused update kernel) as a separate PR, with an Ouro run, sounds right.
  2. Every row runs both boundary stages. That keeps one graph shape, but PRELUDE also runs on retained rows and CODA on rows that are at neither endpoint, and both outputs are discarded. At 64 requests this may explain part of the small gain. Not blocking; a sentence in the docs would help readers of Table 1.
  3. restore_pipeline on the slot pool. For Dory this path stays closed until async preemption is allowed, but Ouro async pipeline preemption already uses it on main. test_pipeline_graph_async_preemption_combination in tests/test_speculative.py covers it on CUDA. Did that test pass on H100 with the slot-pool rewrite? The existing request_id in self.states check runs before free_slots.pop(), so a double restore cannot leak a slot. Building values with a pageable torch.tensor(..., device=...) inside the compute stream is fine off the hot path.
  4. Merge order. The engine gate block and the Self-speculation part of docs/dory.md will conflict with the 3C change (sync preemption for Dory self-speculation), which edits the same lines. Either order is fine; whoever lands second resolves it. The #164 order you mention is the larger one.
  5. Tested on H100 only. That is enough for this PR in my view.

PRELUDE and CODA of a pipeline step built two depth-0 views with the same
positions and tables. Only the write masks were different. CODA now
reuses the PRELUDE view with its own write mask, as synchronous steps
and asynchronous block rounds already do. This also removes 11 kernel
launches per generated token in eager async rounds at one request.

Fresh rows reach the draft depth, so the CODA mask is the PRELUDE mask
plus the active rows that reach the target depth.
… work

Say that PRELUDE and CODA of a pipeline step use one depth-0 view, as
the other speculative paths do, and which boundary outputs the runtime
discards. This extra work can explain a part of the small gain at 64
requests. Add the new launch count and ABBA timing for eager async
rounds at one request.
@XinhaoTheo

Copy link
Copy Markdown
Author

@0z5a Thanks for the review.

  1. Done: option 1, and option 2 in d74adfa. CODA now reuses the PRELUDE view with its own write mask, as sync steps and async block rounds already do. Async makes 113 more launches per token than sync (before: 124). In two new ABBA containers, async against sync was −0.2 % and +0.4 %. The time between containers varies by a few percent, so I do not claim that the gap is closed. Option 3 goes in a separate PR with an Ouro run.
  2. Done: docs/dory.md now says which boundary outputs the runtime discards, next to Table 1.
  3. Yes. With the slot-pool code, the 8 Triton cases of test_pipeline_graph_async_preemption_combination passed on H100. The FA4 cases did not run, because flash-attn is not installed there. The full GPU run at d74adfa also passed (610 passed).
  4. Agreed. Whoever lands second resolves the conflict.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants