Skip to content

PT graph capture, compiled and captured eval steps (torch_cuda_graph eval) - #1995

Draft
albertz wants to merge 5 commits into
masterfrom
albert-torch-cuda-graph-eval
Draft

albertz wants to merge 5 commits into
masterfrom
albert-torch-cuda-graph-eval

Conversation

@albertz

@albertz albertz commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

New torch_cuda_graph option "eval": True (default off):
the eval steps (Engine.eval_model) run Inductor-compiled and CUDA-graph-captured like the train steps.

  • A second, forward-only program per GraphCapturedTrainStep, train flag off, no backward / optimizer step.
    It follows the same compile / capture options as the train step.
  • It reads the same static data / lens buffers, so every eval batch must fit the train bounds
    (batch_size_bound, dim_capacity, packed_total_bound), checked per batch as for train.
    This is also what JAX jax_static_shapes and TF tf_static_shapes do for eval.
  • Its graph is captured into the memory pool of the train graph (and vice versa, whichever is captured second),
    so the eval graph needs no extra pool memory.
    This is safe as the two never run concurrently, and the engine reads each one's outputs right after its replay.
  • Per-program state (compiled fn, graph, result ctx, loss output layout) is in a small _Program holder,
    as the eval step can mark a different set of losses than the train step.
  • Without the option, eval runs the dynamic-shape path as before (bound shapes switched off around it).
    Off by default, as an existing setup with dev seqs beyond dim_capacity would fail otherwise.

Tests (CUDA-only, so they skip in CI), RZ H100 test node:

  • New test_torch_engine_cuda_graph_eval: LM step with BOS / EOS, bf16, mask_padding_seqs,
    padded and packed with compile, and padded with eager capture; 2 epochs with a dev eval after each.
    Dev scores of both epochs match the eager run, incl a loss marked only with the train flag off,
    as do the train steps of epoch 2, which replay after the eval graph;
    the eval graph shares the pool of the train graph.
  • Negative controls (torch 2.12): on master the new test fails (unknown option eval);
    with the eval program traced with the train flag on, it fails with KeyError: 'dev_loss_seq_ce_sq'.
  • -k cuda_graph: 10 passed with torch 2.12 and with torch 2.7 (before the rebase onto PT graph capture: release switches bound shapes off #1996);
    after the rebase, torch 2.12: 11 passed (incl the PT graph capture: release switches bound shapes off #1996 test);
    after the merge of master f640634, torch 2.12: 11 passed.
  • Full test_torch_engine.py, torch 2.12: 76 passed.

Measured (H100 test node, torch 2.12): conformer encoder 12 x 512, 8 heads, CTC (native op), bf16,
train step compiled + captured (capture_optimizer, warmup_steps 0, mask_padding_seqs), padded, batch 30k frames / max 40 seqs,
400 synthetic dev seqs of 300 to 1600 frames.
Columns: wall time of Engine.eval_model (3 runs, after the first eval which compiles / captures),
dev CTC, graph pool reserved after train + first eval, peak allocated during the timed evals.

eval time (s) dev CTC graph pool peak alloc
eager (master behavior) 4.42, 4.51, 4.68 377.9429 14.22 GiB 1.82 GiB
"eval": True 2.20, 2.30, 2.32 377.9435 14.22 GiB (shared) 1.29 GiB

The same probe packed is not usable yet: the packed graphc training itself gets non-finite params after the first step,
on master as well, independent of this PR (separate issue).
Not measured: a production setup.

…eval)

New torch_cuda_graph option "eval" (default off):
the eval steps run compiled / captured like the train steps,
as a second, forward-only program with the train flag off,
on the same bounds and static buffers,
captured into the memory pool of the train graph.
Without it, eval runs the dynamic-shape path as before.
Off by default: with it, every eval batch must fit the train bounds.
…-cuda-graph-eval

# Conflicts:
#	returnn/torch/util/capture_lock.py
#	returnn/torch/util/graph_capture.py
# Conflicts:
#	returnn/torch/util/capture_lock.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant