Skip to content

PT graph capture: compiled warm runs and capture on side stream - #2007

Draft
albertz wants to merge 4 commits into
masterfrom
albert-graph-capture-side-stream
Draft

albertz wants to merge 4 commits into
masterfrom
albert-graph-capture-side-stream

Conversation

@albertz

@albertz albertz commented Oct 15, 2026

Copy link
Copy Markdown
Member

torch_cuda_graph with compile and partitioned:
the capture failed in the RETURNN test env with
CUDA error: operation would make the legacy stream depend on a capturing blocking stream (cudaErrorStreamCaptureImplicit)
at outs[0].backward() in GraphCapturedTrainStep._capture_compiled,
after the warning "The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient".

Cause:
the compiled warm runs (_ensure_compiled and the plain warm run in _capture_compiled) ran on the legacy default stream.
An AccumulateGrad node stores the stream it was created on, and the param refers to it only weakly.
When something keeps the autograd graph of a warm run alive, the backward in the capture reuses that node and runs it on the legacy stream.
In the test env, better_exchook.replace_traceback_format_tb (tests/_setup_test_env.py) keeps the frames of traceback.extract_stack,
and torch FakeTensorMode stores such a stack, which holds the first compiled call's outputs.
With warmup_steps > 0 it worked, as the eager warmup creates the accumulators on the side stream.

Fix:
the compiled warm runs and the capture run on _eager_stream, the side stream the eager warmup already uses.
cuda_graph_capture gets a keyword-only stream arg for that (default: as before, the default capture stream of torch.cuda.graph).
Warm runs and capture on one stream, so a kept-alive accumulator does not even need a cross-stream join in the graph.

Affected: only compile with partitioned (the unpartitioned compiled step binds the grads, no autograd backward in the capture).
Production main processes do not patch StackSummary.extract, so this was seen in the tests only;
it is still a dependence on whether some library keeps a graph alive.

Test: test_torch_engine_cuda_graph_compile_partitioned_train (partitioned with capture, warmup_steps default 0).
CUDA-only, so it skips in CI.
Run on the RZ GPU test node, H100:

  • master + new test: fails with the error above (torch 2.12 and 2.7).
  • branch: passes (torch 2.12 and 2.7), no AccumulateGrad stream warning in the test's process (torch 2.12).
  • branch, full tests/test_torch_engine.py: 78 passed (torch 2.12); -k "cuda_graph or optimizer_step": 25 passed (torch 2.7).

The compiled warm runs ran on the legacy default stream.
An AccumulateGrad node stores its creation stream,
and when an autograd graph of a warm run is kept alive,
the captured backward runs that node on the legacy stream:
cudaErrorStreamCaptureImplicit at the backward in the capture (partitioned mode).
The test env keeps such graphs alive (better_exchook replace_traceback_format_tb keeps frames).
Now the warm runs and the capture run on the eager side stream,
cuda_graph_capture gets a stream arg.
…e-side-stream

# Conflicts:
#	returnn/torch/util/capture_lock.py
…e-side-stream

# Conflicts:
#	returnn/torch/util/capture_lock.py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant