Repository navigation
Conversation
The compiled warm runs ran on the legacy default stream. An AccumulateGrad node stores its creation stream, and when an autograd graph of a warm run is kept alive, the captured backward runs that node on the legacy stream: cudaErrorStreamCaptureImplicit at the backward in the capture (partitioned mode). The test env keeps such graphs alive (better_exchook replace_traceback_format_tb keeps frames). Now the warm runs and the capture run on the eager side stream, cuda_graph_capture gets a stream arg.
…e-side-stream # Conflicts: # returnn/torch/util/capture_lock.py
…e-side-stream # Conflicts: # returnn/torch/util/capture_lock.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
torch_cuda_graphwithcompileandpartitioned:the capture failed in the RETURNN test env with
CUDA error: operation would make the legacy stream depend on a capturing blocking stream(cudaErrorStreamCaptureImplicit)at
outs[0].backward()inGraphCapturedTrainStep._capture_compiled,after the warning "The AccumulateGrad node's stream does not match the stream of the node that produced the incoming gradient".
Cause:
the compiled warm runs (
_ensure_compiledand the plain warm run in_capture_compiled) ran on the legacy default stream.An
AccumulateGradnode stores the stream it was created on, and the param refers to it only weakly.When something keeps the autograd graph of a warm run alive, the backward in the capture reuses that node and runs it on the legacy stream.
In the test env,
better_exchook.replace_traceback_format_tb(tests/_setup_test_env.py) keeps the frames oftraceback.extract_stack,and torch
FakeTensorModestores such a stack, which holds the first compiled call's outputs.With
warmup_steps > 0it worked, as the eager warmup creates the accumulators on the side stream.Fix:
the compiled warm runs and the capture run on
_eager_stream, the side stream the eager warmup already uses.cuda_graph_capturegets a keyword-onlystreamarg for that (default: as before, the default capture stream oftorch.cuda.graph).Warm runs and capture on one stream, so a kept-alive accumulator does not even need a cross-stream join in the graph.
Affected: only
compilewithpartitioned(the unpartitioned compiled step binds the grads, no autograd backward in the capture).Production main processes do not patch
StackSummary.extract, so this was seen in the tests only;it is still a dependence on whether some library keeps a graph alive.
Test:
test_torch_engine_cuda_graph_compile_partitioned_train(partitioned with capture,warmup_stepsdefault 0).CUDA-only, so it skips in CI.
Run on the RZ GPU test node, H100:
tests/test_torch_engine.py: 78 passed (torch 2.12);-k "cuda_graph or optimizer_step": 25 passed (torch 2.7).