Skip to content

PT graph capture: release switches bound shapes off - #1996

Merged
albertz merged 1 commit into
masterfrom
albert-torch-graph-capture-release-bound-shapes
Oct 14, 2026
Merged

albertz merged 1 commit into
masterfrom
albert-torch-graph-capture-release-bound-shapes

Conversation

@albertz

@albertz albertz commented Oct 13, 2026

Copy link
Copy Markdown
Member

GraphCapturedTrainStep.release (called from Engine.finalize) now switches the bound-shape dim state off (set_bound_shapes_enabled(False)).

The bound-shape dim state is process-wide:
the global batch_dim capacity / size and the dim_capacity capacities on the template dims.
It is enabled in GraphCapturedTrainStep.__init__ and per train_epoch,
and was only switched off by eval_model at the end of each train epoch.
A graphc training which stops in the middle of an epoch (an exception, e.g. inf/nan loss) left it on after finalize,
so a later eager engine in the same process saw the dims at their bounds.
The upcoming torch_cuda_graph eval option (#1995) re-enables the bound shapes for eval and would leave them on as well.

Before, an eager engine after such a graphc engine, padded or packed:

RuntimeError: The size of tensor a (10) must match the size of tensor b (4) at non-singleton dimension 0

(10 = batch_size_bound, in Dim.get_mask via copy_masked -> torch.where).
After finalize, the dims are now as before the graphc engine.
Affects only processes which run several engines, e.g. tests, not a normal RETURNN run.

Test: test_torch_engine_cuda_graph_finalize_then_eager (CUDA only, so it skips in CI):
graphc training stopped by an exception after the capture, finalize, then an eager packed engine with the same dims.
Run on the RZ GPU test node (H100, torch 2.12):

  • fails on master with the error above (also a padded eager engine instead of packed fails the same way, probe not in the PR),
  • passes with the fix,
  • full tests/test_torch_engine.py with the fix: 76 passed.

The bound-shape dim state (global batch dim size / capacity, template dim capacities)
is process-wide and was only switched off by eval_model at the end of each train epoch.
A graphc training which stopped in the middle of an epoch (exception) left it on after finalize,
so a later eager engine in the same process saw the dims at their bounds.
@albertz
albertz marked this pull request as ready for review October 14, 2026 16:10
@albertz
albertz requested a review from a team as a code owner October 14, 2026 16:10
@albertz
albertz merged commit 2b09f02 into master Oct 14, 2026
71 checks passed
@albertz
albertz deleted the albert-torch-graph-capture-release-bound-shapes branch October 14, 2026 16:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant