Skip to content

PT graph capture: real batch size as compiled step input - #2006

Draft
albertz wants to merge 1 commit into
masterfrom
albert-graph-capture-batch-size-arg
Draft

albertz wants to merge 1 commit into
masterfrom
albert-graph-capture-batch-size-arg

Conversation

@albertz

@albertz albertz commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Follow-up to #1980 (torch_cuda_graph mask_padding_seqs).

With mask_padding_seqs (explicit, or the default since behavior version 36),
the real batch size buffer _batch_size_buf was closed-over state of the compiled step,
while _compiled_call_args passes the data, seq lens and step buffers as explicit inputs.
With partitioned: True, its value was frozen at trace time:
every later step with another real batch size masked the padding seqs by the traced batch size,
so losses and grads were wrong.
Whole-step compile was not affected (losses and grad norm match eager on master, see below).

Now _compiled_call_args passes _batch_size_buf as an explicit input when it exists, before the step tensor (still last).
step_core_buf_inputs and step_core_buf_inputs_v212 both swap it in around step_core and restore it in the finally,
like the data, lens and step buffers.
Without mask_padding_seqs, the argument layout is unchanged.
The scalar stays device-resident (no host sync per step, no recompilation).

Test: test_torch_engine_cuda_graph_mask_padding_seqs (CUDA only, so not run in CI) now also runs the partitioned compiled step
against eager, padded and packed, with per-step changing real batch sizes and real empty seqs, losses and grad norm.
The partitioned arm runs with capture: False: the compiled step inputs are the same,
and partitioned capture fails in the test env for an unrelated reason
(better_exchook.replace_traceback_format_tb keeps the frames of the first compiled call alive,
its autograd graph then holds AccumulateGrad nodes of the legacy stream; not the case in the main process of a RETURNN run).

Tested at RWTH, H100, torch 2.12 and 2.7:

  • master + new test only: fails, padded partitioned True step (1, 1) ce: 5.965 vs 5.619.
  • this PR: the test passes; full test_torch_engine.py 77 passed (torch 2.12).
  • rerun on master f640634 + this diff (torch 2.12): master + new test only fails the same way, the test passes,
    full test_torch_engine.py 94 passed (master alone: 94 passed).
  • with PT graph capture: compiled warm runs and capture on side stream #2007 applied as well and the partitioned arm under capture (capture=not partitioned removed):
    master + PT graph capture: compiled warm runs and capture on side stream #2007 + new test only fails the same way, the test passes.
    Whichever of the two merges second drops capture=not partitioned.
  • probe beyond the test, under capture (stock traceback.StackSummary.extract, as in the main process):
    the gradient of every parameter after the last step of the epoch (smallest real batch, capture on a larger one), against eager,
    with mask_padding_seqs: True and with behavior version 37 and the option unset (same numbers):
arm max abs loss diff (ce / seq_ce) max rel L2 grad diff over params
master, partitioned, packed 0.791 / 15.881 0.379
this PR, partitioned, packed 0 / 0 0.0012
this PR, partitioned, padded 0 / 0 0.0016
this PR, whole-step, packed 0.002 / 0.013 0.011
this PR, whole-step, padded 0.002 / 0.010 0.0087

Same numbers with torch 2.7, and for the partitioned packed arms with capture: False (test env).
Whole-step on master: the existing test (losses, grad norm) passes there as well.
Not tested: distributed training.

With mask_padding_seqs, the real batch size buffer was closed-over state of the compiled step,
unlike the data, seq lens and step buffers, which are explicit inputs.
In partitioned mode, its value was frozen at trace time,
so steps with another real batch size masked the padding seqs by the traced one (wrong losses and grads).
It is now an explicit input as well, before the step tensor, in both compiled wrappers.
@navverriani

navverriani commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Solution looks good to me. Got a similar problem.

Additionally, I tested distributed training (2 ranks sharing one GPU, Gloo, grad_explicit, different data per rank):

  • Both whole-step and partitioned compilation with capture match eager training within approximately 1e-7.
  • Both ranks end with identical parameters.

Regarding "partitioned arm runs with capture: False": checked it as well, it really is the test-env traceback patch.
Maybe the test could disable the traceback patch for this case and run the partitioned arm with capture on,
then it would also cover the captured path that real trainings use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants