Repository navigation
Conversation
With mask_padding_seqs, the real batch size buffer was closed-over state of the compiled step, unlike the data, seq lens and step buffers, which are explicit inputs. In partitioned mode, its value was frozen at trace time, so steps with another real batch size masked the padding seqs by the traced one (wrong losses and grads). It is now an explicit input as well, before the step tensor, in both compiled wrappers.
Contributor
|
Solution looks good to me. Got a similar problem. Additionally, I tested distributed training (2 ranks sharing one GPU, Gloo, grad_explicit, different data per rank):
Regarding "partitioned arm runs with capture: False": checked it as well, it really is the test-env traceback patch. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1980 (
torch_cuda_graphmask_padding_seqs).With
mask_padding_seqs(explicit, or the default since behavior version 36),the real batch size buffer
_batch_size_bufwas closed-over state of the compiled step,while
_compiled_call_argspasses the data, seq lens and step buffers as explicit inputs.With
partitioned: True, its value was frozen at trace time:every later step with another real batch size masked the padding seqs by the traced batch size,
so losses and grads were wrong.
Whole-step compile was not affected (losses and grad norm match eager on master, see below).
Now
_compiled_call_argspasses_batch_size_bufas an explicit input when it exists, before the step tensor (still last).step_core_buf_inputsandstep_core_buf_inputs_v212both swap it in aroundstep_coreand restore it in thefinally,like the data, lens and step buffers.
Without
mask_padding_seqs, the argument layout is unchanged.The scalar stays device-resident (no host sync per step, no recompilation).
Test:
test_torch_engine_cuda_graph_mask_padding_seqs(CUDA only, so not run in CI) now also runs the partitioned compiled stepagainst eager, padded and packed, with per-step changing real batch sizes and real empty seqs, losses and grad norm.
The partitioned arm runs with
capture: False: the compiled step inputs are the same,and partitioned capture fails in the test env for an unrelated reason
(
better_exchook.replace_traceback_format_tbkeeps the frames of the first compiled call alive,its autograd graph then holds AccumulateGrad nodes of the legacy stream; not the case in the main process of a RETURNN run).
Tested at RWTH, H100, torch 2.12 and 2.7:
padded partitioned True step (1, 1) ce: 5.965 vs 5.619.test_torch_engine.py77 passed (torch 2.12).full
test_torch_engine.py94 passed (master alone: 94 passed).capture=not partitionedremoved):master + PT graph capture: compiled warm runs and capture on side stream #2007 + new test only fails the same way, the test passes.
Whichever of the two merges second drops
capture=not partitioned.traceback.StackSummary.extract, as in the main process):the gradient of every parameter after the last step of the epoch (smallest real batch, capture on a larger one), against eager,
with
mask_padding_seqs: Trueand with behavior version 37 and the option unset (same numbers):Same numbers with torch 2.7, and for the partitioned packed arms with
capture: False(test env).Whole-step on master: the existing test (losses, grad norm) passes there as well.
Not tested: distributed training.