Skip to content

PT distributed, all collectives on the sync device - #2086

Merged
albertz merged 2 commits into
masterfrom
albert-torch-distributed-sync-device
Nov 20, 2026
Merged

albertz merged 2 commits into
masterfrom
albert-torch-distributed-sync-device

Conversation

@albertz

@albertz albertz commented Nov 19, 2026 •

Copy link
Copy Markdown
Member

The train loop control sync (has data, complete_frac) runs on the device the level syncs on:
the train device, or the CPU with sync_on_cpu.
Some collectives were still fixed to the CPU, independent of sync_on_cpu:

  • the max of the counts in _sync_state (the train step, the step tensors of the optimizer state, integer buffers),
  • the broadcast of the optimizer state values which are no tensors,
  • the eval control broadcast (has_data from rank 0),
  • the broadcast of the eval scores at the end of eval_model.

Now they all run on that same device, so one backend (e.g. NCCL) does all of it.
Counts which live on the CPU (the train step, the step tensors of a non-capturable torch optimizer)
are moved there as one flat buffer per dtype.
For that, the flat reduce gets the device as an argument,
_all_reduce_flat(..., sync_device=...) (before: sync_on_cpu).
No change in results.

What stays on the CPU:
the hostname gather at init (on purpose, before the device is selected),
and the seed which the torch DataLoader shares per iterator (inside torch, a CPU scalar).
So a backend for CPU tensors is still needed.

Tests: the epoch-end sync test records the device of every all_reduce and broadcast
(the counts of three dtypes, the Python values, the eval control broadcast, the eval scores; it gets a dev dataset for that)
and checks that all are on the train device.
New CUDA variant test_torch_engine_distributed_param_sync_at_epoch_end_cuda (two gloo ranks on one GPU),
which fails on master (the counts and all broadcasts on the CPU) and passes with this PR.
On the CPU the check holds trivially, so the CPU tests pass on master as well.

Run with torch 2.12:

  • CPU (RWTH login node): the full tests/test_torch_engine.py passes (62 passed, 35 skipped, the skipped ones need CUDA).
  • CUDA (JUPITER login node, GH200): all 23 distributed tests of the file pass.

Not measured: the timing. Not run: NCCL over several GPUs or hosts (the CUDA tests use gloo, with both ranks on one GPU).

Master is merged in after #2085 (the same line of the object broadcast); both runs above were repeated on the merge result for the distributed tests.

The train loop control sync (has data, complete_frac) runs on the device the level syncs on:
the train device, or the CPU with sync_on_cpu.
Some collectives were still fixed to the CPU, independent of sync_on_cpu:
the max of the counts in the state sync (train step, optimizer step tensors, integer buffers),
the broadcast of the optimizer state values which are no tensors,
the eval control broadcast, and the broadcast of the eval scores.

Now they all run on that same device, so one backend (e.g. NCCL) does all of it.
Counts which live on the CPU are moved there as one flat buffer per dtype.
The flat reduce gets the device as an argument (sync_device, before: sync_on_cpu).
No change in results.
…-distributed-sync-device

# Conflicts:
#	returnn/torch/distributed.py
@albertz
albertz marked this pull request as ready for review November 20, 2026 08:56
@albertz
albertz requested a review from a team as a code owner November 20, 2026 08:56
@albertz
albertz merged commit d2db30b into master Nov 20, 2026
65 checks passed
@albertz
albertz deleted the albert-torch-distributed-sync-device branch November 20, 2026 09:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant