Skip to content

PT distributed state sync, non-tensor values cost per value - #2085

Merged
albertz merged 2 commits into
masterfrom
albert-torch-distributed-sync-state-values
Nov 19, 2026
Merged

albertz merged 2 commits into
masterfrom
albert-torch-distributed-sync-state-values

Conversation

@albertz

@albertz albertz commented Nov 19, 2026

Copy link
Copy Markdown
Member

For the optimizer state values which are no tensors (e.g. a Python step per param of a custom optimizer),
which every rank takes from the first rank of the group, _sync_state

  • flattened every param's state a second time to collect the value,
  • broadcast the values as a list of objects, and broadcast_object_list pickles and unpickles every list item on its own,
  • rebuilt the state (tree.flatten + tree.unflatten_as) for every value at every sync,
    also when the received value equals the local one, the normal case for ranks which did the same updates.

Now the values of the first pass are kept, they go out as one object,
and a state is rebuilt only when the received value differs from the local one.
The comparison is done for plain int, float, str and None values of the same type;
anything else is written as before.
No change in results, with one exception by construction: a -0.0 of the first rank does not replace a local 0.0.

Affected: the optimizer state sync (sync_optimizer_state, sync_at_epoch_end) with an optimizer
which keeps non-tensor values in its state. The torch optimizers do not.

Timing of the value part of _sync_state (the tensor reduces stubbed out),
2583 params with a Python step and two tensors each, 4 gloo ranks on one host,
RWTH login node, torch 2.12, min of 7:
57.1 ms on the first rank and 83 ms on the others on master, 15.3 to 15.5 ms with this PR.
Of the time on master, the broadcast was 29 ms (first rank) and 55 ms (others), the rebuild 12 ms;
what remains is the pass over the state (tree.flatten_with_path, 14 ms).

Tests: the optimizer state sync test gets two Python values per param,
a count which differs on one rank only (equal on the others, so both paths run)
and a value which differs only in its type (1.0 on the first rank, 1 on the others):
every rank holds the value of the first rank, with its type.
As this is a pure speedup, it passes on master as well; the control is the timing above.
The full tests/test_torch_engine.py passes (CPU, torch 2.12: 62 passed, 34 skipped, the skipped ones need CUDA).

Independent of #2083 and #2084, merges in any order (checked with a three-way merge of the shared files).

For the optimizer state values which are no tensors (e.g. a Python step per param),
_sync_state flattened every param's state a second time to collect them,
broadcast them as a list of objects, which pickles every value on its own,
and rebuilt the state for every value, also when the received one equals the local one,
the normal case for ranks which did the same updates.

Now the values of the first pass are kept, they go out as one object,
and a state is rebuilt only when the value differs
(compared for int, float, str and None of the same type, anything else is written as before).
The exact type list excluded bool (and other subclasses, e.g. numpy.float64),
so an equal bool was always rebuilt.
The type equality check before it already keeps True and 1 apart.
@albertz
albertz marked this pull request as ready for review November 19, 2026 13:37
@albertz
albertz requested a review from a team as a code owner November 19, 2026 13:37
@albertz
albertz merged commit 9bedb2c into master Nov 19, 2026
65 checks passed
@albertz
albertz deleted the albert-torch-distributed-sync-state-values branch November 19, 2026 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant