Repository navigation
PT distributed state sync, non-tensor values cost per value - #2085
Merged
Merged
Conversation
For the optimizer state values which are no tensors (e.g. a Python step per param), _sync_state flattened every param's state a second time to collect them, broadcast them as a list of objects, which pickles every value on its own, and rebuilt the state for every value, also when the received one equals the local one, the normal case for ranks which did the same updates. Now the values of the first pass are kept, they go out as one object, and a state is rebuilt only when the value differs (compared for int, float, str and None of the same type, anything else is written as before).
The exact type list excluded bool (and other subclasses, e.g. numpy.float64), so an equal bool was always rebuilt. The type equality check before it already keeps True and 1 apart.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
For the optimizer state values which are no tensors (e.g. a Python
stepper param of a custom optimizer),which every rank takes from the first rank of the group,
_sync_statebroadcast_object_listpickles and unpickles every list item on its own,tree.flatten+tree.unflatten_as) for every value at every sync,also when the received value equals the local one, the normal case for ranks which did the same updates.
Now the values of the first pass are kept, they go out as one object,
and a state is rebuilt only when the received value differs from the local one.
The comparison is done for plain
int,float,strandNonevalues of the same type;anything else is written as before.
No change in results, with one exception by construction: a
-0.0of the first rank does not replace a local0.0.Affected: the optimizer state sync (
sync_optimizer_state,sync_at_epoch_end) with an optimizerwhich keeps non-tensor values in its state. The torch optimizers do not.
Timing of the value part of
_sync_state(the tensor reduces stubbed out),2583 params with a Python
stepand two tensors each, 4 gloo ranks on one host,RWTH login node, torch 2.12, min of 7:
57.1 ms on the first rank and 83 ms on the others on master, 15.3 to 15.5 ms with this PR.
Of the time on master, the broadcast was 29 ms (first rank) and 55 ms (others), the rebuild 12 ms;
what remains is the pass over the state (
tree.flatten_with_path, 14 ms).Tests: the optimizer state sync test gets two Python values per param,
a count which differs on one rank only (equal on the others, so both paths run)
and a value which differs only in its type (
1.0on the first rank,1on the others):every rank holds the value of the first rank, with its type.
As this is a pure speedup, it passes on master as well; the control is the timing above.
The full
tests/test_torch_engine.pypasses (CPU, torch 2.12: 62 passed, 34 skipped, the skipped ones need CUDA).Independent of #2083 and #2084, merges in any order (checked with a three-way merge of the shared files).