A learning project. I wanted to understand how network design affects AI training performance, so I built a testbed that lets me change the topology with a command-line flag and measure what happens. I need to thank my Claude for some great assists!
Training a large model means thousands of GPUs exchanging gradients dozens of times per second. If the network stalls, the GPUs idle — and idle GPUs are the largest cost line in an AI cluster. That makes fabric design a genuinely expensive engineering decision, and it is a corner of networking I had no hands-on experience with.
Rather than read about it, I wanted to measure it.
- Fat-tree — the general-purpose datacenter design. Many equal-cost paths between any two hosts, load-balanced by ECMP hashing.
- Rail-optimized — purpose-built for AI (NVIDIA DGX SuperPOD). Each GPU has its own NIC, and GPU rank r in every server connects to a dedicated "rail" switch r.
The question I set out to answer: for a realistic AI training workload, how much does the choice actually matter, and why?
The packet simulator is not mine. It is the Alibaba HPCC fork of ns-3, published alongside the HPCC paper (SIGCOMM 2019), which implements RoCEv2 queue pairs, DCQCN, PFC, ECN and a Broadcom shared-buffer switch model.
What I wrote is everything around it:
| Component | What it does |
|---|---|
Dockerfile |
Reproducible build of ns-3.18 + the HPCC fork. One command, works on Apple Silicon. |
topo/fat_tree.py |
Generates a k-ary fat-tree of any size in HPCC's topology format |
topo/rail_optimized.py |
Generates a rail-optimized fabric (servers × GPUs × rails + spines) |
traffic/allreduce.py |
Ring all-reduce traffic, rail-aware or deliberately rail-oblivious |
traffic/incast.py |
Synchronized many-to-one incast |
experiments/run.py |
Turns one command line into a complete, reproducible experiment |
analysis/ |
Flow-completion statistics and the figures below |
All runs: 128 GPUs, one step of ring all-reduce, 100 Gbps links, DCQCN congestion control.
Rail-optimized removed all variance. Every one of 128 flows finished in exactly 91.0 µs — 1.03× the theoretical floor. On the fat-tree, the median was 178.7 µs but the worst flow took 959.9 µs
| Fabric | median | p99 | worst | worst slowdown |
|---|---|---|---|---|
| Rail-optimized | 91.0 µs | 91.0 µs | 91.0 µs | 1.03× |
| Fat-tree | 178.7 µs | 539.0 µs | 959.9 µs | 10.42× |
Ring ordering mattered as much as the hardware. Running the same collective on the same rail-optimized fabric, but ordering the ring naively (0 → 1 → 2 → …), made every hop cross-rail and forced traffic up to the spine tier. Estimated collective time went from 2.73 ms to 18.0 ms — 6.6× worse
The advantage only appears with large messages. Sweeping flow size from 1 KB to 4 MB, the fat-tree's penalty is invisible below ~10 KB and severe above ~1 MB
| Bytes per flow | Rail median | Fat-tree median | Fat-tree worst |
|---|---|---|---|
| 1 KB | 4.2 µs | 10.6 µs | 12.7 µs |
| 100 KB | 12.7 µs | 25.4 µs | 42.2 µs |
| 1 MB | 91.0 µs | 178.7 µs | 959.9 µs |
| 4 MB | 352.0 µs | 689.0 µs | 1.59 ms |
The measurements are the evidence; these are the things I actually came away understanding.
Rail-optimized wins by removing path choice, not by shortening paths. I expected the benefit to be about hop count. It is not. When same-rank GPUs share a rail switch, there is exactly one route between them, so ECMP has nothing to hash and two flows physically cannot be assigned to the same link by accident.
ECMP is great for many small flows and bad for a few large ones. Hashing distributes well when there are thousands of flows to average over. AI training produces a handful of enormous ones, so a collision doesn't average out — it halves the throughput of both flows for the entire transfer.
Tail latency is the only number that matters for a collective. A collective finishes when its slowest member finishes, so 127 fast flows and one straggler is exactly as bad as 128 slow ones.
There are two distinct regimes. Below ~10 KB, completion time is round-trip propagation and hop count decides. Above ~100 KB, it is bytes divided by bandwidth and contention decides.
Good congestion control makes the emergency brake unnecessary. PFC never fired in any run. DCQCN reacts to ECN marks and throttles senders before switch buffers are ever in danger. PFC only exists for when that fails — and PFC storms are the classic way production RoCE deployments fall over.
Sanity-check simulator output against arithmetic. Bytes divided by bandwidth, hops times delay. If the tool disagrees with physics, the tool is being misused.
The build. The HPCC fork is a 2018-era research codebase. Its build system needs Python 2, which Ubuntu 20.04 no longer provides as python, so ./waf configure died twice with increasingly obscure errors before I pinned Python 2 alongside Python 3 in the image.
A units bug that invalidated my first two experiments. HPCC reads flow size into a variable named maxPacketCount, so my generator divided byte counts by the packet size. The results looked plausible until I checked them: a "10 MB" transfer was finishing in 12 µs, which is faster than a 100 Gbps link can physically move 10 MB. The field is actually wired to an attribute documented as "the number of bytes to write." Fixed in ad2e414; the first two runs were silently 1000× too small.
More reasoning behind each design choice is in docs/DESIGN-NOTES.md.
Requires Docker. On Apple Silicon, enable Settings → General → Use Rosetta for x86/amd64 emulation.
docker build -t rdma-sim .docker run -it --rm -v "$PWD":/work rdma-sim bashInside the container:
python3 -m experiments.run --topo rail --n-servers 16 --gpus-per-server 8 --traffic ring-allreduce --tensor-size 16000000 --ring-mode parallel --sim-time 2.05 --no-packet-trace --name my-runbash experiments/sweep.sh && python3 -m analysis.plotspython3 -m analysis.fct_stats results/*/fct.txtBuild takes about 4 minutes. A 128-GPU all-reduce step simulates in 12–30 seconds.
topo/ topology generators + HPCC topology.txt writer
traffic/ traffic patterns + HPCC flow.txt writer
experiments/ orchestrator and sweep driver
analysis/ statistics and figures
docs/ design notes
plots/ committed figures
results/ simulation output (gitignored)
Dockerfile ns-3.18 + HPCC RDMA fork, pinned