Summary

  • Adel Iazzag’s NANOG 98 slides describe Mixture-of-Experts training as uneven AllToAll exchanges twice per expert layer. Many senders converge on one port, and the congested destination can move.
  • His operating answer is to test the actual AllToAll collective, give the scheduler network topology, and design the fabric for many-to-one congestion instead of assuming that a single routing algorithm will absorb every workload.

The hottest port in an AI cluster may be a temporary appointment. In Adel Iazzag’s NANOG 98 presentation, Mixture-of-Experts training sends tokens toward selected experts on other GPUs. Several senders can converge on one destination; at the next expert decision, the gathering point may move. A network team that chases yesterday’s port has learned the wrong lesson from its own alarm.

Iazzag’s original 32-slide deck is useful because it separates four traffic shapes that the phrase “AI workload” tends to flatten. Data parallelism synchronizes gradients with AllReduce and can light every uplink together, then go quiet. Pipeline parallelism exchanges activations and gradients between adjacent stages; link utilization can look low while a stage waits for a slow handoff. Tensor parallelism talks frequently among the same small group, at each layer. Iazzag wants that group inside the scale-up domain. Mixture-of-Experts produces the awkward fourth shape: AllToAll twice per expert layer, many flows, uneven destinations, and incast that changes ports.

These patterns ask different things of the fabric. The deck’s 256-GPU ring-versus-tree example counts 510 communication steps for the ring and 16 for the tree. Those are algorithmic steps in an illustration, not elapsed seconds from a production run. The practical point is that collective choice, message size and topology alter what the switches see. A single average-utilization chart cannot choose placement or congestion policy for all four.

Iazzag’s design list puts rail alignment first: connect the same NIC index across servers to the same leaf so predictable groups have predictable paths. He calls for one-to-one subscription when every GPU can transmit at line rate, lossless behavior with congestion control tested for many-to-one arrivals, and packet or flowlet spraying where ordinary ECMP leaves too little path diversity. The scheduler must know topology. It decides which GPUs become neighbors in a pipeline, which tensor group remains inside scale-up, and where an expert group gathers traffic. The switching fabric then has to carry the decision the scheduler made.

The slides give no named production incident or measured idle-GPU bill. A before-and-after job trace would put a price on the effect. Iazzag’s test instruction is concrete: run the AllToAll collective on the actual fabric and observe the shifting port-level congestion. A capacity plan built only on bulk throughput can buy expensive links while the training job still queues at a moving many-to-one choke point. The operator should measure the traffic shape that the job creates, then make placement and congestion controls answer that shape.

Sources: NANOG 98 session · Iazzag’s original slides.