AI and machine learning have actually altered how networks act under load. The packages look the very same, but the traffic patterns and timing tension every presumption we made in classical data‑com connectivity. Training clusters press elephant streams between countless GPUs. Function stores and stream processors hammer east‑west links with small, latency‑sensitive operations. Information consume rises in bursts. A network that looked idle at 40 percent usage will still drop frames, trigger microbursts, and stall tasks if it can not take in short-lived fan‑in and offer deterministic tail latency.
I've helped groups update brownfield environments and develop greenfield training clusters. The frustrations follow a pattern: overconfidence in nominal bandwidth, underinvestment in optics and cabling health, and control planes that can't assemble quickly enough when a link flaps at the wrong moment. The wins follow a pattern too: tidy physical layers, consistent optics, thoughtful buffering, and open, automatable changing that treats telemetry as a first‑class signal. Redesigning for AI/ML workloads is less about hero hardware and more about peaceful, predictable behavior under pressure.
What AI/ML Workloads Do to Networks
Data engineering pipelines create high‑fan‑in flows when numerous workers compose checkpoints or pull model artifacts. Training tasks with data parallelism create integrated bursts as gradients pass through the material throughout all‑reduce operations. Reasoning at scale sends out numerous little RPCs with rigorous latency spending plans. These do not mix with dignity with background backup tasks or warehouse inquiries. The outcome is unpredictable lines, head‑of‑line stopping, and PFC storms if you make it possible for lossless habits without surgical control.
The enemy is not typical throughput. It is queue depth plus difference. Training stalls when the 99.9 th percentile latency spikes, even if the mean looks fine. That's why on a whiteboard a 3:1 oversubscription sounds economical, yet in practice those oversubscribed spinal columns and buffer‑poor ToRs turn small microbursts into retransmits and FLR drops at the NIC. Mistake budgets vanish not since the network is sluggish in aggregate, however because it is unpredictable at the worst moments.
Rethinking the Physical Layer: Optics, Fiber, and Cleanliness
People love to talk about software, but when a cluster goes sideways at scale the source frequently lives under the raised floor. An outstanding fiber optic cables supplier makes their keep by making really boring problems stay uninteresting. I imply constant loss spending plans, clean ports, and recorded serials mapped to ports so your automation can reason about them.
Start by choosing where you want single‑mode versus multi‑mode. Multi‑mode OM4 can make good sense for short‑reach intra‑row perform at 100G, 200G, and even 400G with SR4/DR4 in particular geographies, however your future self will appreciate the reach and standardization of single‑mode for leaf‑spine and spine‑super‑spine. Blended estates are common-- simply do not let them become unintentional. Traceability matters when you're debugging an intermittent 1 dB charge on a critical link.
Optics choice is more than cost shopping. AI clusters are pressing toward 400G and 800G in the material, while lots of storage and data prep nodes still sit at 25G or 100G. When deploying QSFP‑DD 400G, check FEC compatibility between NICs and switches. Reed‑Solomon KP4 is not optional; it is table stakes. I have actually seen stores lose weeks chasing phantom package loss that was merely a mis‑matched FEC mode in between transceiver and switch ASIC. If you rely on compatible optical transceivers from 3rd parties, qualify them thoroughly. Excellent vendors publish DOM telemetry compatibility matrices and hold spares with matching firmware standards. Poor vendors leave you guessing when you check out absolutely nos throughout TX predisposition on half your links.
Polarity and MPO hygiene deserve unique attention. Training tasks amplify small physical issues: a slightly dirty ferrule can press you over a vulnerable margin when queues rise. Invest in appropriate cleaning tools and impose a no‑touch policy on adapters unless they've been cleaned up and inspected. Label trunks, test end‑to‑end insertion loss before you turn up the material, and keep a running stock that maps optics serials to switch and port. When a transceiver starts creeping up in mistake counters, you wish to change the specific system without a scavenger hunt.
Open Network Switches and Why They Matter
Open network switches used to be a hobbyist's option. That altered as ASIC suppliers and NOS ecosystems grew. For AI/ML connection, openness translates into 3 advantages: standardized telemetry, flexible buffer setup, and automation hooks that let you iterate quickly.
On the silicon side, Tomahawk, Spear, Jericho, and NPU‑driven platforms each have strengths. Training materials lean on deep buffers in some spots and shallow, fast buffers in others. I have actually had success pairing deep‑buffer spinal columns with shallow‑buffer TORs when combined with disciplined traffic engineering, but the reverse can work if you strictly control incast at the edge. The key is consistency in how the NOS exposes buffer profiles. With an open NOS, you can specify queue disciplines per class, pin PFC to the few flows that really need it, and leave whatever else to ECN with well‑tuned RED thresholds.
The other benefit is visibility. If you can't see queue occupancy, ECN marks, PFC pause durations, and per‑flow drops at line rate, you are operating on faith. Open platforms now support INT or alternative in‑band telemetry, streaming gNMI, and sFlow or equivalent at scale. Tie that into a time‑series database and you can associate training loss spikes to queue events within seconds. When your SRE on call awakens to a page, their chart needs to reveal which leaf saw consistent ECN marks https://search.google.com/local/reviews?placeid=ChIJY08Y5ojLj4ARlkjHwWCIKj0 and which next-door neighbor link began flapping, not a generic "package loss increased" alert.
The NIC and the Change: Where Ethernet Becomes a Fabric
The love with RDMA and lossless networks is easy to understand. Nobody desires CPUs burning cycles on TCP when GPUs sit idle. But lossless Ethernet is not totally free. PFC can collapse if you are sloppy with buffer planning or oversubscription. I have actually viewed a single misbehaving storage node flood pause frames upstream up until half the rack went dark.
Most training stacks take advantage of a hybrid method. Usage RDMA where the vendor environment is stable and you can constrain it to a well‑understood class, then count on well‑tuned ECN for the majority of east‑west traffic. Raise ECN marking limits enough to prevent early downturn while keeping headroom for incast. Modern NICs expose innovative congestion control; use it. Set per‑Q thresholds and verify with packet captures, not folklore.
Algo choice matters for your fabric overlays as well. If you ride IPv6 with eBGP and EVPN, keep the control plane simple. Dampening and BFD timers need to reflect the reality that a microburst‑prone fabric is better off with a little hysteresis than trigger‑happy merging. The objective is behavioral stability. The perfect theoretical topology does not assist if it flaps under real traffic.
The Topology Concern: Leaf‑Spine, Dragonfly, or Something Exotic?
Most enterprises run leaf‑spine because it's easy to reason about and simple to automate. As GPU densities and task sizes climb, the east‑west pressure presses teams towards denser interconnects. Dragonfly and its variations guarantee fewer hops and better international bandwidth, but they punish sloppy implementation and demand fine-tuned traffic engineering.
What I advise is to understand the task graph initially. If your training jobs hardly ever exceed a single row or pod, an easy two‑tier or three‑tier Clos with predictable oversubscription is fine. Keep the course length short and uniform. As tasks span pods and the number of GPUs per task grows, update the inter‑pod bandwidth first instead of leaping to an exotic geography. Introduce localized super‑spines and think about 400G or 800G uplinks for pod‑to‑pod. Just when your task scheduler can not pack work within pods and the cross‑pod traffic ends up being primary should you consider more complicated materials. Every new topology is a new set of failure modes and functional runbooks.
Cabling at Scale: Pragmatism Beats Perfection
I as soon as viewed an implementation stall for two weeks due to the fact that the group demanded zero‑patch panels in a mixed‑vendor cage where the range tolerances moved daily. We lost more time chasing ideal direct‑run cables than we would have invested confirming high‑quality trunks through panel fields. Perfection is not the goal; stable, functional, and traceable is.
Work carefully with your fiber optic cable televisions provider to standardize on a small set of SKUs. If your warehouse team has to identify 10 visually similar 400G jumpers under pressure, you will mis‑patch. Choose color schemes that align with speed or media type. Keep spare optics at every row for the specific designs in production, including compatible optical transceivers that you've currently burned in. For DACs versus AOCs, there's no single response. DACs are cost‑effective and easy within a rack, but thermal and bend radius restrictions show up at odd times, especially in high‑density chassis. AOCs clean up air flow and reach, at the expense of cost and often greater failure rates if managed roughly.
DOM telemetry is your buddy. Standard receive power, send power, temperature, and predisposition existing right after turn‑up. Trend them weekly, not simply when something breaks. A sneaking RX power drop across numerous links in the exact same tray points to a cable pathway issue, Fiber optic cables supplier not a switch issue.
Buffering, Burst Absorption, and Tail Latency
Ask 2 engineers how much buffer a switch requires, and you will get 3 responses. With AI/ML, the trick is to line up buffer policy with the task's burst patterns. Incast during all‑reduce or checkpointing can create microbursts at a leaf that overwhelm shallow shared buffers. Deep buffers assist, however they can likewise mask blockage and extend lines, which gets worse tail latency for small latency‑sensitive RPCs.
I go for a separated approach: shallow, fast lines for control and small RPCs with aggressive ECN, and larger shared pools for elephant flows, with ECN limits set higher to permit brief absorption. If you should run PFC, fence it to the minimum variety of classes, and test failure circumstances that include one stuck sender. Watch for traditional PFC "deadlock triangles" where 3 devices pause each other into a standstill. Instrument time out counters and alert on sustained pause times. Also, think about enabling vibrant buffer allowance if your ASIC supports it, however only after you've gathered enough telemetry to set sane minima and maxima per queue.
Where Open Network Switches Shine: Automation and Repeatability
Building a material is easy when. Running it through four generations of optics, three NOS upgrades, and a dozen geography changes is the hard part. Open network switches coupled with a declarative NOS let you treat the fabric like code. Version control your intent. Verify modifications in a laboratory with traffic generators that simulate incast and microbursts. Use canary racks to stage changes, then roll out slowly based upon telemetry limits, not dates on a calendar.
I have actually pertained to depend on continuous validation. Every night, synthetic tasks encounter the material, setting off common traffic shapes: small RPC storms, large all‑reduce surges, blended storage reads with background compaction. The CI pipeline checks ECN marks, microburst drops, BFD stability, and DOM anomalies. If any metric deviates beyond a band, the pipeline obstructs setup promotion. That workflow is easier when your switches expose streaming state through gNMI or OpenConfig and you can push incremental changes with idempotent APIs.
Storage Courses and the "Slow Side" of the House
Teams frequently optimize the GPU‑to‑GPU course and neglect the course to storage and function shops. Then the training task spends half its time waiting on checks out that traverse a motley set of 10G links hidden behind a firewall appliance. If your information aircraft for storage is an afterthought, you'll never ever see the ROI from expensive accelerators.
Enterprise networking hardware on the storage path must meet the exact same requirements as your training material: consistent optics, clean ECN and buffer policies, and deterministic routing. Separate the storage network or at least the class of service. Make sure the feature store and object shop endpoints have enough parallelism to match GPU ingestion rates; otherwise, you'll observe integrated downturns that look like network issues however are truly application‑level throttling communicating with buffer behavior.
On the NAS side, confirm NFS or SMB habits under deep lines. Some NFS servers behave improperly when customers encounter routine ECN marks; their blockage reasoning is far from contemporary. Test with the exact kernel and client stack you prepare to run. On the object side, confirm that your GW tier is not hair‑pinning traffic in between pods. A one‑hop detour sounds small until you add it to every read for a thousand GPUs.
Vendor Option Without Lock‑In
No one constructs an entire material from a single part number anymore. You will mix optics, cable televisions, NICs, and switches. The trick is to keep adequate uniformity in each plane that your failure blast radius is bounded. A reliable fiber optic cables supplier is a partner in that technique, not simply a box shipper. Demand documented interoperability with your picked optics and NIC lines. Need pre‑burn‑in and serialized reports. Keep a short list of compatible optical transceivers that you have lab‑proven with your open network switches and your NICs.
On the changing side, weigh the NOS environment and support posture as much as the ASIC. If your team lives in automation, choose platforms that expose consistent, well‑documented designs. If your material needs line‑rate INT or high‑resolution line counters, validate them in a laboratory under tension. Do not accept slide‑ware; make the supplier or integrator show you a live counter increasing under a genuine traffic generator.
Security That Does not Torpedo Performance
AI/ ML clusters are magnets for sensitive data. Security controls frequently get here late and then break performance because they were not tested versus practical traffic. Inline home appliances hardly ever survive AI/ML east‑west volumes, and their added latency damages tail habits. Prefer host‑based enforcement integrated with L3 division and light‑touch ACLs in hardware. If you require encryption on the wire, take a look at NIC‑offloaded IPsec or MACsec in hardware, and confirm the influence on ECN and PFC behavior. Some early MACsec executions connected inadequately with time out frames and caused bizarre congestion artifacts.
Telecom and data‑com connection principles still apply: decrease choke points, keep policy deterministic, and collect logs close to the source. Circulation logs connected to EVPN/VXLAN identifiers help you trace habits throughout overlays without learning NAT or stateful middleboxes that don't comprehend your encapsulation.
Power, Cooling, and the Quiet Horrible Things
High density optics and firmly loaded switches run hot. When an 800G optic throttles due to temperature, you get periodic loss that appears like a routing concern. Work with facilities to keep front‑to‑back air flow unobstructed. Avoid flexing AOCs in front of fan trays. Keep blanking panels set up. Optics with onboard diagnostics can notify you before thermal derate begins. Connect that alerting into the exact same telemetry system as your network counters so you see domino effect in one pane.
Power occasions expose control aircraft fragility. If a PDU drops and half a row restarts, your routing convergence needs to be orderly. Staggered boot timers for TORs, elegant restart for BGP, and dampening tuned for your fabric's size turn a possible storm into a peaceful recovery. Check it. Pull a PDU breaker in the lab and view what happens to your lines and your training jobs.
Operating Model: SRE for the Network
AI/ ML stacks iterate rapidly. Your network needs the same posture. Deal with the fabric as a service with SLOs lined up to the real requirements of training and inference. Mean latency is not an SLO. P99.9 under burst is. Packet loss under microburst is. ECN mark rate during all‑reduce is. Develop control panels that display these particular signals per pod and per class, and after that connect them to canary jobs that run hourly.
When an event happens, have a runbook that starts at the top: what job, what rack, what class, which lines. Examine ECN and PFC counters before you touch routing. Confirm DOM telemetry before you reroute. Most agonizing occurrences wind up being a couple of mis‑seated optics or a new storage workload colliding with a vulnerable line configuration. You will not understand which without the best very first look.
Procurement That Assists, Not Hinders
Procurement cycles can wreck consistency if every turn‑up pulls from a different lot, supplier, or firmware. Develop costs of materials that define not simply speeds and feeds, however precise optics models, compatible ranges, and acceptable substitutes. For business networking hardware, record the NOS version, ASIC generation, buffer profiles in usage, and the laboratory recognition artifacts. If a supplier proposes a brand-new suitable optical transceiver to address accessibility, run it through the exact same burn‑in: heat soak, light loss variation, DOM telemetry baseline, and a 24‑hour microburst test.
An undervalued practice is keeping a "golden rack" in the laboratory that mirrors production carefully. Every procurement modification gets installed there first. Push the same automation that production utilizes. Drive traffic up until you have enough confidence to promote.
Cost, Where It Matters and Where It Does n'thtmlplcehlder 102end. You won't save cash by purchasing the least expensive optics and after that staffing an army to chase flaps. Spend where concealed labor expenses hide: optics dependability, correct fiber trunks, good labeling, and telemetry platforms. You can economize on cosmetic features and even on chassis fanciness. I'll take fixed‑form factor, tiring open network changes with excellent counters and buffer control over a feature‑rich platform with opaque behavior. Calculate TCO in months, not years, for GPU clusters. If a network change can recover five percent training throughput by shaving the tail, that often spends for a round of optics upgrades. Do the mathematics with real task traces; eyeballing averages will not catch it. A Brief Field‑Ready Checklist
- Confirm FEC compatibility and mode across NICs, switches, and optics; lock it in automation and audit weekly. Baseline and trend DOM telemetry for every optic; change units that wander outdoors your normal band before they stop working hard. Configure ECN and PFC with intent; limit PFC to the tiniest necessary class and test deadlock circumstances deliberately. Stream queue, ECN, and pause counters into a TSDB; correlate with synthetic and genuine task timelines. Standardize SKUs with your fiber optic cables supplier, including pre‑burn‑in for compatible optical transceivers you intend to use.
Where We're Headed: 800G, Optical Switching, and Smarter Schedulers
The near term is clear: 400G is mainstreaming throughout leaf‑spine, with 800G uplinks appearing in dense pods. Co‑packaged optics will move out of the slide deck and into production for a narrow set of power‑constrained use cases. Optical circuit switching keeps blinking on the horizon; it makes good sense for large, steady circulations but asks a lot from job schedulers that should position work to exploit it. I anticipate hybrid fabrics where most traffic trips packet‑switched Ethernet while inter‑pod "elephants" get pinned to reconfigurable optical courses during long training windows.
More intriguing is how software application will help the network instead of simply consuming it. Job schedulers currently think about GPU region; they can likewise include fabric telemetry. If all‑reduce jobs might choose pods with low ECN mark rates and consistent tail latency, the scheduler would end up being an ally in keeping the network calm. That needs tidy metrics, stable semantics, and trust in between teams.
Bringing Everything Together
Redesigning data‑com connection for AI/ML workloads is about developing a material that takes in volatility without drama. That suggests respecting the physical layer-- proficient cabling, vetted optics, reputable providers-- and accepting open network switches that let you see and shape lines. It suggests acknowledging that lossless is a tool with sharp edges, that ECN is a friend when tuned, which oversubscription needs to be proved, not presumed. It suggests choosing business networking hardware for its operational clarity instead of its brochure gloss, and constructing a culture where telemetry drives change instead of anecdotes.
The reward is concrete. Training curves stop wobbling. Schedulers place larger jobs with self-confidence. Night pages drop. Many of all, the network fades into the background, which is where it ought to live while your groups focus on models and data. When you can explain your fabric as tiring under tension, you've upgraded it well.