Why Everything Breaks at Scale

When I worked on Slingshot networking, my world revolved around one thing: the network.

The problems were certainly challenging, but they were contained within a single domain. If I saw congestion, packet loss, retransmissions, or latency spikes, I had a pretty good idea where to begin looking. My focus was understanding how the network behaved and making it as reliable and performant as possible.

Working at Microsoft has completely changed that perspective.

Instead of focusing on a single subsystem, I now get to see the operational side of some of the largest AI training clusters in the world. What surprised me wasn’t necessarily the size of these systems because we built large systems in HPC too. What surprised me was how interconnected everything is and more importantly, how fragile.

The network is only one piece of the puzzle.

For a training job to run successfully, the GPUs have to communicate efficiently. The NVSwitches have to move data reliably inside each server. The backend Ethernet network has to deliver traffic without introducing instability. Storage has to keep up with the workload. Firmware, drivers, scheduling, power, and cooling where every one of these components has a role to play. None of them operate in isolation.

When you’re training a model across thousands of GPUs, the entire system behaves like one giant computer. If a single component starts misbehaving, it doesn’t just affect that one server. There’s a good chance it affects the entire training job.

That shift in thinking has probably been the biggest lesson for me.

Failure Is Part of the Design

One of the things I found most interesting is that these training applications are written with the expectation that they’re going to fail.

At first, that feels backwards. As software engineers, we’re conditioned to build applications that don’t crash. But distributed AI training operates under a different set of assumptions.

When you’re coordinating hundreds of thousandsof GPUs, something is eventually going to fail. A GPU may experience a hardware fault. A server may disappear. A network path may become unstable. Hardware wears out, firmware has bugs, and unexpected things happen.

The people building these training frameworks understand that. That’s why they constantly write checkpoints throughout training. Those checkpoints exist because everyone assumes that, sooner or later, something somewhere is going to interrupt the job. Instead of pretending failures won’t happen, the software is designed to recover from them as efficiently as possible.

It’s a completely different mindset. Reliability isn’t about creating a system where nothing ever breaks. It’s about building one that can keep making progress even when things inevitably do.

Looking for the Signal(s)

This is the part of my job that I find the most interesting.

My focus today is understanding telemetry and trying to identify signals that tell us a system is beginning to become unhealthy before an entire training job falls apart.

That sounds straightforward until you consider the scale.

Every server is constantly producing telemetry. GPUs generate metrics. NVSwitches expose counters. The backend network has its own statistics. Add in firmware events, driver logs, power consumption, temperatures, hardware counters, storage metrics, and dozens of other data sources, and you quickly end up with hundreds of metrics coming from every server.

Now multiply that by thousands of servers running simultaneously plus a 1 to 5 second interval for data digestion.

The amount of telemetry being collected is staggering. Somewhere inside all of that data are indicators that a failure is beginning to develop. The challenge is figuring out which signals actually matter.

Sometimes it might be a GPU that’s slowly accumulating correctable errors. Other times it could be an NVSwitch counter that’s behaving differently than normal. It might be something happening on the backend network, or it might only become obvious when several independent signals occur together. Sometimes, maybe none of those things actually matter. That’s the difficult part. False Positives are everywhere.

At this scale, it’s incredibly easy to mistake correlation for causation. You can spend days investigating what looks like the perfect indicator, only to discover it was nothing more than noise. Then another pattern appears, and you start the process all over again.

Some days it feels like searching for a needle in a haystack. Other days it feels like trying to figure out whether the needle even exists.

The reality is that nobody has operated AI infrastructure at this scale before. There isn’t a massive dataset containing every possible failure with neatly labeled root causes waiting for an AI model to learn from. Many of the problems we’re seeing are genuinely new. Someone has to determine whether two events are actually related or simply happened at the same time. Someone has to develop a hypothesis, test it against real-world data, eliminate false positives, and slowly build confidence that they’re moving in the right direction.

Why I Find It So Interesting

The reality is that nobody has operated AI infrastructure at this scale before. There isn’t a massive dataset containing every possible failure with neatly labeled root causes waiting for an AI model to learn from. Many of the problems we’re seeing are genuinely new. Someone has to determine whether two events are actually related or simply happened at the same time. Someone has to develop a hypothesis, test it against real-world data, eliminate false positives, and slowly build confidence that they’re moving in the right direction.

This is where my role is currently focused, and honestly, I couldn’t be more excited about it. Opportunities like this don’t come around very often. Getting to study systems operating at this scale, understand how all of these independent components interact, and search for meaningful signals before failures occur is a fascinating engineering challenge.

There isn’t one metric that’s going to explain every failure, and there probably never will be. Progress comes from collecting evidence, challenging assumptions, testing ideas, and slowly building a better understanding of how these systems behave under real workloads.

Coming from an HPC networking background, I naturally started by looking at the network because that’s what I knew best. But this experience has taught me that the network is just one voice in a much larger conversation.

To me, that’s what makes hyperscale AI infrastructure so fascinating. It’s not just about building bigger systems. It’s about understanding how thousands of independent components behave as one system and figuring out how to make that system just a little more reliable than it was yesterday.