Halley Uses Predictive Control to Make Optical AI Networks More Resilient

Why optical reliability is becoming a training issue

Large AI training clusters increasingly rely on optical links between racks. Tens of thousands of accelerators may exchange data at the end of every training step, turning a network impairment into a direct threat to training progress. The reliability challenge is not limited to isolated component failures. Lasers that share an enclosure, cooling path, or power source can fail together, creating correlated failures that a fixed spare ratio cannot fully address as the pool grows.

Halley, the system presented in the paper, approaches this problem as a predictive control-plane task. Rather than waiting for a failure and then applying a recovery policy, it estimates the next state of network resources and acts before the failure occurs. When its prediction is uncertain, it defers to the policy already deployed in the network.

Four control mechanisms

• LightReserve sizes a shared laser pool and provides a coverage bound intended to limit the impact of correlated failures.
• FoldFit schedules optical switches. The paper gives it a scheduling guarantee within 11/9 of the optimum.
• ThermaLead controls microring devices using one-step-ahead thermal information.
• TriageGuard protects the modules assessed as being at highest risk first, allocating limited protection or recovery resources where they matter most.

Together, these components cover resource provisioning, switch scheduling, thermal management, and risk-aware triage. The architecture is notable for combining prediction with defensive fallback rather than assuming that a predictor will always be correct. The paper also reports formal verification of Halley’s fallback circuits, an important property for a control layer that can influence network-wide training behavior.

What the simulations show

The paper evaluates effective goodput, defined as the share of ideal throughput retained by training. When exhausting the spare pool causes an entire training window to be lost, the reported goodput rises from 65.2% under the baseline approach to 76.5% with Halley. In a less severe case where only the bandwidth of dark channels is lost, goodput reaches 67.4%.

The authors further translate the result into infrastructure efficiency. Per megawatt, the system can deliver up to 1.03–1.17 times the training throughput. For the same training workload, the required data-center capacity can be reduced by as much as 3.3–14.7%. These are simulation-based results, however, not measurements from a long-running production deployment. They therefore indicate the potential value of predictive control rather than guaranteeing the same gains in every cluster.

Why it matters

Halley’s broader contribution is to frame optical-network reliability as a cross-layer control problem. Hardware redundancy alone may not be sufficient when failures are correlated and when thermal, power, and scheduling constraints interact. A controller that anticipates resource deterioration can use the available margin earlier, while a verified fallback path limits the consequences of wrong or missing predictions.

The remaining deployment questions are substantial: prediction quality, control latency, device interoperability, and behavior under combinations of failures all need validation in real systems. Still, the paper points toward a useful direction for AI infrastructure. As accelerator clusters scale, preserving effective training throughput may depend as much on predictive network control as on adding more compute.

Source: Hugging Face Daily Papers