Arista targets AI data centers with new liquid cooled pluggable optic module

Arista Networks this week announced that it has developed a 12.8 Tbps liquid cooled optics module that it says will help address the power and performance needed for AI data center network development. The module, called the eXtra-dense Pluggable Optics (XPO) offers 12.8Tbps of bandwidth using 64 electrical lanes and includes an integrated liquid-cooled cold plate capable of supporting 400W+ module power consumption, Arista stated. The networking vendor said it has also assembled some 45 leading optics module suppliers under the auspices of a multi-source agreement (MSA) to build and support XPO—though it only identified three of those vendors—Lightmatter, Eoptolink Technology and TeraHop. Arista says XPO creates a new category and option for customers of pluggable optics beyond ZR/ZR+/800ZR optics but particularly targets Octal Small Form-Factor Pluggable (OSFP) modules found in many data center optical networks from hyperscalers to enterprises. “More than 100 million OSFP are projected to ship this year, making OSFP the most important optics module form factor of all time,” co-wrote Andreas Bechtolsheim, co-founder and chief architect and Vijay Vusirikala, Cloud and AI Networking leader both with Arista in a blog about XPO. “OSFP will continue to thrive as the highest volume optics module form factor for the foreseeable future. That said, the relentless increase in bandwidth demands of large AI data centers are exceeding the OSFP design envelope in terms of bandwidth density, cooling capacity, and reliability.” XPO offers 4X the front-panel density of OSFP, integrated liquid cooling that supports any kind of optics, and a large reduction in failure and a single XPO module replaces the work of eight OSFP modules, Bechtolsheim and Vusirikala stated. “In short, XPO allows customers to build large AI data centers with one quarter the switch racks. This is hugely important for both scale-up and scale-out applications, where without XPO the number of traditional switch racks would exceed the number of GPU racks,” Bechtolsheim and Vusirikala stated. To prove their point, the authors imagined a 400 MW AI datacenter with 1024 GPU racks of 128 GPUs each for a total of 128,000 GPUs. “Assume 12.8T scale-up and 1.6T scale-out bandwidth per GPU. With OSFP switch racks that have a density of 1.6 Pbps per rack, this would require more than 1,400 switch racks for scale-up and scale-out fabrics. With XPO, this would require 75% fewer racks, saving over 1,050 racks or 44 % of the floor space,” Bechtolsheim and Vusirikala stated in the blog. “Eliminating 75% of switch racks translates to massive reductions in construction and infrastructure costs, including power distribution, plumbing and installation costs, while accelerating deployment timelines,” Bechtolsheim and Vusirikala stated. Arista said the water-cooling capability of XPO is also an important feature. “All large AI data centers will be liquid cooled and the switches that go into these data centers also need to be liquid cooled,” Bechtolsheim and Vusirikala stated. “While one can add liquid cooled cold plates on flat-top OSFP modules, this does not substantially improve thermal performance.” XPO solves this problem by integrating a liquid cold plate inside the module, with two 32-channel paddle cards sharing the common cold plate which can cool both low power as well as high-power optics such as 8x1600G-ZR/ZR+ with up to 400W of power, Bechtolsheim and Vusirikala stated. XPO modules are much simpler than OSPF modules which improves reliability as well. “Each 32-channel paddle card has only one microcontroller and one set of voltage converters, a 75% reduction in common components versus 4 OSFPs,” Bechtolsheim and Vusirikala wrote. “In large-scale AI fabrics comprising tens of thousands of optical links, component failures become statistically inevitable. However, the hard and soft failure rates of today’s optical modules remain higher than desirable for the operational reliability required in such environments,” Arista wrote in a white paper on XPO. “A single failure can disrupt or halt a multi-million-dollar training job, resulting in wasted compute cycles and significant financial impact. Diagnosing and replacing a failed module within a fabric containing 50,000+ optical links presents a major operational challenge, often triggering cascading effects on job scheduling and leading to severe resource fragmentation across the network,” Arista wrote. Power consumption is a critical constraint in modern data center design, Arista stated. ‘High-density racks operate within a finite power budget, and every watt consumed by the network is a watt unavailable for revenue-generating compute resources,” Arista wrote. “Consequently, optical interconnects must deliver significantly lower power consumption per transmitted bit. High efficiency is essential not only for reducing operational costs but also for maximizing the computational density and overall performance of each rack.” “XPO supports the most power efficient optic designs in two ways,” Bechtolsheim and Vusirikala wrote. “First, it provides a clean electrical channel to the switch chip that supports a low-power linear-interface. Second, it supports the most power efficient photonics technologies, as well as other technologies such as RF-Microwave that are even lower power.” Mary Jander, senior analyst with Futuriom wrote that Arista’s new XPO module marks a fresh and important take on optical transceiver technology. “If it performs in production as planned, it could substantially reduce capex and operating costs for hyperscalers, neocloud providers, and large enterprises. Not to mention advancing the capabilities of Arista’s products,” Jander stated.