Google owns the most AI compute, and it built it its way

It’s official: Google is the largest single owner of AI compute, and it’s doing it largely without Nvidia.

According to new analysis from Epoch AI research institute, more than 60% of global AI compute is owned by US hyperscalers, and Google holds about one quarter of it. And while the search giant relies heavily on its own custom tensor processing units (TPUs), many of its peers are still bound to Nvidia.

That early concentration of compute and infrastructure among a mighty few could dictate the pace of AI evolution, analysts note.

“No one doubts the massive capital investments required to be a hyperscaler in the first place,” noted independent tech analyst Carmi Levy. They can provide the economies of scale that “smaller players can only dream of,” he noted.

“But when they are essentially the only game in town, it’s difficult to ignore their ability to influence pricing, terms, and availability on a market that literally has no other choice,” he said.

Biggest capacity holders highly-reliant on Nvidia

Epoch AI evaluates compute capacity in what it calls “H100-equivalent (H100e) units,” defined as a cloud or company with enough TPUs, graphics processing units (GPUs), or other accelerators to match the output of an Nvidia H100 processor.

By this measure, Google holds the equivalent of about 5 million H100 GPUs in compute capacity, roughly 4 million of it in its custom TPU chips. The tech giant only hosts about one-quarter of its compute on Nvidia GPUs.

This is “considerably less” than its competitors, notes Matt Kimball, VP and principal analyst at Moor Insights & Strategy. “It shows how comfortable the company is with relying on its TPUs for AI,” he said, adding that the company is heavily using its version 7 Ironwood TPUs to power Google Cloud.

Microsoft is a distant second in capacity, holding the equivalent of just under 3.5 million H100s in compute capacity. Redmond relies mostly on Nvidia infrastructure, with a small amount of its compute powered by AMD.

Amazon is in third place, with the equivalent of roughly 2.5 million H100s; Meta is in fourth with 2.25 million; and Oracle is in fifth, with just over 1 million H100e. According to Epoch, Meta uses a mix of Nvidia and AMD infrastructure; Amazon is powered roughly equally by AMD and its own AWS Trainium chips; and Oracle relies strongly on Nvidia.

The on-premises share of the pie continues to shrink

In a similar analysis, Synergy Research Group found that hyperscale operators now account for nearly half (48%) of all worldwide data center capacity, and will likely hold more than two-thirds (67%) of the market by 2031.

The firm reports that 60% of hyperscale capacity is now in hyperscaler-built and owned data centers, and enterprise on-premises data centers account for just 32% of total capacity. This is in “stark contrast” to 2018, when 56% of data center capacity was in on-premises facilities.

Today, on-premises data center capacity is receiving “something of a boost” thanks to genAI applications and GPU infrastructure, after a “sustained period of essentially no growth,” according to Synergy. However, it predicts that the on-premises share of the total will continue to drop at least two percentage points per year, hitting 19% by 2031.

“Overall, the world is racing towards a situation where hyperscale operators are responsible for the bulk of global data center capacity,” said John Dinsdale, a chief analyst at Synergy Research Group.

Nvidia, Google on top, but the market is shifting

Clearly, Nvidia remains a dominant element of the AI-forward stack.

The company has “rather brilliantly ridden the wave, and has deservedly been rewarded for delivering processor-level solutions that address the needs of an increasingly compute-hungry AI-powered world.” said Levy.

That said, over-reliance on one chip vendor “puts everyone else at unnecessary risk,” he noted, incentivizing platformers like Google, Meta, Amazon and others to seek their own closer-to-home solutions. Whether that involves developing their own silicon or diversifying their access to it is “almost irrelevant.”

“What matters is that they recognize the advantages of indigenous development and the deployment of compute capacity, and the risks inherent in allowing someone else to set the terms of engagement,” Levy said.

Google, for its part, will continue to be “one of the largest, if not the largest,” consumer of compute resources, said Bill Wong, research fellow at Info-Tech Research Group.

“Its business model drives that global demand, specifically through the widespread use of Google search and Gemini, which it provides for ‘free,’” he pointed out. However, that same level of traction for enterprise customers is unlikely, as both Microsoft Azure and Amazon AWS have stronger footprints in enterprise.

AI infrastructure is also being influenced by the emerging trend toward sovereign AI, where the preferred AI stack is more locally controlled or on-premises, Wong pointed out. Countries like Denmark are looking to migrate both AI and non-AI workloads away from US providers, particularly Microsoft and Google.

But let’s see what inferencing brings

It’s also important to note that these numbers largely reflect infrastructure buildouts targeted at large-scale training, a realm that Nvidia has dominated with its chips and its CUDA parallel computing platform.

But market share will likely shift as inference begins to mature, Kimball predicted. Providers like AMD and Cerebras will begin to gain because they are “equally impressive,” and have different price and performance profiles, he said.

The rankings also don’t account for some custom accelerators, including AWS’ Trainium, Microsoft’s Maia, and Meta’s MTIA. Cloud providers will likely deploy their own silicon “whenever and wherever possible,” because there will be considerable price and performance advantages, Kimball pointed out.

“So yes, Nvidia dominates today, Nvidia will lead tomorrow,” he said. But “let’s see what this looks like as inference establishes a meaningful presence in the market.”

There is no doubt that the Nvidia story can be confusing for enterprises consuming AI in the cloud, Kimball observed. “Everything they read and see tells them to use Nvidia because that is the architecture that has built all of the models they are using,” he said.

But inference is different, he pointed out. The right inference platform has many dependencies: different model types and sizes, inference patterns, portability, memory architectures. And, given that inference will occur across the enterprise (in the data center, at the edge, on devices), IT buyers must consider software stacks and portability.

Ultimately, Kimball noted, enterprise IT needs to look at AI as “a clean sheet project,” rather than being bound by what exists in today’s data centers. “You do not want to be locked into a single stack and/or a single chip,” he advised.