The Data Center Story: Part 2, Portfolio Change
So I finished the Data Center episode by stepchange yesterday. But let me offer a correction from yesterday’s piece first.
I quoted “stepchange” saying, “Many enterprises had PUE numbers around 2, meaning twice as much power went to lights, cooling, and everything else compared to the amount of power that actually ran the IT hardware. Google pushed it down to 1.1, meaning only 10% of the power was not directly used to power the computers.”
The quote was a bit sloppy. As a reminder, PUE is total facility power divided by IT power. A PUE of 2.0 actually means the total power used by the facility is double the power used by the IT hardware. If the computers use 1 MW, the facility uses 2 MW total. The overhead (cooling, lights, etc.) accounts for the remaining 1 MW. Therefore, a PUE of 2.0 means the power used for overhead equals the power used for IT hardware (a 1:1 ratio). Similarly, A PUE of 1.1 means roughly 9% overhead and 91% to IT. Thanks to the couple of readers who pointed it out.
Speaking of PUE, as indicated yesterday, it’s not just Alphabet or Meta anymore, the entire industry has become much more efficient now. Over the last decade, compute power actually increased by 40x and yet power consumption grew by only 2.5x, thanks to all these efficiency gains.
One of the surprising stats is hyperscalers energy consumption doubled in just four years during 2017 to 2021 period even before the AI boom took place.
From the podcast:
One study found that the cloud is 93% more efficient than running your compute power on-premises because you have less wasted assets combined with everything we talked about with PUE and driving down that efficiency.
Well, now all of a sudden you had COVID plus ZIRP, where you're not waiting to get more efficient; you're just trying to build more and more.
We'd also kind of run out of that low-hanging fruit, right? PUE is at 1.1.
if you look at the big four that we talked about before — Amazon, Microsoft, Google, and Meta — between 2017 and 2021, in those four years, they doubled their energy use to 72 terawatt-hours in 2021.
So much greater demand, combined with a loss of the PUE benefits and the cloud efficiency, means we're in a new paradigm of power usage, and the industry is having to reckon with that.
I knew mining crypto was energy intensive, but never appreciated the extent of it. I would not have guessed that by 2022 crypto was half of the energy consumption of the data centers (at the high end, that’s double the energy consumption of hyperscalers):
Let's talk a little bit about crypto because it's not just a hobby, but actually ends up being a competitive buyer in the data center infrastructure build-out.
By 2022, 100 to 150 terawatt hours. So that's almost 50% of what the rest of the data centers are consuming now going to crypto mining. Go back to 2009, Bitcoin launches, and it's pretty much just a dark corners of the internet thing, right? Someone mining on a PC in their bedroom.
Then, finally, folks figure if you buy GPUs off the shelf, you can go faster, and people start even designing custom ASICs that are designed to do nothing but mine Bitcoin and other coins. By the mid-2010s, mines had become significant power loads, and folks had started to build these out wherever they could find cheap electricity.
Of course, the AI data centers (which are only ~10% of total datacenters today) have some distinct characteristics from the traditional data centers which make the power question the key focus:
One of the things driving this power conversation we've been having is, again, not only the size of these data centers, but now the density of power required within these data centers. This is what makes an AI-driven data center so much different from a traditional data center.
Just 20 of these training racks in a pod is one megawatt of power. That's on the order of a small, 800-person town powering electricity. So, if you get 200 of these racks, you get to an 8,000-person neighborhood very quickly. You get to a gigawatt of Seattle-scale power.
As you pack these GPUs that have higher power density, they run hotter, and you can't let them run hotter. So now, how you cool your GPUs and your racks needs to evolve.
We've crossed the threshold of physics where air can cool this amount of heat being emitted at this level of density from the racks. And so now it is a liquid-cooling world.
You don't really want to think about it as one GPU or a rack of GPUs or a building of GPUs. You want, as much as possible, the whole data center to act together. Because the way that a transformer actually trains is that it makes guesses about what should come next, and then it compares that to reality, and then it tunes the weights across it. And you're doing that collectively across the whole model. And to do that, you need all of the computers working on the problem to be able to communicate quickly. Otherwise, the whole thing is training slowly. The faster those exchanges can happen, the better. And so you want the connectivity between chips to be as fast as possible. You want the connectivity between boxes of chips to be as fast as possible. And so it does not work if all of a sudden half of your computers are on the East Coast and half are on the West Coast, because the speed of light across the country is going to slow you down by multiple orders of magnitude than within one campus, within one center. And that's why Meta wants to build a 5-gigawatt campus because that's going to get them the biggest training model possible.
While data centers are expected to contribute much more to overall electricity use in the next 5-10 years, it is only 4% today. But looking at this number on a national level may be misleading as it masks the very real implications in the local context, especially when you take into account aforementioned “You want, as much as possible, the whole data center to act together.” From the podcast:
it's worth pausing there for a second to talk about that 4% electricity use. When I hear that number, it actually seems shockingly small. For all of the discussion of data centers and electricity, it's like, okay, it's 4%. Let's say it goes to 8%. Compared to industry, compared to cooling and heating buildings. It all feels small. So why might this be such an issue of conflict?
It might have to do with the fact that it's misleading to think about it in terms of total national electricity use, because a data center has a localized and concentrated impact, right? You're not spreading this load across multiple utilities. And not only is it localized and concentrated, it's localized and concentrated in similar areas. Because, as we've been talking about, the network effect of the value of a data center being positioned near the undersea cables and near other network points is where you get a lot of performance gains. And so you end up concentrating yourself in Virginia, in California, in Texas. There's just basically 10 states where you're seeing new data centers come online. And so there's a tremendous impact at a local level—at the electricity prices for that community, that county, and in that state—but not necessarily at a national level.
There are 11,800 data centers in the world, and nearly half of them are in the US, followed by Germany, UK, China, and Canada. But who owns all these data centers?
…you can kind of think of it in four broad categories. That's representative of the US but also it fits globally.
And so the first category are…the colocation centers, the ones we've talked about: Equinix, Digital Realty, and others. There are about a dozen of these that are building 10 to 100 megawatt blocks in the US and around the world. The second category are the hyperscalers: the Facebooks, Amazons, Microsofts; you can add in Apple, Oracle in there. They account for almost half of global data center capacity, and they're the lion's share of new growth. But importantly, you still have thousands of private server rooms in banks and retail facilities and public institutions that are out there. So while large in count, they're relatively smaller in capacity, but today still make up about 35%. And finally, you've got legacy telcos that are still owning and operating data centers.
Now the interesting thing about this is the trend, right? You're seeing enterprise and public sector data centers trending down, and new build increasingly going to hyperscalers and these purpose-built colocation facilities.
I do disagree with one thing with Stepchange guys though. At one point, Anay Shah (one of the hosts) lamented about a “new digital divide, exacerbating global inequality” due to the disparity of data center locations and loss of “national security” for many countries. I think he got it completely backward. It is certainly very much in the interest of the governments of many weak or non-democratic countries to want to have data centers within their national territory so that they can seize control if it’s necessary, but is that actually in the interest of people living in those countries? I don’t live in Bangladesh anymore, but I can tell you I would rather not want to give any Bangladeshi government such power even if it means my internet connection would be a little slower than Americans. “Inequality” is one of those persistent problems that almost all the solutions basically ignore the unintended consequences and more often than not, these “solutions” are much worse than the problem itself.
However, I do wholeheartedly echoed with them that these datacenters are indeed “modern marvel”. Without them, the podcast or MBI Deep Dives itself would not exist. Of course, we take all the technological miracles for granted and our ability to adapt to new technological paradigms may be close to infinite (although AI may test this hypothesis), but it is quite striking just how “invisible” the datacenters are in our everyday lives even though it has quickly become the core foundation of our lives. From the podcast:
…I think about data centers that started off occupying closets, growing to whole floors, then buildings, then warehouses. And I start to think of the cloud as a building, as a factory, where a bit comes in, gets massaged, gets put together in the right way, and then it gets sent out to its destination, and it really brings the internet home.
For me, this connects to one of my reflections, which is threaded through the invisibility of this infrastructure and what makes data centers and the connecting internet almost the perfect abstracted infrastructure. And what I mean by that is you can access them, you can use them, you can get all of the power of them without ever seeing them or touching them.
Now, with us living in a mostly wirelessly blanketed internet ourselves, you have access to all of these warehouses and all these buildings via all these undersea cables. And if you're an engineer developing software, you can deploy to all these regions around the world and never actually look at a CPU or a hard drive in your life.
It is, I think, the best-abstracted physical infrastructure that humanity has built. It is physical, it is silicon, it is electricity, it is fiber. There is no real magic. It is all physics.
But the only infrastructure I can think of in humanity that is as well abstracted is maybe money, but money is actually not physical anymore. In the same way, this actually is still doing physical work. And that struck me.
Invisible, and so frequently in our life. Even though it is invisible, our entire day is mediated by this infrastructure…if your entire day is spent on the train, you're very aware of where the tracks are, and you can see them, and you're on the train.
Even fire, we probably used it selectively throughout the day. The railroads, we got on and off. The car, we got in and out of. Electricity, we turned the lights on and off. This is a modern marvel that we're only not using when we sleep. And even then, it's doing stuff for us…It's tracking my sleep. I'm wearing a ring that's tracking my sleep, sending bits to Oura's servers.
You're living in this infrastructure, but we never see it.
In addition to "Daily Dose" (yes, DAILY) like this, MBI Deep Dives publishes one Deep Dive on a publicly listed company every month. You can find all the 62 Deep Dives here.
Current Portfolio:
Please note that these are NOT my recommendation to buy/sell these securities, but just disclosure from my end so that you can assess potential biases that I may have because of my own personal portfolio holdings. Always consider my write-up my personal investing journal and never forget my objectives, risk tolerance, and constraints may have no resemblance to yours.
I did make one change to my portfolio yesterday which I will mention behind the paywall.