"Anti-Data Centers is a Chinese Psyop" | How Many Will Actually Get Built? | Positron AI Co-founder

I would say I'm overall opposed to the pace of the frontier direction it's going in.

Yeah, great to be here. Thanks, Harry.

Yeah. So Positron is a fabulous semiconductor startup that's building hardware. So really everything from the chips, the software directly running on top of that, all the way up to the full systems and rack-scale deployments to power generative AI inference. So effectively everything in the hardware and sort of the low-level, direct-talking-to-hardware software stack that powers all the applications, you know, everyone in the world's excited about right now. So everything from, you know, the likes of ChatGPT, Claude, etc.

So training, I would say, from the underlying compute level, fundamentally, you know, is a compute-bound problem. So it's a workload that the more flops that you have, and if you look at, you know, be it from the regulatory and some of the, you know, export control frameworks are heavily focused on just flops required. So the how many floating-point operations per second can be done. And more or less, you know, the amazing thing that the scaling laws of the past decade have shown is that the more parameters you add to a network, and the more flops you dedicate to that during training, the better that model is going to become. A big difference with inference and so the deployment of those models is the fact that for the actual math and the steps that you're doing is about half of what you're doing during training in terms of the steps, not shouldn't be thought of as like the actual compute involved. But what it turns out to be is that that forward pass, that inference portion of it, is heavily, heavily memory-bound due to the fact that basically for every single token that's generated, every little bit of output that requires going through the weights, the parameters, you could kind of, you know, from a biological sense, think of the neurons, you have to read the values of that for every single individual token. And so the way that that's, I think, very interesting about this, and I don't know if it says anything about the value or kind of actually saying that the inference is somehow, I don't want to say more important because of course you have to train, but fundamentally when you're training you already have the corpus, you already have all the training data, and so with all that data you can massively parallelize the token inputs, all of the sequences of words and sentences, paragraphs, etc., that are going into that. So that's something you can just crush through with a bunch of compute. But when you're inferring, because that's actually generative, you don't know what the token is, you know, five words down the line. And so you have to generate each and every one autoregressively or in order without, you know, foresight. And so that becomes a hugely memory-bound problem that can't just be massively parallelized like training.

Partially. I mean, the memory wall as a phrase has been around for a long time before, you know, all the hype around AI. And really what it's come down to is if you look at the past 50, 60 years of computing, we've been able to, you know, have Moore's law giving us more transistors per, you know, square millimeter of silicon, you know, consistently. And while that has been able to result in, you know, greater raw compute, flops, etc., the improvement of the memory technology has not kept up at the same rate. So roughly speaking, you know, between, you know, 2014, you know, just very early innings of the new AI era till 2024, you had about a 120x improvement in the flops of GPUs. So a single Nvidia GPU had about a 120-fold improvement in flops. You know, and that's what enabled a whole lot of the improvements over that decade. The improvement in memory bandwidth is only 17x. So I would say like the real embodiment of this is we had massive improvements on a per-device basis of the flops, and then a whole bunch of, you know, elements on the periphery of improving the connectivity, etc., etc. But just like the ratio of the compute to memory bandwidth had this divergence, and so you had cases where if a problem was memory-bound and you couldn't just scale the compute linearly with that, you were getting more and more memory-bound as the decade progressed.

Well, it comes down to a lot of, you know, technical implementation details, like the fact that if you look at the lowest level, the type of memory that is used on the silicon itself is called SRAM, static RAM, and SRAM is made out of, you know, six transistors with a bit line and word line, some other control logic around it. But that SRAM cell has not scaled in terms of the sizing of that with Moore's law over the past about 15 years. So they have grown or shrunk, I should say, very, very much slower than just a group of transistors that you'll use for other purposes. And I would say that there's just been a lot more architectural advancements that could happen on the compute side, while an SRAM more or less has not changed in 30 or 40 years from, like, an architectural, you know, primitive perspective. And so on the input side of, like, raw technical capabilities on fabrication, etc., have not been able to improve. But I would also say that there wasn't the right motivations for most of that decade. So with convolutional neural networks, so things that powered, like, AlexNet, which was, you know, really launched the deep learning revolution in 2012, and then ResNet and, you know, all of the, I would say, the advancements during the 2010s was in the realm of machine learning models that were fundamentally compute-bound. You could just throw more and more flops at CNNs and get better results. And you didn't really need all that much memory capacity or memory bandwidth. But it was really with the Transformer, and even though the 'Attention Is All You Need' paper came out in 2017, I would say it did not really get the attention, you know, pun intended, it deserved until 2020. Well, GPT-1 and GPT-2 came out prior to that, 2018, 2019, but it was really GPT-3 showing that, okay, you go from a billion-ish parameter up to 175 billion parameters and you actually get this massive improvement in capability. And that's really where I would say the Transformer revolution started. And most people didn't really catch on to that until the end of 2022 when ChatGPT came out.

I do find it, like, compared to a year or two ago, there are now different prices listed for cached versus uncached tokens. But I don't think people realize how many providers that charge the same amount, and even with a lot of people's cached prices, how high margin that is. It's, it's like insane. You make all of your money on selling cached input and output tokens. So as a...

Oh, because for, like, when we discussed earlier, that server processing a cached token is essentially free. It's 1/1,000th of the cost. Yeah, I'm, you know, order of magnitude of actually having to generate, to recompute and generate that token. So there is so much you can juice out of selling those cached tokens. Basically all the providers, they charge you to cache a token. They charge a higher rate than just, you know, the normal processing fee for, like, an input token. And then they charge you a lower rate when you read from that. And it's great when you're paying that lower rate, but they're making obscene margin on that cached read. And yeah, there's a reason why, you know, Anthropic is reported to have, you know, 80 points of gross margin right now on their, you know, API business.

I was impressed.

Well, I'm impressed by 80 points of margin in basically any industry. It's difficult to get that margin, and the great thing about capitalism is those margins will compress with competition. So I'm confident and happy for that, even though those people are, you know, theoretically my customers and my margin is sort of based on their margin, but I care more about, you know, a healthy ecosystem long term. I would say that it's more surprising to me how many people still today think that these are horribly unprofitable businesses and that the whole market's going to zero. It's like, it's absurd to me that the meme of these, you know, OpenAI, Anthropic, etc., are just burning cash and eventually they'll run out of cash that they can burn. Like, if they stopped training, they'd be massively profitable overnight. And there's a ton of other levers that they have without, you know, pacing the frontier, as Dario just had in his essay. I mean, I jokingly think that, you know, a little bit of the pacing the frontier discussion is, oh, this is a great way to reduce costs ahead of IPO. But I don't think Anthropic or anyone needs to do that. I think they're amazingly profitable businesses with their scaling rate and would reduce costs just so I'm saying because they would spend less on training because they would be slowing down the speed.

Yeah. So I don't think that's actually the intention or anything, but yes, that's the bulk of their costs.

I have mixed feelings on the safety topic. I am a human, so I, you know, would like to live to old age and more so than that, have humanity continue to the stars and beyond. But I believe much more in the ability for this technology to revolutionize every part of humanity in a positive way. And I do worry that pacing, in a lot of the ways that's being talked about, not necessarily how it will be implemented, has two big risks. One, a major pause, you know, at stopping and sort of that playing into, for lack of a better word, the late sentiment that exists, and that by pushing for pause, it's actually giving ammunition, giving a better basis for those that actually just want to stop the technology completely. And I see that as a major risk for humanity. The second piece is I'm very, very, like, my actual, the thing that I'm most worried about of any, you know, AI outcomes is that technology and capability being concentrated to relatively few people. And having a lot of what's being discussed from a regulatory framework and limitations on technology, etc., I think is like the modern, you know, road to serfdom. It's like the concentration of technological capability and, like, making legal to do matrix multiplications is like the thing that will set us back to, you know, pre, not just industrial revolution, like pre-enlightenment, you know, capabilities like that. That is like the biggest attack on classical, like, liberal freedom concepts that I can think of because while I do think that the vast majority of tokens are going to be produced by the big, you know, players, if the technology itself is restricted to just those, then they're going to be the new, you know, lords and kings and everyone else is back to serfs.

And Elon wants time to catch up as well. And there's...

Yeah, I completely agree, and I think that is the biggest internal reason for probably everyone other than Dario. I like Dario, and I would say the vast majority of people in Anthropic are true believers, both in all the promise and capabilities of the technology and the risks. And I mean, if I were in their shoes, I would also be taking the massive amount of responsibility. And, you know, for that, yeah, there's a lot of strategic reasons of saying, okay, by having these auditors, etc., that removes some potential responsibility, culpability from, like, a legal perspective. The risk that I just don't think that any of, like, I think this is a problem with a lot of very smart people, especially when they've amassed large wealth and power, etc., is they think that they're going to be able to keep that. And the scariest thing, and kind of my point, like the centralization of technology, like if it gets concentrated with companies, governments, etc., is you've got people that think that they're the smartest people in the room, not realizing that they are not going to be the ones to actually control it when they put these measures in place. Like Dario basically on one hand verbally begging for governments to take over Anthropic. He's in part saying that because he doesn't think it will actually happen. And I would love to see his reaction if and when that actually happens and he realizes, oh, I thought that if I, you know, was begging for regulations, they would then make me the regulator in some form or fashion. And when that doesn't happen and it just becomes a bureaucracy that halts all progress and the capabilities that currently exist basically get squandered to select bureaucrats, that's the worst outcome I can imagine.

I guess when I said that the worst possible outcome, I wasn't counting that as, I wasn't counting the Terminator outcome and I wasn't counting that. So I think, of course, everyone can agree Terminator or similar is very bad, but I think it's extremely low probability. And just, I'm not a believer in that sort of doom scenario. For the vast majority of people, would result in the same level of serfdom that I worry about with the scenario that I described. Would be significantly worse for some number of people in a, you know, Chinese, CCP-controlled, you know, super intelligent AI scenario. I think that, you know, on one hand, their strategic angle right now is to have technology proliferate through open source, etc. I think as soon as they get into, you know, pole position, the ladder gets pulled up with them in some way. I don't think they actually want the technology to be, you know, easily accessible to everyone. Now, I don't know if they will decide that it's okay if the rest of the world has some access to the technology, but they definitely will not let the billion people that are not CCP party members, you know, benefit equally from the technology. So...

You can't pace the frontier unless the global AI community paces the frontier, and I don't see Putin signing up.

And yeah, I think this is a little bit the same naivety that I described by these company leaders and in general people in the Western world thinking, oh, we're so great, we're so advanced, so far ahead that we can't get caught up to. I mean, on paper, is the US the greatest military force in the world? Yes. If we had to all of a sudden have a drone incursion the same level of, you know, what's happening in Ukraine, Russia, you know, coming up from Mexico, and if you take Mexico and just say that they developed, you know, very naive drone technology and etc., like on the level of what's happening in Russia, Ukraine, and Iran. How would we respond to that as a country if we had that coming up across our border? Like, it doesn't matter our amazing military might. We built our military to fight the last war. And I think geopolitically our thinking is, oh, we're the big dog still, and that when it comes to AI technology, there's not the acceptance that export controls and all the other elements that theoretically would allow us to pace and have people keep pace behind us just aren't good long-term solutions.

I am very much a strong believer in free trade and free exchange of ideas. The exception to that is, you know, a little bit, I think China has been a free rider of all the benefits of, you know, a liberal free trade order for the rest of the world while they get to keep everything closed off. So I am very, very happy and think that any government, societies, people that want to embrace free exchange of ideas and trade and everything else, we should have a very vibrant economy and ecosystem. But totalitarian regimes should not be able to participate with that, especially in the case where they get all of the benefits of that and get to export themselves, you know, things that make them better able to have that totalitarian, you know, system keep up.

Based on what my overall beliefs right now, as probably evidenced by the conversation so far, I would say I'm overall opposed to, you know, the pace of the frontier direction it's going in. I appreciate anyone that is adding to the discussion that is, I think, being realist about the benefits and risks, but you always have to take that with a grain of salt of what are the motives of anyone that's discussing it. And I would say I probably appreciate Zuck or Dario's comments infinitely more than a random politician, and not just random, the quote-unquote leading politicians that don't actually understand the technology. And I would, the scariest thing to me on the political spectrum and the way all this is being treated is that it's now become almost a unifying issue on left and right about being anti-data centers. And I think that is almost entirely a Chinese psyop.

Well, just on the ugly and all that, I totally am in support of a, we need to have beautification campaigns and really turn them into centerpieces of our society. I think if thousands of years from now, you know, a future history looks and they should see these massive data centers, like the really, really massive impressive ones, should have, you know, be like the great pyramids or like they, we need to dress them up to be the world wonders that they are technologically. The thing from water usage and like the amount of power they consume, etc., like so much of the early information that went out by unsophisticated writers or things that are just patently false, like, you know, a single In-N-Out uses, you know, more water than the largest data centers in the United States. And like it's, you know, all golf courses are orders of magnitude more. Like these are closed-loop, you know, liquid-cooled systems that, you know, you don't even want to use water in a lot of these cases. From the Chinese psyop perspective is they're not, to your point, they're not pacing the frontier. They're adding, you know, gigawatts of, you know, new electricity generation capacity, most of it being dirty. They're building massive new data centers, horribly displacing people. Like it just, it irks me so much that we have the freedom in the Western world to criticize companies, governments, everything and slow down and based on false information. And I love the freedom elements of that. But it is a strategic disadvantage when China can just say, yeah, we're going to just bulldoze all these people's homes and do rolling blackouts wherever in order to serve the greater good of new training capacity.

No, you're right. I think the greatest advantage the US has in that regard is that there's still a lot of land, a lot of places that do not have all the same levels of restrictions. I don't agree with a lot of things of most, you know, administrations of my lifetime. The current administration gets attacked for saying like they're destroying our environments and, you know, destroying national parks, etc. Like the vast, vast majority, 90 plus percent, I don't know the exact numbers of national federal land is just open empty desert in the west that is not part of a national park or anything. And the fact that there are so many restrictions to utilizing, you know, BLM land for building data centers where it's literally hundreds of miles from, you know, a person from any, you know, populated area. My great state of Nevada has plentiful geothermal, solar, all these green energy technologies and we could, you know, build, you know, nuclear and other things in the middle of the desert where it won't impact anyone. And that there's restrictions to that is completely absurd to me. And I will say there has been some political will and push to solve these things, but literally just in the past year, you have, you know, Republican governors and other politicians that at least had part of their platform to be pro-growth and all these things backing away because they see from their own, you know, political base being anti-data center based on completely false premises. And one of the points I want to go back to that you brought up was like that people would have higher electricity costs if we increase, like this is the most basic supply and demand, if we increase generation capacity. And no one's saying we want to be taking energy from what's reserved for people's homes. Like one of the regulatory problems I see is power companies have to have this buffer of energy availability that is baked into the cost and capabilities for everyone. There is absolutely zero cases where a data center could be potentially pulling power from anything that's already been allocated. So that's just an impossibility. And all these data centers that are getting built right now are coming with generation capacity that covers their own use and beyond that. And we're just not allowing them to hook up to the grid where they could actually be lowering the prices for everyone. And then you've got people on the power company side. They're lobbying against new generation capacity because that will actually, you know, market forces, more capacity will decrease prices, which would be good for consumers.

I would say that the capacity that they have planned, they may be in different locations. I mean, you've had some local communities that have successfully, you know, stopped facilities going in there, but then those data centers just move. I don't think a year ago the major data center builders and operators were thinking that the political problems were as bad as they were. And so there is a lot more effort being put into education in those communities now, which I think will, you know, turn the tide a bit. But I mean, it's also just going to mean that those data centers move to local that aren't going to have those problems as well. And like I said, we've got large tracts of land that can support it. So I'm not too worried that it's going to be like an existential threat in capacity buildout. And then of course there's space if Elon's successful.