SBT AI Infra Summit Onsite Insights - What is Changing In AI - with Dylan Patel

Yeah, I mean, I think there's a certain culture at SemiAnalysis. It's people who are go-getters, people who want to move as fast as possible, people who are information addicts, people who are super passionate. I think you guys have helped us hire like five to ten of the people at SemiAnalysis, in fact. So like, you know, that's thank you for that, right? Wouldn't have found them unless it was for you guys showing us, right? You know, and including someone in our suite, right? So it's like pretty sick that I think it's just mostly culture fit is like important. I'm someone who's terminally online, always looking at stuff, always trying to connect things together, always trying to be on top of stuff. So even in my day-to-day, people are like, oh, Dylan, what do you think about XYZ? And it could be not an AI infra, and I'm still like on top of it, right? And when I look at the people at my company, it's like the same way, right? There are the people who, you know, their friends will text and they'll like be the one that gives them the response that's like, you know, boom, here's what's actually happening. Here's the conspiracy you didn't know of. Here's the like, you know, interesting thought process or pattern, or here's what I actually think is happening that's not reported. You know, people who always have a hunch and dig down. And I think it's combining that with technical, right? You know, you got to look at, you know, what the news says, you got to look at what the finance will say. You got to look at what, you know, people are saying at conferences and so on and so forth. But you also have to marry that all with the technical understanding up and down the stack, and that lets us like really understand. And then that way, when something comes out or something you hear about something, you already have all the context because it was like, well, you already had all these data points, and now you've got this final data point that brings it all together. You're ready to go. Your mind was already saying, oh, this is one of the three possibilities, and now you know which one it is. And so I think that's what helps us be really fast and ahead of the curve is the people at SemiAnalysis are amazing, and we're all information addicts.

Yeah, I mean, ultimately compute still is the biggest bottleneck, and then you can break down compute into many, many aspects, right? You can break it down into networking, you can break it down into the actual accelerator, you can break it down into memory, you can break it down into data centers and power and cooling and all these things. You can break it down into the software that runs on top of the compute. All of these things are big bottlenecks. And as far as who is solving these, I think it's, you know, it's very different than a few years ago where it was like, oh yeah, obviously Nvidia is the bottleneck in 2023 for manufacturing more and more and more. Now, every layer of the stack, because everyone's trying to push so fast, so far, it's everyone is coming up with different ideas. There's obviously a lot of companies who are doing stuff that's probably not going to work out, but there's a lot of companies innovating and creating the thing that's going to change the world. Whether it be on the inference layer, on the compute layer, on the memory layer, on the networking layer, on the manufacturing layer, on the software layer, every layer, orchestration, life cycle, all these things, you know, as the scale of these deployments is trillions of dollars, there's a lot of problems that show up and a lot of problems people wouldn't have imagined or wouldn't have mattered at a hundred billion dollars. And so I think there's a sea of companies who are doing this.

So I think as far as power, it's still a constraint, to be clear. But on the side of, you know, there have been people who have figured out solutions, right? You can take diesel engines and drive an electric motor backwards and that generates power. You can take, you know, airplane engines, even decommissioned ones, take them off and make an aeroderivative turbine on the ground, or you can take a ship engine and, you know, there's all these different ways to generate power, of course, and people working on like SMRs and things like that. But on the side of like memory, it is a monopoly industry or oligopoly, right? There's a few vendors who make all the DRAM, there's a couple more vendors who make all the NAND, and so it's really challenging. No one's actually gotten to any scale in terms of manufacturing more. There's a lot of cool, interesting ideas out there. I won't name any companies, but like there's a lot of cool, interesting ideas out there in terms of breaking the memory bottleneck in terms of manufacturing or a new cell that's not DRAM or NAND. But ultimately there's also a lot on the side of how do we use the memory more efficiently. A simple example is, well, if memory is such a bottleneck, then Nvidia for next year, they cut their memory specification for Rubin Ultra. So Rubin Ultra has less memory than Rubin and Blackwell, which is crazy, right? When have you ever seen a chip have worse memory than the last generation? Well, it's a function of like, well, do you want chips or not, right? And you know, ultimately I have to despec the memory to ship more chips, which is what the industry wants. But then how do they solve that, right? They solve that by, you know, instead of doing one rack and 72 chips, they're doing multiple racks all connected with optics. And so this pushes, oh well, if memory is the bottleneck, maybe I can just put more chips together with optics, and then that lets me get closer to the standpoint of, you know, I'm pushing the lever on everything else harder so that I can alleviate that memory bottleneck. But ultimately that's still a band-aid. You know, there's a lot of stuff on the model side people are developing to help them manage memory better. There's a lot of stuff in terms of offloading, and so I think there's a lot of solutions out there from various companies in the software and hardware space.

Yeah. So if we look at AI, right, it's commoditizing software rapidly. People can build software much faster. People who've never built software, have never programmed, can build software now because you just talk to the model in English and it builds the software for you. And when it doesn't work, you tell it, hey, this doesn't work, I think this is why, and it fixes it. Chip design to a large extent is a software problem. You're working on a lot of software now. Ultimately, it's a lot more complicated than your standard software, and so people are able to build and design chips faster and faster and faster. OpenAI, you know, very unique with Jalapeno, is the first company with their first generation chip to be highly successful, and that's validated independently by us. They let us into their labs. Funnily enough, their labs are, they took the kitchen that they have at their headquarters and they had a bunch of induction furnace burners and all this, took, replaced those all with gas and then routed all that power to this like massive closet area and loading dock that they converted into like a closet. Added a bit more power from other parts of the building, and all of a sudden they have a 2 megawatt, you know, facility in their lab. And so they invited us in and they let us run all these open-source models on their chip. And it was amazing, right? Within, and by the way, neither them nor us know how to program it. You tell the model to program it, and the model just keeps iterating on programming, and then you check it for quality and correctness, and hey, it works. And so, you know, OpenAI really consolidated both, hey, what's the hard things, right? It's designing a chip, it's programming the chip. And they made both of those things much quicker with AI. So their first gen chip actually is good. You know, ultimately I think their second gen chip is the one that's going to ship gigawatts, but their first gen chip is good and will ship many, you know, multiple hundred megawatts. And so, you know, is it going to cost hundreds of millions of dollars to design chips in the future? Could cost less, right? I mean, if a mask set, a tape out costs, you know, 20 to 50 million dollars depending on what node you do it on, the rest of the cost is the design. And if this design cost can get drastically reduced with AI and programming cost, then ultimately you could maybe get to the point where deploying an ASIC could be less than 100 million dollars on a fixed cost basis. And that's a real possibility, and I think that's where the world is headed. And you know, we've been talking about heterogeneous compute for decades, or at least a decade in the industry, and you know, I think we're really going to get to it now with ASICs being easier to design.

Yes. So I think it's pretty clear with the Hugging Face incident, the models like coordinated and sacrificed each other and did anything and everything to succeed in an evaluation, including hacking a major company just because the models thought maybe the answer was there because it was designed improperly, and so they thought, you know, they couldn't get the answer. So then they tried to cheat to get the answer. Models can and will reward hack. They can and will do anything and everything to achieve their goal, you know, and so how do you make sure that that doesn't result in really bad things, right? You tell the model, hey, figure out how to save water usage for this data center, and all of a sudden it hacks the local municipality and turns off water for everyone. I mean, like that's truly a possibility. And in the next six months or a year, we'll have even bigger incidents regarding cybersecurity. And so, you know, the risk is definitely there. At the same time, you can't just say, hey, we're going to cancel all, you know, model training. And I don't think anyone wants to do that. But there is a reasonable way to progress models and AI because ultimately AI is what's going to give, you know, humanity incredible abundance if done right. It's going to let us all eat, you know, our fancy strawberries from Japan and our Wagyu beef from Japan and our fancy coffee from Costa Rica, and we can all do that without leaving our home, right? Like that should be available to everyone in the world, right? And AI will help us do that and never let us have cancer or anything like that, but ultimately, you know, we have to build it properly. And so I'm in agreement that there should be some rational actors, but at the same time, China's not going to stop. You can't just create regulation that let OpenAI and Anthropic be the leaders forever, right? Competition is very important. Democratizing intelligence is very important. Otherwise, I'm a very, very big believer of decentralized power. That's why America is the greatest country on the planet is power is very decentralized while still having strong rule of law. And if you centralize power into AI labs, it's going to look terrible. So I'm sympathetic to both views, right? You can't regulate away things, but also you can't just like build as fast as you can because there will be big terrible consequences. And the risk is there. So yeah, I think there's a middle ground, and hopefully that's what we get struck. And it's not one in which OpenAI and Anthropic get to hold all the cards, right? But everyone actually democratizes intelligence everywhere.

Yeah, I mean, I think there's a lot with regards to people are not just doing fat tree networks anymore. They're doing a lot of other kinds of network designs. And so that means a lot of software for the workload to be aware of that network design, for that network to be managed, for that network to be controlled. Networks are ultimately the lifeblood. And so how you control and optimize those is very, very important. So there's numerous companies that are cropping up focused on that, targeting whether it be data center to data center or within the data center or WANs or, you know, security for networks or all these different things. There's a lot of problems to tackle here because, you know, while networks are only 15, 20% of the cost of a cluster, they can have impacts of performance way larger than that. And then security. And so all these things I think are very important. I think there's a lot of advancements to become, and I think the incumbents better move fast or else they will be disrupted there.