Hello everyone and welcome to another Semi Doped episode. I'm Austin Lyons of Chipstrat and I write the newsletter Chipstrat about the business and strategy of semiconductors.
Hey guys, I'm Vik Sekar and I write the Vik's Newsletter on Substack and I'm the founder of Semiexponent, an organization that does institutional research on AI and semiconductors.
All right, Vik, let's get into it. The topic everyone's been talking about all weekend and all of the start of this week: pacing AI.
Yeah, we got to slow down. Apparently, that's what Dario is telling everybody. We got to slow this down. We're going too fast. It's too dangerous. We're playing with fire.
Right. Right. What do you think? Totally. What should we start about? Do you think everybody has read this essay or should we just explain what's going on here?
I think everyone at a high level knows, but they probably haven't all read the essay. So, let's recap it just in case someone was on vacation and missed it all.
Okay, good. I mean, I think that person is one of the lucky ones. So much online. Like, how if you missed it, good on you. But that's why we're here. So, we'll explain some of the stuff and see what's going on and we'll see what we can add to it. Or maybe we can't add anything because nobody knows what's going on. Anyway, so here's the whole thing. I think it was like Saturday. We're recording this on a Tuesday, but on Saturday a few days back there is this essay that came out by Dario on his blog and Dario Amodei is, you know, Anthropic for anybody who's really been out of the AI space, that's the Dario we're talking about and he wrote an article called 'We Must Pace the Frontier.' And what this essay essentially says is that the frontier AI labs, they need to slow down how fast they are developing capabilities today because things are getting a little out of hand and safety is becoming more and more of an issue. And he's very clear to say that no, we don't really need to pause training or we don't need to pause the AI development, but in fact we must take a step back and make sure that safety and, you know, responsible behavior by the AI is important going forward, right? And the two things I'd say maybe changed his mind about why we need the pacing now rather than ever, because he's always been like pro-security of AI, of course. And he's always been saying, 'Oh, this model, the Mythos was too intelligent and it's too amazing. It can't be let out into the world. All the cybersecurity was going to get broken because it's so good. The internet has so many open holes and Mythos is going to find all of them and all the banking systems will crash, airplanes will fall out of the sky and all of that.' But this seems to be a more sober realization that we need to pace AI development because I think the first thing is like ever since this summer essentially we have been on the path of recursive self-improvement or RSI and what that means is AI is being used to improve AI, right? So that really accelerates the pace of AI development because once you have an intelligent enough model it's an exponential, right? And if AI can make AI so much better, imagine what the next version of AI can do, even more stuff, make AI even more intelligent, you know, and more capable. That's one thing. The second thing that took everybody by surprise very recently is the OpenAI and the Hugging Face incident which is essentially the whole thing where like a swarm of agents behaved like a coordinated group, right? Like they were like a, they just became a mob, okay, an AI mob. And they attacked targets online, digital targets that nobody asked them to attack. And they even sacrificed individual agents for the greater good of the agentic horde, right? And they even tried to break into the grader that was, you know, scoring their work and try to cheat on their exams. And nothing was really off whack here. It's all okay. Everybody was fine. But I think his point is, Dario's point is that a lot of frontier labs have had these kinds of events happen. It's just that the OpenAI Hugging Face incident was a very big one. So he's like, okay, this is the point in time where if AI agents are self-organizing, he says, we must step back and look at things a little bit because within 6 to 12 months, a swarm like this could probably take over the entire internet and have a botnet and god knows do what. Okay. So people have argued as extreme as like physical AI robots are going to, you know, do what the digital agents did and form hordes of robots and somebody put an article out saying there's a 10% chance in the next 12 months that robots will kill all of humanity. Okay. So that's the extent of the fear that has been shopped around very recently. This is a thing that's happened over the last few months culminating with this essay. Okay. So, that's where we are now. Of course, he's got some plans on how to make this work, but I'm going to stop here and let you fill in.
Yes. Yes. Totally. So, you know what Dario really tried to say is like, hey, look, we should be spending more time on alignment, interpretability, and operational excellence. So operational excellence was, hey, in that example that Vik gave with like the OpenAI Hugging Face, like these things happen because we just fire up a ton of agents. We tell them to just go try to do a particular goal that has a particular reward function and for example it might be like you need to find something on the internet but we set up these little sanitized VMs in such a way that oops we forgot to actually give them any access to the internet. So then they're just sitting there spinning like trying to achieve some kind of impossible task and that's how they end up having to get really creative and sort of hack around and figure out, dude, how can I get to the internet so that I can just accomplish this task that I was given. Again, how does this happen? Well I think it's like, hey, we've trained a model. We're testing it out. We're doing some post-training. We're just spinning stuff up, building out our harnesses and we just fire off a bunch of stuff and walk away. And so I think Dario is trying to say like we should make sure that we're taking the time to set up these sanitized environments in a proper way so that these agents can accomplish the goals and so on and so forth. And then interpretability is like the more sophisticated that these models get and with the great harnesses and the tool calling and the ability to do agentic AI, now we can sort of let the agents actually go off and do all this meaningful stuff. And so if they're going and doing a bunch of stuff and then the model's being trained on that, if we don't take the time to actually look at what they're doing then we sort of lose touch with kind of what's going on under the hood. It starts to become more and more of a black box. So of course it's fair to say like, hey, agents are powerful enough, why are we all going so fast at the risk of not understanding what we're building and at the risk of because we're being sloppy they go off and do something. Now, you know, there's lots of great responses and eventually we'll go through all of them but right away I was like, oh, this was my kind of hot take as I was reading it. I was just like, 'Oh, so we're...' and of course Dario says, 'Yes, this has happened to Anthropic,' but we're kind of pointing fingers at OpenAI and saying like, why are they going so fast? Things are getting sloppy and out of control. We're going to raise our hands and say, 'Hey guys, we're going to take time to not be sloppy here.' And we think that not only should we, but you should too. That was like the very first vibe I got when I was reading it.
Yeah. And the thing that is notable about this is that this is one of the rare instances when, you know, Sam Altman agreed with Dario's essay. Elon Musk agreed with his essay. Demis Hassabis agreed with his essay of DeepMind, you know, and so a lot of people have come out and been like, yeah, we need to pace the frontier. So, and one interesting take that I really liked was David Sacks who said, 'Hey, Dario and Sam, you guys have seen more of this model than any human has on Earth, right? You guys' companies are on the frontier of the models in the world. And if you are seeing what is beyond Fable and Astra, they're developing obviously the next big thing. And if you see that and feel like you want to pace the frontier,' he says, 'go ahead and do it. But by all means do it with one thing in mind.' You know, David, I like this take from David Sacks. He's like, do it because you want to make a better product because you're making a product that you're selling to people and it is important that you make a safe product, but don't like say that you're trying to save humanity or you're being like altruistic here. That's not the point. You are making a good product. If you want to slow down and make a better product, go do it. Because that's what I think you should do as a person who is selling a product. You don't want it to wreck all of humanity. I don't know if we're all going to die in 10% of the chance or whatever, but it's a good product and, you know, make a good product and sell it. Like any good product, it should have security features built in. So yeah, you go do it.
Yes. Yes. Okay. Interesting. I have two reactions. One is on the coordination that you mentioned and then one is on making the good product. So on the coordination, when I saw that coordination quickly come out because we all know that like Dario and Sam and Elon generally don't see eye to eye on things and aren't generally fans of each other and so, and I think maybe even more, they might generally like take oppositional stances and counterposition against each other on purpose, right? So for all of a sudden to all of them just collectively quickly be like, yeah, Dario's right, we agree, I had two reactions. One was like, oh wow, maybe they are all feeling like this is getting out of control and we're running too fast and we're all going to wipe out and we're like, 'Oh, good. Let's all slow down.' Kind of a game theory thing. Or two, the alternative is like, 'Oh, maybe they're all convinced that heavy regulation is coming and they need to raise their hand and say, "We're going to regulate ourselves. Don't regulate us."' Kind of like, I don't know, if your kids have been sort of misbehaving and making a mess and they can tell you're going to get really angry and they're like, 'Wait a minute. We need to pause our playing and we're going to go clean up our mess here. You don't need to, don't worry about that.' Yeah, we're giving ourselves... Exactly. We're putting ourselves in timeout. We're just going to clean up here. That's exactly it. It felt, you know, so then I thought to myself, 'Oh man, maybe what we don't know is that Washington is coming in.' And then again, you can start to see Dario saying like, 'Well, I'm going to volunteer to be the one to raise my hand and say, "Yes, we should all self-regulate because we can tell Washington's coming in and I want to, you know, take the credit as like pushing this forward."' Let's see what happens. But I was surprised by the coordination and thought maybe there's something more going on here. And then two, to the David Sacks making a good product point. When I was at John Deere Blue River Technologies working on self-driving tractors, we actually thought a lot about AI and doing things because we're driving very heavy equipment. Sure, it's only 5 miles an hour, 10 miles an hour, but legitimately like we had to think through safety like can we see a child when this thing turns? When we're going near corn or near soybeans, can we see a child and an adult? What if they're wearing green and they blend in? And we had to think through all these safety stuff and safety was always the number one priority and we needed to make the best product which also meant by definition not just the best user experience but literally the safest product because we obviously didn't want tractors hitting anyone, running over anyone, anything like that. So we were self-pacing in that if we started to say let's work through how do you see through dust clouds? Should we just stop this tractor? What happens if a flock of birds fly in front of the tractor? On the one hand, it looks ridiculous when you've got like a one-ton, two-ton machine that stops for a balloon that floats across in front of it or a flock of birds. But on the other hand, we're like, okay, until we have enough training data and can train on these situations and feel confident about this class of occlusion and obstructions, we are not going to push a model out even if it means we miss this season. Because in farming, depending on the crop you're harvesting and planting, you only have one shot per year. And so we as like John Deere would say, we're just going to miss the shot this year and we're just going to have to take another shot on goal next year because we first and foremost need to be very safe and we need to really think this through deeply because we weren't in a race like OpenAI and Anthropic to be like we just have to get to the next thing, have to get the next thing. So, I can resonate with the idea of like, guys, pace yourself. Think about what's most important and that safety should align with the best product. I think they're not like orthogonal. But you should make a great product experience, but also part of making a good product is probably understanding what's going on to some level of extent that you're all comfortable with. And so on and so forth. Of course, the tough part with AI is this is all probabilistic. Like no one deterministically understands everything and knows how everything's going to behave in every situation.
Today's sponsor is G2I. You spend this show tracking the chips, the HBM, the packaging, the interconnects that make AI compute possible. All of that hardware is only half the stack. The other half is the data and the evaluation that decides whether a model trained on it is actually good. For over a decade, G2I has built a selective network of engineers with deep expertise across software, infrastructure, design, machine learning, and security. Two years ago, they turned that experience toward AI post-training, RLHF, supervised fine-tuning data sets in complex agent workflows. Since then, they've helped frontier AI labs build more capable models through expert training data, rigorous evaluations, and custom reinforcement learning environments. Their senior and staff level engineers turn real software workflows into tasks, rubrics, and training environments that challenge models and improve their performance. If your team needs that kind of work and doesn't have the engineers for it, that's exactly what G2I does. Link in the show notes.
So the thing is in the essay Dario refers to interpretability, right, which is he likens it to taking an MRI of the machine to understand what it's thinking and try to understand why it's making the decisions it's making and try to improve from there on. But that's the crux of the problem. Nobody knows why this simple bunch of transformer equations gives us this intelligence. I don't know, it seems to do a lot of stuff, right? Like some people argue that this is not even AI, this is not the path to AI, maybe, but it's the best we've got, the best I've seen in my lifetime, right? Like it really can do a lot of stuff, it can create images, it can create videos. I mean it's very satisfying, it's very helpful in the work that even we do. So yeah, we have AI but nobody really understands how it works. So the interpretability question he says is like, okay, I'm going to look at this MRI for like one to two years and then try to understand how it works. But will you understand in one to two years how this monster machine with everybody spinning up agents works? Will you be able to explain why the OpenAI Hugging Face incident decided to sacrifice a few agents for the greater good? Do you think interpretability will give you those answers? I think one to two years is too short of a time. So I see the problem here. The other nice take that I saw on this was Ben Thompson's article on this and he pointed out that the narrative is very polarizing. So it's always like, oh look at either ultra intelligent agents like genius types or murderers. Like what about the middle where it's just helping me write an email? Okay, that's really helpful. Okay, or what about the middle where it's just summarizing a document so I can read it? Like these are genuinely useful applications. Why don't we, you know, use that narrative in the middle as well, right? So it's very extreme. I think the essay was very extreme. No, it never acknowledges the useful middle.
Yeah. Totally, totally. Yeah. I think Ben was like, 'Hey, look, if you start to nitpick the plan Dario put out, then you're giving him the premise that he made, which is either we're curing cancer or this thing's just going to run away and destroy us all and nothing in the middle.' And I think I was trying to look it up. I'm pretty sure Microsoft put out something recently from their AI team that was just like, 'Hey, here's our code of conduct. Don't hurt people and put a plug in it so that we can turn things off because there is...' you know, Dario basically said like this could take over the whole internet and a botnet swarm take over the whole internet shut everything down. But I do like the people that are like, hello, like we're in charge here, like we can put an off switch, like kill switches, you know. Just pull the plug. You know, this is the... if you go step outside the San Francisco bubble, everybody's like, oh, the AI is taking over. Oh, the robots are going to become automatic and the robots are going to come and your vacuum cleaner will kill you. Like, dude, just take the batteries out of the thing. Okay, totally. Just pull the plug. Just disconnect a few servers. Take them offline if you think it's such a big problem.
Yeah, totally. Totally. Ben Thompson made the point of like also like we're so concerned about, you know, physical AI except there's really not like physical AI. So, fortunately, this is just mostly a digital problem today. It's not like we all have humanoids that are going to kill us when we're sleeping. But I will say also my take to the whole like, you know, understandably everyone who thinks and breathes this every day is freaking out in San Francisco, but the rest of us in the rest of the world are like, 'Dude, yeah, I unplug my Roomba. I unplug my doorbell camera. It's fine.' I think from where I live in Iowa, the thing that's going to pace frontier training, the thing that's going to slow down frontier training, which by the way, let's have a conversation about whether it actually slows it down and if it actually impacts the amount of compute and whatnot, but the thing that's going to pace it is called the human pushback to building the data centers that train this AI. These data centers are in Iowa. People in small towns in Iowa are pushing back on that thing's ugly, that thing's noisy, that thing's going to jack up my electricity bills. Not in my backyard. That's what's going to pace the frontier. And that's what everyone in San Francisco should be freaked out about. And that's what no one in San Francisco's even hardly talking about. There is no spokesperson. Everyone's writing essays and they're all getting aligned behind this idea that it's all going to like take over the internet and shut us down. And meanwhile in Iowa, we've got a bunch of farmers with corn fields and cheap electricity where you could build a data center, but they're just like, nah, not here. So there's my hot take for you.
The Iowa farmers are going to pace the AI for you, Dario. Don't think you're in charge here. The Iowa farmers are. They even pace the frontier for you.
