If you regulate AI too early, you actually don't solve anything. You still just kind of have the same risk ultimately.
You will the thing into being, but you haven't figured out how to control it.
The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions: sloppy, and you're not complete in what you're telling us happened.
An employee says there's a 10% chance of species extinction. Agent swarms completely flip that. These are just roaming drones, but like times 10,000, and they will easily mistake a good task for a bad one.
So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The US about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation has just moved. This is the signal that...
Guys, welcome back to the podcast.
Didn't think we'd ever do this again. This is — I can't believe it. This is great.
I mean, Martin's just building these like hundred-billion-dollar companies. Too busy for this podcast. So...
At least taking credit for, as VCs do.
Exactly. Um, we have a lot to discuss today, but first, look, Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there and just the discourse that's followed?
Oh boy. I think we should start with Martin on this one. He — you were fighting lots of good ground wars. Um, I maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe kind of fracture off. Um, I think we would agree that any AI lab right now at the frontier should be building, you know, in the safest way possible, with, you know, the highest degree of governance and security, and, you know, kind of whatever your definition of alignment is. Like, this is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. Like, you're not going to have AI diffusion without extremely high-quality products that can be, you know, trusted by enterprises and that aren't kind of constantly hacking systems. So when, at least I read the Dario post, um, I actually didn't disagree with almost anything. Um, because it was all about how do you have better, you know, security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of the testers and like, do you agree with who those are and does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is does this get sort of used or leveraged to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of, you know, not make it easy to compete, you know, with the frontier labs. Um, or do politicians kind of end up sort of taking the message and run with it and maybe even worse outcomes happen, like, you know, it's used to ban data centers, you know, far faster and whatnot. And so I think the actual like substance of the topic is actually incredibly important and I think very important for AI advancement in general. And then the question is what do you do about it, especially what do you do about it from a regulatory standpoint. And that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on like, let's make sure that we don't use this for regulatory kind of capture. Um, which I also agree with. But I think the ideas in the pacing conversation are important. Um, you know, again, like it's a little bit of a funny concept because maybe it's not even pacing as much as just like good hygiene and good engineering. And so with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would, you know, trust using it. So the post is very reasonable, but the atmospherics are not right. Like, so an employee is like, this is going to kill whatever, 10% chance of species extinction, and you know what Dario says...
I agree with him more than I disagree with him, right? Like...
On TV, on TV, the same day that he landed these things. And so in some way you can't have these conversations in isolation, which is of course if he's going to agree in species extinction...
This post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics. And then I have this quibble, but it really bothers me because I'm impeded, which is...
I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.
Right. So like you can very slowly build a nuclear weapon and that doesn't make anybody feel better that it's slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually, you know, addressing it. So, you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy.
Because the pause people are like, well, that's not a pause. That's just pacing. And then the regulators are, you're still doing the thing, right?
And so I just feel like they're trying to — my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction, which are doomers, pause, and then the regulators on the other side. And the problem is they're making both of them unhappy. And I think what they have to do is they need to come out, they need to address the x-risk question directly. Like they need to say, no, we don't think this stuff we're going to do is going to cause extinction. And then I think this becomes this very sensible...
And what would you do, just to play the other side for one second, what do you do? Do you make room for the one possible Venn diagram, which is the lab researcher that is both simultaneously super scared but also works on advancing the state of AI because they believe that it's so important to get right that they want to pursue that? And then obviously the language is right now very, you know, problematic, but that person does exist and that is a real kind of person in our industry, which is like, we have to be at the forefront of AI. I'm also very scared of it, and so that's why I'm working on this.
So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have access.
You were the first pacer.
We were part of the first pacers. So if a constituency within the labs that are the most knowledgeable people believe the stuff has existential risk...
The answer is to nationalize it and actually put controls that we know that work. Right now, if they don't actually believe that, and in the private conversations I have, the most sensible people don't. It's a small fraction that do. This is an HR problem. Right? So to me, an HR problem is a company problem. Like if they are worried that they can't recruit people, right, or they can't retain people, which it seems to me a lot of this is just that concern, it's almost more of this kind of researcher currency if that's the case. I think that is the wrong reason to cause a national-level lockdown on a very promising technology. So listen, I think the post again, I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on x-risk with the proposal that was put out. You just can't reconcile those two things, and that has always been my primary...
Right. Right. Okay. Unleash. Okay.
Holding it back. So, okay. The first thing is you can't have — is there a schedule that they've published that says when all this stuff, whatever bad stuff is going to happen, is? They haven't. So you can't pace it because nobody knows when it was supposed to finish in the first place. It also just seems disingenuous. It's completely...
This is like when the press reports on Apple's latest iPhone is late.
The phone from what nobody knows exists that they haven't told anybody about.
Yeah. My Apple car was very late.
Yeah. Like I don't understand. Like in order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense, and you can't escape that. And then...
By the way, this is another problem that, like again from the — my little quibble on PR is nobody believes it anyway, right? And so if that's the thing, the pacing, right?
Which part going to pace?
That they're going to pace. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your...
I see. I see. Yeah. Well, the internal issue obviously is that the models that everybody has internally far exceed what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point, pacing versus what, right? Like, we don't know if the one that scares everybody also doesn't, you know, work for a legal brief. Like, it might actually screw all that stuff up because it's so — who knows, right? So there's that. And that's just — it's just disingenuous to claim that you're pacing. Second, why do they have to announce all of this and ask the government to tell them to pace? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop.
Like, I like this is the — I worked at a missile factory in college and we had nuclear missiles and I walked the floor.
What's with you guys in missiles? I don't — I'm just here with software.
You were one of two people in college when we were — which was either you were protesting to keep them off campus or building them. So that was your choice. And like we had like nuns from the Catholic Church show up and pour blood all over our missiles, you know, and I'm busy just wheeling PCs around on carts saying, here's the secure PC, and I'm like scared to die. I have no idea what's going on. And I'm like, it's a nuclear missile. And then you find out that that's what stopped the Cold War. Like that literally — it was a Persian missile and that was what did it. And so, but you said something I think is super interesting, which is pacing is this sort of fuzzy non-word between like going and not going. Yeah. And the problem is you're exactly right, like there's no one is going to be happy with the middle road. And so one of the things that people — I think it's almost fun for me to watch as a sport — they think that all these people saying to the government, do this, don't do this, that they think they're going to get what they want. And it's a complete 100% misunderstanding of how government works, which is when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone. And no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that all the inputs can make it into the government, the output never makes everyone happy, right? 100% of the time it either doesn't go far enough or it goes way, way too far. Yeah. And so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity. Well, my favorite thing is like, what was it? David Sacks was like...
I think it was David Sacks, but someone from the government said, 'Uh, you're asking us to regulate you.' No. Facebook. The answer was no.
Well, but the thing that...
Probably just Trump, I think, but...
The thing that they know now that you just know from experience is once the wheels start on regulating, you can't slow that one down. And now it's become an election issue for every party in every jurisdiction up and down the whole government stack. So there is now this whole basket of regulatory approaches.
Well, the next election will 100% be a referendum on AI. So like it has to happen that 2028 is like the AI election and you could basically run on — the problem is it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just basically varying degrees of how much do you regulate it or at least try and avoid the topic. Yeah. But it is too bad that we as a country are in a spot where like the pro-AI case is just like it sounds too — it's like it just takes too many words, you know, it's way too nuanced.
It's defensive and we own none of the vocabulary, right? So the whole debate is pause. It's swarms. It's rogue. Every word has been chosen by the people who don't want to do AI.
Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that...
Yeah. So we got to be like union jobs and, you know, cancer and like, you know, there needs to be another word cloud that emerges. What I don't understand is why the labs have not taken a position on x-risk. Like short of that, I don't think this goes in any direction other than heavy-handed regulation.
Yeah. Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with it. Like, how can a government not do — have you — he had a regulation?
Well, but what — but what would anybody really knowing this ecosystem though, I don't know that you would be able to pin anybody down on that other than something?
No, I would say Dario said it on...
No, but I'm saying you're not going to pin anybody down on a lower. Well, in fact, he does the worst thing about it, which is he agrees with people who claim that they believe that there's an x percentage of extinction happening, but he specifically goes out of his way to say, 'I'm not going to put a percentage on it.'
Which I just think is the weirdest...
No, but that — to be fair, to be fair...
Okay. No, nobody could be fair. To be fair, that's like — it's not 100% like disingenuous or whatever. Like he probably doesn't specifically agree that it's 10%. Or maybe he does and it's just too scary if he were to say...
Let's just play binary search. I mean, is it more or is it less?
But the whole thing is a made-up concept that we just created. But like nobody can — you can't quantify any of this.
Well, but that's sort of Martin's point.
Well, but you just had an official position though. So the only official position would be like — I think the only official position you could possibly get would be the PR version of this. That would be the only thing that would be intellectually honest would be: there are real risks with AI. There's incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.
I don't think that's true. Listen, so what do you think that...
We've been through multiple epochs of technology. We've been through compute. We've been through the internet. We've been through the web. We've been through social networking. We had a discussion about the risks without talking about x-risk. Right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is.
But what if they do think it is higher?
Okay. Okay. Yeah. Like I think the answer is they do think it's more than just the internet.
One of two options. You believe that we're going to go extinct and we shut it all down. Like shut it down or they believe that this is just a chance that we got...
Dario thinks less, but these Cold Warriors — no one like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war. Great movie, War Games, about the whole thing and all of that. But the thing was it was nonzero.
Yeah. And so once you said it was...
Can they say — can they say non-zero then? Is that allowed?
I think as soon as you think it's non-zero...
The problem is if you say non-zero, that could be like, you know, the guy thinks 93%. But let's wait. So just so we're all having a clean conversation, let's talk about marginal risk here. We're not talking about absolute risk, right? Okay. It's just that I think if once you say it's non-zero, the only answer of like catastrophic — the only answer is you have to nationalize it. Yes. And so what he's trying to — what the labs in general that have that view are trying to thread is they want it to be non-zero as a license to do a certain set of things without the burden of just becoming a national...
Well, do you have — I actually don't know all of the precedents, you hopefully do, but like there must be some things that are non-zero, let's say x-risk, x-risks that are not particularly nationalized, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on like — I'm sure like to develop anthrax you have to go to a particular kind of lab.
Well, BSL4 labs are — but they're nationalized. They are national. Okay. But...
Ethics tends to be nationalized. Yes.
Normal human safety, not so much. Right. Like industry oversight that over time becomes federal regulation.
Right. And the progression, which I think is just super important to this discussion, has been since the post-World War II era, most industries that are critical to the infrastructure — power and banking and healthcare — the trend has been to basically be nationalized. Yeah. By just like what you — like your examples of KYC and all the other stuff. The banks are for all practical purposes nationalized and the financial crisis.
Okay. But you said all practical purposes, right? They're literally not nationalized, right? So maybe that — but this might be the intent of the labs is to look like JPMorgan or look like Verizon. And it's just like we're critical infrastructure. We get heavily regulated. It's not good for open source or at least frontier open source, but it is like a plausible outcome for this industry. Good for innovation is the problem.
I think even that's fine, just don't use species extinction as your — literally is like the difference between the things that are like stuck in the lab. And let me just say something. I actually think the labs are moving in the right direction. I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders, like they understand that they have like this tension and they're going to reconcile that. So I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think that grew so fast and just trying to figure out how this machinery works. And I think Sinofsky really hit the nail on the head.
The political process is its own thing. And I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.
Well, I think the tech industry has literally over 100 years consistently relearned the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured it out. Both got sued for antitrust, both got substantially and structurally changed as a result, and they had hundreds of lawyers in the 1960s navigating them. And that, you know, Microsoft came along like we were just like, what? Yeah. Like we had no idea what was happening to us. And there's, you know, there's Bill Gates playing golf with Bill Clinton and then we get slapped with an antitrust lawsuit from his administration and Bill was like, I was playing golf, here's the picture, and that doesn't help. And like, isn't that what I was supposed to do was go and play golf? That's a shortened version of the whole thing. But I think — and I think it's just — it's — I always use this example, which is Hollywood got together during the Red Scare and all the censorship when they were worried about the government censoring movies for sexual content, for adult themes, and they all got together and of course they were never going to be able to win in court if they tried to, but they were threatening to do it and they got together and they formed the Motion Picture Association.
Yeah. And movie ratings and all that, and they police themselves.
So how do you — do you guys like that? Do you like the FINRA proposal?
No, because FINRA is — well, that's as close to NPA as you can get with more.
No, it's not, because FINRA becomes legislation which comes with direct oversight.
Yeah, I think MPAA might not have as much consequence in like how society functions, the First Amendment.
Um...
Did I win that? No, I — first of all, fantastic. But I still don't know if you want it. Like I agree speech is really important, but like — I just think...
There was no societal risk.
I just think what's in our movies will be like — we'll survive like on like a different continuum of...
All these — every history is always relative and at the time being a communist was a really bad thing and 30% of Hollywood got fired for, you know, it was all this stuff. Yeah. So it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings and that's because like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV and so we got to grow up with HBO and all this other stuff.
But the problem with FINRA is that is a perfect example of essentially nationalizing risk.
Yes. Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated.
Okay. Wait, sorry. Do you think we're going to end up with a situation that is better than FINRA? FINRA appears to be the best case scenario, but it is the best case scenario...
But not even anymore. Like I do think this technology is in the limit again so powerful relative to what it — I mean for what it can deliver that it would be impossible for eventually Congress not caring about that. Like it's just like not possible. Question: if it's eventually — at what point...
If it's eventually making every recommendation in your healthcare process and it's inside of your medical device as an open-weights model and it's on every trading system for high-frequency trading and it's on an airplane — like there's no chance that the government doesn't say we need something where FINRA actually is like probably the best case scenario of what that looks like. Right.
Yeah. And so now that's absolutely the most crucial point because if you — all the people in the Senate now were in the Senate when the Communications Act was passed and liability was not passed through to ISPs and to social networks and they sat around — the debate at the time was they would literally say to us, the internet is so big, how did we not have anything to do with it? And that's why Al Gore gets a bunch of abuse for saying he created it, right? Because he was actually trying to get out in front of that and say, no, the government was instrumental in it, and it all backfired because it made it look like he was a crazy person. But it is absolutely the case that they felt like they missed their chance to be on top of the internet.
With Al Gore being on the positive side of innovation.
That was their support. So who is the Al Gore?
There's no one. There's a couple — well, Bessent seems to be that, you know, the defense people sort of want it private but not, which is exactly where they were on the internet, right?
And so it's very interesting that a lot of this is just this like — Elizabeth Warren, Senator Warren's tweets were all like, we missed this for social networks, and it's like...
There's a lot of pent-up energy against tech that could end up just all siphoning into AI right now.
That is exactly what it is. That's what's happening.
Yeah. I think there's another problem, which is we're all trying to predict what's bad, which is not how we've normally done things. Like by this time on the internet, we'd taken out like tens of billions of dollars of like economic — well, we've caused tens of billions of dollars of economic damage. We'd had worms that took out 10% of the infrastructure.
The internet infrastructure which was running critical infrastructure. We had hospitals go down.
We really should have blocked the internet in like '97.
How do we — yeah. Yeah. Okay. So we had — I mean, I remember a time — you like — I remember when you would buy your CD of Windows 95 and by the time it was done installing...
You would have a worm potentially.
No, no, this was the reality — the reality of the PC until 2001 was you could not install a PC connected to the network without getting infected. But viruses were everywhere. And there were two rounds of congressional hearings. You had all of the...
That is actually — okay. Okay. I'll grant you this is a very interesting point. This type of zeitgeist in '94 would have — we would probably not have the internet.
Listen, listen. I mean, the automotive, the airline industry — you always have these periods of kind of like teeth-cutting where like you learn about the dangers and you learn about the technology. And the internet — I mean, we were on the ground floor of this. I mean, things were down all the time. There's economic damage all the time. Like there was like novel — there was new viruses. There were new worms. They're all over the place.
Y2K. Y2K was going to destroy.
But here's the interesting thing about this.
But what I just want to say, like so when we created policy and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do and so you know that the policy actually fits the fact pattern. And in this case, I mean, even when you were talking, you're like, well, what if, you know, yada yada yada. It's very hard to predictive policy.
Security risk. Great. And what is nice about the discourse that is actually starting to evolve around that — like you actually hear people from the lab saying novel cybersecurity risk, this is an engineering problem, we're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date. That's a very...
Perfect, perfect point. Because if you look in 1986, the Computer Crime and Fraud Act got signed. It was out of a very, very specific scenario, which was two groups of hackers broke into GTE Telemet. And...
Sure, well, it wasn't you. It was the other guy.
Oh, he worked at Cisco later. And...
And the problem was that was 1983 and there was no crime.
