If you regulate AI too early, you actually don't solve anything. You still just kind of have the same risk ultimately. You will the thing into being, but you haven't figured out how to control it.
The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions: sloppy and you're not complete in what you're telling us happened. An employee is like 10% chance of species extinction. Agent swarms completely flip that. These are just roaming drones but like times 10,000 and they will easily mistake a good task for a bad one. So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The US about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation has just moved. This is the signal that...
Guys, welcome back to the podcast. Happy to be here. Thank you.
I mean, Martine's just building these like hundred billion dollar companies. Too busy for this podcast.
At least taking credit for, as VCs do.
Exactly. We have a lot to discuss today, but first look, Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there and just the discourse that's followed?
Oh boy. I think we should start with Martin on this one. He was fighting lots of good ground wars. Maybe I'll say one thing that we probably all agree with and then we can figure out where we maybe kind of fracture off. I think we would agree that any AI lab right now at the frontier should be building in the safest way possible with the highest degree of governance and security and kind of whatever your definition of alignment is. This is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. You're not going to have AI diffusion without extremely high-quality products that can be trusted by enterprises and that aren't constantly hacking systems. So when I read the Dario post, I actually didn't disagree with almost anything. Because it was all about how do you have better security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of the testers and like do you agree with who those are and does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is does this get sort of used or leveraged to do things that maybe we don't agree with which would be like a slowdown of AI dramatically because of regulatory controls that would sort of not make it easy to compete with the frontier labs. Or do politicians kind of end up taking the message and run with it and maybe even worse outcomes happen like you know it's used to ban data centers far faster and whatnot. So I think the actual like substance of the topic is incredibly important and very important for AI advancement in general. And then the question is what do you do about it, especially what do you do about it from a regulatory standpoint, and that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on like let's make sure that we don't use this for regulatory kind of capture. Which I also agree with, but I think the ideas in the pacing conversation are important. Again, it's a little bit of a funny concept because maybe it's not even pacing as much as just like good hygiene and good engineering. And with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would trust using it. So the post is very reasonable, but the atmospherics are not right.
Like so an employee is like this is going to kill whatever 10% chance of species extinction and you know what Dario says...
I agree with him more than I disagree with him, right?
On TV, on TV the same day that he landed these things. And so in some way you can't have these conversations in isolation, which is of course if he's going to agree in species extinction, this post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics. And then I have this quibble, but it really bothers me because I'm impeded, which is I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.
Right. So like you can very slowly build a nuclear weapon and that doesn't make anybody feel better that it's slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually, you know, addressing it. So, you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy. Because the pause people are like, well, that's not a pause. That's just pacing. And then the regulators are, you're still doing the thing, right? And so, I just feel like they're trying to, my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction which are doomers pause and then the regulators on the other side and the problem is they're making both of them unhappy and I think what they have to do is they need to come out, they need to address the x-risk question directly. Like they need to say no we don't think this stuff we're going to do is going to cause extinction and then I think this becomes this very sensible...
And what would you do, just to play the other side for one second, what do you do? Do you make room for the one possible Venn diagram which is the lab researcher that is both simultaneously super scared but also works on advancing the state of AI because they believe that it's so important to get right that they want to pursue that. And then obviously the language is right now very, you know, problematic, but like that person does exist and that is a real kind of person in our industry, which is like we have to be at the forefront of AI. I'm also very scared of it and that's why I'm working on this.
So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have access.
You were the first pacer.
We were part of the first pacers. So if a constituency within the labs that are the most knowledgeable people believe the stuff has existential risk...
The answer is to nationalize it and actually put controls that we know that work. Right now if they don't actually believe that and in the private conversations I have the most sensible people don't. It's a small fraction that do, this is an HR problem.
Right? So to me an HR problem is a company problem. Like if they are worried that they can't recruit people...
You know, or they can't retain people, which it seems to me a lot of this is just that concern. It's almost more of this kind of researcher currency. If that's the case...
I think that is the wrong reason to cause a national level lockdown on a very promising technology. So listen, I think the post again, I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on x-risk with the proposal that was put out. You just can't reconcile those two things and that has always been my primary.
Right. Right. Okay. Unleash. Okay.
So, okay. The first thing is you can't have, is there a schedule that they've published that says when all this stuff, whatever bad stuff is going to happen is? They haven't. So, you can't pace it because nobody knows when it was supposed to finish in the first place. It also just seems disingenuous. It's completely disingenuous to claim. This is like when the press reports on Apple's latest iPhone is late.
The phone from what nobody knows exists that they haven't told anybody about.
Yeah. My Apple car was very late. Like I don't understand. In order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense and you can't escape that. And then by the way, this is another problem that like again from the like my little quibble on PR is nobody believes it anyway, right? And so if that's the thing, the pacing, right?
Which part? Going to pace?
That they're going to pace. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your...
I see. I see. Yeah. Well, the internal, the issue obviously is that the models that everybody has internally far exceed what anybody else has access to. So, it's really just the pacing of external releases. But again, back to your point, pacing versus what, right? Like, we don't know if the one that scares everybody also doesn't, you know, work for a legal brief. Like, it might actually screw all that stuff up because it's so, who knows, right? So, there's that.
And that's just, it's just disingenuous to claim that you're pacing. Second, why do they have to announce all of this and ask the government to tell them to pace? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop. Don't do it. Like, this is the, like I worked at a missile factory in college and we had nuclear missiles and I walked the floor.
What's with you guys in missiles? I don't, I'm just here with software.
You were one of two people in college when we were, which was either you were protesting to keep them off campus or building them. Building them. So that was your choice. And like we had like nuns from the Catholic Church show up and pour blood all over our missiles, you know, and I'm busy just wheeling PCs around on carts saying here's the secure PC and I'm like scared to die. I have no idea what's going on. And I'm like it's a nuclear missile and then you find out that that's what stopped the Cold War. Like that literally, it was a Pershing missile and that was what did it. And so but you said something I think is super interesting which is pacing is this sort of fuzzy non-word between like going and not going. Yeah. And the problem is, you're exactly right, like, no one is going to be happy with the middle road.
And so one of the things that people, I think it's almost fun for me to watch as a sport, they think that all these people saying to the government do this, don't do this, that they think they're going to get what they want. And it's a complete 100% misunderstanding of how government works, which is when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone. And no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that, all the inputs can make it into the government, the output never makes everyone happy, right? 100% of the time it either doesn't go far enough or it goes way way too far. Yeah. And so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity.
Well, my favorite thing is like what was it? David Sacks was like...
I think it was David Sacks, but someone from the government said, you're asking us to regulate you. No. Facebook. The answer was no.
Well, but the thing that...
Probably just Trump, I think, but the thing that they know now that you just know from experience is once the wheels start on regulating, you can't slow that one down. And now it's become an election issue for every party in every jurisdiction up and down the whole government stack. So there's now this whole basket of regulatory approaches. Well, the next election will 100% be a referendum on AI. So, like it has to happen that 2028 is like the AI election and you could basically run on, the problem is it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just basically varying degrees of how much do you regulate it or at least try and avoid the topic. Yeah. But it is too bad that we as a country are in a spot where the pro-AI case is just like, it's too, like it just takes too many words, you know, it's way too nuanced.
It's defensive and we own none of the vocabulary, right? So the whole debate is pause. It's swarms. It's rogue. Every word has been chosen by the people who don't want to do AI.
Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong which takes so many words that...
Yeah. So we got to be like union jobs and cancer and like, you know, there needs to be another word cloud that emerges.
What I don't understand is why the labs have not taken a position on x-risk. Like short of that I don't think this goes in any direction other than heavy-handed regulation.
Yeah. Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with it. Like, how can a government not have a regulation?
Well, but what, who, but what would anybody really knowing this ecosystem though, I don't know that you would be able to pin anybody down on that other than something.
No, I would say Dario said it on...
No, but I'm saying you're not going to pin anybody down on a lower. Well, in fact, he does the worst thing about it, which is he agrees with people who claim that they believe that there's an x percentage of extinction happening, but he specifically goes out of his way to say, I'm not going to put a percentage on it.
Which I just think is the weirdest.
No, but to be fair, to be fair...
No. Nobody could be fair.
To be fair, that's like, it's not 100% like disingenuous or whatever, like he probably doesn't specifically agree that it's 10%. Or maybe he does and it's just too scary if he were to say...
Let's just play binary search. I mean is it more or is it less?
But the whole thing is a made-up concept that we just created. But like nobody can, you can't quantify any of this.
Well, but that's sort of Martin's point.
Well, but you just had an official position though. So the only official position would be like, I think the only official position you could possibly get would be the PR version of this that would be the only thing that would be intellectually honest would be...
There are real risks with AI. There's incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.
I don't think that's, I don't think that's true. Listen, so what do you think that...
We've been through multiple epochs of technology. We've been through compute. We've been through the internet. We've been through the web. We've been through social networking. We had a discussion about the risks without talking about x-risk. Right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is.
But what if they do think it is higher?
Okay. Okay. Yeah. Like I think the answer is they do think it's more than just the internet.
One of two options. You believe that we're going to go extinct and we shut it all down. Like shut it down or...
They believe that this is just a chance that we got...
Dario thinks less but these Cold Warriors. No one, like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war. Great movie War Games about the whole thing and all of that. But the thing was it was nonzero.
Yeah. And so once you said it was...
Can they say nonzero then? Is that allowed?
I think as soon as you think it's nonzero...
The problem is if you say nonzero that could be like, you know, the guy thinks 93%. But let's wait, so just so we're all having a clean conversation. Let's talk about marginal risk here. We're not talking about absolute risk, right? Okay.
It's just that I think once you say it's nonzero, the only answer of catastrophic, the only answer is you have to nationalize it.
Yes. And so what he's trying to, what the labs in general that have that view are trying to thread is they want it to be nonzero as a license to do a certain set of things without the burden of just becoming a national...
Well, do you have, I actually don't know all of the precedents, you hopefully do, but like there must be some things that are nonzero, let's say x-risks, that are not particularly nationalized but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements. Like I'm sure to develop anthrax you have to go to a particular kind of lab.
Well, BSL4 labs are, but they're nationalized. They are national. Okay. But ethics tends to be nationalized. Yes.
Like normal human safety not so much. Right. Like industry oversight that over time becomes federal regulation.
Right. And the progression which I think is just super important to this discussion has been since the post-World War II era most industries that are critical to the infrastructure, power and banking and healthcare. The trend has been to basically be nationalized.
Yeah. By just like your examples of KYC and all the other stuff. The banks are for all practical purposes nationalized after the financial crisis.
Okay. But you said all practical purposes, right? They're literally not nationalized, right? So maybe that, but this might be the intent of the labs is to look like JP Morgan or look like Verizon. And it's just like we're critical infrastructure. We get heavily regulated. It's not good for open source or at least frontier open source but it is a plausible outcome for this industry. Good for innovation is the problem.
I think even that's fine, just don't use species extinction as your, literally is like the difference between the things that are stuck in the lab. And let me just say something, I actually think the labs are moving in the right direction. I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders like they understand that they have this tension and they're going to reconcile that. So, I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think they grew so fast and just trying to figure out how this machinery works. And I think Cernoski really hit the nail on the head.
The political process is its own thing. And I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.
Well, I think the tech industry has literally over 100 years consistently relearned the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured out. Both got sued for antitrust. Both got substantially and structurally changed as a result and they had hundreds of lawyers in the 1960s navigating them. And then Microsoft came along like we were just like what? Yeah. Like we had no idea what was happening to us. And there's Bill Gates playing golf with Bill Clinton and then we get slapped with an antitrust lawsuit from his administration and Bill was like I was playing golf, here's the picture, and that doesn't help. And like isn't that what I was supposed to do was go and play golf. That's a shortened version of the whole thing. But I think, and I think it's just, I always use this example which is Hollywood got together during the Red Scare and the censorship when they were worried about the government censoring movies for sexual content for adult themes and they all got together and of course they were never going to be able to win in court if they tried to but they were threatening to do it and they got together and they formed the Motion Picture Association.
Yeah. And movie ratings and all that and they police themselves.
So how do you, do you guys like that? Do you like the FINRA proposal?
No, because FINRA is, well that's, that's as close to an SRO as you can get with more...
No, it's not because FINRA is going to, FINRA becomes legislation which comes with direct oversight.
Yeah, I think MPAA might not have as much consequence in like how society functions. The first amendment.
Did I win that? No, first of all, fantastic. But I still don't know if you want it. Like I agree speech is really important, but like I just think there was no societal risk. I just think what's in our movies will survive on a different continuum of...
All history is always relative and at the time being a communist was a really bad thing and 30% of Hollywood got fired for, you know, it was all this stuff. Yeah. So it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings and that's because actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV and so we got to grow up with HBO and all this other stuff.
But the problem with FINRA is that is a perfect example of essentially nationalizing risk.
Yes. Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated.
Okay. Wait, sorry. Do you think we're going to end up with a situation that is better than FINRA? FINRA appears to be the best case scenario, but it is the best case scenario.
But not even anymore. Like I do think this technology is in the limit again so powerful relative to what it can deliver that it would be impossible for eventually Congress not caring about that. Like it's just not possible.
Question. If it's eventually, at what point...
If it's eventually making every recommendation in your healthcare process and it's inside of your medical device as an open weights model and it's on every trading system for high frequency trading and it's on an airplane, like there's no chance that the government doesn't say we need something where FINRA actually is like probably the best case scenario of what that looks like. Right.
Yeah. And so now that's absolutely the most crucial point because if you, all the people in the Senate now were in the Senate when the Communications Act was passed and liability was not passed through to ISPs and to social networks and they sat around the debate at the time was they would literally say to us the internet is so big how did we not have anything to do with it and that's why Al Gore gets a bunch of abuse for saying he created it, right? Because he was actually trying to get out in front of that and say no the government was instrumental in it and it all backfired because it made it look like he was a crazy person. But that it is absolutely the case that they felt like they missed their chance to be on top of the internet.
With Al Gore being on the positive side of innovation.
That was their support. So who is the Al Gore?
There's no one. There's a couple, well Bessent seems to be that, you know, the defense people sort of want it private but not, which is exactly where they were on the internet, right.
And so it's very interesting that a lot of this is just this like Elizabeth Warren, Senator Warren's tweets were all like we missed this for social networks and it's like there's a lot of pent-up energy against tech that could end up just all siphoning into AI right now.
That is exactly what it is. That's what's happening.
Yeah. I think there's another problem which is we're all trying to predict what's bad which is not how we've normally done things. Like by this time on the internet we'd taken out like tens of billions of dollars of, well we've caused tens of billions of dollars of economic damage. We'd had worms that took out 10% of the infrastructure. The internet infrastructure which was running critical infrastructure. We had hospitals go down.
We really should have blocked the internet in like '97.
How do we, yeah, okay. So, we had, I mean I remember a time you like, I remember when you would buy your CD of Windows 95 and by the time it was done installing...
You would have a worm potentially.
No, no, this was the reality. The reality of the PC until 2001 was you could not install a PC connected to the network without getting infected. But viruses were everywhere for the, and there were two rounds of congressional hearings. You had all of the...
That is actually, okay. I'll grant you this is a very interesting point. This type of zeitgeist in '94 would have, we would probably not have the internet.
Listen, the automotive, the airline industry you always have these periods of kind of teeth cutting where like you learn about the dangers and you learn about the technology. The internet, I mean we were on the ground floor of this. I mean things were down all the time. There's economic damage all the time. Like there was novel, there was new viruses. There were new worms. They're all over the place.
Y2K. Y2K was going to destroy.
But here's the interesting thing about this.
But I just want to say, so when we created policy and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do and so the policy actually fits the fact pattern. And in this case, I mean, even when you were talking, you're like, well, what if, you know, yada yada yada. It's very hard to do predictive policy.
security risk. Great. And what is nice about the discourse that is actually starting to evolve around that. Like you actually hear people from the labs saying novel cybersecurity risk. This is an engineering problem. We're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date.
That's a very perfect point because if you look in 1986, the Computer Crime and Fraud Act got signed. It was out of a very, very specific scenario which was two groups of hackers broke into GTE Telenet.
Sure, well, it wasn't you.
Oh, he worked at Cisco later. And the problem was that was 1983 and there was no crime, and it took two and a half years for the bill to make it through. That made it very, very specific. And that's the law that says you can't access unauthorized computer systems.
So I think we'd all agree that we probably have like 90% of laws already in the applied layer, like you can't hack systems etc. Do you think there's anything that should be from a liability standpoint in the model layer? Which of course then...
How are you making a tech legal?
No, the legal, everything my read of Hugging Face, OpenAI, they're all absolutely blatant computer crime acts except for the fact that there's, they carved out amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes and they didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said, we're not going to prosecute for this. And we're also not going to prosecute if you just violate the terms of use, right, of a system versus actually try to breach it.
And so I think the law is ample for this scenario. And it was all written, this GTE Telenet was used by NASA and Livermore and all the labs. And so that's why it caught the attention of DC because it was federal systems that were being broken into. And the problem we have now is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions: sloppy and you're not complete in what you're telling us happened. And so of course the CVE process came about in the 1980s, the computer virus and vulnerability reporting stuff out of CMU. And for years they worked on very structured reporting with obligations and how to, and for some reason they're not using any of that to do this reporting. And so there is this very basic stuff that I look at and say well until they're doing that they really should...
Stop talking like they shouldn't do a postmortem on a breach that looks like an intern wrote it and it's not a postmortem, it's this selective memory it looks like.
Exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff because you don't have to give the lawyers you hire all the information about what happened. Like where's all the Slack messages? Where's the actual details of what happened?
Can I just interject an annoying aside? So, which is very in line with this, which is I've been in the security, like the actual cyber security community for a long time and it is, you know, it's always been one of these things where like they just don't like practical solutions. So even if you build like, you know, a secure system, like well what if somebody, you know, shows up like, you know, like you know, can Russell Crowe can like break the encryption or something like that. It's Mission Impossible. Some crew, there's always like these kind of like whatever. So my favorite thing that happened recently was like Noam Brown, some podcast, we've all said stupid stuff in podcasts. I've already said...
Oh, you didn't like this one.
No, it was great. I thought it was fun.
No, it was fantastic. No, I'm setting it up. Yeah.
So, so Noam Brown was like, listen, you don't know what a super intelligence could do. It could maybe use like the heat of a CPU to exfiltrate itself to another computer, which this brought me back to my... now I'm very comfortable right now, I could have endless pointless discussions on this, but what is interesting is you basically have the X-risk people saying something they thought was plausible and then you have like the security people having this kind of endless discussion. And so I think at some level, now these communities are being bridged, which is like, you know, I actually think that was a very reasonable thing for Noam Brown to... I think you can pick holes in it, but we all say weird stuff on podcasts. I actually think exfiltration risk is real. I mean, I worked in secure computing environments that were highly classified that you would, the covert channels were unbelievable. You actually, I mean, so these are very real comments, but like we actually have a real discourse and it was the first time I saw really hardcore systems people having pretty, like I would say constructive...
Calculating the bit rate and...
Yeah, it was great, but it was a constructive discussion and like, you know, you had like the kind of the typical X-risk people were engaging. Of course, it was Twitter so there's like a lot of name calling this and that, but it was actually, I felt like a real discussion for the first time. So I hope we see more of this. I hope we see more, you know, cyber related things. I think the labs should talk more about it. I think it'll engage the community and I think once that happens, we can...
Actually, yeah, Greg's been out there a lot more on this topic, which has been good. But I think the takeaway is Noam Brown should be doing more brainstorms on podcasts.
I thought it was actually just, it opened people's eyes to the fact that actually there's a lot of risks in security that most people don't understand. Yeah. And so that means that there's more stuff that like you can't make baseline risk, you know, like how does your authentication layer work? You can't just wave your hand and say that should go away and then bring up, oh but you know, space aliens can invade, and that's a little bit of what was going on that I felt uncomfortable with. But like, what do you mean? Like it was sort of like, but you know, there's also this risk, and is how I viewed that of the heat thing. But at least we're now in a domain we're comfortable, like I can talk about entropy and like we can actually have a concrete discussion that's not, oh well, it's super powerful, like at least we've reduced it to like the laws of physics and the laws of systems. And most people did, I would say most people had no idea that that kind of risk was real.
