If you regulate AI too early, you actually don't solve anything. You still just kind of have the same risk ultimately.
You will the thing into being, but you haven't figured out how to control it.
The problem we have now is this rift between the labs and the security community that keeps coming to two conclusions: sloppy, and you're not complete in what you're telling us happened.
An employee says there's a 10% chance of species extinction. Agent swarms completely flip that. These are just roaming drones, but like times 10,000, and they will easily mistake a good task for a bad one.
So now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. The US about 15 years ago stopped leading in tech antitrust. The problem is that Europe is going to lead with that because they have nothing to lose. This could change the nature of software fundamentally. The center of innovation has just moved. This is the signal that...
Guys, welcome back to the podcast.
Happy to be here.
Thank you.
Your beard.
Didn't think we'd ever do this again. This is... I can't believe it. This is great.
I mean, Martin's just building these like hundred-billion-dollar companies. Too busy for this podcast. So, at least taking credit for, as VCs do.
Exactly. Um, we have a lot to discuss today, but first, look, Aaron, why don't we start with you? Pacing the frontier. How have you reacted and reflected on what's happened there and just the discourse that's followed?
Oh boy. I think we should start with Martin on this one. He, you were fighting lots of good ground wars. Um, I maybe I'll say one thing that we probably all agree with, and then we can figure out where we maybe kind of fracture off. Um, I think we would agree that any AI lab right now at the frontier should be building, you know, in the safest way possible, with, you know, the highest degree of governance and security, and, you know, kind of whatever your definition of alignment is. Like, this is an incredibly important area of research. It's an incredibly important area for the diffusion of AI. Like, you're not going to have AI diffusion without extremely high-quality products that can be, you know, trusted by enterprises and that aren't kind of constantly hacking systems. So when, at least I read the Dario post, um, I actually didn't agree with, didn't disagree with almost anything. Um, because it was all about how do you have better, you know, security of these systems, sandboxing, better testing. There's going to be some debates around the embedded nature of the testers and like, do you agree with who those are and does the industry all align on that? But I think actually all of the major points were probably salient and appropriate. Then the only question is does this get sort of used or leveraged to do things that maybe we don't agree with, which would be like a slowdown of AI dramatically because of regulatory controls that would sort of, you know, not make it easy to compete, you know, with the frontier labs. Um, or do politicians kind of end up sort of taking the message and run with it and maybe even worse outcomes happen, like, you know, it's used to ban data centers, you know, far faster and whatnot. And so I think the actual substance of the topic is actually incredibly important and I think very important for AI advancement in general. And then the question is what do you do about it, especially what do you do about it from a regulatory standpoint. And that's probably where the industry is going to land on very different points in the continuum. But Martin was putting up a good fight on like, let's make sure that we don't use this for regulatory kind of capture. Um, which I also agree with, but I think the ideas in the pacing conversation are important. Um, you know, again, like it's a little bit of a funny concept because maybe it's not even pacing as much as just like good hygiene and then good engineering. And so with good engineering, obviously there is a slight slowdown, but it's a slowdown that obviously allows acceleration of your diffusion because you wouldn't be able to have any of the AI be diffused if nobody would, you know, trust using it. So the post is very reasonable, but the atmospherics are not right. Like, so an employee is like, this is going to kill whatever, 10% chance of species extinction, and you know what Dario says...
I agree with him more than I disagree with him, right? Like...
On TV, on TV, the same day that he landed these things. And so in some way you can't have these conversations in isolation, which is of course if he's going to agree in species extinction...
This post that he has looks like this milquetoast capitulation that's totally not adequate for the task at hand. And so I think the atmospherics are totally broken. And a lot of my comments were on the atmospherics.
And then I have this quibble, but it really bothers me because I'm impeded, which is...
I think pacing is the wrong way to describe this. For one, yes, it is orthogonal to security.
Right. So like you can very slowly build a nuclear weapon and that doesn't make anybody feel better that it's slow versus fast. So that's one. The second one, it just feels like a capitulation to the pause folks without actually, you know, addressing it. So, you're saying, well, we're not going to pause. We're going to pace to make them happy, but we'll also somehow make the regulators happy. And I think it makes them both unhappy.
Because the pause people are like, well, that's not a pause. That's just pacing. And then the regulators are, you're still doing the thing, right?
And so I just feel like they're trying to... my sense of what's happening is the labs are actually trying to do the right thing. And I applaud them for that. I think this is a pragmatic proposal and I applaud them for that. I think the messaging is wrong because they're trying to like split the difference between an internal kind of fringe faction, which are doomers, pause, and then the regulators on the other side. And the problem is they're making both of them unhappy. And I think what they have to do is they need to come out, they need to address the x-risk question directly.
Like they need to say, no, we don't think this stuff we're going to do is going to cause extinction. And then I think this becomes this very sensible...
And what would you do, just to play the other side for one second, what do you do? Do you make room for the one possible Venn diagram, which is the lab researcher that is both simultaneously super scared but also works on advancing the state of AI because they believe that it's so important to get right that they want to pursue that. And then obviously the language is right now very, you know, problematic, but like, that person does exist and that is a real kind of person in our industry, which is like, we have to be at the forefront of AI. I'm also very scared of it, and so that's why I'm working on this.
So let me address this directly. I used to work for Lawrence Livermore National Labs on a weapons program, nuclear weapons. I know what it's like to work on things that have access.
You were the first pacer.
We were part of the first pacers. So if a constituency within the labs that are the most knowledgeable people believe the stuff has existential risk...
Yes.
The answer is to nationalize it and actually put controls that we know that work.
Right now, if they don't actually believe that, and in the private conversations I have, the most sensible people don't. It's a small fraction that do. This is an HR problem.
Right. So to me, an HR problem is a company problem. Like if they are worried that they can't recruit people, right, or they can't retain people, which it seems to me a lot of this is just that concern, it's almost more of this kind of researcher currency if that's the case.
I think that is the wrong reason to cause a national-level lockdown on a very promising technology. So listen, I think the post, again, I think the post is actually very sensible. I think there are real concerns around security. We've had many compute epochs that have real security concerns. I don't think you can reconcile discussions on x-risk with the proposal that was put out. You just can't reconcile those two things, and that has always been my primary...
Right. Right. Okay. Unleash. Okay.
Holding it back.
So, okay. The first thing is you can't have... Is there a schedule that they've published that says when all this stuff, whatever bad stuff is going to happen, is? They haven't. So you can't pace it because nobody knows when it was supposed to finish in the first place. It also just seems disingenuous. It's completely...
This is like when the press reports on Apple's latest iPhone is late.
The phone from what nobody knows exists that they haven't told anybody about.
Yeah. My Apple car was very late.
Yeah. Like I don't understand. Like in order for something to be slower, you need to know the rate at which it was moving in the first place. So it's all just utter nonsense and you can't escape that. And then...
By the way, this is another problem that, like, again from the, like, my little quibble on PR is nobody believes it anyway, right? And so if that's the thing, the pacing, right?
Which part going to pace?
That they're going to pace. They've been at a dead run. They've raised more money than ever before. They've grown faster than ever before. There's no indication that they're pacing. So if this is what you're going to hang your...
I see. I see. Yeah. Well, the internal, the issue obviously is that the models that everybody has internally far exceed what anybody else has access to. So it's really just the pacing of external releases. But again, back to your point, pacing versus what, right? Like, we don't know if the one that scares everybody also doesn't, you know, work for a legal brief. Like, it might actually screw all that stuff up because it's so... who knows, right? So there's that.
And that's just, it's just disingenuous to claim that you're pacing. Second, why do they have to announce all of this and ask the government to tell them to pace? Like, that's the part that starts to go, well, this is really spooky. If you are the most afraid of how everything is going to go because of your product, just stop.
Don't do it.
Like, I like this is the, like I worked at a missile factory in college and we had nuclear missiles and I walked the floor.
What's with you guys in missiles? I don't... I'm just here with software.
You were one of two people in college when we were, which was either you were protesting to keep them off campus or building them. So that was your choice. And like we had like nuns from the Catholic Church show up and pour blood all over our missiles, you know, and I'm busy just wheeling PCs around on carts saying, here's the secure PC, and I'm like scared to die. I have no idea what's going on. And I'm like, it's a nuclear missile. And then you find out that that's what stopped the Cold War. Like that literally, it was a Persian missile and that was what did it. And so, but you said something I think is super interesting, which is pacing is this sort of fuzzy non-word between like going and not going. Yeah. And the problem is, you're exactly right, like there's no one is going to be happy with the middle road.
And so one of the things that people, I think it's almost fun for me to watch as a sport, they think that all these people saying to the government, do this, don't do this, that they think they're going to get what they want. And it's a complete 100% misunderstanding of how government works, which is when you talk to the government, they actually know how these things work. And they know they're just listening to you and they're listening to everyone. And no one is going to get what they want because they know in order to do anything in our system, it's a compromise. And so everything that all the inputs can make it into the government, the output never makes everyone happy, right? 100% of the time it either doesn't go far enough or it goes way, way too far.
Yeah.
And so you can't take the view that you're going to talk to the government and talk your way into the solution you want. So the bottom line is if they're asking for pacing, they're going to get the wrong velocity. Well, my favorite thing is like, what was it? David Sax was like...
I think it was David Sax, but someone from the government said, you're asking us to regulate you. No. Facebook. The answer was no.
Well, but the thing that...
Probably just Trump, I think, but...
The thing that they know now that you just know from experience is once the wheels start on regulating, you can't slow that one down. And now it's become an election issue for every party in every jurisdiction up and down the whole government stack. So there is now this whole basket of regulatory approaches.
Well, the next election will 100% be a referendum on AI. So like it has to happen that 2028 is like the AI election and you could basically run on... The problem is like it's not obvious who would run on the pro-AI story because it's going to be too nebulous to tell that story. So then it's just basically varying degrees of how much do you regulate it or at least try and like avoid the topic. Yeah. But it is too bad that we as a country are in a spot where like the pro-AI case is just like it sounds too... it's like it just takes too many words, you know, it's way too nuanced. It's defensive.
Yeah. It's defensive.
It's defensive and we own none of the vocabulary, right? So the whole debate is pause. It's swarms. It's rogue. Every word has been chosen by the people who don't want to do AI.
Yeah. And so it means the first thing you have to do is invent new words and say that their words are wrong, which takes so many words that...
Yeah. So we got to be like union jobs and, you know, cancer and like, you know, there needs to be another word cloud that emerges.
What I don't understand is why the labs have not taken a position on x-risk. Like short of that, I don't think this goes in any direction other than heavy-handed regulation.
Yeah. Well, it would just be negligent of the government to be like, there's a 10% chance of species extinction. The CEO of the top company says he agrees with it. Like, how can a government not do... have you... he had a regulation?
Well, but what would anybody really, knowing this ecosystem though, I don't know that you would be able to pin anybody down on that other than something?
No, I would say Dario said it on...
No, but I'm saying you're not going to pin anybody down on a lower... Well, in fact, he does the worst thing about it, which is he agrees with people who claim that they believe that there's an x percentage of extinction happening, but he specifically goes out of his way to say, I'm not going to put a percentage on it.
Which I just think is the weirdest...
No, but that, to be fair, to be fair...
No.
Okay. No, nobody could be fair. To be fair, that's like, it's not 100% like disingenuous or whatever. Like he probably doesn't specifically agree that it's 10%. Or maybe he does and it's just too scary if he were to say...
Let's just play binary search. I mean, is it more or is it less?
But the whole thing is a made-up concept that we just created. But like nobody can... you can't quantify any of this.
Well, but that's sort of Martin's point.
Well, but you just had an official position though. So the only official position would be like, I think the only official position you could possibly get would be the PR version of this. That would be the only thing that would be intellectually honest would be...
There are real risks with AI. There's incredibly positive benefits as well. We are working to mitigate the risks so they are as reduced as humanly possible.
I don't think that's true. Listen, so what do you think that...
We've been through multiple epochs of technology. We've been through compute. We've been through the internet. We've been through the web. We've been through social networking. We had a discussion about the risks without talking about x-risk. Right? So for example, one thing you can say is we do not think the marginal risk for species extinction is different than it is.
But what if they do think it is higher?
Well, then we should...
Okay. Okay. Yeah. Like I think the answer is they do think it's more than just the internet.
One of two options. You believe that we're going to go extinct and we shut it all down. Like shut it down or...
They believe that this is just a chance that we got...
Speaking.
Dario thinks less, but these Cold Warriors... No one, like the Pentagon, you know, they ran a lot of simulations on the chances for nuclear war. Great movie, War Games, about the whole thing and all of that. But the thing was it was nonzero.
Yeah. And so once you said it was...
Can they say nonzero then? Is that allowed?
I think as soon as you think it's nonzero...
The problem is if you say nonzero, that could be like, you know, the guy thinks 93%. But let's wait. So just so we're all having a clean conversation, let's talk about marginal risk here. We're not talking about absolute risk, right? Okay.
It's just that I think if once you say it's nonzero, the only answer of like catastrophic, the only answer is you have to nationalize it.
Yes. And so what he's trying to... what the labs in general that have that view are trying to thread is they want it to be nonzero as a license to do a certain set of things without the burden of just becoming a national...
Well, do you have... I actually don't know all of the precedents, you hopefully do, but like there must be some things that are nonzero, let's say x-risk, x-risks that are not particularly nationalized, but the regulatory environment around them is so heavy that it might as well be nationalized because of the KYC requirements on like... I'm sure like to develop anthrax you have to go to a particular kind of lab.
Well, BSL4 labs are, but they're nationalized. They are national. Okay. But...
Ethics tends to be nationalized. Yes.
Like...
Normal human safety, not so much. Right. Like industry oversight that over time becomes federal regulation.
Right. And the progression, which I think is just super important to this discussion, has been since the post-World War II era, most industries that are critical to the infrastructure, power and banking and healthcare, the trend has been to basically be nationalized. Yeah. By just like what you like your examples of KYC and all the other stuff. The banks are for all practical purposes nationalized and the financial crisis.
Okay. But you said all practical purposes, right? They're literally not nationalized, right? So maybe that... but this might be the intent of the labs is to look like JPMorgan or look like Verizon. And it's just like we're critical infrastructure. We get heavily regulated. It's not good for open source or at least frontier open source, but it is like a plausible outcome for this industry. Good for innovation is the problem.
I think even that's fine, just don't use species extinction as your... literally is like the difference between the things that are like stuck in the lab. And let me just say something. I actually think the labs are moving in the right direction. I actually think the actual statement was really good. You know, in my discussions with, you know, executives and leaders, like they understand that they have like this tension and they're going to reconcile that. So I actually am quite optimistic that the labs are both doing the right things and trying to do the right thing. I just think that grew so fast and just trying to figure out how this machinery works. And I think Sinoski really hit the nail on the head.
The political process is its own thing. And I don't know if it's naivety or hubris, but I just don't think that they kind of know how to navigate it.
Well, I think the tech industry has literally over 100 years consistently relearned the lesson at each technology wave that we don't understand the regulatory climate and we can't navigate it. And even the companies like AT&T and IBM that were born out of basically being government monopolies from the start never figured it out. Both got sued for antitrust, both got substantially and structurally changed as a result, and they had hundreds of lawyers in the 1960s navigating them. And that, you know, Microsoft came along like we were just like, what? Yeah. Like we had no idea what was happening to us. And there's, you know, there's Bill Gates playing golf with Bill Clinton and then we get slapped with an antitrust lawsuit from his administration. And Bill was like, I was playing golf, here's the picture, and that doesn't help. And like, isn't that what I was supposed to do was go and play golf? That's a shortened version of the whole thing. But I think...
It's... I always use this example, which is Hollywood got together during the Red Scare and all the censorship when they were worried about the government censoring movies for sexual content, for adult themes. And they all got together and of course they were never going to be able to win in court if they tried to, but they were threatening to do it. And they got together and they formed the Motion Picture Association.
Yeah. And movie ratings and all that, and they police themselves.
So how do you... do you guys like that? Do you like the FINRA proposal?
No, because FINRA is... well, that's as close to NPA as you can get with more.
No, it's not, because FINRA becomes legislation which comes with direct oversight.
Yeah, I think MPAA might not have as much consequence in like how society functions, the First Amendment.
Yeah.
Um.
Did I win that? No, I... first of all, fantastic. But I still don't know if you want it. Like, I agree speech is really important, but like, you know, like the... I just think...
There was no societal risk.
I just think what's in our movies will be like, we'll survive like on like a different continuum of...
All these, every history is always relative. And at the time, being a communist was a really bad thing and 30% of Hollywood got fired for, you know, it was all this stuff. Yeah. So it's always a risk to bring up something in that kind of way because it sounds so dumb. Like nobody thinks about movie ratings and that's because like actually they were ruled basically you can't constitutionally mandate them. So they couldn't regulate them on cable TV and so we got to grow up with HBO and all this other stuff.
But the problem with FINRA is that is a perfect example of essentially nationalizing risk.
Yes. Because even though the banks all pay money into it and that's how it's run and stuff, it's all mandated.
Okay. Wait, sorry. Do you think we're going to end up with a situation that is better than FINRA? FINRA appears to be the best case scenario, but it is the best case scenario,
but not even anymore. Like I do think this technology is in the limit again so powerful relative to what it can deliver that it would be impossible for eventually Congress not caring about that. Like it's just like not possible. Question: if it's eventually, at what point...
If it's eventually making every recommendation in your healthcare process and it's inside of your medical device as an open weights model and it's on every trading system for high frequency trading and it's on an airplane, like there's no chance that the government doesn't say we need something where FINRA actually is like probably the best case scenario of what that looks like. Right.
Yeah. And so now that's absolutely the most crucial point because if all the people in the Senate now were in the Senate when the Communications Act was passed and liability was not passed through to ISPs and to social networks and they sat around, the debate at the time was they would literally say to us, the internet is so big, how did we not have anything to do with it? And that's why Al Gore gets a bunch of abuse for saying he created it, right? Because he was actually trying to get out in front of that and say, no, the government was instrumental in it, and it all backfired because it made it look like he was a crazy person. But it is absolutely the case that they felt like they missed their chance to be on top of the internet.
With Al Gore being on the positive side of innovation.
That was their support. So who is the Al Gore?
There's no one. There's a couple. Well, Bessent seems to be that, you know, the defense people sort of want it private but not, which is exactly where they were on the internet, right?
And so it's very interesting that a lot of this is just this like Elizabeth Warren, Senator Warren's tweets were all like, we missed this for social network, and it's like...
There's a lot of pent-up energy against tech that could end up just all siphoning into AI right now.
That is exactly what it is. That's what's happening.
Yeah. I think there's another problem, which is we're all trying to predict what's bad, which is not how we've normally done things. Like by this time on the internet, we'd taken out like tens of billions of dollars of like economic... well, we've caused tens of billions of dollars of economic damage. We've had worms that took out 10% of the infrastructure.
Yeah.
The internet infrastructure which was running critical infrastructure. We had hospitals go down.
We really should have blocked the internet in like '97.
How do we... Yeah. Yeah. Okay. So we had... I mean, I remember a time you like... I remember when you would buy your CD of Windows 95 and by the time it was done installing...
You would have a worm potentially.
Way to go, Stephen.
No, no, this was the reality. The reality of the PC until 2001 was you could not install a PC connected to the network without getting infected. But viruses were everywhere.
And there were two rounds of congressional hearings. You had all of the...
That is actually... Okay. Okay. I'll grant you this is a very interesting point. This type of zeitgeist in '94 would have... we would probably not have the internet.
Listen, listen. I mean, the automotive, the airline industry, you always have these periods of kind of like teeth cutting where like you learn about the dangers and you learn about the technology. And the internet, I mean, we were on the ground floor of this. I mean...
Things were down all the time. There's economic damage all the time. Like there was like novel... there was new viruses. There are new worms. They're all over the place.
Y2K. Y2K was going to destroy.
But here's the interesting thing about this.
But what I just want to say, like, so when we created policy and we did create a lot of policy, it was kind of like with a bunch of very specific data points on what you're trying to do. And so you know that the policy actually fits the fact pattern. And in this case, I mean, even when you were talking, you're like, well, what if, you know, yada, yada, yada. It's very hard to predictive policy.
Security risk. Great. And what is nice about the discourse that is actually starting to evolve around that, like you actually hear people from the lab saying novel cybersecurity risk. This is an engineering problem. We're talking about that. So the more this becomes concrete around real identified risks, I think we can all fall in line. But that's not been the discussion to date. That's a very...
Perfect, perfect point. Because if you look in 1986, the Computer Crime and Fraud Act got signed. It was out of a very, very specific scenario, which was two groups of hackers broke into GTE Telemet.
And...
Sure, well, it wasn't you. It was the other guy.
Oh, he worked at Cisco later. And...
And the problem was that was 1983 and there was no crime.
And it took two and a half years for the bill to make it through. That made it very, very specific. And that's the law that says you can't access unauthorized computer systems. So I think we'd all agree that we probably have like 90% of laws already in the applied layer, like you can't hack systems, etc. Do you think there's anything that should be from a liability standpoint in the model layer? Which of course then...
How are you making a tech legal?
No, the legal... everything, my read of Hugging Face, OpenAI, for they're all absolutely blatant computer crime acts except for the fact that they carved out amendments later that if you're a white hat, it's not illegal anymore. And that was because people kept making mistakes.
And they didn't want to go around arresting everybody who was actually trying to make the system better because they messed something up. And so the Justice Department wrote a memo that said, we're not going to prosecute for this. And we're also not going to prosecute if you just like violate the terms of use, right, of a system versus actually try to breach it.
And so I think the law is ample for this scenario. And it was all written... this GTE Telemet was used by NASA and Livermore and all the labs. And so that's why it caught the attention of DC because it was federal systems that were being broken into. And the problem we have now is this rift between the labs and the security community that keeps looking at all their postmortems and coming to two conclusions: sloppy,
and you're not complete in what you're telling us happened.
And so of course the CVE process came about in the 1980s. The computer virus and vulnerability reporting stuff out of CMU. And for years they worked on very structured reporting with obligations and how to. And for some reason they're not using any of that to do this reporting.
And so there is this very basic stuff that I look at and say, well, until they're doing that, they really should...
Stop talking like they shouldn't do a postmortem on a breach.
That looks like an intern wrote it, and it's not a postmortem. It's this selective memory. It looks like exactly the kind of postmortem you do when you hire outside lawyers to investigate some random thing and you only give them certain stuff.
Because you don't have to give the lawyers you hire all the information about what happened. Like, where's all the Slack messages? Where's the actual details of what happened?
Can I just interject an annoying aside? So, which is very in line with this, which is I've been in the security, like the actual cybersecurity community, for a long time, and it is, you know, it's always been one of these things where like they just don't like practical solutions. So even if you build like, you know, a secure system, like, well, what if somebody, you know, shows up like, you know, like...
You know, can Russell Crowe, can like break the encryption or something like that? It's Mission Impossible. Some crew, there's always like these kind of like whatever. So my favorite thing that happened recently was like Gnome Brown on some podcast. We've all said stupid stuff in podcasts. I've already said...
Oh, you didn't like this one? No, it was great. I thought it was fun. No, it was fantastic. No, I'm setting it up. Yeah.
So, so Gnome Brown was like, listen, you don't know what a superintelligence could do. It could maybe use like the heat of a CPU to exfiltrate itself to another computer, which this brought me back to my...
Now I'm very comfortable right now. I could have endless pointless discussions on this, but what is interesting is you basically have the X-risk people saying something they thought was plausible, and then you have like the security people having this kind of endless discussion. And so I think at some level now these communities are being bridged, which is like, you know, I actually think that was a very reasonable thing for Gnome Brown to...
I think you can pick holes in it, but we all say weird stuff on podcasts. I actually think exfiltration risk is real. I mean, I worked in secure computing environments that were highly classified, that you would... the covert channels were unbelievable. You actually, I mean, so these are very real comments, but like we actually have a real discourse, and it was the first time I saw really hardcore systems people having pretty, like, I would say constructive...
Calculating the bit rate and... yeah, it was great, but it was a constructive discussion. And like, you know, you had like the kind of the typical X-risk people were engaging. Of course, it was Twitter, so there's like a lot of name calling this and that, but it was actually, I felt like a real discussion for the first time. So I hope we see more of this. I hope we see more...
You know, cyber-related things. I think the lab should talk more about it. I think it'll engage the community, and I think once that happens, we can... Actually, yeah, Greg's been out there a lot more on this topic, which has been good. But I think the takeaway is Gnome Brown should be doing more brainstorms on podcasts.
I thought it was actually just, it opened people's eyes to the fact that actually there's a lot of risks in security that most people don't understand. Yeah. And so that means that there's more stuff that like you can't make baseline risk, you know, like how does your authentication layer work? You can't just wave your hand and say that should go away and then bring up, oh, but you know, space aliens can invade. And that's a little bit of what was going on that I felt uncomfortable with. But like, what do you mean? Like it was sort of like, but you know, there's also this risk, and is how I viewed that...
...of the heat thing. But at least we're now in a domain we're comfortable. Like I can talk about entropy, and like we can actually have a concrete discussion that's not, oh, well, it's super powerful. Like at least we've reduced it to like the laws of physics and the laws of systems. And most people did, I would say most people had no idea that that kind of risk was real.
I mean, like when I'm walking around the missile factory, I actually had to test the graphics cards and PCs because a DoD requirement is that the screen memory not be sustained when you pulled the power, but not for zero time. Like the minute that the power went off, the memory image had to go. And if there was like a three-second delay... Yeah. Oh, you had that could be read.
Oh, see, we had people that came into our offices and would literally measure the distance of the monitors to each other because of Tempest attacks, which is 100% a way to use electromagnetic radiation to leak information. I've seen the same thing with spread spectrum from the BIOS. I've seen it from audio speakers, like we had to remove the speakers out because it's a very high channel.
Our building had just music speakers aimed at the windows just to produce interference. You need to go run safety at one of these labs. This is like, I've never seen you at more excited than heat-based communication.
Can I tell you the craziest covert channel I've ever seen? So it turns out, remember the old CRT... I know this is like one of my... So I'm glad that someone's older than me. Not really, but acting it. So remember the old CRT? So it turns out if like, let's say it's night and you're using a CRT terminal in your room, the lightest thing in the room is actually the pixel that the raster beam is on. So most people think it's like the glow of the monitor, but it's actually that given pixel. Mhm.
So somebody figured out that like, let's say you're in like, you know, a hotel room and you're on your computer. If you have something that can sample the color of the window, you can reconstruct the screen. Oh, that's crazy, right? You just do it at the same hertz that the raster beam is moving.
Then somebody else figured out if you can subvert three pixels, you can use those, you know, because it just looks like bad pixels. You can use those to basically send a message. So like you could literally like sit out in whatever, like you sample the message and you could... it was a relatively high-bandwidth one-way communication. And so like what Gnome Brown was saying, like maybe heat is not the way to do it, but that level of sophistication is actually real. That's a thing.
When I was at the missile factory, like I had to lock my keyboard up. I... that's really an impolite word, but that's what we call something. I had to lock my keyboard up at night because they didn't want the custodians who didn't have clearance walking by and just noticing which keys were dirtier or cleaner. And I once left it out and there's like a note from security, you know, telling me to report to security and pick up my keyboard.
Like the guard who walked the hallways just took the keyboard that night off my machine. And that's like basically, I was not cleared. I was just that sensitive, you know? I was like nothing. I was an intern.
The big problem with this conversation is now this is all in the training data of every AI model in the future. So... but that's also the threat model. Well, that's the threat model for these things is always you've got a trusted side and an untrusted side. NIST has 500-page manuals. Every untrusted side you assume basically an oracle that can do and know everything, and then the question is can you get information off the trusted side, which by the way is what Gnome was saying, which again is actually quite...
Which I do think brings up a super interesting point, which I'm going to bridge to, which is just that I think the thing that people really aren't wrapping their heads around and are using the language that's really confusing is just that what AI can do is it can try all of those things in a very short time. Yeah. Yep. And it doesn't get tired, it doesn't get bored, and you know, and so...
But the interesting thing about it is there's a whole layer of security that you now have to go look at every single API, every single service you're running internally on your network as like, you know, like nobody thinks that their internal GitHub or their internal Slack or internal finance expense tool is vulnerable to a denial-of-service attack, but swarms do it, look literally look like a denial-of-service attack.
Yeah. And so now we need a whole layer internally that just is tracking way more about what authentications are being done, what APIs are being done. Yeah. And but that's just like now it's just going to be basic. And all the old-timers that are old are like mailing me like, why are we explaining this to everybody? It's so basic because nobody did it internally.
Yeah. And well, you didn't have to worry about it for your people, and that's the like... but now your person is just a piece of software. Yeah. And like unlimited, and has a credit card.
Yeah. I mean, we kind of got by with information security like to some extent on the fact that most people will do the right thing 95 to 99% of the time. Oh, wait. Is that the malicious employee is like one in 10,000? Yeah. Yeah. Yeah. Yeah. So like, so all these systems are basically open to whoever wants to access them or like one tap on the shoulder and then you have access. And agent swarms completely flip that because they will just... these are just roaming, you know, drones. Yes. And but like, yeah, times 10,000, and they will easily mistake a good task for a bad one and vice versa.
So the data security in our systems is a... this is going to be a huge upgrade moment. I actually, like, I think that actually we're going to need a different access and security model going forward. Where are we going to go on that? Because the model we have is not granular enough and it's not performant enough to handle this stuff.
So I want to step back. I want to say like a meta point, which is I think this is how these conversations should go. We've identified a novel risk, which is like cybersecurity, which we actually have proof points, and now we're talking about solutions. I think the entire discourse around AI can be of that form, and that the biggest mistake is that's not what it's been like.
I think the entire industry and community is very happy to engage exactly like this, and I have something to say about exactly what you're asking. But I say like, I think this is where the conversation should be. So we're known for having healthy conversations that everyone should learn from, and so that's what we do.
So here's the thing. I don't think there's a technical limitation here. Like these threat models are very well understood and have been in the literature for a long time. I mean, like the operating systems research, the MLS, the multi-level security research has considered these sorts of things from an academic lens. The reason it hasn't been adopted is it just tended to be a usability issue. Like it's just really hard to maintain, and you didn't have to.
So you could argue that AI solves the usability issue because AI is using it. So maybe now is going to be a renaissance in operating systems and network and computer languages, and we should like go back to the old research and we should kind of start rebuilding systems that are secure by design. And oh, by the way, if we don't think that, you know, these things are safe to put out, we don't put them out until we have these systems built. And oh, by the way, the AI is very smart, so they can help us build it.
And so I think again, like, computer owes a... like, you know, how much of the stack we had to change for the internet. Mhm. Everything. Right. Tell. And you know, you know how not vulnerable and how vulnerable everything was. Like, we could be in one of those moments, like, oh, we got to rethink everything, and that's fine. We've done that before, but I think that's a conversation we should have. So I agree, it's time to think about evolving these things.
Well, like, look at, like, here you mentioned earlier, like, this booting a PC and getting a virus in 30 seconds or whatever. So if you look at how, like, you take an iPhone out of the box, which a lot of people are doing this week, you know, what happens is the whole network is basically shut down except for getting the latest version of the operating system because it doesn't even, though it was pressed, you know, six weeks ago, you know, some zero-day thing has been discovered since. And so actually the whole out-of-box process now involves first step update, doing an update where the device can't do anything else, and it can never do anything else until it's updated.
That's the, like, there are all these benign things, completely benign, that we turned off in all of this desktop software of the era that used to be good things. Like it was having a macro for words so you could build automatic citations or something was like this super cool thing until it became a virus. We had a thing where you could put a CD in and it would just arbitrarily run a program. And so then what somebody did was like, oh, well, I'm going to burn a clone of that CD and replace the program with my virus, but it's going to look like the thing that's supposed to run, and it's just collecting all stuff and being evil.
Yeah. Then we disabled that. And so what one of the things that's happening right now is we, there's a whole bunch of stuff that happens on, like, your own corporate network that actually is going to have to just change as the standard procedure.
Two-factor auth five years ago was not standard in most places. This whole, your whole, like, your whole SaaS world, like, I remember in, like, 2015 or so when you would talk to a new company about their, oh, we're going to do enterprise pricing and whatever, and then they realized the first thing they had to do was go do Okta integration or Google Auth because they could not have their own directory of how to manage it. And then that just became a thing, and now there's not a SaaS program anywhere that doesn't just launch right with managed authentication, right? And so there's just so many things that need to happen before you're even software now.
Yeah. Well, yeah, we're, I mean, there's probably every layer of the stack has to evolve a bit on this. Like even the lack of granular nature of, like, you know, you have these modes of, like, the agent will either ask you every single time, you know, if you want to give it permission to do something, or the exact opposite of, like, it can just, like, delete your entire computer, right? And, like, our OS probably wasn't built for the right level of granular set of tools you want to give the agent.
We've kind of done a lot of work in this space because obviously, like, you know, do you want to give an agent access to your entire file system? Probably not, right? Maybe in some cases you do, but oftentimes you want to have granular controls of, like, in this folder you can do read-write, and in that folder you can only do read. And so how do you kind of make this all intuitive for the user? So it's very difficult, and so there's going to, like... Yeah. What you just said is a very deep comment. I know exactly. Yeah.
Which is basically the conclusion of 40 years of... No, it's like you actually can't make it. But maybe with AI you actually can. Like maybe there is, like... If you ask me, like, walking in, like, what I wrote down on my note was my biggest fear. Yeah.
Is that Europe decides that GDPR was the best thing ever. Uh-huh. And they're going to just GDPR AI. Yeah. And the AI will be fine. It'll have one prompt for when text gets emitted that just says this vendor is emitting text and it's probably wrong, yes or no. That's going to be because you can't really, at least in North America, you're not going to send your speech in Europe. They still will, and they'll have filters and keywords and block lists.
But then on any verb, anytime that an agent or a background agent or a frontline agent touches a third-party product, I'm really worried that they're just going to say we need a GDPR prompt on that, and it's going to be, oh, back to the user agent, every single write or every single non-lookup, yeah, becomes like a safety warning, like the airbag thing in your car.
And the regulators love that because it's a liability assignment, and so it has this sort of legal precedent. And I really worry that that's actually the middle ground where we're going to end up. And unfortunately, because the US about 15 years ago stopped leading in tech antitrust, the problem is that Europe is going to lead with that because they have nothing to lose.
Yeah. Like there, and so I don't want to be sad and down about it, but I just can't get out of my head that they love prompts. They, I mean, look, I had to put in that browser choice thing. No, they, it, because it's the same. Look, get into a new car. Yeah. Which I haven't done in years, but it, you know, like you're pulling stickers off, you've got, you know, you have all of these things, and someone thinks that that was success. Yeah.
And it's like, has anybody ever read, like, what is this, if there's a baby in this seat? And it's like, well, that's relevant for some people some of the time, but it's this entire fabric. It's attached to the seat, and they love that. That is a very particular thing that they just love. And so I would say if I were an AI now, the one thing I would be trying to avoid, and look, we added it, okay, FINRA for AI, we'll create that standard.
I mean, look, we had to put this, we, on the, when the internet was new, the big thing was to download a program and run it. And of course, if your machine is running in administrator mode, it was download a virus and take all your files forever. And so with Windows XP, which was in 2000, we added this thing that prompted you and stopped, like literally your machine stopped, user account control. And like it was absolute assault. And we also did it in Word, the stupid little macro to help you write your thesis also came with a warning every time you opened your thesis saying this has macros.
Everybody would just click. And everybody, and so you end up in this world where just like with GDPR, everybody is numb. And so then they're like, well, it needs to be bigger. Although Mac, I mean, Mac kind of has... Nobody downloads software, that's the thing. It's a very different usage pattern on Mac. So I wish I... I don't have, like, I don't have, like, 10 applications on my Mac, but well, that's, but 10 and then you're done. Yeah. And it's, but it's a lot of people still do all this stuff. Imagine if instead it was every time you go to a new website. Yes. Which you do now because that would be bad.
Yeah. I mean, to start from here, but to bring it kind of back to the macro, like I wish this was the discussion we were having, which is like, I feel like we've dealt with a lot of these problems. I feel like when we talk about philosophical ex-risk, we're not solving these very pragmatic problems. I actually think this is actually a constructive conversation to have. Maybe prompts would help. I don't know.
And I think part of the problem is like people don't remember, like, you know, how unfettered access was and how bad it was and just how relatively benign that ended up being. Like I even remember, like, there's at Stanford during our PhD, I remember the oscilloscope was kind of janky. I'm like, what's going on with this oscilloscope? Like it's, you know, it's a little slow. And I was measuring the network traffic and it was like more network traffic than you would expect. And I'm like, why is there network? I didn't know the thing had a TCP stack. Somebody like broke in because there's an old version of Windows CE and was running a porn server, you know, and so like...
I noted that. It used to be... For the record, nothing. It used to be the case that like anytime you turned over a stone, yeah, yeah, somebody had broke into something and like was it this like worst-case malicious whatever, and it was actually very rarely, even though the capability was there.
And so if we could just somehow tone down the rhetoric and put it in context, and these are still computer systems, and yes, there's very serious stuff, and people have definitely died because networks have gone down, there's been real issues, but like, and then can we just quibble about GDPR? That would be amazing. But the problem is like that's not the discussion. It's not about GDPR and prompts. It's about species extinction and philosophy.
And philosophy and irrefutable things, and it's just... It's very soon. Like I think that's such a great point, and I think you hear people now talk about we regulate airplanes and we regulate cars, and you forget like, well, the first cars were at the turn of the century, and Unsafe at Any Speed was in the mid-1960s, totally.
And you know, people had been selling pharmaceuticals during the gold rush, and thalidomide, you know, 50 or 75 years later, and not even in the US, and then there was the FDA. And you know, people, the first pilots were flying obviously at the beginning of the 20th century, and it wasn't until the 1920s that you had to get a license to be a pilot, and you literally showed up with your own plane and you got a certificate if you had it. It was like getting, it was literally no different than driver's ed is today.
And then it was 20 more years until they had anything to do with airworthiness and looking at your plane, but it was minimal. And then it wasn't until way after World War I that they got involved in like what you think of as the modern FAA. And so you're looking at 40 years of innovation. And they were not moving slow. I mean, if you've ever seen that video in black and white of like all the different planes that crashed and everything, that was 20 years after the Wright brothers.
And it's incredibly effective. And right, it's incredibly... It's the safest form of transportation, like, you know, and so I do think that this process... It's also the slowest, most difficult form of innovation. And so if you start the FAA, if you had started the FAA in 1910, you never, you never, you never... Well, I was listening to a Nick Bostrom podcast just a couple days ago. No, no, I... it was...
It was interesting. It was interesting, and but no, but he actually makes his point. If you regulate AI too early, you actually basically don't solve anything, and you still just kind of have the same risk ultimately, but you don't understand the system. Then you will the thing into being, but you haven't figured out how to control it.
Well, we have to figure out what it actually is. Yeah. Like there are new things coming out all the time, and like casting AI completely differently than we thought of just six or nine months ago. And I think that like all the innovation that's going to happen at the application layer is going to cause things to move in and out of the models in different ways. And like we thought up until last week, I think, that, you know, text prompts and text coming back was going to be the best way to interact with... I know, then Jeff... and then Jeb... so good too, and then...
So talk about that because I think... why you find it so remarkable. Yeah. Well, okay. So the way I think about it is, so okay, so LLMs were kind of text in, text out, right? They generate text, and they came from chat, right? It was to communicate with a human. And we've spent the last few years trying to take this thing that spits out text and cram it into a traditional program.
Right? And but traditional programs don't really speak text, right? And so then you end up doing this janky thing where you're like in the prompt you're like, here's the schema, here's the schema, but like the thing is generating text and it kind of ignores it, and it's just been super janky.
And so what Jeff basically said is that, listen, you know, generating the text is a very expensive thing, but it's also kind of, you know, it's more complicated than you need. So why don't we, we'll read text and we'll have all of that kind of knowledge to read the text, but then rather than generating text, which is very expensive, we will just, if you give us a set of options, we'll choose the best option.
We can do that incredibly, we can do it incredibly fast, incredibly cheaply, but also we can do it with much more accuracy because we can train just for this. And so for all of the use cases that are not talking to a chatbot but are actually trying to put it in traditional software, this is a great fit. And so this has probably been the fastest adoption of an AI model since ChatGPT. It's just been remarkable because we're all primed for this.
I just want to pile on this one because I can't tell you how much I love seeing this exact form of innovation, and because what it does is it does the thing that's bugged me from the very beginning, which is there's been no user study ever that shows like interacting with the computer using full natural language is efficient. It's literally always the least efficient way, and it's very simple.
And it's just like ask yourself how many people are really, really good at asking questions, and immediately that's like less than half the people can ask a good question in a meeting. Yeah. And then how often do you look at the answer and get really frustrated before it's finished, but you have to pay all this money to watch the seven paragraphs come out and then apologize that it's only a little. And so that's one, like having a different model.
And then the other, of course, is my favorite, which is the output of it is designed for probabilistic programming. Yeah. And so instead of saying like, is this a customer service question, then route to customer service, otherwise route to general help desk or whatever, it's like, well, this is 80% customer service. And that's exactly simulation. And it turns out there's like 50 years of computer science research in literally like probabilistic if statements.
And so suddenly the coolest place to be in computer science is going to be in probabilistic programming, which was like all of computer science in the 1960s and 70s. So it was basically how do, because all of computers started with doing math and it was all simulation. So it was like, let's launch the missile and hit that target, but it's windy, but wind isn't constant. So like, let's model the wind and decide where to put the thrusters in order to do the arc.
And so most programming through like say 1970 before it got to accounting was probably... and then we ruined everything. No, but accounting, there's no probability in accounting, right? But most programming was basically this modeling kind of thing. And so most programming language design was trying to figure out how to put probability into if statements or into while loops, like do this until something happens, maybe most of the time.
And like, so my first CS class, like the very second assignment or so, was a simulation about like waiting online at a store. And I didn't know it at the time. I actually looked all this up when I was reading about Jeb, which was like the whole thing was my professor wrote the book called The Theory of Simulation in like 1960. And I just didn't know that because it kind of died by the 80s because it was all replaced by hyper. And so this notion of probabilistic...
And that slide deck you shared about the future, what's different about language models and stuff that you said was super good. Oh, Halper Flags, one phenomenal, Thomas, great. But the part that I felt was missing was that like, oh wait, this is not all new. Like all of computer science was this probabilistic stuff, and so it's going to be very interesting to dust off all of that work because it's exactly what's going on. Like it's not an if statement now, it's if X%, not if always.
And so the way that Jeb worked is just to like, you basically, it's a custom programming language almost, which is here's the prompt, come back with a percentage. Yeah, that's... and then you put that in the if statement. But the consequential thing is like finally we have a way to integrate these language models into a traditional software.
And I think it's kind of funny because it like, you ask the question like, why haven't the labs done this, right? And it's kind of like a, not an indictment, but a reflection on how they think. Like they're trying to create beings, and beings speak. If you're trying to create God, God speaks in natural languages or whatever, where this is really about something that's for traditional software, which is kind of not the direction that they've been taking.
But one of the reasons the uptick has been so dramatic is because a lot of us software people have been trying to integrate these models into software. It just hasn't. So even before you get to the probabilistic, like if I want a language model to drive an if statement, like it's really hard today. With this model, it makes it much, much, much easier. And then of course this could change the nature of software fundamentally to make it more stochastic.
Well, I think, but I absolutely, and I think that what's so cool is that it is happening outside the models because that's what I think is just going to happen, which is the center of innovation has just moved. Yeah. And it's just, and now people need to like, it turns out that the... I mean, outside of the lab, the big... the platform providers, 100%.
You know, basically reach a point of critical mass where the innovation stops happening at the platform layer. And then, you know, Apple, there's this famous expression in the Apple community called sherlocking, where Apple looks around and the things from the outside world become features, and people complain, and it's a real, but that's sort of how the innovation works because once you're a platform, you're overwhelmed.
No matter how many people you add, you're overwhelmed with just keeping the thing running and compatibility and stuff like that. And so I think that this is the signal that now people have figured out that there's innovation to be done to the model, but outside the model. Yeah. Yeah. Guys, thanks for coming. This is great. Okay, great.
