When AI's builders sound the alarm

Arya, welcome back from a summer extravaganza of sabbatical with the family in Europe. It's awesome that you're back.

You know, as you know, what I tend to say is, hey, it's less about acceleration, deacceleration, and that actually frequently is a false dichotomy of various sorts, and either one, especially an extreme, being deeply wrong, but more about steering. Now, part of steering is sometimes when you get to a particular curve, you slow down when you go around that curve, but you're slowing down with a purpose of where you're going to. Like, not 'I'm just slowing down, I'm kind of going around now.' You know, obviously you push the metaphor and you say, well, if you're heading towards a cliff, you stop, right? If it's a cliff. But this kind of gets to kind of what are the actual ways I think we should analyze, you know, kind of what are the call it safety concerns. The first is kind of existential risk. And I think that actually in fact the kind of the X% in a decade, etc., etc., are all in the kind of histrionic category, and they may be in some cases deeply well-meaning, like, eg, I don't know how to call the percentage, and it's a little bit like, you know, you're exploding the Hiroshima bomb and it's going to split the earth's core and we're going to be one molten, you know, thing, and I don't know how to articulate percentage so I just put a solid number on it as a way of doing it.

Yes. Exactly. And so whereas, you know, when I kind of chart out the existential risk, I kind of go, look, there's only a very specific set of scenarios where it gets there in that kind of time. There's multiple checkpoints along the ways. There's multiple things we can do. There's multiple telemetry. And so I tend to think that the X-risk is the way of kind of saying the existential risk is the way of saying, look at me, look at me. It's really important. This is a key moment in history and it's really important you talk to me about this. And so I broadly across the whole set, including the Anthropic people who are driving the frontier along with OpenAI on this, which is, you know, part of the, you know, well-commented irony of all this thing, I tend to think that's well-meaning, but the sky is falling, the sky is falling, the sky is falling. Now we get to the three real ones, which is biorisk, cyber risk, and jobs. Biorisk is the closest that you get to kind of existential risk of various forms. And by the way, it's doable without AI. It gets amplified with AI, but also the only defense to biorisk, as I can say, is AI. And so one of the reasons why you kind of go this pause stuff is malformed, the deaccelerationist, the extreme deaccelerationist is basically by being ahead and by formulating the defense is the basic way to counter the biorisk side. So that's the first reason why I'm kind of pause opposed, right, and steering, steering with a general sense of acceleration and speed, not the kind of precautionary principle.

The second one, cyber, is the one that we have real line of sight to. We know from Mythos and we know from other things that there is like a real upping of the arms race in capability. And part of the problem is we have so many frail systems, and the frail systems can be, you know, hospitals and utilities and banking and a whole bunch of other things. And again you say, well, why not pause? It's like, well, the real risk from cyber is the open-weight models and their deployment in ways because, yes, you know, it's like, hey, we haven't trained the frontier models that they are perfect and never do anything wrong on the cyber stuff, and we can come back to the Hugging Face side. That's not that. That's, by the way, there's one way of telling the Hugging Face story, which is sure that was a kind of startling way of doing rewards hacking and all the rest, but also what they had is bad password security that the AI discovered. It was not undiscoverable by a human. It wasn't a superhuman cyber hack. Right. And by the way, that means that it's demonstrating what the open-weight models, which are not that far behind, six to 12 months, you know, can be doing and can be doing in the privacy of iterating offline in a non-connected cluster and then be used whether it's by rogue state actors, whether it's by criminals, whether it's by crazy people, you know, just people who want to up the ante. And so that we know, we see it. It's line of sight. It's not like, oh, let's speculate. It could be A, it could be B. Let's tell some science fiction stories. It's there. Once again, when you say, well, why do you say not pause? Why not deaccelerate? It's like, well, what we need to be doing is taking the fact that we have a lead on frontier models and doing the hardening and the rewriting and the defense side of this in a really intense way. So that's another reason why I tend to be a smart accelerationist, right? You know, strategic optimist about how to do this.

Now the third, which is, you know, kind of roughly, you know, kind of Bill Gates most recently said, hey, we don't have anyone working on, you know, what the job transformation will be. We don't know what the job transformation will be, but it could be kind of sudden and quick and important. And I think that's right. And you know, we use careful point out that there's all these positive features around, you know, kind of AI for this kind of, you know, kind of what can be done and going to agriculture and a bunch of other things. And I think that people should be thinking about what is a combination of two things on jobs. One is, you know, kind of telemetry, like where do we see that it's actually in fact, you know, measuring where it's actually beginning to really happen, and we should go to that area, customer service, sales, etc., and really try to help a lot in particular. And the second is what does a tool set look like? And on the key thing on the tool set is, and I think there's a few people who have begun working on this. You know, former Secretary Gina Raimondo is doing some really great work here. There's a few people, but one of the key things that I think is lacking here is your solution has to have the same speed, scale, cost of the problem, of which the only thing I see right now is AI. So you need to have AI on the ledger. It has to have the same impedance match. The AI has to be on the solutions ledger. So it's like, what are we doing to shape AI to be helping with a job transformation? Now, part of the reason I don't say, well, why don't you just flow or why don't you just do the Bill Gates human reserve thing where 90% of human beings are going to argue my job should be in human reserve? Hell, venture capital should be in human reserve. You know, like everything should be in human reserve. And you know, so it doesn't really solve any particular problem other than you might slow a specific thing down for a bit. You know, it's kind of like the equivalent of like autowork changing and all that kind of thing. But I think that the question is like, okay, we're going to have job transformation. It's going to be people don't per se like they say I have a job, I don't want job transformation, and it's like, yeah, you know, look, same thing for the weavers in England. You're going to have job transformation. How do you make that easier? And I think there's two vectors. One vector which is really under-commentated is that you must be giving people something that they're at least getting something. It's not like, oh, you're going to create job transformation that's going to make your tech companies rich and give me a bunch of problems that I don't really want and I'm getting nothing other than having to scramble. Of course people are very opposed to that pitch.

So you say, okay, how about a medical assistant? So only really, really wealthy people have 24/7 good medical assistant. How about that for everybody? And how about that for free for everybody in terms of navigating the stuff? And it's you, your kids, your parents, your pets, the whole thing. Like, okay, how about a legal assistant, right? Oh, most people can't afford lawyers to look at their rental agreement, to look at these other things. Now you have that. How about, you know, educational tutors? Not obviously help a lot with not just having an AI system to help you do new jobs, but also to help you learn stuff and help you find things. And by the way, how about, you know, AI for government services so you don't have to go to the DMV anymore? You know, it can all be handled. So all of that I think is really important to go, oh look, I'm getting some things. I still am unhappy about my job transformation, but I'm getting some things while I'm doing it. And then I think the other part of this is to say, how do we kind of put in some fact that we're, you know, the advantage, this is why the industrial revolution parallel to the fact that we're leading to say, hey, let's create some incentives for tech companies creating things that help, you know, your kind of middle-class Americans, you like make the job transition work well, right, in terms of things. And you know, that will be invented as we go. You'd like to know it's going to be exactly this. It's going to be like we're going to have care jobs, and the care jobs for 30% of the people coming out of sales will be helped this way. And then we're going to have, you know, kind of like other kinds of like, you know, kind of like context awareness and kind of AI training jobs, and that will work. The problem is, look, you could say those things. The real thing is we have to be adaptive in terms of where we're going. So I think that's the key thing. And I'll say one last point. Part of the reason why when it gets to the jobs thing, I'm still a, you know, kind of a smart accelerationist, a kind of a steering accelerationist, is that when America is getting, or country of your choice is getting the prosperity of having intelligence at the scale and speed and cost of electricity applied to a whole bunch of different things, that creates the economics by which we can sort this problem out. If we don't have those economics, then we're unable to sort the problem out. And you know, this isn't a, hey, you know, help the tech people get rich while everyone else suffers. I think it's really important we have this dual mandate where like 60% plus of Americans are having challenging times right now. We need to be helping them. And you say, well, what's a return to manufacturing jobs in the US? It's AI and robotics. It's the only way we're going to get there. So as kind of an instance probe, look, if you look at the data centers, all the unions are going, this is creating a bunch of great, great union jobs. We're positive on this. We want to do this. This capital inflow gives us a bunch of work while we're doing it. That's actually really important. And that kind of prosperity is an important thing for the 60-plus percentage Americans who are, you know, in turbulent times right now.

You know, maybe for the first time in a decade, I partially agree with Sacks. I mean, since most of his activities have been pretty, shall we say, suboptimal for society, to be polite. And you know, he's also kind of like as a kind of a sociopathic political actor. You know, those concerns, those concerns. But what he's right about is to say, look, that the people who are sounding the alarm are the people who are out on the frontier and in the front. So you go, okay, why are you saying you need us to slow things down? Because you could slow down, right? And if you wanted to put a particular point on it, is say, how about we say we define the frontier where you're at and we say this is the point where everyone has to slow down and gives other people a chance to catch up. Are you happy with that? And the answer will be not really, right? And so that's, I think, the substance of his position. Now, that being said, I think part of the more detailed thing is to say, look, part of it is there's this question about, you know, OpenAI and Anthropic not just being the world leaders, but also national champions, and to say if you're doing a slowdown, you should get at least China bought in in some kinds of ways. And so it doesn't really make sense given that China's, you know, call it six to 12 months behind us to kind of do a unilateral slowdown just for that. And so it's a call for, you know, kind of international treaty on this, which, you know, Sacks and his normal like completely political, you know, call it disinformation, it makes it entirely an internal thing versus actually in fact this is what they're substantively trying to kind of say and kind of call for. And that is a thing that fits within it, and it's not an antitrust law, it's a question of international treaty. And you know, this is what both Anthropic and OpenAI have been calling for for a year or two in terms of, you know, and then Demis came out recently also in this as a way of navigating and in particular with recursive self-improvement. You know, that's, I think, the thing that brings the urgency for why they're calling for it again, because you go, well, we, you know, recursive self-improvement, we don't know. And this is part of gets back to, you know, Hubinger's, you know, kind of call is like, well, we don't know where the recursive self-improvement fully goes. It's a unique thing. And it's one of the things where you say, well, your steering and acceleration and deacceleration metaphor has some challenges because it's like the car is doing itself accelerating and it's accelerating and it kind of taking over the steering a little bit as it accelerates. So what's that? And it's like, well, and we don't know how to 100% like completely do that. Now we still do know how to pull the plug, you know, like requiring a kill switch, requiring telemetry, those are good things for governments to do. This is one of the things that, you know, I've been trying for eight years to kind of say this is how governments should plug in. And it's not the same kind of thing as regulatory capture. It's not the same thing as slowing everybody down because those things can be done with de minimis, you know, kind of like it doesn't really affect your speed at all and gives you at least some ability to do that. And so I think those are, you know, good things to do, but it's, you know, here, you know, historic moment, you know, I don't 100% disagree with, I double-digit percentage disagree with Sacks and disagree with his actual intent here by mischaracterizing the other side. But, you know, like there's a point here, right? There are some things they can do.