We've seen incidents in which, you know, groups of misaligned AIs all work together to accomplish malign outcomes. Most notably the Hugging Face incident. Suppose that we proceed on what the current default path looks like in front of us. Maybe there's about a 50 or 60% chance that misaligned AIs would end up taking over the world, and then if that did happen, there would be a, you know, significant chance that many or all humans would die.
I am here with Ryan Greenblat. Ryan, thanks for joining me.
So, you're the chief scientist at Redwood Research, which is one of these AI safety research firms that analyzed the recent Hugging Face fiasco incident — a terrifying phenomenon. We'll get into it, but how did you come to this work? What was your path to focusing on AI safety?
Yeah. So, in college during my junior year, I was sort of in my apartment alone because it was, you know, COVID, all classes were remote. And I was sort of in a contemplative mood, thinking about what I should do with my life, listening to various podcasts. And in one of those podcasts, someone made an argument that was like, you know, basically I would summarize the argument as like, it doesn't make that much sense to be selfish because, you know, what's really the distinction at a material level between your future self and other future people. And I was like, that argument kind of makes sense to me. I should really consider how I want to lead my life and what I should do. And ended up getting pretty sold on being much more focused on altruism. And then after that I spent a bunch of time thinking about what I should do with my life. Eventually ended up deciding that working on technical AI safety was like a very important thing to do, was like a very critical issue that we would face. Got sold on those arguments applied to various places and then started working at Redwood, where I've been working on AI security and AI safety research for about five years.
And is your background in computer science?
Yeah. Computer science, also some math.
So it sounds like you might have come up through the effective altruist community. Is that — I mean, do you consider yourself a part of the EA community?
I would say that I'm definitely within the EA community. I'm not sure I would self-identify as an EA, but that's I think to some extent just because I'm a bit reluctant to self-identify with any label that's going to associate you in some — I feel like I don't know if I like — when I think about myself, I don't think like, I'm an EA. I'm sort of like, yeah, I have a bunch of properties that other people in that community have. I'm in touch with that community, but I wouldn't necessarily say like I'm an EA per se. I do think that I'm interested in effectively pursuing the impartial good.
Right. Right. Well, as you know, and as I've talked about in the podcast of late, effective altruism has come in for some abuse. Much of it unfair, some of it might be fair. And there's a — I think there's a reason why I, like you, have not hung up an EA shingle in my life. Perhaps we can talk about that, but it sounds like — well, you tell me, where on the spectrum of concern are you? I would put the most worried people over there with Eliezer Yudkowsky and his recent co-author Nate Soares. Maybe Max Tegmark is over there. Nick Bostrom is not quite over there, but still on that side of the spectrum. And then all the way on the other side opposing them, you have the people who say that there really is no real concern here, that it's all a hoax, that any thought that artificial intelligence might get out of our control and destroy us, that is a weird form of marketing being employed by the frontier labs. It's an effort to get regulation imposed and achieve something like regulatory capture, or it's an EA scam. And there are people like, you know, I would say Marc Andreessen or David Sacks or even the president himself, who's recently called it a hoax in so many words, who are just fundamentally not worried about any significant downside risk here and just see dollar signs stretching out to the horizon. Where do you locate yourself on that spectrum?
So I would say I'm very concerned, but maybe more optimistic that we'll make it through without sort of averting the course of AI development. So I would say that I'm sort of like — suppose that we proceed on what the current default path looks like in front of us. Maybe there's about a 50 or 60% chance that misaligned AIs would end up taking over the world. And then if that did happen, there would be a, you know, significant chance that many or all humans would die. So I think that's — I would say I'm very concerned and don't think the situation is on track to go well. And I would also note that in addition to sort of these misalignment concerns, there's other risks with building extremely capable AI systems, especially around sort of concentration of power and, like, you know, who controls these systems. And one answer is no one controls the systems, they're misaligned. Another possible answer is the power is very concentrated in a few people and it sort of overturns our institutions and our ability to have a broad distribution of power, democracy, etc.
So what keeps you from being among the absolutely most worried? What do you disagree with them about, or what do you think they're getting wrong?
Yeah. So, I would say that it seems pretty plausible to me that if it turns out that the technical alignment problem is somewhat easier, which I think is plausible, and we do a reasonably good job aligning and utilizing AI systems up through roughly human levels of capability, we could then get those systems to automate technical safety work. And that could potentially sort of get us into a positive feedback loop where these systems make themselves more aligned. They produce a new version of the system that's even better at carefully figuring out what to do, better at trying to pursue our intentions, and that sort of is self-reinforcing rather than getting worse and worse over time. And that could be fast enough to keep up with the rapid growth in capabilities. So that's one reason, and I would say this comes down to sort of more optimism about prosaic or relatively empirical and iterative methods. I wouldn't say I'm hugely optimistic about those methods, but I think that there's a decent chance they're working. It's just that when I say a decent chance they're working, I also implicitly mean a decent chance of, you know, losing control over the future. And I'm more like 50/50 on that. And then I also think it's plausible that we'll develop very, very capable AI systems and the worst misalignment concerns will mostly not materialize, even with not very advanced methods, even independent of this automation. I think that's a minority, but it's possible. Like, I don't think there's extremely strong reason to believe that when building significantly superhuman systems, those systems would be so misaligned that they would take over. I think there's definitely pretty strong evidence pointing in that direction, and I think that case gets more concerning the more capable these systems are. But I think that I could imagine basically people not really taking much of a precaution, proceeding through the AI development trajectory, patching problems as they come up, and that ending up getting you very high levels of capability while things are still fine, and then there would be time for society to react.
So you've mentioned a few numbers here with respect to probability and just kind of gestured at a range of likelihood. I'm wondering how people should think about statements of that kind. I think it seems to me that any actual probability we would assign to this is pretty much made up. Maybe you have a more rigorous way of making an estimate here, but whatever the estimate — I mean, unless it was infinitesimally small, like, you know, well below 1%, which is really never the number that you're hearing. Some people will say 10%, 20%, 30%. I mean, that seems to be the — you know, you just talked about 50% takeover. I don't know how that translates into the ruination of everything. But these are enormous numbers. And so in the normal case, if we were developing a technology where the people who were closest to doing the work said things like, 'Yeah, I think maybe there's a 10% chance we're going to destroy the world here on our present course,' the only rational response to that range of outcomes is you stop immediately, right? I mean, if the Manhattan Project scientists said, 'Yeah, we've run our calculations, we got the smartest people in a room together, we thought about it, and there's a 10% chance that when we execute this first test at Alamogordo, we ignite the atmosphere and destroy the future,' the only sane response to that is you don't do this initial test. But that doesn't seem to be what's happening here at all. We have a lot of people saying that the probability of some extremely bad outcome is quite high, certainly within range of the roll of a normal die or even a coin toss, and yet the work is proceeding more or less under an arms race condition at full pace. We'll talk about the recent statements of Dario and others that we could somehow pace this better than we are. But I mean, it just does not seem like the response anyone would have if they thought the probabilities of doom were really that high.
