CEOInterviews.AIThe Ideology Behind AI | Robert Wright & Connor Leahy

C

Connor Leahy0:00

There's a fundamental tension in the heads of basically all of these rationality EA etc people which includes many of the best engineers at these companies. They want to build super intelligence. They've wanted to do it since they were like 12 years old. They've read sci-fi stories. And then the second thing is they are unfortunately smart enough to understand that doing that wrong would be a bad idea. So what they need is someone like a strong fatherly character, you know, in the psychoanalytic sense that tell them hush hush, don't worry. We're going to do it well. If you're in the in-group, we're the good guys. So, the messaging of Anthropic isn't for you. It's for future hires.

R

Robert Wright0:35

Anthropic does not want me to work for them. Is that what you're telling me, Connor?

C

Connor Leahy0:38

You're not polyamorous enough. I'm sorry.

R

Robert Wright0:40

Oh my god.

Hey, Connor.

C

Connor Leahy0:48

Hey, Bob.

R

Robert Wright0:49

How you doing? Been pretty great today except the wildfires here in DC have turned the city quite an apocalyptic shade of gray.

C

Connor Leahy1:31

Yeah, it is funny. My one of my best friends lives in one of the most beautiful parts of southern France and he always makes fun of me for how bad the weather is in DC. You know, it's hot, it's humid, you have horrible snow storms in winter and now we also have, you know, like smoke and hellfire. So, it truly is.

R

Robert Wright1:48

This is his lucky day. He should call you and ask you immediately. So, this will probably post several days after the day we're talking about and the air will once again be, you know, spring-like and clear in Washington DC. But anyway, you are Connor Leahy, well-known figure in the AI community, in part because you co-founded a well-known AI research collective called Eleuther AI. Maybe we'll have a chance to talk about that actually, eventually. It's relevant to the thing we are going to mainly talk about. Since you're at Eleuther, your concerns about the direction that AI is headed in have led you to become the US executive director of Control AI, a nonprofit which has a branch there in DC and also one in London I think. And it's a group that looks at AI policy and you do advocacy and you meet with politicians and various other people. In fact, the two people I saw later that day, I guess I won't mention their names, but both say they know you, that you've talked to everybody. This is impressive because these are like known people that I was talking to, more known than I am, and they know you. So anyway,

We are going to talk about the world view that is helping to shape the quote AI race because we're going to try to kind of flesh out the world views that prevail within the companies that are leading the AI race, notably Anthropic, OpenAI, maybe to some extent Google DeepMind, although it is only over the last few months it has come to seem to many observers, I don't know, but more like a two-horse race between Anthropic and OpenAI. These things change, who knows what Google will release tomorrow, but in terms of the two frontier models right now it seems like Mythos/Fable, Fable being the Mythos with guard rails that we hope will not be circumvented, but I think if I asked you would say guard rails often get circumvented. Right. And then Opus, not Opus, but ChatGPT 5.6 and OpenAI said those are the two. Before we get into the larger questions, I nodded toward those are like the two big models available to the public now. Right.

C

Connor Leahy4:33

Yeah. I think most people would say that those are the strongest currently available models.

R

Robert Wright4:38

And let's just spend a minute because the Fable Mythos thing got so much attention. First Mythos comes out and it's like this is a super hacking thing and there was argument oh that's just hype. I think there's more and more consensus that this has non-trivial cyber capabilities. It is still not really out, it's being used by certain institutions including the private sector to help tighten up their infrastructure before a model like it gets out or becomes so readily available that it's being used by the bad guys as they see them to break into their infrastructures. And then they released this guardrail version called Fable. The Trump administration basically took it off the market to evaluate it, then put it back on. What's your quick take before we get into the whole worldview slash ideology of these organizations, because that is a fascinating story that I don't think many people appreciate and you are very conversant in partly by virtue of your own history and the fact that you know these people and so on. But before we get into that, what's your take on the Mythos Fable thing?

C

Connor Leahy6:05

I think it's one of the most interesting things that's happened in AI in quite a long time, especially when it comes to the politics, but also, you know, speak a little bit about the ideology of the kind of people you're talking about. In many AI safety circles, there's this obsession with what they call the warning shot, where some minor catastrophe or moderate catastrophe will happen that doesn't quite lead to human extinction and this will make people wake up. Well, it has happened, and it was Mythos. As far as I'm concerned, we had the director of the NSA, the actual guy, General Rudd, director of the NSA, saying to the Gang of Eight, which are the top ranking members of the oversight committees, that Mythos was capable of breaking into almost all their classified systems in hours, not weeks. Now, a statement like this, it's hard to really overstate how much that is a statement that a national security person never wants to hear in their life.

R

Robert Wright7:00

And you think it was an accurate statement there? I generally think that when the director of the NSA speaks to the Gang of Eight, he's providing accurate information.

C

Connor Leahy7:11

Okay. I think there are nuances to this. There are always nuances. Of course there are. You know, it probably was as part of a red teaming exercise. It probably etc. But that doesn't mean that this wasn't a crazy thing to have happened. Since then, there's also been other crazy things that have happened. You know, recently the director of the CIA, not to be left out, said that it wouldn't be wrong to compare AI to digital nukes.

R

Robert Wright7:36

Like these are strong statements from some of the most informed and powerful people in the military and intelligence apparatuses. This is not random people. This is not doomers. These are the guys. And to say words this strong on the record, well, my interpretation personally is that it's much worse than we think it is.

C

Connor Leahy7:58

Like to get them to say that is probably means much worse things have happened. And you know, I can't say I know such things and if I did, I wouldn't confirm it. But this is not to be underestimated how big of a deal this is to get people to act this strongly on this. Two months ago, people would tell me regularly, all the time, Connor, you should give up. Why are you even trying? It's impossible. Trump said he will never

R

Robert Wright8:30

It being what goal that they say is impossible.

Oh, to regulate AI. They said it's impossible. And I should say in particular, one of your organizations' stated goals is to stop and even make illegal super intelligence, which is a thing we haven't reached. Maybe make illegal is not quite the way you'd put it, but that's you know, some groups are like, 'Okay, our main goal is pause.' I'm pretty sure that wouldn't bother you a lot, but your main stated goal is to stop before super intelligence, right? You're not trying to ban all AI or anything, but that's the thing you do want to ban.

C

Connor Leahy9:14

That is correct. So, we at Control AI are very much single issue in the sense that we are not against AI in general. We just want it to be controlled and in particular, we are against super intelligence. These are fully autonomous AI systems that are vastly more competent than humans across all relevant tasks. Such a system would outcompete humans at all relevant economic, military and political goals and tasks, and we would have no idea how we would control something like that. So such a system is so dangerous that it could lead to literal extinction, as has been warned by literal Nobel Prize winning scientists including some of the people who invented modern AI such as Geoffrey Hinton, who won the Nobel Prize related to this. So this is not a niche concern.

R

Robert Wright10:03

Okay. So you were saying when your friends, I think I interrupted you, when you were saying your friends say give up on this goal, Connor.

C

Connor Leahy10:11

Yeah. So what they say is, Connor, you should give up on trying to get regulation or so on because Trump has said he will never regulate AI. And I always answer, yeah, we'll see. And in fact, what we have seen is that the thing to understand about politics is that something is completely impossible until it very suddenly isn't. So what I find most interesting about the Fable and the Mythos saga is how quickly something that was considered completely impossible, unimaginable that the government would intervene on the AI market, can happen very suddenly when people realize, oh, this is a big deal. This is a very dangerous thing. What the hell is going on here? So I think there's a lot of stuff in the background that happened there. And I'm not saying all of it was super rational or well thought out or necessarily legal, but I think it's very indicative of seeing that the quote unquote adults in the room, the military, the government, etc., are waking up to, wait, what the hell is going on here? This is getting crazy.

R

Robert Wright11:15

Yeah. And I would say Trump then went on to take OpenAI's kind of rival of Fable, the aforementioned 5.6 ChatGPT 5.6, and took it off the market for a briefer examination, I would say. And I know what your view is on how close it is to being comparable to Mythos/Fables capabilities, but it's now on the market and people who really, connoisseurs like you, I assume you'd agree, say that yeah, Fable and most of them I think say 5.6 as well. This is a significant step. You just feel it. It's just a smarter thing even if you're not trying to use its cyber capabilities, right?

C

Connor Leahy12:08

Yeah. No, absolutely. I mean, the same way is the difference between Opus 3, Opus 4, then you go from Opus 4.8 to Fable. It's just much smarter. It is more capable. It can run for longer autonomous periods. It can perform more complex tasks. It makes better decisions more reliably. Does it make zero mistakes? No, of course not. It's not perfect, but it is better.

R

Robert Wright12:28

Yeah. Okay. So, as you said, this was in a certain sense some people's dream warning shot, right? I mean, because a lot of people, and look, I think you and I would both say that to really get the world to focus, you're going to have to have probably some subsequent dramatic developments. But it's true that people have long said, I just hope the catastrophe it takes to get the world's attention doesn't get a ton of people killed. Right. Well, this so far apparently got nobody killed. It was just a really well publicized capability. And I think personally it's one reason, and maybe the main reason, that Trump and Xi during the Beijing summit wound up initiating a formal dialogue on AI. So, look, that's a lot of bang per buck. Nobody dies. And you start moving toward taking this stuff pretty seriously. Even though, and I wrote a piece in the Washington Post about this, that look, it's not crazy to take a model of Fable's capabilities and take it off the market and take a good look at it. I'm afraid the government should have that kind of power. At the same time, going forward we need to have a much more systematized approach in this and it shouldn't be the caprice of whoever the president is. There needs to be a system. Congress needs to lay it down and I would say even possibly need to be involved in some sense in oversight of the oversight mechanism. But that's all. I guess we've waited too long. That's just my view.

We've waited too long to get to what I promised to get to, which is I don't want to speak ill of anyone when I say that some of the leaders in the field, Dario Amodei, maybe Sam Altman, and I'm not sure about Demis Hassabis, those are the leaders of the big three, as well as other super prominent important researchers. At least a fair number of these developed their worldviews in a milieu that I think many Americans if they knew more about would go, man, that's super weird. These guys, you know, wait, I thought corporate CEOs were these people who just joined country clubs or whatever. And even the Silicon Valley variety, it's like they're just these super ambitious entrepreneurs, founders, and so on. They're intense, blah, blah, blah. This was very, very different. It has to do with Eliezer Yudkowsky, now known as the doomer chief, and a culture he was prominent in when he started his LessWrong blog, which I think you were an early contributor to. And you know these people, have interacted with him. So, you're nodding your head as if I am saying something that you recognize. So, why don't you flesh that out? Tell us a little about the milieu that shaped some of the leading figures here.

C

Connor Leahy15:46

Yeah. So I think there's a lot of really fascinating history here because in many ways it often looks from the outside, and for some many people it is, that AI in particular AGI and super intelligence is like a pure

R

Robert Wright15:59

That's artificial general intelligence.

C

Connor Leahy16:02

Yes.

R

Robert Wright16:02

Those are a little hard to define super precisely but you've already done super for us and that'll do for now. Go ahead.

C

Connor Leahy16:07

Exactly. Like these terms change regularly. Nowadays we say super intelligence. Back in the day we said AGI, whatever, it doesn't matter. The big thing, the thing that can replace humans, that's what matters. And these ideas didn't come out of a vacuum. This is not like the ruthless market decided this is the optimal thing to go for. These are ideas that actually grew out of a very specific ideological lineage that I would say really started with what is called the extropians.

R

Robert Wright16:41

So. Whoa. Yeah, that's hardcore. There was a weird story a few months ago about some extropians who were involved in some weird crime, right? But we don't need to tar them all. Extropianism per se is a worldview. Now what is it?

C

Connor Leahy17:05

Extropianism is a particularly intense form of what's called transhumanism more broadly. Transhumanism is a very broad set of philosophies that in various ways are about the belief that humanity should be improved or transcended in various ways. This can be through technological means in particular, not just through meditating.

R

Robert Wright17:23

Yes. Generally especially by some of them are also very into meditating. Especially some of the newer ones are very into meditating. We'll get to that.

Yeah. Are you thinking of Beff Jezos by the way? He's like a classic example of the archetypical SF meditation accelerationist guy.

C

Connor Leahy17:42

Total accelerationist, like the faster the better, but he's always telling us about the Jonestown stuff he's realized, you know.

R

Robert Wright17:56

Yeah, we'll get there in the story. So like another thing to understand about San Francisco is that San Francisco as a place and Silicon Valley is a really weird place. And it really has, I would say, three main ideological strengths that kind of all mix and mutate in this stew. And these three are the transhumanist techno-utopian kind of visions, also coming from golden age sci-fi and stuff like that which we're talking about. The other one is just cutthroat capitalism, libertarianism. And the third is true new age communist, crazy hippie ideology. All of these are core to the soul of Silicon Valley. And if you neglect any one of them, you won't understand what's actually going on. So all of these cultures, of all these companies, of people who live there, of people who grew up there, should be understood as being influenced by these three large ideological clusters to varying degrees. Some people are way more into the hippie stuff. Some people are way more just capitalists that don't really care. Other people are really into the techno-utopian kind of vision of various kinds. And then there's the overlaps, like open source to a large degree is an ideology that's between techno-utopianism and libertarianism.

It has a lot of this like anti-authoritarianism while this belief that technology is good and will improve humanity, will improve the world, etc. So there's a lot of various mixtures of these ideologies to be had. Okay. And

C

Connor Leahy19:33

Yeah, so this is where I think the origin of this culture kind of comes from. And then the modern version of what you now see in many of these AGI religions or cults almost is coming from the extropians, who are a particularly intense form of transhumanism that kind of emerged in the 80s by a guy named Max More, I think was the one who first coined the term. Extropianism was all about exponential growth. It was this idea that we could get exponential amounts of energy and growth and intelligence and improve everything at unbelievable levels. Not all transhumanisms are that intense. Some are quite mild. They're like, 'Hey, we should have better medicine, brain chips,' and we use brain chips therapeutically for certain kinds of maladies, so it kind of happens. But there are these relatively mild forms. Extropianism is a pretty extreme form. It was one of the first to really seriously grapple with some very intense ideas such as whole brain uploading, immortality, and AGI and super intelligence as concepts. They were some of the first people to really think seriously about some of these ideas in the 80s and the 90s.

R

Robert Wright20:52

And so Max More and his wife in the 1990s got a visit from a 19-year-old Jewish boy named Eliezer Yudkowsky, who was on their mailing list. And who by his own account, if I've got it right, his parents were orthodox, but in any event it was a religious family. He, I think by his own account, has said science fiction kind of became his religion. I'm not sure if he said that, but anyway, he did not become conventionally religious. He got into science fiction at an early age. I like him. I've had him on the podcast. And look, we both talked about this a little yesterday. I think that he really deserves high praise. He has some very distinctive accomplishments. Maybe we'll get into that. He foresaw some things. But anyway, I know.

C

Connor Leahy21:43

Yeah. Absolutely. So Eliezer was someone who was very early on the extropian mailing list and so on, talking about these ideas of super intelligence. They didn't exactly call it that back then, but that's what they were talking about. This idea of if you had this recursively self-improving AI that could improve itself, you could very quickly build these incredibly powerful super intelligent beings that could cure all diseases, solve all problems, turn the whole universe into whatever nanobots or whatever, really intense sci-fi stuff.

R

Robert Wright22:17

And I don't mean sci-fi dismissively. I think there's a lot of valid philosophy that was going on here. I'm okay with philosophers talking about crazy things, right? For example,

C

Connor Leahy22:29

Doing them that you need to pay close attention. Exactly. I'm all fine for people thinking about crazy ideologies. Implementing them is when it becomes tricky. So growing out of this and Eliezer's interest in this topic in the late 90s and early 2000s, he actually splinters off from the extropians and creates what's called a mailing list called SL4, which is where we see some of the first overlap with some names you might otherwise recognize. SL4 was a mailing list for talking about the most intense, even more intense than extropianism, the most intense forms of future sci-fi speculation.

R

Robert Wright23:12

And what are some of the names we might recognize?

C

Connor Leahy23:14

Shane Legg, co-founder of DeepMind, Nick Bostrom, the popularizer of the term super intelligence.

R

Robert Wright23:21

He's been on the podcast and he wrote the big book Superintelligence which in turn shaped a lot of thinking. I think Shane Legg came up with the term AGI, didn't he?

C

Connor Leahy23:30

I think he did. Yes, if I remember correctly. Okay. So, go ahead. Any other big ones?

So, that's big. So, there were definitely some people around back then. There's also one person who was there who invented the precursor to Bitcoin, which was quite interesting. Wei Dai was his name. He almost invented Bitcoin. He was so close to it. There's also speculation that Satoshi might have been there or not. I don't know. I think probably not. The inventor of Bitcoin whose actual identity is unknown, whose name may or may not be anything like that anyway.

R

Robert Wright24:09

Exactly. So, you know, still that's why I say it's speculation about who was there or not. Yeah. So, there was a surprising amount of very interesting people that later became relevant on this mailing list every so often. And Eliezer then founded an organization called the Singularity Institute. This was in 2000, I believe, or 2001. So, very ahead of his time.

C

Connor Leahy24:33

And, you know, you might recognize that the goal was to build AI as fast as possible so that we could have AI that can take over the world and fix all the problems. That was the plan. And so, at this time, this is what really would nowadays be called accelerationism. So, you know, the EAs are little babies compared to Eliezer. He was 20 years ahead of them. So, in the early days,

R

Robert Wright25:02

The EAs meaning effective accelerationists. It's a play on effective altruism. They call themselves the effective accelerationists. It's kind of a joke, but the aforementioned Beff Jezos is probably an example. This Beth Jos character is also one. So, go ahead.

C

Connor Leahy25:15

Yeah, just like there's a broader category of accelerationism that I think is not specific to EA. It's more like a Twitter meme which I think died a year or two ago anyways. But there is a broader form of accelerationism which goes all the way back to Nick Land in the 90s.

R

Robert Wright25:31

Nick Land, who I gather we don't need to go down all these things, but I don't know anything about, but people who know go kind of like ideologically he went to some places that some people consider dark, I guess, and we can leave it at that.

C

Connor Leahy25:47

Yeah, I would say Nick Land is someone who tried to optimize to be as edgy as possible as a philosopher, to come up with the most repulsive possible ideology as he could, and he did a pretty good job of it.

R

Robert Wright25:58

Okay. Well, hats off.

C

Connor Leahy26:00

Yeah. But I think he had one insight that I do think is very interesting for the story, which is that in the very early days, Nick Land was a communist. He was a Marxist and he's not anymore. Oh god, is he not? But in the early days, the idea of acceleration, the word accelerationism actually comes from Marxism historically. So Marxist accelerationism is the idea that the thing a Marxist should do is actually do as much capitalism as possible as fast as possible because capitalism is inherently unstable. So if you accelerate capitalism, it will at some point collapse under its own contradictions and then lead to the communist utopia of some kind. So this is the idea of Marxist accelerationism.

R

Robert Wright26:51

Did the accelerationists have something they wanted to collapse? I mean, was it all the institutions constraining geniuses like them or what?

C

Connor Leahy27:03

The Marxists want to destroy capitalism. The modern accelerationists I think inherit that. Like what you're saying is that a lot of what modern accelerationists really believe is that they are extremely annoyed by governments, by laws, by anything that is holding them back. They often feel this is where the libertarian strength comes in. So the original Marxist accelerationists weren't really thinking about technology. They were thinking about destroying capitalism. And then there was this development in accelerationism as a concept. I think to a large degree, this is more my speculation, but basically Nick Land made a pretty good observation about accelerationism where he speculated that in Marxist theory, as you expand the amount of capital that exists and you have less and less labor, more and more capital as part of the pie chart, at some point labor would revolt and the communist revolution would start. But Nick Land speculated, wait, what if that doesn't happen? What if capital just keeps expanding indefinitely? What if there's just no more labor and there's just only capital? What then? And this was in the 80s and the 90s, and he basically predicted AI and AGI. What if you could just have sentient capital that could replace all labor? If you can substitute labor with capital, then labor is no longer necessary. And this is then where things take a very dark turn in Landian philosophy and get very dark because he basically then concludes that this is good and all humans should die. So,

R

Robert Wright28:32

Yeah, and there is well we need I don't go off on that tangent, but anyway, so Nick Land was a prominent person in this. And this was still like an email list among other things.

C

Connor Leahy28:45

I don't think he was ever on the email list. He was kind of more like a parallel offshoot. It was kind of like a parallel idea. There are many interesting people I'm also leaving off here, but yeah, so basically

R

Robert Wright28:56

You know, I gotta say, for people who are younger than me with great literary ambitions, there is a great intellectual history to be done about all this, right?

C

Connor Leahy29:05

Oh yes, oh my god. Like there is that will be unlike other intellectual history. People say where are the Parisian cafes of the 30s or whenever these people for Sartre, maybe the 40s or whatever. But this was a very influential set of very unusual ideas that took shape under the surface, under the table. I don't mean they were hiding it. It's just we didn't know. I mean, I guess you could say Wired magazine was a certain oblique manifestation of some of this and that got people's attention. I remember when it showed up. But now all of a sudden these guys are ruling the world. Anyway, go ahead.

I mean, fun fact, one of the first people to give Eliezer money was Peter Thiel.

R

Robert Wright29:58

I know. And then of course, Eliezer did the 180 and became not a techno optimist, but quite the opposite. And hats off to him for turning his back on the people who supported him. But that's where his ideas took him. So, okay. So, I interrupted you. You can proceed with narrative.

C

Connor Leahy30:25

Yeah. And so, as you just said, Eliezer, to his incredible credit, I really want to give him credit here. There are many things to disagree with Eliezer about, but I think it's immense credit to him that, I think 2004 maybe, I don't know exactly the timeline, Eliezer realized, oh, I made a mistake. How do you control super intelligence? What does that even mean? If you build something that smart, by default this does not go well. And so he pivots entirely to thinking about how do you build safe AI. Back then it was called friendly AI, but what we would call safe or controllable or aligned.

R

Robert Wright31:05

Aligned. Yeah.

C

Connor Leahy31:07

Yes. Alignment became the term.

R

Robert Wright31:08

Can I just ask you quickly? I assume there were science fiction examples to build that worldview on. Obviously science fiction writers had said, right, this AI thing, you wind up ruling the world. So I guess he had reference points to latch on to.

C

Connor Leahy31:28

Yep. There is a lot here actually. So I was quite, I remember reading when I read Eliezer's work. So after his come to God moment about the risk of AI, he started developing this ideology that he would call rationalism, for better or for worse. And a lot of this ideology had a lot of interesting aspects that I found very novel as at least as a teenager. I'd never heard of these things before. But very fascinatingly, I then read one of his favorite 1950s sci-fi books, which was called, I think, The World of A or something like this. I'd never heard about it before. It's quite obscure. And it's about a guy who has learned a kind of mental martial arts where he can rationally solve problems very quickly using systems of rationality. And I'm like, wow, that seems familiar. So a lot of this is drawn from golden age sci-fi ideas, also stuff like Ian Banks, but also stuff like J. Good, for example, who is one of the first people to talk about the intelligence explosion, recursive self-improvement. He was a sci-fi author. So there's a lot of, I think it's worth not dismissing here just because it comes from fiction doesn't mean there isn't interesting philosophy.

R

Robert Wright32:46

Absolutely not. I mean, these people, some of them were visionaries, some of these science fiction writers. So you were in Eliezer's world before he even turned to pessimist, or this is around the time you started paying attention.

C

Connor Leahy33:00

I unfortunately was still too young to be on the internet at this time. So this is before my time. I have just done my archaeology. I have done a lot of internet archaeology on this kind of stuff.

And so there's the interesting part of the story which is usually left out which I think is very important to include. So you said this kind of earlier, like oh there's all these things going on that we just didn't know about and it wasn't like they were hiding it. Well, you would be wrong. They were in fact hiding it. So this was a thing in the early days of this movement. Very early on, there was a strong part of the ideology that is don't tell people, don't let the stupid unwashed masses know about this. Don't let the government know about this because they're crazy, they're stupid. They don't understand it. We the smart elites who are super smart, we're going to solve the problem. We're going to solve everything. We just have to make sure that all these dumb stupid normal people don't get in the way. So this is the dark ring on what actually there's the positive and there was actually discussion of this on the record because it seems like about a year ago there was a controversy, an intramural controversy about this among people from this milieu, right? Wasn't there kind of an argument? Some stuff came out or somebody said here's this email and here's that and there was a whole thing. This feels like it happens every six months. There's a lot of infighting inside of these milieus. And like, I'm also part of this is that one of the reasons I have broken a lot with this group of people is because I think it is extremely bad to hide this kind of information from people in the government. I think people are allowed to have insight and make decisions on this thing. Doing this unilaterally is deeply immoral. And so this is a thing where, and I'm sure we're gonna get into that, where the libertarianism kind of really mixes with this techno optimism where this idea that the few elites, the John Galts, the Randian Ubermensch in Silicon Valley who are so much smarter than all these other people, have to just sit in a basement and figure it out and make sure the government doesn't get in the way. And this will rear its head many times in the future as well. So I think this is a very important part of the ideology to understand, is this worship of intelligence, this worship of the lone genius hacker, the lone genius founder, the Randian character so to speak.

R

Robert Wright35:34

Yeah, it's funny. You know, I co-founded this platform called Bloggingheads.tv, one of the first video dialogue platforms in 2005. And a guy who was helping me line up people brought on people like Eliezer was on even before I did a podcast with him in 2010 or something, but I think by 2006 or 2007 he was on. And I just remember I'm like, who the hell is this? And I got to admit, like many people, I didn't entirely warm up to his persona because I felt he was partly, he's a total autodidact, okay, he taught himself, and as happens with these people sometimes, there's an idea that a lot of people already know about, you don't have to give it a new name. Right? Like I remember back when I was 18 and I thought I had discovered utilitarianism. No, it's not a new idea. Anyway, so by that time he was turning and he becomes the doomer chief. Meanwhile, what the branch that he branches off from eventually begets, I guess you would say Dario, maybe Sam, I'm not so sure about him, you might say Shane Legg, co-founder with Demis of DeepMind. You're nodding your head, so it looks like. So go ahead and let's stick with that branch and keep telling the story.

C

Connor Leahy37:01

Great. Yeah. So, second half of the 2000s, a couple really interesting things start happening. One thing is that DeepMind is founded. DeepMind, including seed capital from Peter Thiel, which was introduced by Eliezer. Eliezer introduced Peter Thiel to Demis and Shane to get some other initiative. I think this is something a little bit mythologized. I don't think all of the money came from Peter Thiel or this one conversation, but it still was a pivotal moment and shows this overlap between DeepMind and Eliezer and the race dynamic thing. And so the ideology kind of believed at this point in time, I don't want to say any specific person believed this, but the ideological belief was something like, still DeepMind is good because if we could just put all the smart people into one company and keep the government and everyone else away, then we can do it right, then we can do it properly because then we will have control. And this is where this idea really came into being. And I've talked to people who were there at the time and they all believe versions of this. I don't want to attribute it to specific people because these are private conversations, but these people really believed a version of as long as we the good people do it in a secret company project, then it can be good.

R

Robert Wright38:22

Pausing here just for a second to compare this to other historical examples such as the Manhattan Project, because this is kind of what happened in a sense. It's kind of like Leo Szilard, or maybe Shane Legg is like Leo Szilard. Szilard is the physicist who is maybe the origin of the letter that was ultimately sent to Roosevelt that Einstein signed, that said we've got this, if you're in the middle of a war maybe you should build a bomb. Basically what happened was that before Szilard, famously Lord Kelvin, a very famous scientist, said that the idea of getting net energy from fission is moonshine, it's impossible, it's silly. And Szilard was so upset by this that he immediately came up with a design for how to build a nuclear bomb and how it would work. And he basically realized, he wrote it down and it just worked out and he was like, 'Oh, this will work.'

C

Connor Leahy39:15

And so what did he do? Well, he ran out and started a startup to have a competition to build as many nuclear bombs as possible at an affordable price to sell them to as many people as possible. Actually, in that case, there's reason to think that this one may not work out as well as that one works because with nuclear weapons, at least deterrence turned out to be this fundamentally simple dynamic. I don't want to get into that, but it's a different kettle of fish. The reason we haven't had a nuclear bomb go off since World War II is easier to understand than to understand how this thing, if it continues to be a headlong race, won't go awry.

R

Robert Wright40:00

But in any event, so you were saying that the people who, I guess the most obvious comparison, you don't want to talk about individuals I'm sure, so you and I won't even tell people whether you're nodding your head although we are on video, but a name that comes to mind with me is Dario Amodei and for that matter Sam Altman. I mean, they both profess to be very concerned about the possibility that yes this could lead to human extinction and they are both just moving as fast as they possibly can toward this fate. All the while saying that there's something about their approach that's not going to lead to the bad outcome or something about where they're leading us that's not going to lead to the bad outcome. Even though I don't know, you tell me. Anyway, you pick up the story here. Do I have that that's kind of the comparison?

C

Connor Leahy40:52

Comparison. So I think we're starting to assemble all the ideological building blocks that lead to the rickety edifice that we're at now, where there's one block missing which is kind of like Oxford University, utilitarianism, Peter Singer, and MacAskill, which led to what is called effective altruism. So there's another strain of thought.

R

Robert Wright41:17

We should say Peter, who's a pretty good friend of mine, he should be absolved. I mean, I'm sure you'd agree that he doesn't get into this stuff. His idea of effective altruism is be very careful about which philanthropy you give your money to. I mean, but he would say, 'Look, by my lights, it's better to save 10,000 kids in Africa with mosquito nets than to build another art museum.' You can argue with that, but that's a very different kettle of fish. And his version of EA is like, 'Let's do those kinds of calculations.'

C

Connor Leahy41:56

Exactly. Okay. Yeah. Exactly. So, the original effective altruism is exactly what described and I think is really not that controversial. Like the idea that we should just evaluate which charities, for example, help the most people. I think is a very reasonable thing to do. So this is why it's interesting because in a sense there's a bait and switch where you read the head on the tin what EA says and you say well that seems very reasonable, but then if you open the tin you're suddenly going to see a lot of crazy stuff. You're suddenly see your anthropics and your whatever, people saying that actually replacing humans with AI is good maybe, or that we have to race to ASI, or many many crazy things. How did this happen? And I think this is very interesting where what happened basically is that this pretty non-objectionable version of EA merged through this shared link of utilitarianism with rationalism and the libertarianism of Bay Area Silicon Valley culture to create kind of the new mutant ideology. And this grew out to a large degree from one of the largest and most important early EA organizations was called GiveWell. So this was an organization that evaluated charities for effectiveness and would help you make better donations, which I think is a very reasonable thing to do.

R

Robert Wright43:17

It was founded by a previous hedge fund manager named Holden Karnofsky. And Holden later then got to know one of the Facebook founders, Dustin Moskovitz. Dustin Moskovitz decided to donate a very large amount of money to a foundation that Karnofsky would lead and this became what is called Open Philanthropy. So it is now called

Now funds for example what range of things? Interesting if you look at all the places its money has gone or is going, it's maybe not a group you, I mean what would you say are some of the highlights of where its money has gone?

C

Connor Leahy44:00

It's gone to OpenAI, to Anthropic, to METR, to which is a safe, well yeah, I mean it does these evaluations, these interesting and useful gauges of the progress of AI, and then some AI safety organizations. Yes, lots of things are known as AI safety. What is important to understand about Open Philanthropy is that it is the beating heart of the financial system of EA, the engine that kind of fed effective altruism to a large degree in Silicon

Valley specifically came through Dustin Moskovitz's money being funneled through Open Philanthropy. It's hard to overstate how big of an influence they were. It's not the only funder, it's not the only thing that was going on, but it was truly the 900-pound gorilla.

R

Robert Wright44:56

And then Carnovski, the I guess former hedge fund guy. He married Dario Amodei's sister. Is that correct? And she now is a well, I forget her title at Tropic but...

C

Connor Leahy45:06

Yes, in fact Dario and Holden were roommates for over 10 years. So before any of this, Dario and Holden...

R

Robert Wright45:15

Why didn't I have a roommate like that? I know, right? You should have been on the right email list. So here we are talking to each other, Connor, because neither of us had a roommate like that.

C

Connor Leahy45:24

Exactly. Exactly. So I think it's really interesting because their third roommate was Paul Cristiano, the inventor of RLHF and a very famous...

R

Robert Wright45:34

An inside name, but in the early debates about AI, he and Eliezer are engaged and so on, and he did not follow Eliezer into doomerland, but is a respected intellect in the realm, I guess.

C

Connor Leahy45:51

Yeah, exactly. So he invented RLHF, the technique that allowed ChatGPT to be possible.

R

Robert Wright45:57

Reinforcement learning through human feedback.

C

Connor Leahy45:59

Exactly. Which allowed ChatGPT to be possible while he was at OpenAI. He later founded an AI safety organization called ARC, and then he became head of the Center for AI Security in the US government. I don't think he is that anymore, but he was for a while.

R

Robert Wright46:16

Okay.

C

Connor Leahy46:17

So he's kind of one of those background characters that actually had quite a lot of influence. And so what you'll see is that the same names keep popping up quite a bit. You can kind of pick and choose almost anyone, there will be a connection to Open Phil or to MIRI, or Eliezer to DeepMind. There's a lot of connections going on. These people all know each other. And this is why I think it's so fascinating, is that there was an AGI superintelligence scene long before the technology existed, long before it existed, and it overlapped very heavily with these other adjacent ideologies like rationalism and effective altruism, and so on. And this world is extremely incestuous, extremely incestuous.

R

Robert Wright47:04

At this point we're talking metaphorically, presumably.

C

Connor Leahy47:08

Hopefully.

R

Robert Wright47:08

Uh, yeah. I mean, the only reason I said this because I know there is this, and this doesn't involve literal incest, even but there is this sexual dimension of the culture of at least some of this. I don't want to say everybody, but this has been common. I mean, for example, the polyamory thing, and I had somebody on to talk about this practitioner and member of the rationalist community years and years ago. But that became intertwined kind of with rationalism in the Silicon Valley econ or in the San Francisco Northern California context. And here is where the third route comes in that I pre-warned earlier about the hippie stuff. This is where it comes in.

C

Connor Leahy47:54

So, and then the missing route to understand this culture is just the absolute most insane, degenerate, out of all containment cult stuff. Like, I'm not saying, I am actually saying this derogatory to a certain degree.

R

Robert Wright48:12

Kind of like you're headed there. Yeah.

C

Connor Leahy48:14

Go ahead. Like what I'm saying here is that consenting adults, I think consenting adults should be allowed to join cults. They're still kind, you know, like I think it should be completely legal for a consenting adult to join a crazy cult. I will never stop them. I might advise them against it. But that being said, it is hard to understand for people who have not been there how culty literally all of this is. And what, what is there? Like we've talked about a number of you know, we've talked about altruism, rationalism, extropianism.

R

Robert Wright48:38

What can you better define the subculture or the fusion of these, the intersection on some Venn diagram that is the universe you're talking about now?

C

Connor Leahy49:01

This extended cluster that is located primarily in Berkeley and San Francisco.

R

Robert Wright49:08

Okay.

C

Connor Leahy49:08

That's the primary part. There is just an extended scene. It's not one ideology. It's a scene. Some of it is EA, some of it is rationalism, some of it is just AI company employees. Some of it is like literal magical alternative healing practitioners are part of this culture. There's a lot of stuff.

R

Robert Wright49:31

That seems not consistent with rationalism. I mean, who knows what's out there in this crazy universe. You could, I guess, but...

C

Connor Leahy49:36

Well, so this is where we get into the cult thing. So one of the things about cults is that no cult calls itself a cult, and smart people like to think they can't fall for cults. And ironically, this actually makes a lot of smart people very vulnerable to cults. So a lot of people who think they're not vulnerable to cult is a risk factor for getting caught into a cult. So what has happened in a lot of these areas is that a lot of these people, one of the things to understand about this extended scene that doesn't really have a name, but the extended cluster of all of these adjacent ideologies and beliefs and so on, is that they have extreme levels of openness. They're willing to try anything, and they think they're immortal. They think they're like Ayn Rand characters, they're so smart, they can solve anything. So for example, drug use is extremely prolific in these things. They will take psychedelics to try to get new insights, they will try crazy meditation techniques. On the tin it says rationalism, but if you go to these actual people, you will see a lot of really esoteric things. These are not like boring conservative people. This is where the hippie alternative medicine meditation culture really comes in and mixes with this to create this strange comp.

R

Robert Wright51:13

And now that I've heard where you're going, I would say meditation there is, and now I can see more an actual rational kind of approach to certain kinds of meditation and so on. I have a better idea where you, and you know who knows what not yet recognized herbal medicines have effects. So you're not talking about telepathic communication, although maybe that too, maybe entanglement or something. But anyway, okay, so...

C

Connor Leahy51:47

Yeah, if you want a telepathy story, I recommend looking up Leverage Research. It's too long for a story for right now, but basically it was a cult that kind of helped start effective altruism and then claimed to have developed telepathy and ultimately spiraled into demon possession and stuff like that. It's a crazy story. We don't have time for it. It's kind of peripheral, but it's a great read.

R

Robert Wright52:11

So, let me ask you, what are the features of this group that make it a cult? Generally, when I think of a cult, I think of blind obedience to somebody or something, for example, and then there are other things. And there are a lot of traits you could attribute definitionally to a cult. What is it you're saying about this group that you think merits that term? I mean, I guess part of it you would say is the aforementioned, I guess you're saying it's a little weird to start out trying to shape the future in a positive direction technologically, which they some of them would say their mission was, and to wind up doing whatever they did. I mean, well, I should also say I got you off on this tangent by talking about sex. We haven't talked about that, but there is the polyamory, right? Like, okay, we can switch mates at the drop of a hat, and regular people may not succeed at this, but we're so rational that things like jealousy won't get the better of us or something.

C

Connor Leahy53:18

Yes, there are literal whole blogs of people talking about how they have used rationality to turn off their jealousy, or there's a term called 'eye hacking' because it's irrational not to like men. So just hack yourself to like men. Why not? It's rational. So this is a real thing. And for me, this is very culty. Whether or not you're following a specific guru, this is messing with your mind in these various ways, part of a controlling social group that has its own weird lingo and weird kind of self-harming activities. What I think is useful to talk about this is not to see it as a traditional cult. A very traditional cult, you have one leader to obey. Don't get me wrong, that does happen. You have your Leverages and your Z and your Z, which are classic cults. But I think there's a new generation of cult that we've seen in the internet age where there's not one specific leader, but still it's self-sustaining. We've also seen this with like 764 cults recently. In case you haven't heard of that story, maybe don't look it up if you're having a bad day. It's one of the most horrific things to have happened in the last years.

R

Robert Wright54:30

Are those the guys who killed people or tried or something?

C

Connor Leahy54:34

It's a group of online groomers who try to get children to kill themselves.

R

Robert Wright54:40

Oh, even worse than the one I was thinking of. Okay. Yeah. Let's get back on the main narrative.

C

Connor Leahy54:46

Yes. So I guess you'd say, I mean, what it sounds like one reason you're using the word cult is like, wait a second, supposedly this was a kind of ideology, granted, but it was supposed to be focused on this technological thing, and there were certain assumptions about the way technology would play out, maybe ways you could help play out that way, and suddenly you've got these seemingly extraneous cultural expressions that don't have any obvious connection to it, and yet they are kind of part of binding this group together.

R

Robert Wright55:20

Is that what you're saying?

C

Connor Leahy55:22

I'll go even further. In the sense that what I mean by cult is really that there's a certain kind of thing that happens: if people go too crazy, if you just put people in a room too tightly, they form a... like, you know what I mean? If people are packed too tightly, they go a little bit crazy as time goes on. And there's a similar thing: there's just certain patterns that emerge.

R

Robert Wright55:44

We should say we are not talking about every single individual we have mentioned who has some connection to this.

C

Connor Leahy55:50

No, by no means. I'm not saying any of these people have a connection to all the things we're talking about, and to varying degrees they probably left the part of the culture you're talking about and have nothing to do with it. So no individual indictments are to be assumed here.

R

Robert Wright56:07

Exactly. No individual indictments are to be assumed here. Most of the people that are involved here are people that the general public wouldn't even know. But yeah, so there's this idea, this thing that happens where people just in practice, empirically, redevelop certain social controlling structures if they are put in certain circumstances. Like controlling each other's speech, struggle sessions is a recurring theme. Where someone has sinned against the group and then other people sit them down and scream at them, which is a thing that these people love to do. It's very common.

C

Connor Leahy56:41

Is that right? I mean, those are associated with Chinese communism, maybe Cultural Revolution, but they do it. You're saying they...

R

Robert Wright56:49

They don't call it that, but it happens. So one important thing, for example, is that a lot of these people live in group houses. It's often seven people in one house that all are part of the same culture. They all live at the same company, they all date each other, often multiple people at the same time, they read the same news sources, they have the same lingo. They often have a large, and this brings in the hero myth from earlier, is that they're often very xenophobic towards people who are not part of the culture. They're often very dismissive: the outside world doesn't get us, they don't really understand the true thing that's going on, we're the real rational people. Which is classic cult talk. You are the true understanders, you have the true knowledge. What is curious about this is that this cult just happened to actually have secret knowledge that turned out to be true. That's what makes this one interesting.

C

Connor Leahy57:40

You mean they understood better than most what could happen with AI?

R

Robert Wright57:45

Yes. This is what I think is so fascinating here. There are a lot of things wrong, but fundamentally they did have secret knowledge which did turn out to be true. If you were in the cult sphere in the early 2000s, you did know things about the future that other people didn't know. So this was very intoxicating to people. This was very, very intoxicating to people. And so, all things equal, you have a lot of young vulnerable people potentially, and you have a few people or field groups that have all the money. You have your Open Phil and whatever that have a lot of authority. You have your Eliezers and so on that have a lot of authority and recognition. It's just like weird things happen, is how I would describe it. I don't think anyone here had bad intentions per se. I don't think any person involved was like, 'I'm going to go create a cult and harm people.' I think it's just an emergent thing that happens as part of culture. If you do crazy mind-altering substances and practices and you're sardined in with eight other people that are all part of the same culture, you just kind of develop your own insular culture. And I think this is what happened to a large degree. And again, I'm not saying by any means that everyone in Silicon Valley or at these AI companies is deeply part of this culture, but you'd be surprised how many are. You'd be surprised how much this culture has been very influential on the thoughts of people. Even the accelerationists who like to present themselves as the anti-EA, they're part of the same culture. They're the same.

C

Connor Leahy59:16

Effective altruists, you're saying?

R

Robert Wright59:17

Yeah. Like these people are the same. They like to say that about themselves, but you're saying they have a certain amount in common culturally.

C

Connor Leahy59:23

Yeah, absolutely. Like people like Bezos or whatever, it's the same thing. There's like two bits of difference, but almost everything is the same. It still comes with this techno-utopian vision, this vision of superintelligence, this openness to a lot of alternative strange things like meditation and so on, which I'm not saying is necessarily a bad thing. It's just empirically.

R

Robert Wright59:46

I should say I'm pro-meditation, as some people know.

C

Connor Leahy59:48

I am too. For what it's worth, I meditate every day. I'm not against it. I just think meditation is something to do responsibly and can be misused. So maybe at this point I should say, okay, we've said we're not talking about everybody, and it's an admittedly not formally defined group of people that you're saying are in some sense part of a cult. So the question is then, what are the parts of this milieu that, leaving aside the question whether any one person at Anthropic, OpenAI, DeepMind is or was associated directly with what you're talking about, independent of that, what is the legacy of this cluster of worldviews in Northern California that took shape? And of course with any intellectual history you run into the same problem. It's like, oh, you mean this kind of Marxist or that kind of Marxist? Which of the subgroupings of the subgrouping are you talking about? So it's always vague, but you're saying there was something discernibly coherent in Northern California that is there still, and you can see its legacy in these frontier AI labs, right? And how would you, what is the most important part of that for our purposes?

R

Robert Wright1:01:19

Yeah, I agree exactly with what you said. Like there's the saying the only thing a leftist hates more than a right-winger is another leftist, right? And so this is definitely true. All these people hate each other, to be clear. All these people hate each other. Of course they do. And I'm being very broad, and they're all going to hate my description of them. But I think what's the interesting legacy is kind of the things I was talking about earlier. Basically what happened was that Eliezer in this world did not go to the government, did not go to the military, and instead decided to keep it secret and keep it private and start the race to create private companies. And here's where I think the core problem is: this is a market failure. This is a classic market failure. There is a class of things which we know markets don't solve. Markets are great, I love markets where my food comes from, they're very good to solve certain types of coordination problems. But we know there are other classes of coordination problems that markets just do not solve. This is well-known basic economic game theory.

C

Connor Leahy1:02:29

Public goods versus private goods, positive externalities. Yeah. It's like this is not controversial in mainstream economics what you just said.

R

Robert Wright1:02:37

Exactly. So very standard. And the problem of public security is not the kind of problem that markets solve well, and the problem of superintelligence risk is a public security problem. It's a public goods problem. It is not a private profit problem. So the fundamental mistake that was made, fundamentally and the reason to a large degree we're in this mess where now we have a billion companies racing to build superintelligence that will kill everybody, is because they tried to solve a public goods problem with private markets, which doesn't work. And it never has. And this predictably so. I'm not saying the government is awesome and everything is great. It's just on pure standard game theory, we know how this is.

C

Connor Leahy1:03:22

And let me get back. I mentioned negative externalities. This is a problem that is created by market dynamics that markets do not address. Classic example is pollution. Car companies compete to get the best car out there, and since no one car buyer is absorbing the pollution, the pollution is just expelled out there. Their car is not a particular problem for them. It's just that all cars collectively are a problem for people collectively in terms of the pollution they create. That's a negative externality. And I think you'd say an AI arms race leads to risky behaviors among other things. That's an example of a negative externality that is not going to be addressed by the market. Quite the contrary, the market seems determined to create as much AI risk as it can by itself. Although there are constraining factors, these companies want to be known as safety conscious, and that leads to a certain amount of behavior that is actually virtuous and to some that is virtue signaling. We won't get into that yet, and maybe if you're up for it, this is going to eventually require a second conversation to explore all of this. But anyway, that's my little aside that is an example of what you're talking about, right?

R

Robert Wright1:04:44

This is not a problem for the market alone to handle, what the problems AI creates for all of us.

C

Connor Leahy1:04:51

Exactly. And this is like, I love complex crazy stories that have a simple punchline, which is what I think this is. I think fundamentally it all adds up to normality. It's just this was never going to be a problem that can be solved by private markets. The fact that they believed that this could be solved with private markets was ideologically driven. It was not rationality driven. It was because they had this libertarian ideology, this idea that the government is a hindrance to geniuses like them, and only if it would get out of the way, these problems would be solved. But no, in fact, it doesn't work for coordination problems. Lone geniuses do not solve coordination problems. I'm sorry. It's just not how these things work. And there's this idea often that is still very common, especially in Silicon Valley, that markets are the solution to everything. And I'm not saying markets are bad. What I'm saying is they are one tool in our tool belt, and this is the wrong problem to be using them on. We have other tools. We should use them.

R

Robert Wright1:05:48

Okay. So you would first of all, before we talk a little more, I know you have to go at some point. You'd be up for a second conversation. It warrants delving into all this because it really warrants it. Because in a way, we haven't gotten to what I suspect you'll agree is something very important, which is a vision of this superintelligence thing and what it's going to be and what's going to be wonderful about it and how you could keep it under control and couldn't keep it under control. Because there are really important engineering questions raised by the worldview. Why don't you say a little about this? The plan, I mean these people have a plan for it working out well in their own heads. They're not sitting there thinking, 'I'm proceeding recklessly, this is going to lead to some extremely dystopic sci-fi scenario.' They think they're going to play a big role in it not happening, right? And that has to do with the way the whole thing was conceived back in the day, 15, 20 years ago. The way they were thinking about this thing, which for all I know is true. But the point is it took shape in this milieu, right?

C

Connor Leahy1:07:32

Yep.

R

Robert Wright1:07:33

And what would you say about that?

C

Connor Leahy1:07:35

I think this is extremely right. I think that these ideas really formed in this very self-reinforcing, intelligence-worshipping, hero-worshipping environment where the hero CEO, the Ayn Rand protagonist, will be the one to solve it. But this is just not how the real world works. If you want to solve large-scale coordination problems at scale, you need boring governments and diplomats and bureaucracies and stuff like this. And I'm not saying it's the best system, but it is the best one we have come up with so far. Democracy is the worst system except for all the other ones. The idea that individual heroic CEOs would somehow be responsible for global security and international stability is crazy. This is actually a crazy belief, and it's so normalized in these cultures. It's like fish in water. People are just like, 'Oh, if only Sam Altman did the right thing.' And I'm like, 'Whoa, whoa, whoa. You're begging the question. Why are we asking Sam Altman? Why is this hero CEO worship going on? Sam Altman is not the general. He's not the president. He's not even part of this.'

R

Robert Wright1:08:48

I had a really funny conversation once where I met someone in Japan who had grown up in Japan, and we were talking about AI safety and superintelligence. I was explaining effective altruism and superintelligence risk and all this kind of stuff to him. Extinction risk from superintelligence all made sense to him. He was like, 'Makes sense.' There's one thing about EA that he just couldn't understand. He was so confused. He's like, 'Okay, makes sense. AI poses a risk to humanity. Makes sense. But why are charities involved? This should be the military. What is a charity doing here being funded by some billionaire? This is crazy.' And I think he's right.

C

Connor Leahy1:09:30

I mean, I will say in theory, charities can address things markets don't address. And in fact, that's almost definitionally true. But what's odd about some of these charities is that they wound up giving money to the companies, right? Like Open Philanthropy. I'm not passing any judgments. I don't know the landscape well enough. But it is an odd thing that a philanthropy was, and of course people know that OpenAI has this odd history where it starts out as a right-minded NGO and nonprofit and winds up a capitalist machine, somewhat I guess the same as Anthropic. But anyway, that's like the corporate behemoths of previous technological revolutions don't start out as charities, right? That's just weird. And it shows in a way something commendable, which is that they understood that they were dealing with something different that did pose distinctive problems. And I think they may have, I don't know. Go ahead. Tell me what you're thinking.

R

Robert Wright1:10:44

I think people are complex. I think people are inconsistent. I think people change their minds. Yes, I do think a lot of people tried and failed in many ways. Charities can address many things that markets can address. Yes. But they also can't address military problems. That's not what they're for. Public security problems should not be solved by charities. There are reasons we don't do that. We don't want charity police, we want state police. There's just different categories. And so, the thing with the charity is that again it's a private thing. It's a Randian thing. It's this idea that you have this private entity run by this genius CEO that will solve the problem separate from the state. There's always this fascination with avoiding the state, circumventing the state in various ways. And we could do a lot of psychoanalysis about why these people don't want to engage with the state. They're so traumatized by interacting with the state. And I think there are many legitimate reasons to be traumatized from interacting with the state, especially in certain countries. I understand. But fundamentally, this is a psychoanalytic reason that led to actual tactical mistakes by their stated reasons. And then it comes back rearing its head as one of the roots of the ideology transhumanism. I think there's a much simpler explanation for all of this. All of these strange things can be explained with one very simple observation. These people want to build superintelligence. That's the fundamental observation. They've always wanted to build it. Dario wants to build superintelligence. If you just take this one sentence, Dario wants to build superintelligence, you can predict basically all of his actions. Whenever Anthropic suggests a new policy or supports something, ask yourself, does this stop Dario from creating superintelligence? And the answer will be 100% of the time no. It might do other things. It might be good press. It might help chatbot security. I don't know. But it will never ever get in the way of Dario creating superintelligence.

C

Connor Leahy1:12:54

But sometimes it seems like it could have that effect. Like they did this paper, Anthropic came out with this paper authored by Jack Clark, this guy at Anthropic, I think a former journalist, who writes a lot. And it was about recursive self-improvement, which is of course as you alluded to earlier, the idea that you get to a point where the AI is just each generation building the next with no human in the loop. Anyway, at the end of it they say, look, we may get to a point where it makes sense to slow down, or even maybe they said even stop, I doubt it, but anyway, and that may require global coordination, and so we're going to establish some sort of process through which we study this, and I guess we'll get back to you. Maybe that's the key part where they say we're going to start a committee. We'll let you know when we find something out. But in any event, ostensibly at least, they seem to be focusing on a problem that you agree is a problem, and even saying we may do this thing that keeps Dario from building superintelligence. That's what it seemed like. So I think this is a perfect example. Did publishing that paper stop him from building superintelligence? Yes or no?

R

Robert Wright1:14:12

I don't recall that exact phrase.

C

Connor Leahy1:14:15

Of course not. It didn't slow him down or harm him or get in the way of Anthropic in any way. They are hiring people on their website. They have job postings for people to work on RSI. Do not listen to people when they speak. This is a very important skill to learn. When you're dealing with adversarial environments such as politics or CEOs, you have to watch the hands, not the mouth. They will say whatever. It doesn't matter. If you want to hear X, they'll say X. Someone else wants to hear Y, they'll say Y. They will just contradict themselves on everything. So the thing you have to watch is what do they do? What are the invariants that their actions uphold? Another great example is responsible scaling policies. This was the previous Anthropic thing.

R

Robert Wright1:15:04

Anthropic's, right? The dear departed responsible scaling policies. They formally abandoned the term, didn't they?

C

Connor Leahy1:15:13

Yes. And so when it came out, I was like, this is obviously not binding. There's nothing binding in here.

R

Robert Wright1:15:19

So these are things like, okay, these are things we won't do out of concern for safety. And I forget how they define them, but this will keep us honest, so to speak. They pledge to you that our machines won't do this, will do this, and they just...

C

Connor Leahy1:15:33

I was surprised. Wouldn't it be smarter just to redefine them than abandon?

R

Robert Wright1:15:38

So, here's the thing. You think that's what they said. That's not what they said. I would recommend reading the paper. That's not what it said. That's what the headline said. But the actual paper was basically, we will evaluate what a future moment could look like in which we might have some restraint, some restrictions of some form.

C

Connor Leahy1:15:55

We'll get back to you on that.

R

Robert Wright1:15:56

Then two years later they came back and were like, 'Ah, actually we're not going to do this.'

C

Connor Leahy1:16:01

So here is a very important thing we haven't talked about so far, which is I think a skeleton key to understand what's going on here. Fear, uncertainty, doubt, FUD. This is one of the most classic corporate strategies and marketing tactics, goes all the way back to big tobacco and so on. There's even a whole Wikipedia page of the big tobacco playbook.

R

Robert Wright1:16:23

When they faced the cancer stats and wanted to evade the implications. Yeah.

C

Connor Leahy1:16:29

So here is how you should understand when you hear people like Dario or Demis or Sam or whatever talk about this. What you have to understand is their goal, they have one goal, which is to stall for time. Because they think they can get to superintelligence in a couple years, and then none of this matters. All they need to do is stall. They don't need to convince you. They just need to make you think, 'That was interesting. They'll get back to us. Let's wait and see.' They just need to make you confused.

R

Robert Wright1:17:01

It's like ink cloud, like an octopus shooting ink. It's just confusion. All this stuff, 'Oh look, they put out this policy, wow, they did this thing, this guy said this on Twitter.' Wow, look at all the shiny jingling keys.

C

Connor Leahy1:17:16

It seems like if I were Anthropic and I had the motivations you suggest, I might be more inclined to say, 'This thing is a long way off. Chill. We're not going to get to AGI tomorrow or super.' But their estimates actually tend to be, same with OpenAI, on the near side.

R

Robert Wright1:17:39

Yes. And the reason for this is because you're not the audience. They want the AI engineers to come to them and pledge allegiance. They want the AI safety super high IQ LessWrong people to work for them as engineers. So they have to signal to them. They have to be like, 'Look guys, you want to build superintelligence, right? So we can do it safe.' Because there's a fundamental tension in the heads of basically all of these rationality, EA, etc. people, which includes many of the best engineers at these companies. And the tension is fundamentally they want to build ASI. They want to build superintelligence. They've wanted to do it since they were 12 years old. They've read sci-fi stories, they've read Iain Banks or whatever, and they want to build superintelligence. And then the second thing is they are unfortunately smart enough to understand that doing that wrong would be a bad idea. So what they need is someone like a strong fatherly character, in the psychoanalytic sense, to tell them, 'Hush, hush, don't worry. We're going to do it well. If you're in the in-group, we're the good guys.' So the messaging of Anthropic isn't for you. It's for future hires.

C

Connor Leahy1:18:47

Does not want me to work for them. Is that what you're telling me, Connor?

R

Robert Wright1:18:50

I'm sorry to say, but...

C

Connor Leahy1:18:51

Look, I know a lot about AI. I could help. I could help.

R

Robert Wright1:18:54

You're not poly enough. I'm sorry.

C

Connor Leahy1:18:56

Oh my god. I'm flexible. I'm flexible. Just tell me what lifestyle changes I need. So, okay. Well, there's so many places this could go, and so I'm glad you're up for a second conversation. I mean, one thing ultimately, I think in the next conversation I'm going to review and recap and try to flesh out, get you to say a little more about some of the connections between the emergence of this culture you've described and the cultures in the companies that are different from one another to some extent, but the legacy of the cultures in those companies and how that matters. But I also know that one thing that matters is this specific superintelligence scenario, where it gets back to what we said about alignment. It's like this idea that if we build the right kind of superintelligence, which means it's built by the right kind of person, which may mean me, you know, we all think we're good, right? Honestly, I think I would be responsible if you put me in charge of superintelligence. I'm not kidding. We all think like that.

R

Robert Wright1:20:10

And there is this assumption, first of all, there are all these assumptions about what can and can't alignment do. Will there be this kind of one machine that as long as you keep it in line you're good? That'll get us back to open-weight open source, which you have changed your views on since you were an important member of that community. So there's so much to discuss here. And I want to give you a chance to say any last thing you want to say. But first, I want to just say I also want to get more into the cultural stuff because I so distinctly remember when Wired magazine emerged and there was the techno-futurism, transhumanism on the cover and stuff. But what's funny is there was also the very much this granola vibe, back to earth, living off the land. And look, I know Kevin Kelly. I really like him. I think he's a good guy. He didn't go off to become part of what we're describing, but he was one of the important members of the community. Of course, Stewart Brand, Whole Earth Catalog. This is before your time. You don't remember Wired showing up, but I imagined the Northern California hippie libertarian vibe that was clearly connected to the tech world not winding up at Palantir. Although actually now there's open war between Palantir and Anthropic. So never mind. Forget I said that because Palantir is not really a direct offshoot. Let me just shut my mouth and give you a few minutes. Say anything you want to say, and I hope that within a couple of weeks we'll tape a part two that reviews, fleshes out, and embraces and extends, as Microsoft described its plan to kill Netscape back in the day. But anyway, go ahead.

C

Connor Leahy1:22:11

I think the most important thing is again to understand that fundamentally they are trying to stall. These people think that they are building a technology that will kill literally everyone on the planet. They will kill you, they will kill your children, they will kill your spouse, they will kill everybody. And they think it's worth it. And they think they have the right to do this. What I think is that they don't. Previously you said that everyone thinks that they would be responsible. I have thankfully grown out of this phase. I do not think that I should be in control of superintelligence. I don't think anyone should be. I don't think this is a decision that anyone gets to make unilaterally. I think even if I built an aligned ASI, whatever that means, I think that's a very bad thing because it means that I'm disempowering other people against their will. People have not consented to this. If I build a superintelligence, that means I disempower literally everybody. I can disempower all the governments, all the private people and their private lives, everyone, the families, etc. And I think power imbalances are very dangerous. Concentrations of power of any kind are very dangerous, and a superintelligence is the most extreme form of that because you don't even need a human in charge anymore. You can just have the power run itself. So what I think is that this is why I stopped working on the technical problems, because I realized even if I succeeded, I've not succeeded. In the sense, I've made the problem worse. The actual problem we are facing is a political problem, not a technical problem. It is the question of how do we want to govern and how do we want to make choices like this as a society. How do we want to decide what experiments get run, what kind of technology is deployed or constructed in the first place, what kind of risk is the general public and our families exposed to? And these are choices that I think individual private people at companies should not be allowed to make. These should be decisions that are made democratically. These are choices that should be made by the people. And therefore, this is why I am part of AI, controlai.org, where we just really encourage people to reach out to their lawmakers, to make their voices heard that this is not acceptable and that this is not what people want. It is in fact not what people want. People don't want unelected billionaires in Silicon Valley gambling with the lives of them and their family. No one wants this, and we should demand that action, and we should make sure that that action actually happens.

R

Robert Wright1:24:45

Okay. And one thing I would say in kind of wrapping up is, tell me if I'm wrong, but you're not just saying that it's your view that superintelligence is this super dangerous thing that no one, that a handful of people with in some cases possibly eccentric worldviews especially should be in charge of. But you're also saying, and this is what I think connects it to the story, you're saying that these people themselves believe, at least according to their own past professions, superintelligence is this sci-fi level incredibly dangerous thing. I mean, it would be one thing for Eliezer to be running up to somebody who runs IBM and saying, 'Don't you understand these things are dangerous?' and the guy's like, 'Kid, I'm just focused on next quarter's profit, shut the hell up.' You're saying these people actually emerge from this culture where they believe this, at least in some cases they believe as strongly as anybody, and they're saying, 'But don't worry, I'm the guy who can make this work out well, trust me.' And again, I know that's not what they're explicitly saying. That's why I brought up that they say this, they say that. And you're saying basically that's PR. But did I just accurately characterize an important part of the characterization between the situation we're in in your view and this culture in which a lot of important people in AI emerged?

C

Connor Leahy1:26:15

Yes, I think that's absolutely the case. I mean, Dario has said before that he thinks it's like 20% up to 20% that AI could kill everybody. This is worse odds than Russian roulette.

R

Robert Wright1:26:26

By one chamber. Yeah. Okay. Well, thanks so much. And again, it's controlai.org. Also, do you spend much time on Twitter anywhere you want to advertise your own thing?

C

Connor Leahy1:26:41

Yeah, I am reluctantly on Twitter. You know, @mpcollapse or just my name Connor Leahy. Other than that, I also have a blog which is linked from there where I post very infrequently.

R

Robert Wright1:26:54

Yeah. Twitter, by the way, Elon has his own fascinating connection intersection with this technology. He didn't emerge in it, but...

C

Connor Leahy1:27:00

Yeah, I didn't even touch on it.

R

Robert Wright1:27:03

And I'm Robert Wright on Twitter, and I also have a book on AI called The God Test which came out not long ago and actually gets into a lot of this, especially the Bostrom book Superintelligence and Eliezer's worldview. And I contend that in some cases I've made Eliezer's arguments more persuasive than Eliezer. Judge for yourself. While being technically agnostic on whether he's right or wrong. But thanks, and I'm glad we're going to see you again soon on this platform, Connor, because this is very important stuff.

C

Connor Leahy1:27:45

Thanks so much for having me.

The Ideology Behind AI | Robert Wright & Connor Leahy

About Connor Leahy