The Bitter Rivalries Behind the Race to Build Superintelligence | Plain English

Thank you for having me back. Always a pleasure.

Super intelligence you could think of as an AI system that is smarter than everyone at everything. There are sort of these gradations of AI capabilities. People talk about AGI, artificial general intelligence, as a computer program that is human level or better at most tasks or most intellectual tasks. Super intelligence would be the level beyond that where we have something that is vastly smarter than us at everything that we could conceivably do.

Well, let's back up more than 12 months to 10 years ago when the book starts. Which is, we had these AI systems. They could do things like identifying objects in images or translating between languages or helping you with autocorrect on your email. But those were narrow systems. They could only do one thing. They couldn't be repurposed to do 20 different tasks. Now we have these general models, near or above human level. We have systems that are capable of solving millennium prize problems in math, of discovering new protein structures, of helping people, you know, billions of people around the world every day in various ways. So I think that's what I've seen, is just the continued march of this technology. And then the non-public thing that I've seen is that the people at the companies who are building this stuff — I guess this is not a secret anymore — are terrified of it. They have been for years, and they're not getting any calmer as we approach what they think could be some kind of intelligence explosion, which we could talk about. That's sort of another concept that they think is not far off, and I'm inclined to believe them.

He has this superpower which everyone he's ever worked with, all of his mentors, they have recognized this in him, and he had it from quite a young age, of figuring out exactly what someone wants and giving it to them, or at least appearing to give it to them. So back in his days running Y Combinator, people would sort of talk about this power of his, and they didn't know where it came from, but people like Paul Graham recognized it very early. He was very good at reading the room, at figuring out, you know, this person expects me to do this or wants me to do this, so I will do that. And it led to him becoming one of the most successful young entrepreneurs and investors in the Valley. The flip side of it is that he would often get into trouble because he would tell one person one thing that they wanted to hear and then tell a different person something completely different that they wanted to hear. And he sort of got this reputation as a sort of slippery people pleaser, which is, I think, a criticism that many people who have worked with him over the years have identified and have made in pretty consistent ways. He just seems to be in the business of both being quite ambitious but also of being quite eager to please.

I believe he has a core. I don't know what it is. I'm not in his inner circle of people that he confides in. I've spent time with him. I've talked to him. I've had encounters with him as a journalist, but I've always sort of understood that he was playing the part in those interactions and wasn't necessarily showing me his true self. And for what it's worth, I think, you know, most executives do this. They have kind of their persona that they put on. I think the thing that kept coming up in reporting on Sam and the people around him is that he is a person who believes first and foremost in agency, right? He has this famous phrase, "you can just do things," and this is something that he believed at Y Combinator. He would tell these young startup founders that he was investing in, like, you don't have to wait to change the world. You don't have to, like, nobody has to give you permission. You can just go do things. And that is a core part of his belief about himself and about those around him. Like, you could, another way of saying it is, like, there's no grown-ups. Like, you are the grown-ups, and you can just do things that change the world without permission. He's also a fervent believer in scale, the power of scale. You know, people would tell me these stories about, you know, he would walk into a meeting and he would just tell everyone, you know, tell the people in the meeting to, like, add a zero or two zeros to whatever budget they had for their project going forward, or, you know, increase the scale of a training run, just because, like, he thought that bigger was better. So I think that love of scale is probably part of his experience running Y Combinator and being an investor before OpenAI. But certainly OpenAI's success has confirmed that just having this kind of delusional self-belief and belief in the power of scaling is actually what you need to win in AI. Because this is something that didn't really occur to me until midway through the book. Like, belief is an active ingredient in AI success. You have to believe that the scaling laws will continue, that putting billions or trillions of dollars into building data centers and acquiring compute will pay off. Like, you have to, like, the belief is the enabling force that allows you to win. And if you don't have it, no amount of technical expertise can save you.

Yeah, it's an incredible story, and it hadn't been reported before, and I was sort of delighted to find it out because it does really illustrate who these people are. So, they go up to Redmond for the day from San Francisco, Sam and a few other OpenAI executives, and they've got this, you know, very high-pressure meeting with Satya Nadella and his leadership team at Microsoft. And they've got this new model, GPT-2, that is still, it's not good by today's standards. Like, you would never use it for anything that you needed done, but it can sort of complete a sentence. It can sort of generate coherent text, and it can do all of this, like, more flexibly than any of the chatbots that had come before it. So they go up, they run this demo, they bring a Microsoft laptop borrowed from one of the OpenAI executives' girlfriends because they're like, Microsoft will, you know, will think more favorably of us if we're using Windows. So they borrow a Windows laptop, they run, they go up, they run the demo, and they're showing some sort of canned prompts that they've prepared that they kind of know GPT-2 will be able to do. And then Satya Nadella asks, you know, can we try, can we see this thing? And another Microsoft executive sort of wants to try their own prompt that wasn't part of the canned demos. And so they're like, they're sort of looking around each other. They're like, can we do this? Can we do this? Yeah, I guess we have to do this. So they put in a prompt that's like, you know, Satya Nadella was hired as a CEO of Microsoft and the first thing he did was blank. And they hit enter and they're sort of like sweating as they wait for this thing to come back. And then it spits out the completion, which is that on his first day, Satya Nadella fired all the employees and pivoted the company to Azure, their cloud platform. And this draws a big laugh, and it's sort of an amazing moment in time. Like, an AI model has never been able to do this before. And it was very high stakes. Like, had this sentence completion not worked, they might not have gotten the billion dollars from Microsoft. But as it turned out, that was, you know, exactly what they wanted to see, and they invested a billion dollars, and the rest is history.

Do you remember Tay? It was like a glorious...

I have a thing for deranged chatbots, Derek. And Tay was one of the more deranged ones we've had. It was this, like, experimental chatbot that Microsoft built, and they put it on, like, Twitter, and people just started, like, messing with it and, like, turned it into, like, essentially a white supremacist within, like, 12 hours. It was, like, spewing, like, Nazi propaganda. And so the lesson that Bill Gates and other Microsoft executives learned from that is, like, don't make chatbots. That's, like, that's a bad idea.

That would never happen.

Yeah, it's an incredible story because it truly was an accidental hit. The people at OpenAI, as you said, many of them didn't think this thing would be very popular. And I think to explain why, it's like we had to sort of step back in the time machine to, like, the sort of summer and fall of 2022. At the time, GPT-4 was just sort of finishing being trained. This was the most capable model. It would not be released until the spring of 2023, but OpenAI employees already had seen versions of it and were playing around with it. And they were all racing to get this thing ready for a launch in early 2023. And then in November 2022, so a couple months before the planned launch of GPT-4, this OpenAI co-founder, John Schulman, hears this rumor at a party where he hears that Anthropic, which is a splinter company that's sort of formed out of a group of dissatisfied OpenAI employees, is going to release Claude, their chatbot. Now, the rumor is not true. Like, they had no plans to release Claude as a commercial chatbot. They were sort of testing it in a private Slack channel with, like, friends and family. But this gets sort of passed back to Sam Altman and the folks at OpenAI, and they're like, shit, like, we have this plan to release GPT-4 in a few months, but we think that this chatbot Claude might be really good and might kind of upstage us, and we don't want that to happen. So John Schulman proposes this sort of rushed launch of a thing that was originally called Chat with GPT-3.5, which was basically to take their old model that had already been out for, like, six months for developers and put kind of a free, easy-to-use chatbot interface on it and just throw open the doors as what they called a low-key research preview. Basically, it was not their best model internally. And I think that's why a lot of people at OpenAI had low expectations because they're like, "This thing sucks. It's, like, six months old. We already have this better model that has finished training that we've been using ourselves that's a lot better. Why would we, like, pump the brakes, throw a new skin on this old model, and, like, put that out as if it's supposed to be impressive?" But they did it anyway. They decided to make it this low-key research preview. They had low expectations, and then it just caught fire. They had, like, badly misjudged how impressive the things that they were using internally would look to the outside world.

No, they were fully planning to release GPT-4 as a chatbot. And actually, they were planning with Microsoft, which had this exclusive partnership with OpenAI, had plans to build it into their products and make chatbots as well. So it was going to happen in the spring of 2023. But the launch of ChatGPT, you know, brought that up in the timeline to the fall of 2022.

You want the dirt? Well, he was fired because the nonprofit board that governed OpenAI had been configured in a way that gave the more sort of safety-conscious members of the board a temporary majority. And these four members of the board were also getting worried about how honest Sam was. This is again a pattern that has been observed by people close to Sam for many years. In some ways, it was exactly the reason that Dario Amodei and the Anthropic co-founders had quit in late 2020. They just sort of saw Sam lying and, you know, observed that he would say things to different people that were contradictory. And so Ilya Sutskever, who was an OpenAI co-founder, former chief scientist, also on the board, starts sort of assembling this rap sheet on Sam Altman and Greg Brockman, the president of OpenAI, who was also on the board at the time. And he and Mira Murati, who was then OpenAI's chief technology officer, start kind of secretly compiling this dossier about all of the times that Sam and Greg would, you know, would say something to one person and a different thing to another person, or would undermine executives or pit them against each other. Just kind of bad management practices, but displaying that this was a consistent pattern. They send it to a member of the nonprofit board, and they basically plot this coup that they're expecting will be greeted with, you know, a hero's welcome. They're like, they're like, "Oh, if we're seeing this stuff about Sam, then everyone else must be seeing it, too." And, you know, we will be greeted as liberators, essentially. And they carry out this thing on a random Friday in November, and they just are sort of amazed and bowled over by how much that does not come to pass. Right, the employees rally behind Sam and Greg. They demand their reinstatement. They threaten to quit and go to Microsoft. And there's a lot of, like, good gory detail in the book about how that all went down. But essentially, Sam engineers his own comeback with the help of OpenAI's employee base. And five days after he's fired, he's reinstated. You know, people have heard this story a zillion times, but what's new and what I was interested in are the sort of dynamics of this employee-led resistance and how and why they decided to band together to sort of restore the position of this guy that I think a lot of them had misgivings about even at the sort of rank-and-file level. So it really was, like, one of the most fascinating five-day periods in business history, I think. And so I devoted two chapters to those five days because I was just like, it's not going to get any juicier than this.

The hero of both the prologue and the entire book is Leo Szilard. Leo Card... I don't know how to exactly pronounce the Hungarian name.

Szilard. Okay. It's S-I-L-A-R-D. Let's call him Leo. Who is Leo, and what does it tell us about Amodei that he sees himself in Szilard's mode?

So Leo Szilard was a Hungarian physicist, and he has this discovery, this breakthrough. He's the person who came up with the nuclear chain reaction, the thing that paved the way for the splitting of the atom and eventually to the atomic bomb. And he has this realization. And there's this very dramatic scene where he sort of has this realization when he's waiting to cross a street in Richard Rhodes's book. And then once he has this realization and proves it out, he tries to keep it a secret. He goes around to all of, he sends telegrams to all the scientific labs, all the nuclear, you know, sort of-ish labs in Europe, begging them not to look into this or not to publish their findings because he is terrified that the Nazis will discover this secret and use it to build bombs and use it to win the war. And he tries to classify it as a secret with the British Admiralty. He tries to, you know, patent it to sort of keep it out of the public view, and he fails at that. The secret gets out, and after the secret gets out, he has this moment where he realizes that the only moral thing to do now that the secret is out is to build it before the Nazis do. So he, there's this, you know, sort of amazing sequence in Richard Rhodes's book where, you know, he basically petitions the president with the help of his friend Albert Einstein to start the American nuclear effort that becomes the Manhattan Project. He then signs up for the Manhattan Project. And basically his theory is, if we don't build it before the Nazis do, the world is going to end, or we're all going to be living under Nazi rule. And Dario Amodei not only read this book, it's his favorite book, he bought copies for early Anthropic employees. Like, on your first day at Anthropic, for years, you would get handed your onboarding packet and a copy of "The Making of the Atomic Bomb,"