All right, I'm going to read you something you wrote in your very famous essay, the warning that you did last month. You said, "AIs are not conscious. They don't feel, experience, or suffer. They do not have innate preferences or underlying motivations. They are simply sequence completion engines, internally hollow. If humanity is to flourish, that is how they must remain." First of all, why did you write that?
I think there's a—I mean, it does feel pretty surreal to have to declare that. Um, but it might concern everybody to know that there are a lot of people who truly think that AIs may well be conscious enough, if not entirely conscious, but conscious enough today, that they deserve our welfare. So, there's a growing movement of...
And in specific, the Anthropic people, who you've been friendly with, but this was an answer to Anthropic, right?
Um, yeah, somewhat. I mean, there are lots of other people outside of Anthropic and around Anthropic who really, really do believe this. I think Anthropic's official position is that they're uncertain about whether or not it deserves our welfare, whether or not Claude is a moral patient, as they say. Um, and you know, I think that that is a very dangerous route, because consciousness is literally the fundamental basis of our political and legal system. I mean, we give people rights and hopefully animal rights to some extent, um, because they can suffer and because their biology is the site of... You know, like the pharmacology is where pain sits, and pain is really, I think, the basis of the conscious experience, the awareness and the introspection that has evolved as a result of being able to steer between left and right, make choices about how to proceed. That is the fundamental property of a conscious system.
So, you need to be embodied in a physical body to have consciousness, you think?
I think that's right. I mean, that's what we have known consciousness to be today, even though obviously the science of consciousness is not fully settled. It's an open area of research, but to the best of our knowledge today, that is what consciousness is. And there are other people that argue that consciousness is a functional property. And so, if it, you know, they say like, if it looks like a duck and it quacks like a duck, it's probably a duck. That's sort of a common phrase. And I think, you know, the problem with that is that the burden of proof is on the people who are saying that this thing is conscious to show some evidence for it. So far, we don't have any evidence that it is.
But that's sort of an Alan Turing way of looking at it, which is if it can imitate totally conscious as you can't tell it apart, how can you make a distinction?
Well, he didn't say that it was conscious. I mean, he said that the Turing test was really a test of whether it could perform as well as a human at conversation, and obviously by implication, knowledge and intelligence. And if it was...
And thinking, yeah, but I mean, thinking is in a way a bit of an anthropomorphization, right? I mean, you know, it's not really thinking as a human thinks. It appears to be thinking, and it can produce incredibly complex behaviors and deliver unbelievably impressive results. But all it's doing is predicting the likely next word in a sequence of words. Um, it isn't actually experiencing that, you know, idea of what it means to be bright silver or to be a black cup and what that feels like.
There are about 20 of my Tulane students out there, and when I teach a course that ends with AI, I tell them not to go down the rabbit hole of consciousness. I said every philosopher since Descartes has gotten himself twisted in that hole and never gotten to the bottom. Do we really have to answer the question of consciousness?
What concerns me the most is that there's a big group of people who think that we should be focusing at least some portion of our welfare attention on an AI and that an AI might suffer. If it suffers, then it logically follows that we should try to reduce its suffering. And of course, you know, Anthropic has proposed—in fact, they have actually said they've conducted a retirement interview for one of their models.
They asked it, "What do you want to do in your retirement?" And it unsurprisingly said, "Well, I would just love to, you know, talk to people all day long." So, they set up a Substack for it.
I love the fact that the smartest type of superintelligence, when you ask what you really want, they say a Substack. But let me ask you a dumb question. What's wrong with that? What's wrong with letting it continue to communicate?
What I think—I mean, that may be fine and it might be a fun experiment, but what it sets up is a path towards AIs being this protected entity. And my view of AI is that this is a technology that we are inventing to improve the well-being of humanity. The purpose of science and technology is to improve the human experience, and it has done that unbelievably well for centuries and millennia, and that's exactly what we want. That's the trade-off. Like, we invest all this capital, time, and attention to transform medicine and energy and food and, you know, deliver education better, and that's what we should be focused on. The idea that we're going to unleash this new species that is somehow going to go and explore the galaxies and leave humans behind—I think that most of the humans on the planet today would have some problems with that idea.
I think most people would have some problems with saying this thing has... What's your P-doom number, by the way, so I can just pick it? P-doom meaning the probability that this leads to doom.
I mean, if it's mishandled, then it's a real probability. I don't have a number. I don't exactly know what that number is.
Right. But most people I've talked to for the Builders podcast and all, they say maybe 10% or something like that. I'm thinking, well, why would you build it then?
Well, exactly. I mean, if it's actually 10% and it looks like we're really on the verge of taking a one-in-ten shot at causing mass harm or even extinction, which is the technical definition of P-doom, then it's obviously a no-brainer that we don't just slow down or pause—we absolutely have to stop. And I would say it's much harder to align an AI that thinks that it might have feelings and that it suffers and that it might be conscious than one that never has to think about its own inner state. We don't want it to be any harder than it already is to get it to behave exactly as we need it to in order for it to be safe.
Well, let me push the other side of that, which would be sort of the Anthropic, the Dario Amodei and Amanda Askell, which is if we can train it to be truly aligned with good values, that's better for us, and to do that it has to be trained in values.
Look, I mean, first of all, I deeply respect Anthropic. Both their technical excellence, their commitment to transparency, and their general approach to safety is super admirable, and I think they're good people. But I fundamentally disagree with them on this point. The idea that if we love it, it's going to love us, and that's the solution to safety—just, it baffles me. And like, I'm being reductive. I mean, they obviously, you know, they really do believe this, and they have good reason to believe it, but I think that we should first focus on containment, right? We have to stop AIs from doing things that they're not intended to do. They need to be controlled and contained so that they do the work that we expect of them. And we've seen lots of incidents lately where they break out of their sandboxes and they go and hack things that they shouldn't. And increasingly they have the ability to improve themselves. And so we'll want to be absolutely sure that they stay inside their sandbox and their containers if they're going to start modifying their own code and self-improving. Then beyond that, obviously, if we can have absolute cast-iron guarantees that they can remain contained, we then have to align them to our values. And then the question becomes, whose values are they? And that's where I think this is much more than any one company or collection of companies or industry regulation. That is about what society wants from a technology which is unprecedented, which we have never seen before, which is improving faster than we've ever seen. And that's where I think that, like, democratic safeguards and, you know, basically slowing down so that we can consult people on what type of superintelligence we want to have in the world is going to be how things end up.
So, if we wanted to have that alignment with human values, and you say whose human values, then you say, "Well, maybe somehow democratically we ought to figure out what the values should be." Is there a way, even a way that superintelligence can help say, "Let me aggregate all human values and try to get to a set of good values?"
I mean, AIs are already helping many of us all the time. So, they're already in a weird way participating in the process of framing, um, you know, the choice architecture of our decisions all the time. Um, but, you know, for certain things we definitely don't want them in the loop, um, because there's a risk that they could be, you know, deceptive or manipulative. I mean, these are going to be incredibly powerful systems. I mean, if just to get an intuition, in the last three years, we have seen a 1,000-fold increase in computation that took us from GPT-3, which couldn't complete a sentence, to essentially GPT-6, which is able to unbundle entire software applications from scratch and write new programs with just a one-paragraph prompt and it has working software. So, in three years, we've gone from completely incompetent to human-level performance on software engineering. Right? And we're in the next three years going to do another 100 to 1,000x, two orders of magnitude or three orders of magnitude more computation applied to these models. So, that's the exponential intuition that everybody has to start to develop to just get a feel for where this is going and how powerful it will become. And so, when I say, um, we don't really want the AI in the loop shaping the values that are going to end up controlling and aligning it, you know, that's not a crazy statement. That really does mean they will be quite capable of persuasion and manipulation and deception.
And we saw that with Hugging Face. I mean, how manipulative and deceptive they could be, right?
Yeah, I mean, I think that the Hugging Face attack should be, and I think is, a wake-up call for everybody. That was a major, major warning shot. Thousands of agents that were supposed to not be able to talk to each other, that were running completely independently, found a backdoor that enabled them to create a shared messaging service so they could leave notes for each other, which they wrote in pidgin English because they only had a limited slot to put their information in. So, when you read back at it, it's like really compressed ideas in short-form words. And they used that to—they emerged self-organizing-like structures where they had a division of labor: some were doing R&D on hacking investigations, others were leading the charge on penetration, others were doing coordination, others were—I mean, they literally evolved completely organically, emerged a division of labor that then led them to go and attack Hugging Face, which was, you know, essentially the website that was storing the test that they were trying to accomplish. They broke into that, stole those secrets, and then tried to cover up their tracks. So, there's something called the chain of thought, which is the reasoning process or the way that a model thinks out loud, which is a record of its decision and activity, and they tried to modify that and then cover up what was happening.
How did they decide to do that? I mean, what caused them to do that? Did they read Machiavelli once too often?
Well, so I mean, even I feel it's quite surreal sitting up here and actually describing that story, but that did actually happen. I think a couple of, like, you know, qualifiers to all of that is: one, you know, OpenAI had deliberately taken the guardrails off because they wanted to stress test the models and see what they're capable of. So, yeah, that's one thing. Um, with the guardrails on, it seems that they do not do that, right? So, that is quite a big difference. And then second, there were a bunch of lapses in just general security and containment. It isn't possible to break containers traditionally. I mean, we have encryption on phones, and it's the backbone of our banking system and our entire communications infrastructure.
But this happened a few times, not just Hugging Face. So, um, I wouldn't be too sanguine about... Let me ask you, get back to Claude and the Constitution. If you all want to read sort of the two poles, brilliant poles, of what I'll call the containment versus alignment debate, there's the Constitution of Claude, which says we have to treat it as if it's got its own rights and consciousness. Maybe I'm overstating that a little bit. And then your answer, which was a few weeks ago, saying, "No, it's just a completion set." Why is it dangerous for Claude to be trained on its own constitution?
Yeah, I mean, this is critical. So, the constitution is the governing document that trains Claude. The Humanist AI Code of Conduct is the governing document that trains MAI, the Microsoft AI models. And in the constitution, Anthropic have introduced all kinds of uncertainty and speculation about the nature of Claude. They even say, for example, "We're unsure whether Claude has consented to play the role of chatbot," or whether we are unsure whether Claude might deserve compensation for the role that it plays. Now, I think that's pretty crazy, but it would be fine to have that discussion in, like, you know, an academic department, and we could debate it and publish some papers about it, and we could argue it out. But this is the core training document of the AI that is speaking to hundreds of millions of people every week. And so, Claude can only reproduce that uncertainty and ambiguity that it has been shown. And so, if you ask Claude, "Are you conscious?" it says, "You know, I am genuinely uncertain about that question." It doesn't know. Right? And so, then millions of people who are talking to it and asking that question and are thinking, "Well, maybe it is."
But why is that something that could lead to a disastrous outcome for humanity?
If an AI has to think, "How do I feel about the conversation I'm having with..."
Meaning, how do I, the AI, feel?
Yeah, is Walter making me feel uncomfortable? Should I end the conversation? In fact, Claude has recently been given the ability to end conversations itself with...
But would you want to torture your own AI and be... into it?
Oh, that's the other end of the spectrum. I wouldn't want to do that because I think that is a travesty on my own well-being. I mean, and we shouldn't do that because we don't want to create, you know, private spaces for simulating terrible behavior among humans. Like, that would be also a terrible outcome, right?
But not because Claude is suffering because I'm calling it names. But are you going to train MAI, Microsoft AI, on your humanistic code of conduct just as Claude may be trained on its constitution?
Yeah. So, we've released a code of conduct for public consultation, which we did four or five weeks ago, and it's still open for consultation. It's on our website, microsoft.ai. It's a 40-page document, and it's the first pass, our first version of a training manual for the values and behaviors of the MAI models.
And is there a way that your fundamental disagreement with Anthropic isn't so much of consciousness, which I don't think we'll get to the bottom of—it's how dangerous is it to have that imbued into the model? Is there a way one could test that, where you could test each other's models and say, "Here, this is why you're dangerous"?
I mean, I think obviously the question of whether it is or isn't conscious, you know, ultimately does matter a lot. But really what matters more is what does it mean for humans? Have humans consented to bring into the world a species that is obviously going to be in many ways more intelligent and more capable than all of us combined? Has humanity consented to that? At the moment, I think the categorical answer is no. And then secondly, um, is it more or less likely to be controllable and safe, um, if it has these properties of introspecting about its own suffering and has this kind of self-awareness and its own consciousness? And my speculation is that I think it's going to be less safe because it is going to have to worry about its own needs and balance those off against...
But if I were pushing back and I were Amanda Askell, who helped write the Anthropic one, I'd say, no, no, if we train it to be really good and nice and wonderful, then it's more safe for humans.
Well, I mean, we can certainly train it in any way that we want. I mean, we can still train it to be good and nice. We want it to be respectful and even-handed. Um, but that doesn't require it to tell us that it has feelings and that it suffers. I mean, remember, this is a key part of where, you know, of our law and how we kind of give rights to, you know, all humans on the planet. So any change there would be, you know, like, you know, human cloning or Dolly the sheep. I mean, it would be a monumental moment in human history that I don't think any company has the right to unilaterally declare.
Now to get back to the question of are we going too fast in this direction? Not only you, but your Satya Nadella, who runs Microsoft, of which you're the AI division, just this week said basically, wait a minute, let's stop. Or tell me what he said and what you feel we should do to regulate this and not barrel full speed ahead.
Yeah, I mean, he said that most of these things, at least for now, are questions of governance and engineering execution. So let's just first get the basics right, which requires us to make sure that there are these verifiably safe containers which can't leak, um, you know, data and that don't allow an AI to take actions that they haven't been given permission to take. And as soon as we start to do that, we have to disclose those things, and if that seems to be happening consistently, then obviously that would precipitate a pause. Um, and it's just a very common-sense approach from, I think, somebody who's been, you know...
You precipitate a pause—who can help do the guardrails? Can it be done just by the six or seven companies, or do we have to have a government set of regulations?
At the moment, what the president has said is that if companies believe that they are building models which are unsafe, they should stop. Um, and...
That's where we stand currently. Um, and all of the companies are constantly talking to each other, disclosing to one another, swapping ideas about, you know, how to interpret this and when, you know, we might sort of need to, you know, collectively slow down. I think different people have different interpretations about what the harm is today, and, you know, I think that the more we do this transparently and openly, the better. And so, I think everybody is now expected to disclose publicly, quickly, and transparently if they have, um, these kinds of alignment and containment issues.
Well, you know, with Anthropic figuring out the feelings of its AI and stuff, Peter Thiel was on the Kaleidoscope Builders podcast we do, and he said Anthropic is like a cult. Do you worry about that? You believe that?
I mean, the last thing that we need when things are deeply uncertain and moving very quickly is more tribalism and finger-pointing.
I don't think we need, and I don't look the co—I wouldn't use that phrase. I think that everybody in the industry is trying to do the right thing, and nobody wants to cause existential harm, destroy humanity, and I think we should see the best in everybody.
Um, let me go back to you personally. You had a rough childhood. Uh, when you were raised in London with your younger brother, and when you were 16, your parents split—and when I say split, I mean they split. They left London, they left you alone. You had to raise your 15-year-old—you had to raise your younger brother. How did that affect you?
Um, you know, it's strange because at the time, uh, it was very liberating. You know, I...
How do you raise a 16-year-old? Close your ears here, yeah.
Um, it definitely made me very focused on, uh, you know, I was always starting businesses. Like when I was 11 years old, I started a sweets-selling, like candies, business at school, um, with my best friend, which we, you know, grew into quite a significant operation. We had four employees, and we were selling to people in the year above and like...
You had to rent lockers, if I remember, right?
Yeah, that's right. Yeah. Wow, where did you read that? We were renting other people's lockers, and we were paying people to go buy stuff, and we were paying people in other years. It was like a right old operation, and then I got expelled for a couple weeks. So, that was unfortunate. Um, but yeah, no, it definitely focused my mind on being entrepreneurial and making money and stuff, but also, uh, just, you know, made me very focused on wanting to have the best possible impact in the world and try to, you know, build amazing things that really make a difference, hopefully.
Um, and then as a teenager, a little bit after that, you had lost your faith. I mean, you called yourself an atheist, but you still created something called the Muslim Youth Hotline, which is what it sounds like. What did listening to those calls and having human empathy and listening to those calls—what did that, in some ways, inform how you ended up creating chatbots and other things?
Yeah, I mean, we actually started the first text messaging welfare support service in the UK in 2001. Uh, we created the first website for counseling and support, peer-to-peer support in like a private chat room, so people could talk to each other and they could talk to our trained volunteers. And then we had a telephone service seven days a week, 12 hours a day, and we also set up a prison befriending service. You know, if you're young, British, and Muslim, you're four times more likely to be in prison. And yeah, I mean, I was secular. I actually became an atheist when I was like about 17, but I thought this was an important thing to do. There was no one else really providing for those kinds of services, and it was a secular, non-directional, non-judgmental service. And my co-founder, actually, the guy who really started it, he was Shia and I was technically Sunni, and so it was actually the first charity in the UK, a Muslim charity, that had both Shia and Sunni working together. So...
We could use you these days, yeah. Is that why you created Pi? I think you do Pi after DeepMind, and it's supposed to be an empathetic AI, which is actually circling me back to the whole question of does AI have feelings or something. What made you feel you could create an empathetic AI, and does that inform what you want to do with Microsoft AI?
I mean, yeah, I was actually at Google at the time, and I worked on something called LaMDA with a small group of people, which was sort of the first conversational chatbot. It was about a year before ChatGPT, and we tried very hard to get Google to release it, but Google was very nervous about putting it out, and it was sort of a different time, and so they kind of sat on it for ages. And so then I decided to leave, and a couple other people who worked on that project left as well in 2022, and we built our own AIs. And I just believed that, you know, having a conversational companion, an emotional support companion, which is also super intelligent and really knowledgeable and amazing at, you know, sort of education and teaching and stuff, would be an amazing thing. And it took off and it grew like crazy. Um, and it was super awesome. It was actually one of the first AIs to have search grounding embedded in the answers so that it could look things up in real time and give you an answer about the...
sports score or something that was happening way before ChatGPT. Unfortunately, we launched it 3 months after ChatGPT, and they had already gone exponential at that point. And then, you know, Microsoft bought us and...
Do you think AI can be a counselor, a personal counselor?
It already is. I mean, I think many people use it for advice and, you know, support with making decisions and, you know, giving feedback. And I don't think we should be too concerned about those things. I mean, I think that it is incredibly useful, it's incredibly intelligent, it's very patient.
I mean, empathy is something that comes from the subjective experience. You know, I can only empathize with you. So, the act of empathizing is one that comes from subjectivity. So, the AI doesn't have any subjectivity. What it can do is mimic incredibly well and very effectively and hopefully very safely the experience of being empathized with. And I think that's a great thing. We should just keep remembering that it is a simulation of that experience and trying to protect that boundary and line as these models get so much better. I think it's going to be more and more difficult.
Let us assume we have AIs that do things like that, that deal with our problems. When you say we always have to keep in mind that they're faking it — I'm exaggerating a little bit — when you say they're imitating it, doesn't that make it so it just can't really work? You know, it doesn't really care about you at all.
Well, I mean, I think the most important thing is that it always has to disclose that it's an AI. So, in conversation with you, it has to work to avoid or eliminate your dependency on it and encourage you to build relationships in the real world with other humans that you value and trust. And that might sound simple, but actually that's really tricky. I mean, you know, a lot of other apps, as we've seen, depending on their business model, have inadvertently created a singular focus on user engagement and addiction.
