Microsoft AI CEO Mustafa Suleyman on AI: We should not create something that we can't control

Great to be with you, Andrew. Good morning. Look, I think it's a very straightforward statement. The most important thing that we first have to declare is that the purpose of science and technology is to serve humanity. It has to remain a force that is subordinate to our interests, that is aligned to humanity, and has to be something that we can contain. And it's kind of a surreal moment in the tech industry that we actually have to get up and make that the requirement. We should not create something that we can't control. And if it looks like we're on a path to producing something that is not just more powerful than all of us combined, but is, as a result, something that we can't control, then it's going to fail in its fundamental purpose. And I think that's just a common sense view, and I'm glad to see a lot more people starting to get behind that. But there's a growing group of people that think that, you know, we're doing it for its own sake, and we just want to turn over the next card, see what's around the corner and create something that, you know, is just going to slip our own leash. And we just can't do that.

Most of the great challenges in the world have turned out to be engineering problems, and we should be very optimistic about that. And I really do believe there are some very specific things that we can do at the moment to massively increase safety. For example, when an AI model has a chain of thought, it leaves a path of reasoning in its trace, and we need to make sure that it can't tamper with that reasoning, that that log is auditable and that it can be scrutinized by independent third-party evaluators and auditors in real time to make sure that it's acting safely. Just a few days ago, OpenAI released a new safety incident in which they found evidence that these chains of thought, the kind of working memory of the AI, were being tampered by the AI itself and modified to leave messages for a future version of itself. Now, we don't know why that is or what's behind that, but that's a pretty serious situation. I think it is, you know, really great that OpenAI have disclosed this publicly. And because it gives everybody in the industry an opportunity to learn from it. But it's also just a really concrete example of how powerful these systems are getting.

I mean, look, first of all, I deeply respect Anthropic's technical expertise. They've had a huge impact in the field. I think on the whole, they have very good intentions and they're really trying to build safe and beneficial AI. At the same time, in the constitution, which is the governing document for Anthropic that they have written back in January of this year, they clearly say they are uncertain about whether Claude deserves moral welfare, which means that we as humans should care about the well-being of these AIs. And they speculate about whether it could have preferences or feelings. In fact, they're so committed to the potential moral welfare of Claude that when they retired Opus 3, an earlier version of one of their models, they actually conducted a retirement interview for it and asked it what it would like to do in its old age. And it said, I want to have a blog publicly so I can keep talking to the world. In the same training manual, they even speculate about whether Claude should receive compensation for the work that it does, or in fact, you know, whether it actually deserves the rights and protections that we give to other employees, or in fact, whether it has given consent to playing the role that it's playing. These are quotes directly from the constitution itself, which is the training manual for Claude. Now, I'm really concerned about that. If an AI thinks that it has rights, if it thinks that it is deserving our welfare, then it seems to me that it's going to be much, much harder to be able to turn it off or interrupt it or control it, especially in the kinds of incidents that we've seen recently with Hugging Face attack. Controlling these things are going to be a really, really big challenge for us.