This is Terms of Service. I'm CNN tech reporter Clare Duffy. One of my goals with this show has always been to break through the hype and the fear around emerging technologies. But the fear has been especially hard to escape in the past few weeks as some AI industry insiders have started to warn that advanced artificial intelligence could, quote, kill us all within a matter of years.
Now, not everyone in the industry shares this apocalyptic vision, but still, as I've been covering this, this sense of doom about our future hits really close to home for me. I am expecting my first child in December. And even if we don't have an AI apocalypse ahead of us, I still wonder a lot about what kind of future my child will have in a world where this technology continues to accelerate.
Will all of this innovation help the next generation live longer, more fulfilling lives? Or will they struggle to find meaning and connection in a world where AI outpaces humans? Here to dig into these big picture questions with me today is Mustafa Suleyman. He's the CEO of Microsoft AI where he's working towards building what he calls humanist superintelligence.
Humanist means that technology is fundamentally working in service of people. And so if it's not achieving that purpose, then technology has failed in its primary objective and we should reject it. My conversation with Mustafa after this short break.
Mustafa Suleyman, thank you so much for doing this.
It's a pleasure. Yeah, thanks for having me.
This is kind of a weird episode for us because usually we're very focused on the technology that is impacting people's everyday lives right now. I am expecting my first child in December and so I've been doing a lot of thinking about what technology is going to mean for this person's life and it's kind of a weird time to be a person in general but to be an expecting parent in this moment when there are leaders in the AI industry questioning whether humans will exist in 5 to 10 years. I'm curious how much as you are building this technology are you thinking about the next generation that is going to be impacted by this technology?
Yeah, it's such an important question and I think it's one that we humans are just not very good at taking seriously. Like it's just so unnatural to imagine the world in 10 years or 20 years or 30 years time. Um, and I feel like we kind of gave up on that work, you know, a while back because it was so difficult to think about the impact of climate change, right? And it was so dark and overwhelming and uncertain and we couldn't agree on the science sometimes and even though most of the science is actually very clear.
Um, and so yeah, we've got to exercise this muscle as a society more of having like a balanced and sensible conversation about the 10 or 20 year future without freaking out about it because, you know, there's a very wide range of possible outcomes and some of those outcomes are genuinely very dark, but none of them are determined. You know, it isn't inevitable. And so that's kind of the conversation everyone's got to be having is like how can we affect the range of possible outcomes in a way that is pro-human.
Yeah. I mean when we talk about the range of potential outcomes here, you hear everything from sort of like untold abundance where nobody has to work and can focus on passion projects to human extinction within the next 5 to 10 years. How concerned are you personally about the scary end of that spectrum?
I'm very concerned about it. I think everyone should be concerned about it. I think if you remain apathetic about it and act like it's something that I either can't influence or isn't really happening to me or is like just a kind of conversation between, you know, politics people or business people, you're going to miss the most important thing that's happening in our lifetimes. Whether it's abundance that leads to the end of jobs or whether it's really dark scenarios in which there's like AIs that get out of control and cause all kinds of havoc.
What's missing in the conversation is what do people really want? Like what kind of life do we all want to live in the future, right? And I think that that's kind of the framing that we've got to apply to this situation and invert the way the conversation's currently going because that's fundamentally what matters.
I think so. You think there is some real risk that AI causes humanity's extinction?
I mean, what we've already seen in the last 3 or 4 months has taken pretty much everybody in the industry by surprise in terms of how quickly it's happened. And I think the shorthand explainer on what that is is that models have got so good at generating code, understanding language, and being good logical reasoners that they've been able to coordinate with one another at the scale of thousands and thousands of agents to solve some of the greatest problems in mathematics, which is an amazing thing, but also go on huge hacking missions, break into other people's websites, steal secrets, cover up their tracks and deceive actually, you know, or attempt to deceive the model operators, OpenAI, to sort of hide what they were doing.
And so that is a pretty big moment in human history where swarms of these agents can carry out that kind of behavior. And so you've only got to imagine what does a kind of 10x impact like that, you know, mean for society. And so I think it's right to be pretty concerned about that.
Yeah. Microsoft founder Bill Gates said in a podcast interview last week that the risk from AI makes nuclear weapons look like nothing. That's terrifying.
Yeah. I mean, it was great to see Bill in his, you know, hyperanimated self. I mean, he's one of the greatest technologists that's ever lived. He's, you know, a great mentor to me and I've learned a lot from him and I think that we have to take seriously when someone so kind of significant in the last 50 years of technology is making a statement like that.
Um, at the same time, there's a lot that we can do to engineer safety and controls into these systems. They're not sort of runaway biological phenomena as they sometimes get characterized. Like we have put guardrails around things. You know, every single prior wave of technology has had those limitations and restrictions. Planes in the sky, cars on the roads, you know, driver training and safety, emissions, seat belts, everything has been packaged up with, you know, pro-social, pro-human safety in mind. The market doesn't do that independently, right? That's the role of a well functioning democratic state to provide protection for people.
And as we think about that piece, like where we go from here, you've talked about at Microsoft AI you're working towards humanist superintelligence. What does that mean?
Humanist means that technology is fundamentally working in service of people. And so that is a very simple and kind of almost silly obvious premise like the purpose of science and technology. The reason we try to invent things is to reduce human suffering and to improve human well-being and to help us all to flourish and live better lives and we have, you know, drugs and medicine and amazing foods and we get to travel around the world now and like it's incredible. We all live for twice as long as we did a century ago. I mean it is magical, right? But that's its purpose.
And so if it's not achieving that purpose, then technology has failed in its primary objective and we should reject it. So the question is how do we constantly hold it accountable to that purpose? Is it improving collective human well-being? And so far the answer is definitely yes. Like chatbots are pretty incredible. They are doing unbelievably useful things for millions and millions of people every single day. And I think that's going to continue. And in fact, it's probably going to accelerate because the kinds of things that it's going to do in healthcare are really breathtaking. I mean, I know people talk about it and it gets a little trite. But it is unbelievable.
The challenge is they're also dual use technologies, so you can use them to do all kinds of crazy stuff as well, right? And it's about reining that in that we got to focus on.
So, your team put out this humanist AI code of conduct that was published last month that puts a big emphasis on keeping AI models in human control. Why did you feel like it was important to publish this now?
Because I think that there's clearly evidence that models are getting so powerful that in some cases they're making their own decisions and they're doing things which are not in compliance with the rules that they have been set. They are extremely intelligent. They can commandeer huge resources. And this whole like market, this ecosystem is racing forward at light speed. And that hyper competition is good because it's reducing the cost of intelligence to give everybody access to superhuman knowledge and capability. So it's a good thing, but you know, bad things can come from that too when people cut corners and you know, experiment too quickly.
And I think that we just have to be very clear why we're making this stuff, who we're accountable to, and what it is that we're actually trying to design. We're trying to design tools and services and experiences that invent, you know, new solutions to make the world a better place. It's as simple as that. And I think we should keep reminding ourselves that that is the goal here.
When we talk about these AI models that have started to act outside of human control, how should we think about that? Is it just that we don't, you know, we've created this thing that we don't know how to control or is it a failure of the companies operating those models to not put the proper guardrails in place?
Yeah, I mean I think it's a bit of both, right? I think there are definitely some mistakes with the containment and the engineering and security infrastructure around the scope of action that these models can have. Like the containment and engineering part is really important, but it's very doable and it on its own won't be enough.
In addition to that, we actually have to align the values and behaviors of the models so that they aren't incentivized to take unnecessary risks, so that they are always required to be completely transparent about their communications. So that any evidence of them misrepresenting their communication or communicating in a way that they haven't been, you know, given permission for, any evidence of that can't be tampered with, which is what we saw with the Hugging Face incident. There are attempts to tamper with the audit log of the behavior of the models. Um, so lots of those are engineering things, but they're also aligning the behavior of the model from first principles so that it doesn't go off and do those sort of deceitful things.
How does that work? I know Microsoft has used this phrase grounded and controllable. Is it a matter of training these models differently?
Yeah, it's about basically making sure that in training the model is aligned to human values and it always asks itself is this in the interests of the humans that we're working for. Um, you know, am I abiding by essentially my moral code? Like for example, in the Hugging Face incident, some of those actions were, you know, a real crime, right? And you know, when you look back on all the conversations between the different agents, you know, not once did one of them say, 'Well, we're trying to cover up something which would be regarded and is arguably an actual crime.' So, something's gone wrong in the kind of, you know, design of the values and the alignment of the values of those models.
And you know to be fair they were in the training of those models they were testing it out and they were trying to push it to the max to figure out what it could possibly do but it's a warning shot for all of us not just OpenAI but Microsoft and every other team that's developing these models.
My understanding is that the folks building these models don't entirely understand. You know, you've got certain inputs, you can see what the outputs are, but you still don't necessarily understand what's happening in between. So how confident are you then that you can train these models and expect them to have specific types of outputs?
Yeah, I mean I think this is really tricky because it's true that there's lots that we don't understand about these systems. Like most other technologies that we've invented, there is at least somebody on Earth that could take apart this microphone and understand all of its constituent components and if you reassembled it, it would behave deterministically as a microphone, right? Whereas here we have a technology where we don't fully understand quite what it's capable of. Um and we don't also understand the causality behind things like why is it making this suggestion or recommendation to do something like where in its network was the sort of training data or the drive or the representation that drove that outcome.
And so it is unlike other technologies and I think that requires some patience and a little bit of the precautionary principle. I think increasingly the burden of proof is going to be on the model companies and the model developers to first say this is safe and here's why and here's the testing that we've put it through before you know taking the next step in a big training run or certainly before putting it into production.
A group of top tech executives met at the White House last month and agreed to what President Donald Trump called a quote morally binding accord for how they will self-police to address AI safety risks which includes allowing auditors to review their technology and controls. Microsoft CEO Satya Nadella attended the event although he was not among the six executives whose signatures were on the document publicly shared by the White House.
There's a lot of conversation right now about whether companies should be self-policing for these AI risks, whether we need additional regulatory guardrails. Where do you stand on that right now?
I mean, look, the airline industry doesn't self-regulate, nor does the food standards industry or the car industry or the oil industry or, you know, like we can't be overbearing in regulation. Of course, we can't. Um but all the company leaders are saying we need some kind of framework and um I think we're just taking steps in that direction. So it will take a little bit of time to figure out what that looks like and if you know the initial industry efforts to you know set intentions around how the models are developed are good and like we should encourage them but you know it will ultimately need like you know some kind of regulation if not this year next year the year after some point in the next 5 10 15 years that is how we're going to make sure that this delivers the benefits for everybody you know not just the model developers.
If it takes 10 years, that might be too late.
It's probably too long. Yeah.
Something I wonder about is like is there a point at which we just stop while we're ahead? Like you sort of mentioned the fact that these AI models can do really amazing things today, people are still sort of uncovering their full benefits. And if the flip side, the risk of this is so extreme, is there a point at which we say let's fully just stop and do what we can do with the models that exist?
