A conversation with Mustafa Suleyman. Mustafa Suleyman is the CEO of Microsoft AI. The following are excerpts from our recent conversation recorded on December 5th, 2025, which have been lightly edited for clarity.
On the modern Turing test. Mustafa, when last we spoke circa 2015, I think that was perhaps three years post DeepMind and five years pre language models are few-shot learners. Agents, agentic AI was nowhere to be seen at the level of what we see now. Since you've written about your vision, what you've socialized as a modern Turing test, the idea of economic benchmarks for autonomy by agents, I'd love to hear where are Microsoft's economic benchmarks for these agents. If the agents are about to take over the economy or take over so many economically useful functions, why are we stuck with benchmarks like VendingBench rather than Microsoft leading the way with Microsoft's economically autonomous benchmarks for its agents?
Mustafa, it's probably just worth adding the context that we met in 2015 in Puerto Rico at the AI safety conference.
Many, many of those in the field now were there at the same time.
Alex, it was a seminal moment.
Mustafa. Yeah. Was it the day after New Year's Eve or somewhere around New Year's?
Alex, it was pretty cold out everywhere except Puerto Rico.
Mustafa. Yeah, exactly. It was pretty cool. It was a quite surreal moment.
Alex, it's like a Salamander right before it all happened.
Mustafa. Yeah, totally. And that modern Turing test was something I proposed. I guess it was 2022 when I wrote it. It was basically making a pretty simple prediction. If the scaling laws continue with more data and compute and adding an order of magnitude more compute to the best models in the world every year, then it's pretty clear we would go from recognition, which was the first part of the wave, to generation, which is clearly we're now in the middle of or maybe ending that chapter, to then having perfect generation at every time step, which in sequence is going to produce assistive agentive actions. And actions would obviously look like an intelligent knowledge worker or a project manager or a strategist or a startup founder or whatever it is. And so then how would we measure that performance? Rather than measuring it with academic and theoretical benchmarks, one would clearly want to measure it through capabilities. What can the thing do in the economy in the workplace? And how do we measure the economy? We measure it by dollars and cents. And so what would be the first model to make a million dollars?
Alex, given as I recall, $100,000 in starting capital.
Mustafa, that's right. Yeah. Which model could turn it into a million dollars?
Alex, 10 times return on investment by an agent.
Mustafa, exactly. And so I think that's a pretty good measure of performance and capability. And certainly we've kind of just breezed past the Turing test, right? It's kind of been passed. No one's done a big AlphaGo moment.
Alex, the Loebner Prize wound down before we breezed past Turing.
Mustafa. Yeah, and no one celebrated it. Where was the big Kasparov Deep Blue moment?
Alex, can we clink virtual glasses right now and celebrate that we won? It happened.
Mustafa. Yeah, exactly. And that's what it feels like to make progress in a world full of these compounding exponentials where we just get desensitized to 10 times. So much so that you can be like, 'Guys, why haven't you done it yet?'
Alex, where's my Microsoft Loebner Prize for the modern Turing test?
Mustafa, exactly. Like someone said to me earlier on, 'But you know this AI thing, it's still in its infancy, isn't it?' And I'm like, 'Man, if this is infancy, wow, I can talk to my computer fluently in real time.' Exactly. Obviously, at the same time, agents don't really work yet. The action stuff is still progressing. It's getting better and better every minute, but it's pretty clear that in the next couple of years, those things come into view, and they're going to be very, very good.
Alex, can we get together again after the modern Turing test has been passed just to celebrate and recognize it?
Mustafa, virtual glasses again, Alex. Absolutely. Mustafa, hopefully we can pop a champagne or something.
Alex, I think we should. On timelines for AI solving science and math. Alex, I think you've made already a little bit of news in this conversation with the expectation that in the next two years, I read that as 2027, we'll see agents start to pass your modern Turing test. We'll see them be able to 10 times $100,000 as a return on investment. I'm curious about the next surprises to come. AI for science. Microsoft Research has an AI for science initiative. Do you have timelines in your mind for AI solving math in which we're seeing a whole bunch of startups right now tear through Erdős problems? AI for physics, chemistry, medicine, material science. What do you think happens and when, Mustafa?
Mustafa? Yeah, actually you've just reminded me. The more recent thing that has blown my mind is the fact that these methods could learn from one domain, coding, puzzles, maths, the essence of logical reasoning. So just as AI learned the essence or the conceptual representation of a number seven, it's clearly learned the abstract nature of a logical reasoning path and then can apply that to many, many other domains. And so that's interesting because it can apply that as well as the underlying hallucination/creativity instinct that it has which is more like interpolation. But those two things combined are like a lethal combination for making progress in new mathematical theorem solving or new scientific challenges. Because that's basically what humans do all the time. We combine these two capabilities. Some people want to put dates on those things. It's hard to put a date on those things because they are very, very fundamental, but it feels like they're definitely within reach. It would be very odd to bet against them.
