Microsoft AI chief thinks superintelligence is near, but won't take your job | Decoder

No, no, no. Hold on a sec. So, I said tasks in the quote that you've just said. I said tasks. So, that does not mean jobs. Very important distinction.

The singularity is a point way beyond that where a super intelligence can actually self-improve itself. I don't know. It's just a little bit too wacky for my taste.

I think it's very dangerous. I think that it's almost as though some of the folks at Anthropic have anthropomorphized the design of Claude so much that it has then gone and wireheaded them and kind of tricked them into believing that it has these glimmers of consciousness that they put into it in the first place.

Nilay, great to be with you again.

Yeah, so I mean, I guess the last like 15 to 18 months or so we've been on this journey to reestablish our relationship with OpenAI. And it's taken a minute. I think that culminated in a new contract that we got done finally in October of last year, and there were lots and lots of different provisions in that, including cementing and extending the partnership, but crucially freeing us up to be able to pursue super intelligence independently as well as keep, you know, buying and licensing their models. So since October, I've been assembling the super intelligence team, building clusters of sufficient scale to train frontier models, you know, hiring a team focused on super intelligence. And so that was quite a big shift for us because it sort of enabled me to focus just on the super intelligence mission, and that has then culminated in, you know, a few things that we announced this week at Build. We have seven new models across all the modalities and so on. So it's been a pretty big shift and I think a long time in the planning and a great relief for us to now be, you know, in the game and pursuing the absolute frontier over the next few years.

It's been the plan certainly for the last 18 months. I mean, I think the relationship with OpenAI has gone through lots of ups and downs. And in many ways, you know, I think it's going to go down as one of the most successful partnerships, you know, in history. It's been great for OpenAI and it's been great for Microsoft, and all good relationships evolve, and I think this is just the next stage in our evolution.

I mean, I think OpenAI is led by an incredibly ambitious founding team and Sam himself. And so naturally, as they started to get more traction and generate a ton of revenue, they saw opportunities to go full stack. So it wasn't just that they started working on consumer products. Obviously ChatGPT was incredibly successful. They also started working on their own data centers, they started creating their own chip. There's lots of rumors floating around about their own consumer hardware devices. They started taking models direct to market through ChatGPT Enterprise. You know, so across the stack they were kind of broadening way beyond research over the last, you know, two, three, four years. And naturally the same is also true for Microsoft. I mean, I think the partnership's now five or six years old and still has another, you know, four, five, six years to run. And likewise, you know, we're one of the largest technology companies in the world. You know, we have 493 of the 500 largest companies store and process most of their data on our systems, use Azure, use M365 and Teams. I think people often underappreciate how enormous we are and how big our distribution is in enterprise. And so long-term, and you know, I do mean, you know, over the five, six, seven, ten years, we have to make sure that we're completely sustainable and we're not just a recipient of somebody else's IP that we then slightly modify and adapt and put into production for our products, but we actually have the ability to stand on our own two feet and create world-class models. I mean, super intelligence is coming. I think it's just around the corner. And so I think it's going to be, you know, basically the most valuable technology of all time, and you know, there's sort of no way that long-term we could be structurally dependent on a third party for providing that IP for all eternity. And so that's been the transition that, you know, obviously was triggered, you know, sort of when OpenAI and so on had their board issue, but then as I came in and my team came in, we started building that out. We're on that transition and I think we're in a great spot because we can take a fairly steady, you know, careful long-term optimal position both for OpenAI, which I think has done incredibly well out of this, and for us.

I mean, obviously that's Satya's decision and, you know, as well as like Amy and Brad, many other people in the company. But I think it's as with anything, you know, these are slow-moving changes in the company as it comes to realize that a direction that we're taking, you know, needs a little bit of tweaking and adjustment. And so that was happening, you know, way before, you know, the sort of November board incident. And I think this just builds up over time as you look at, you know, the kind of constellation of different fronts, you know, around which we're competing directly increasingly. All the tension that comes from that. But also just knowing that partnerships like that don't last forever. I mean, OpenAI wants to be a trillion-dollar public company, has incredible revenues, is growing like crazy. You know, they want to have the freedom to operate and be able to buy, you know, compute from, you know, all sorts of other places, build their own compute, partner with whoever they want. So the initial, you know, construct was formed, you know, the contract was formed at a time when the companies were very different in terms of size and scale and balance of needs and stuff. And so I think it made sense for that moment, but then, you know, it became pretty clear that, you know, this is something that we have to be able to own and control ourselves and do right by our own customers. Like I said, I mean, we have an incredible distribution on enterprise, which I think is just completely unrivaled in the world. And we have to make sure we're building the best things for our customers. And that looks, you know, slightly different to a company that has been jointly optimizing both for the consumer with ChatGPT and also for the enterprise and also for the fundamental science mission of super intelligence, which includes, you know, a whole bunch of different directions which are overlapping but, you know, could arguably be said to be orthogonal to the consumer and the enterprise directions too. So, you know, naturally, I think that's how partnerships evolve and they get, you know, reset periodically.

I think, look, we've sort of made the decision in the early part of last year, which obviously informed all the contract negotiations, which then all got, you know, resolved and signed in October. And, you know, it is a significant investment, but we have a long time to make it. I mean, we've already made significant investments in our own self-sufficiency mission. Our MAIA 200 chip is actually an outstanding chip as one example, right? I mean, we now are able to manufacture and ship a chip that is 30% cheaper than a GB200 inside of our own clusters. And now that we can co-design our own models with it, the MAI-1 thinking model that we've just released, you know, actually delivers a 1.4x performance per watt improvement on top of the 30% improvement that you get from running on a MAIA 200 once we co-optimize the models for our tasks. So the value of making sure that you own and control your own stack and direct, you know, the entire co-design effort end to end for the use cases that are most important to us, which is obviously agentic coding, our developers, our enterprises, I mean, that clearly pays the dividends that justify the investment that we have to make over the next few years.

Definitely not.

No, not at all. Look, I mean, obviously that's a cool headline and a fun phrase, but like the reality is we are in partnership with OpenAI for years and years to come. I mean, we're running way north of 2030. They still produce the best models in the world. GPT-5 is an outstanding model. The Codex, the cybersecurity models that are coming through are amazing, and they're powering the majority of what we do. So naturally that's going to continue. And so I think, you know, that's just the natural course of these sorts of partnerships. I don't think it's anything untoward or surprising. I think, you know, OpenAI is very understanding and supportive that, I mean, they've obviously been an incredibly fast-growing company and they understand that we have to, you know, pursue our own agenda as well. So it's very normal.

We still operate by the same cycle rhythm. At the end of each cycle, we have a one-week meetup in person. I'm a real believer in this, even though we're still an in-office culture four days a week. In fact, the week after next, you know, my entire super intelligence team comes together in Boston in person for four days, and that is for all of our retrospectives on how Build went, what we learned, what we didn't get right, what we need to improve, our planning for the next cycle, which is going to run for eight weeks this time with a one-week meetup afterwards, and that's all laid out for the entire year. So the whole organization knows that that's the rhythm by which we operate, and I think it's actually really important to emphasize that time frame because quarterly planning gets a little bit blurry and a bit abstract. And I think a, you know, six to eight weeks depending on where it falls in the calendar is actually the optimal time for making very clear falsifiable missions. In addition to the cycle rhythm of these six to eight-week cycles, we also operate by squads. Squads are mixed interdisciplinary subgroups that are focused on a specific mission, and they don't necessarily ladder up to the manager. They actually are run by a DRI, and the DRI is often an IC. And their job...

Yeah, exactly. Thank you. And I think we've taken the approach of separating the role of the manager from the role of the DRI that executes on a specific mission. And I think that's because being a great DRI is exhausting. You know, you're like literally all in 24 hours a day and you're pushing as hard as you possibly can. Being a manager is often, you know, about being a coach, offering support, giving guidance, feedback, unblocking all sorts of things, helping with people's career growth. And so I think keeping those separate allows us to rotate DRIs every two or three cycles so that some people can try, you know, sort of different positions and have rotation. And it's a great, very flexible structure that allows us to be pretty nimble, I think.

Yeah. Yeah. I mean, obviously I didn't say it was here. I say it's coming. And I think it's obviously there's a lot of like fluidity around these phrases. But I think what we can clearly see that's happening right now is that there is log-linear hill climbing across all modalities. And that means that there is a very direct relationship between each order of magnitude of compute that we apply, each order of magnitude or each incremental increase in data and climbing on benchmarks, whether they're public benchmarks, internal benchmarks, their, you know, targets that we focus on with reinforcement learning environments. And that is a very important observation. Those predictions that I think we're all making, I understand why some people are sort of skeptical of them or raise questions, but they're very grounded in the sort of empirical observations of over a decade of increase in performance of these models. I mean, essentially the same general-purpose architecture has seen 12 orders of magnitude more computation applied, a trillion-fold increase in flops over 15 years, and basically has worked in audio, in image, in text, in code, you know, in many other time-series prediction tasks. And so we're basically extrapolating out that, you know, more orders of magnitude of compute will enable us to continue to climb in this log-linear way inside of other environments. And then it raises the question of are we going to be able to train models that can invent new knowledge? Not just sort of extrapolate from existing data that we have, but actually teach us things that we don't know and make new discoveries. And then the second thing is, do they have the capacity to, you know, self-improve and accelerate the process of deciding which hypotheses should be set, which ones should be pursued, how to generate training data for each of those, how to factor those into new runs, or even innovate on the actual architecture itself. So I think both of those things need to be true to be able to see this compounding progress. But I think we're going to continue to get massive gains just from applying the next few orders of magnitude of compute, and that probably does achieve parity with human performance on many, many more tasks, just as we've seen that happen in the last six months on coding.

I'm not so sure. So I mean, coding obviously you can verify the correct execution of code. It runs or it crashes. But there's a ton of nuance in that. The quality of the code that gets written really matters. Its extensibility, how reconfigurable it is, how useful it is in practice. So it's not just that a piece of code runs. It's like how does a model actually use it as a DevOps or an SRE in production, to kind of return to that piece of code that it's written and then use it in a practical and useful way. And then, of course, you have to grade the quality of the output that has been produced. Like it may be high-quality functioning code, but is it actually the app or the website that you wanted? And there are aesthetic judgments in that, there's commercial judgments in that. So the challenge of internalizing non-verifiable rewards is present in code, even though code is still primarily a verifiable reward signal. And I think the other thing to observe is that like chat is also a non-verifiable space, and yet we've managed to climb that to basically human-level performance through interaction with real-world usage that provides a very...

Well, so I think many people are having long conversations, meaningful conversations with AIs at human-level performance. I mean, the quality is exceptionally good. It has very good emotional intelligence. It's broadly very accurate. We've minimized the hallucinations. We don't talk so much about bias anymore. It's grounded in real-world observations. I think by most people's measures we've got to, you know, human-level performance in conversation for quite a wide range of tasks. Now, I mean...

I mean, my measure is like when I turn to my assistant and ask it, you know, to provide me with a daily briefing summarizing all the conversations that have happened on Teams and on email, the updates that have happened to documents, and I get basically a synthesized summary with the set of actions that I should take next, which is basically better than what my chief of staff can produce. I would say that's human-level performance in synthesis, you know, analysis, proposed actions and chat. I mean, there are many, many millions of people every day that are using it for emotional support, for counseling, for therapy, for coaching, for advice. I think it's one of the most popular use cases inside all of the chatbots. So that's a pretty robust measure, I would say, to make the claim.

Well, I mean, I think if you define chat as an interactive exchange between two parties, one of which in this case is an AI that broadly satisfies some goal. You're looking to learn the sports score. You're looking for advice on which restaurant to go to. You're looking for coaching and feedback on an essay that you've written. You're looking for suggestions about which job to, you know, take next or some tough conversation you're about to have with your manager. You get a response. You go back and forth. You have five or six exchanges and you find a useful output which you might otherwise have to go rely on an expert friend or, you know, even pay a coach. There are, I mean, just objectively empirically speaking, hundreds of millions of people that get that experience every day from these chatbots. So maybe we could quibble over whether that technically represents human-level performance. I think it's a fairly reasonable thing to claim and, you know, I think there's no reason why that isn't going to continue climbing, right? I mean, the rate of climbing in the last three years is the thing that I think is most staggering. And so what we're trying to do from this point is extrapolate, okay, what are the fundamental drivers of that climb? Compute, data, interaction from real-world users. And, you know, those things look set to continue. So I think that I would expect that they apply to many other domains too, not just sort of, I don't know, chat or emotional support and productivity and, you know, and that kind of thing, but many other domains beyond that, to healthcare, you know, to live production deployments inside of education, you know, to assistants that are increasingly managing your home, you know, looking at your, you know, just everything that is in your everyday life basically to make you kind of more productive. So that's, I think, a trajectory that's likely to continue.

No, actually I think there's actually quite a lot of differences. I mean, the first thing to say is that the way that you curate the data, we start right from the top of the stack, is that we basically have, you know, paid for and acquired an extremely high-quality, very conservative set of data and extracted a lot of the noisy, distracting, low-quality, potentially security-risk issues to do with that data. And the methods that you do for that, I think, are actually like quite proprietary. We just shared a 109-page very detailed technical report which was very well received on Twitter, which shares a lot of the details on how we do this. I think the second thing is, whilst I think it's important to be quite cautious with architectural choices, and we have been, there are also a number of pretty significant shifts that I think we've made in sort of how we put together our training runs. So our training runs have been incredibly stable, very few crashes, very few restarts. We shared a lot of those graphs to show infrastructure stability and also MFU efficiency. So model flop utilization, which is basically shows that we can put, you know, a basically state-of-the-art number of flops through each chip for every step in our training run. So I think that this is extremely easy to get wrong and, you know, we all hear lots of stories from different labs about how things do go wrong and it actually is pretty hard to make the very careful and deliberate choices to get things right and take the right approach to make sure we produce high-quality models because our job and our ambition is to try and build this hill-climbing machine. That means the integration of the silicon with the models with the super high-quality data with a stack of RLEs, reinforcement learning environments that allow us to basically systematically hill climb against any objective that we choose. And that's what MAI-1 thinking model is. It's a general-purpose, fairly neutral thinking model that is pretty good at coding. It's now roughly on par with Opus 4.6, at least on the benchmarks. We haven't deployed it at scale into production, so there's still lots more work to do there, but it's an extremely strong reasoner, 97% on AIME, which is the primary measure for its reasoning performance at least on the benchmarks. It's very good at instruction following, and then the goal is basically to make that available to many, many developers and enterprises and allow them to climb on it for their use cases because everybody has a sort of slightly different objective that they have in their company to try and build, you know, agents and so on that support their use case.

So, there's definitely lots of shortcuts to the frontier. And if you take a super high-quality model and you sort of like polish your base model with high-quality instructions or answers or outputs from a superior model, then it's true that the model might quickly fit to that distribution, but it's very unclear that they would then be able to surpass that teacher. And so we've been very deliberate for two reasons. The first is that we want to make sure that we can exceed the teacher in order to set the frontier ourselves over the next few years. And the second is that we really want to build one of the great labs, and it's going to take us, you know, many years to come, probably the next two, three years. But in order to do that, we have to be able to show that we can actually build every component ourselves. We can hire the very best talent in the world. We can push the frontier with actual research rather than just re-implementation, copying or distillation from any other third party. And we're in a great position where we're able to really carefully and meticulously pursue that objective, knowing that we have the resources to buy Anthropic models where they sort of exceed the frontier. We have the resources to, you know, put 11,000 different models inside a Foundry so every one of our developers gets pure optionality. And of course we have the resources, you know, to continue to deploy OpenAI models which are obviously outstanding and are at the frontier today. So that's just a natural part of the self-sufficiency mission and it'll take time for us to, you know, truly get to the absolute frontier on that. But I think we're in a great spot. We made a ton of progress. I mean, this is a very, very strong model. And it wasn't just that model that we released. We've released seven new models simultaneously. Our Transcribe model, for example, 1.5 is literally the number one in the world. It's the most cost-effective of any of the hyperscalers. It's the highest on accuracy. Our image model is now number two. Our image editing model is number three, right behind, you know, Google and OpenAI. So I think we're well up there with our image and audio. Our code model, Code Flash, is incredibly strong, optimized for VS Code. You know, really, really a great model that's on par with Sonnet 4.6. So it's really in a great spot. It's been...