Hi everyone. My name is Arjun Karp. I'm senior technology correspondent at CNBC and it's my pleasure to speak to Kevin today. Kevin, good to meet you and good to speak to you. Look, really big topic, agentic AI, but I've heard about 101 different definitions for agentic AI, AI agents. What is it in your view and how do you envision this playing out?
I think simply put, what we want with AI agents are things that we can delegate increasingly complicated work to, the same way that you would think about delegating work like a task or a very complicated project to a collaborator or someone who works with you in your enterprise or something that you might seek assistance for in just normal ways like a consultant or whatnot. So the agentic web is a collection of these agents with all of the platform infrastructure available so that they can not only complete these tasks in the narrow sense by making calls into models to do inference but to also communicate with each other, to be able to make calls to effect action on things in the real world, to pull information into them that they need to complete the task, to change state in other systems. And so in a sense, the agentic web is really a lot like the set of infrastructure and the platform that emerged with the web in the early days.
And I know you've been thinking recently a lot about the path there, particularly as it pertains to infrastructure. So what to you at the moment are some of the big challenges that are ahead that need to be solved?
Well, I think the thing that we've seen over the last handful of years is the models are certainly becoming increasingly capable at reasoning and reasoning and planning and coordinating relatively complicated things. But the things that are missing are, if you are a provider of a piece of content or a service, like you have a website or you have a product or an application and you want them to be accessible to agents, some of the plumbing there doesn't exist. It's beginning to emerge right now. There's this beautiful thing called the Model Context Protocol, MCP, that was started by Anthropic that is one of those really essential plumbing components that's emerging, and you can just see the fast adoption of that by lots and lots of people is telling you that that is an essential piece that was missing. The thing I think we're all going to hear a lot about over the coming year is memory, because if you think about delegating work to an agent the same way that you think about delegating work to a human being, if the agent doesn't have any memory, it's a weird limitation on what is possible for it to do. So the ability for the agents to remember past interactions with you, to remember the solutions to problems that it's already solved, is super important. And I think increasingly you're seeing memory talked about by infrastructure providers and becoming features of products. And the important thing with memory, the thing that I spend a bunch of my time thinking about, is how you can ensure that this agentic web is as open as humanly possible. So one of the things with memory is you want memory in agents to be kind of like a document. You want it to be a thing that you own as the person who's created these memories by interacting with agents, and that you can store it, you can move it around into other contexts, that it can't be locked up inside of one place where you can never get at it and there's no way to build tooling around it.
There's also this idea though that you want the agent to be able to have that memory but also not become an echo chamber. So how do you get to that part?
Well, it depends on what you mean by echo chamber, but I think having portable memory where you can combine memories from multiple different interactions with multiple agents that you're having ensures that you've got some diversity of context that the agents can reason over. One of the things that our teams at Microsoft have been doing a bunch of research on the past four or five years is the psychology of agents. And you can get into very weird spots with agents, like these echo chamber effects where they get siloed and can't break out of a strong attractor so to speak, and the thing that you've asked them to do without being able to really have some control over the memory context that they have.
And to get to this idea of agents that you're laying out here, do we need the models to get even bigger than they currently are?
Well, I think the models are going to get bigger than they already are just because there's this very clear path and we're not yet at the point of diminishing marginal returns here on scaling models up to make them more capable at reasoning against more complicated things. So it does actually that there are features of models that aren't necessarily about how big the model is in terms of number of parameters that are very helpful for dealing with memories. So you'll hear people talk about context windows, the amount of tokens that you can send into a model at inference time. Long context windows are kind of a hack, I think, in terms of getting a real complete memory. What you eventually want is models that are really good at planning how to accomplish a task, and they can construct a memory on the fly by accessing a bunch of resources that contain fragments of memory, in the same way that you reason. If someone asks you a question about a particular thing, you don't consult every memory that's stored in your brain in order to answer a particular question. You just consider the bits and pieces and fragments of memory that you have that are relevant to the thing that you're trying to answer. And the other really interesting thing I think about agentic memory that will be borrowed from biological memory is we have a way to recall things relatively imprecisely in the beginning but we can refine our recollections to become very precise. And most of that refinement of memory to get them to be precise is about reasoning.
We were talking about model sizes and you mentioned scaling here and I know there's been this big debate whether we're reaching the limit of scaling laws at this point. You don't believe that, do you?
Well, I don't think we have any concrete evidence in the modeling that we're doing that we're at the point of diminishing marginal returns yet. And the scaling laws with AI models is a bit like Moore's law was with semiconductors. For most of my career as a computer scientist it was the dominant thing that we were thinking about. It drove all of the progress in the world. But it wasn't one continuous process. It was a series of breakthroughs that the chip designers and the material scientists and the physicists and the computer architects made in order to continue that doubling of price performance every 18 months or so that we saw for so many years. And the same thing still looks to me from things that I have visibility into like we have at least two more very major turns of the crank left for the scaling laws. Probably more but it's hard to see past the next two, and that's the next four years. And I think what we've all seen is what can happen with these AI systems in just a year, much less four, is kind of extraordinary. So tons of space for improvement left.
If we think about infrastructure, you mentioned semiconductors. Let's go down to the picks and shovels, to the data centers. Compute underlines all of this. So I want to approach how Microsoft is dealing with compute. I know you're investing billions of dollars in data centers. You're also getting capacity from others, Nebas for example as well. So is there a big massive crunch on capacity right now and how are you solving that?
Massive crunch is probably an understatement. I think we have been in a mode where it's been almost impossible to build capacity fast enough since ChatGPT and GPT-4 launched. So we have this enormous demand for compute. And we've been very aggressive about prepositioning capacity, building data centers, contracting for power. We're doing a whole bunch of increasingly complicated things to make sure that we can service this demand. But even our most ambitious forecasts are just turning out to be insufficient on a regular basis. And so I think we have deployed an incredible amount of capacity over the past year and it will be even more over the coming handful of years. We just lit up a data center called the Fairwater Project that is a gigantic training facility that will be used for nothing other than training the next set of frontier models. And I spent hours and hours and hours a week for two years planning this data center, trying to make sure that it's the right architecture. It's a very unusual data center. It doesn't look anything like the ones that preceded it.
