Kevin Scott, CTO @ Microsoft: An Evaluation of Deepseek and How We Underestimate the Chinese

This is the best time to be alive if you have an entrepreneurial spirit. I can very clearly see what we're doing now and what we're doing next, and I don't see the limit to the scaling laws. I don't believe in this one agent for everything sort of theory. I think you'll have a lot of agents, and the reason I think you're going to have a lot of agents is because your product managers are probably going to have to be domain experts. The agents will definitely be less transactional, less session oriented going forward.

I think the thing that you just described, which is all of a sudden things have gotten a little less clear than they have been, is exactly the thing that happens at the beginning of every big technological paradigm shift and every new cycle that's driven by it. So it was super confusing in the early days of the internet, and I think it was super confusing in the early days of mobile, where everybody had these ideas about what was going to be valuable, and very few of those ideas were actually the durable ones that proved all the way through. In those moments of transition where there is this confusion, what have you learned as the right action to do? Is it to be active, to iterate and learn, but you'll make so many mistakes that you regret, or should you sit on your hands and watch others make those mistakes? Oh, God no, you definitely shouldn't do the latter. So this is the best time to be alive if you have an entrepreneurial spirit. The thing I think that you have to do in these moments is not forget the things that you've learned from the past moments about what works. It's not like, okay, do this specific thing, but it's how you go about doing that exploration that you just described, which is product matters. I've been saying this for the past couple of years: models aren't products. Because everybody was just so fascinated by the infrastructure itself. This is also a characteristic of the beginning of these cycles: you have technical people who get just swept up in the technical bits and they kind of forget that the only thing that really matters is making good product. That's where we're at right now. You have to make good product. You have to have ideas and have conviction, and then you have to go get stuff done really fast so that you can see whether you're full of crap or not about the conviction that you have. You have very few patterns at the beginning of a cycle to snap to. You're not looking at someone else's success and saying, okay, I'm going to do that but just a little bit better. You're trying to figure out something completely new, and the only way to figure that out is you got to launch stuff and gather data and iterate, and be super super brutal with your own self about what you're seeing. You can't love your idea so much that you overlook what it is you're seeing about the data and the feedback that you're getting.

No, no, no. They're super valuable, but they're only valuable to the extent that you can connect them to things that users need via product. So in the limit, I think product is the most important thing. Now, if you build good models and you build good infrastructure around models and you have good efficient compute and all of these other things, you're going to get lots of ability to monetize all of those things because as people build those products, they will need to consume your platform and your infrastructure. That's all good. But most of the value has to be in the products. We don't build infrastructure just for the sake of infrastructure. We build infrastructure so people can make product.

Again, if you look to past cycles, you've got a pretty good mix of where value gets created across startups and new ventures and existing enterprises. So I think everybody's kind of doing the same thing. You're trying to discover the new. If you are a big company like Microsoft with a long tradition and a bunch of successful things already in the market, the thing that you are trying to do is figure out what of the things that you already know super well and which of the customers that you're already serving super well can you do for them with this new set of capabilities that you can provide. Hopefully, I run among other things Microsoft Research, so I also have a charter of, hey, can we go try to shine flashlights in places that no one else has shown them before and try to discover some super disruptive brand new things. But that's kind of the job of the startup ecosystem as well. I also am an angel investor and I advise startups and I've worked at startups. So I think it's just really important that you've got lots of people hunting for those interesting new things. I have super high conviction on that in the AI platform transition that we're going through right now because it's impossible for any entity like Microsoft or any other big company to have enough imagination and enough perspective to know what every interesting thing is. So having just this vibrant ecosystem, lots and lots of people sort of exploring where value exists, I think is incredibly exciting and necessary. I also think that there has never been a moment where the tools, the infrastructure, and the platform are as cheap and accessible and available and easy to use as they are right now. So it's just super easy to pick stuff up and just go get crackin'.

That's a ridiculous statement. I can very clearly see what we're doing now and what we're doing next, and I don't see the limit to the scaling laws. If you're just thinking about the raw capability of the models and how well you can condition them to reason over increasingly complicated things, I'm sure there at some point will be maybe a limit. I intuitively feel like there must be. There are some people who don't believe that there's a limit, that the limit that human beings have on intelligence is you've got so many neurons packed into your skull and you got about a 20 watt power envelope, and that's the limit. Some people believe that with AI, there is no such limit and things will continue to scale into weird territory. I don't really like that. I don't necessarily believe we will get to some point where we'll hit a scaling asymptote and there'll be diminishing marginal returns, and it's so expensive that we will decide it's not worth spending that next dollar to make this thing one unit smarter because we haven't figured out how that translates into something that's useful for the people who are using the tool. I think that point will come, I just don't see it yet. It's not in the viewfinder right now.

Yeah, the mix of synthetic data is going up. High quality data is becoming much more useful, especially in the post-training parts of the model production pipeline, than low quality data. So I think we're clearly at the point now where if you have the right infrastructure and you have super high quality data and super high quality expert human feedback, you can amplify that into the right set of tokens for training bigger and bigger models. That stuff has way more value than just the undifferentiated tokens that are floating around on the web.

There's a super interesting thing right now that we don't have around assessment. It's very hard to know quantitatively what the incremental value of a token of data is to the quality of a model that uses it in its training. In other words, if you're asking, okay, I think my data is super valuable, and if this gets used in a model it's going to make the model better, most of those assertions that people make are unfounded in any kind of science. No one's gone and done the measurements. The measurements we do have show that there's a pretty big disconnect around what some people think valuable data is versus how valuable it actually is to producing capabilities and models that are legitimately useful. Most of what's legitimately useful is models. People who think of models as repositories of factual information are treating them like the world's worst and most expensive databases. That's not super useful. We've got search indices and databases, and those things are plenty good enough for retrieving information. What you want models to be able to do is to be able to reason over information. So if you give them access to information, how well are they able to reason over a set of information to go do something that's useful for you? You just need different tokens for training a model to make them good at reasoning than you do at making them recallers of facts.

I think the thing that most people miss, although the DeepSeek R1 launch a few weeks back kind of clued everybody into it, is that we just have an incredible track record over the past handful of years of repeated year-over-year mind boggling progress in optimizing the performance of models so that inference is just better and better. Over time, the models have gotten bigger and the API calls have gotten cheaper. A little bit of that is because you get maybe a 2x benefit in price performance from hardware every generation if you're lucky, but you get a much bigger improvement in price performance from all of the things that you're doing in the software stack. There's just a ton of work happening there. The DeepSeek R1 stuff, which was good work, is the way you should think about it: it's a point on a line of price performance improvement that maybe was invisible to everyone else but not invisible to the people who are neck deep in optimizing these systems. And it's not the last point. It just marches on.

We've had models more interesting than DeepSeek R1 that we chose not to even launch. I was surprised at how interesting people thought that it was. They did good work, so don't get me wrong. It was good solid technical work, and it was super cool. They chose to release this thing and make it open source. It's really interesting seeing how the public reacted to what they did.

Even when you've made it as easy and cheap as humanly possible for folks to go do something, they still have super strong preferences about the how. We're paying very close attention to that. We have to give people more how than we have been doing, because I think developers want lots and lots of choice.

When I was a graduate student, I was a complete open source zealot. As I've gotten older, I've become a lot more pragmatic. It's probably more important for me to make a set of pragmatic decisions about how I'm going to go build these things rather than singly optimizing for my curiosity.

I think there's going to be lots of both. Part of it is, let's just forget about AI, which is the controversial thing at the moment. It's a thing where industry structure hasn't settled yet and we don't know exactly what it's going to be. But you just sort of pick your previous things, like search for instance. There's a whole bunch of open source search engine projects out there. People who want to have a search feature in their application or who want to go build a search engine themselves have lots of options. They can go grab something open source as a starting point, they can stand up a product, they can go load their data into something like Azure Cognitive Search, which is a search as a service platform. Google has one, Amazon has one. They're readily available. And you still have search engines like Bing and Google that are out there. So they all exist. All of the economics in search go to somebody who stood up a gigantic infrastructure and who's running a whole search business with its own feedback loop. I think we're probably going to have similar sorts of things happening here. For the infrastructure layer, you're going to have lots of open source infrastructure products and people are going to use them in lots of different ways. But you're also going to have a lot of people who don't want to have to go stand up their own infrastructure from scratch or to take an open source project and go build it out where it's lacking for the things that they need it to do. It's good to live in a universe where you have both of those things.

I think it's a reasonable step in the right direction. The thing I've been saying for a few years now is that I think one of the most interesting things happening with AI is we've had one paradigm for using computing devices for effectively 200 years since Ada Lovelace wrote the first program. If you want a computing device to go do something for you, you have to be a programmer yourself, which is a pretty high barrier to entry for a lot of people, or you have to rely on the fact that a programmer has anticipated some need that you might have and packaged up a piece of software into an application that you were able to run. Those are the two ways you can get a computing device to do something for you until now. The thing that changes with AI is it can understand a thing that you want your computing device to go do for you, and it can figure out a way to go make that thing happen, and you don't have to be a programmer. It's kind of a profound change. I don't think this is next year, but it's probably not going to be 10 years. This whole notion that you have teams of people whose job is to go anticipate a bunch of very granular user needs in some narrow space, and then they're going to go write a bunch of code and figure out how to hang that code onto some user experience, and they hope that they've done a good enough job and they've anticipated the needs in the right way and they've designed the user interface in the right way and they've gotten all of the code right, and they just sort of grind away on figuring out what that feedback loop is—that's going to change. You just aren't going to need as much of that anymore. What you're going to need instead—it doesn't mean that completely goes away—you will still need all of the capabilities that these applications provide, but you're probably going to want some kind of agent actuating those capabilities on your behalf rather than you having to do this weird impedance matching that we've got right now between how a user has a set of expectations and how a product team has imagined what those expectations are.

You're still going to have to have people who build capability infrastructure to make this thing happen in the real world, provide access to this weirdly situated repository of information. There are just a bunch of capability things that people will need to build. But the user interface that surfaces those capabilities will probably be agents. Product managers—I don't believe in this one agent for everything sort of theory. I think you'll have a lot of agents. The reason I think you're going to have a lot of agents is because your product managers are probably going to have to be domain experts, people who deeply understand something like medicine or drug discovery or early round venture investing. They will have to deeply understand the idiosyncrasies of that domain and help set up the feedback loops that help agents assisting people doing those tasks get better and better. So a little bit like the combination of the product manager and the users of the agents teaching the agents how to be better and better at the things that you're trying to get them to assist you with.

I think usage always follows utility. You make useful things, they get used a lot. Clearly with software development agents, we're getting a lot of adoption right now. We've gone very quickly from developers being skeptical about these tools to, you will get this from my cold dying fingers. I think of this as one of the most essential tools in my toolkit, and I will never give it up. The agents are becoming more and more powerful.

There's no lock-in in search. You can send your next query to a different search engine than the one you're using right now, and yet you don't. The reason that's true is it is our job building these agents to grind and grind and grind, and go every day try to make the agent better and better and to do more and more of value for our users. If you do that and you do it well, they will continue to choose you.

I think the thing that's missing right now with our agents is they are conspicuously missing memory, which makes them awfully transactional. Even in the places where agents have memory, it's a pretty limited form of memory. So I think one of the things that's going to happen because I know lots of people are working on it right now is memory is going to get a lot better over the next year or so. That means as you're using an agent, it remembers more and more about your past interactions with it, it will be able to conform itself more and more to your preferences, and it will be able to do things that we do very naturally, which is you solve a problem once and you record the solution and then you don't go and solve it from first principles over and over again. Memory even gives you the ability with these agents to have some kind of abstraction and compositionality where you can just sort of build up more and more powerful ways of doing things inside of the agent over time because it's remembering the past things that it's done and learned. So I think the agents definitely will be less transactional, less session oriented going forward. I hope we get more asynchronous things happening over the next 12 months. Right now it's very interactive: you go to your agent and you send a prompt in and it goes and does something immediately and gives you the response back. I think there's going to be more over the next year of you sort of dispatching your agent to go do something and it goes and works while you are not paying attention to it. The thing you want with agents—we should just never lose the plot on where we're going. The first generation of agents are good at five-second tasks, and then the generation after that are good at five-minute tasks. What we're going towards are things that you can delegate increasingly complicated tasks and increasingly beefy work to over time, the same way that you would to a coworker. That's what everybody's going to want, and that's where the capabilities are headed. So you need to think about how to build product around where the future is almost certainly going to be, and what you need to go augment these systems with to allow them to do more of this thing that ultimately we want. You don't want a thing that's just a good email summarizer. You want something that you can sort of tell it, I get up every morning at 5:00 a.m., please at 5:00 a.m. every morning digest all of the email that came in overnight, draft responses to anything urgent, and show them to me while I drink my coffee. That ought to be an entirely possible, doable thing.