How a 3 Trillion+ Company Thinks About AI Microsoft CTO Kevin Scott South Park Commons

Yeah, I think maybe the most interesting minus one moment I had was when I was nearing the end of my PhD. The thing that I was working on was super intellectually stimulating. I was working on this stuff called dynamic binary translation. And I was just like, yeah, I got to go find something to do where the thing that's the first priority is impact. So, I left academia. I got super lucky. Got a job at Google.

Yeah.

Yeah.

Yeah. I mean, I think maybe the most interesting minus one moment I had was when I was nearing the end of my PhD and I thought I was going to be a computer science professor and it was all I wanted to be from the time I was 16 years old until I was in my late 20s and I just had this epiphany that the thing that I was working on was super intellectually stimulating. It was really really interesting to me and not really all that interesting to anyone else. So I was a compiler optimization programming language computer architecture person. I was working on this stuff called dynamic binary translation and I thought it was great. And in success this thing was going to make a bunch of stuff temporally like a few percentage points better on a benchmark. I was going to publish some papers like a few dozen people were going to read them and cite them and I was going to have spent all of this energy having marginal impact. And you sort of couple that with the fact that I was broke. I was making 18 grand a year, trying to take care of my mom and brother, pay my rent, and my car payment, and I was just like, yeah, I got to go find something to do where the thing that's the first priority is impact. And so I left academia. I got super lucky and got a job at Google before the IPO. It's really funny like I didn't even know why search was an interesting problem. I just knew that a bunch of my compiler buddies, ors and Allan Eustace and Jeff Dean, Sanj, just an inordinate number of system software people had gone to work for Google. I didn't know why but I was like all right well resume in. When I joined the thing that I saw at Google was the same thing that I saw for myself as an academic that people were gravitating towards these intellectually interesting problems that if you just looked at them even a little bit you were going to notice that they weren't going to have any impact. And so I was just determined when I got to Google it's like I'm going to go find a thing and I don't care how intellectually interesting it is like this thing is going to have impact. And it was this weird thing. It was called the ads approval bin automation system. And the problem that it was solving was at the time every ad that was going to run on Google had to be reviewed by human beings. And so we couldn't hire people fast enough to do all of the reviewing. And the review stuff was very simple. Most of the time it was like you can't use superlative in your ad copy. Like you can't have repeated punctuation, like you can't have three exclamation marks at the end of something. But some of it was complicated like you can't have advertising that is for adult content if the keywords aren't adult content. So you can't have deceptive redirects on clicks. It was blocking $50 million worth of inventory a day from running. So it was like a big dollar value problem. And some of it wasn't super sexy to fix like the no superlatives and repeated punctuation was really like a regular expression. So you just needed to build some plumbing that was going to solve a workflow problem and then you needed to build a little bit of machine learning. And so it was...

Yep.

Yeah.

Like a bunch of the things I can say here just pithy, not novel insight. I mean I think you actually do have to, when you're pre-revenue and you're hunting for product market fit, you just have to be really really relentless about your willingness to pivot. Because effectively what you're doing is exploring this optimization landscape. You have no idea exactly where you're going and you just have to be quick about running experiments. Yeah. A super super dangerous trap that people get into all the time when they're pre-product fit is getting super in love with an idea or super in love with a piece of technology. And you just sort of spend way way way too much time working on that idea in a vacuum of real feedback about whether or not it's useful. And so just getting to the point where you can try things as quickly as possible. But that's not unique, right? Like everybody should know that at this point. The thing that I've always tried to do, maybe to my own detriment, is when I left Google and did a startup, I could have done a whole bunch of things and the thing that I chose to do was a thing that just very clearly made sense to me was going to have impact. So, it was mobile advertising and this was right before the iPhone. I knew the iPhone was coming. And I had a suspicion that it was going to change the entire landscape. And you kind of knew that if you just sort of looked at the technology trends, you knew that not necessarily the iPhone was going to happen, but your wireless networks were getting faster, batteries were getting faster, screen technology was getting better and better. You knew that at some point a whole bunch of the power of computers was going to converge on this mobile form factor. And if you believe that, you know that people are going to be building mobile apps and as soon as you've got mobile apps, you need distribution for them and you need monetization for them and so I'm going to go build an ad network. It was real clear to me that I should go join this company because I knew a bunch about advertising from working at Google and I knew nothing at all about a whole bunch of other things that that startup. And so I had a little bit of comfort zone on here's a problem space I understand. So I know I can make progress on that and that will give me an opportunity to go learn a whole bunch of other things that I don't know whether I'm good at or not.

Yeah. 2007 and we got bought by Google in 2010.

Yeah. And look, it was the fastest growing thing I think still I've ever seen. Like we were doubling everything about the business was doubling once every four and a half months. It was just super challenging just to keep the infrastructure up and running. I think there's great advice in there. I think one of the things that I find, because the natural temptation is you come out of Facebook, you come out of these other places, you're like, 'Oh, let me go make sure I talk to enough customers.' Your proxy for impact becomes near-term customer validation. I think that's very important. But I actually think the first thing you have to answer is having a very strongly opinionated point of view on where the world is going to be 5 years from now.

Yeah. And look I think you also I mean it's a bunch of things that come with that. So you have to be very honest with yourself about the difference between what you wish is going to happen in the future and what must happen in the future. And they're very very different things. And a lot of us, I think it's especially hard for entrepreneurs because entrepreneurs are used to being able to force their will on the world. And so you can convince yourself that you can make a whole bunch of things happen. And thank goodness you actually can. But that's different from what must happen, the thing that's going to happen whether or not you apply your will to it. And it's really really good I think to have a point of view about what must happen because that's going to be the landscape in which everything you do is going to have to operate. Correct.

Yeah.

Well, I think it gets back to this trying to understand what must happen in the world. And so there were a handful of technological things that had happened in machine learning leading up to the point where we did that first investment in or the first big investment in OpenAI. That were the first glimmers that I had ever seen in my career of these AI systems becoming generalizable platform components, things where you're going to invest a ton of time and energy in training a thing and it was going to be useful for more than one narrow thing. So, for a while I ran the team at Google that did the CTR click-through rate prediction models for advertising, which is at the time was the singular thing that made the ad system work well.

Yeah. You basically it was the quality signal for the ad auction. And if you didn't have it, the ad system just wouldn't have worked. And it was an extraordinary system. It still may even today be the most valuable machine learning system. Meta's probably got one that's relatively equivalent. But that thing just prints money, relatively unsophisticated algorithm, hugely complicated infrastructure, only useful for that one thing. No matter how much money you invest, it's only ever going to be good at predicting the click-through rate of not ads in general, but the ads in this particular context. But everybody had been looking for, is transfer learning ever going to work? Are we ever going to be able to have generalizable things? And it had begun to work. There were a handful of papers that were demonstrating the ability of these models to generalize and OpenAI just happened to have a very very credible theory about how as you apply more compute and data scale to training these systems, these are the ways in which they become more general and more capable. They had run a couple of rounds of these experiments and they were on prediction for the experiments they ran and it's a little more complicated than this but effectively the conversation was we need a billion dollars to turn over the next card. And for us, the possibility that this was going to be the most efficient way to make progress towards getting to generalizable models where you could have a world where you really didn't need a 100 teams at Microsoft each independently building their narrow vertical model for a particular thing, but you could collapse a whole bunch of those down into a single piece of platform infrastructure that you could invest in that had composability that acted like the rest of the software engineering universe that we know and love. That seemed like not a large amount of money to me at least to go run that experiment.

Yeah. So to give a billion dollars at that point I think is a story that is underreported on the early days like 2017 2018 and certainly more than I think Google was giving to the deep learning folks and Google Brain at that point.

Yes.

Could not agree more. I think that the technology is best when it's essentially positive sum and it just creates more abundance and that can flow through to all the different constituents. Moving on perhaps more specifically to the AI landscape. You've been around for a while now. What do you think is different about starting a company in this unique moment in time that we are today where it's so loud out there? There's so much money flying around. There's so much being written about companies that some of them do or don't have. It's just very loud, right? And there's also a lot of stuff changing up and down the stack in terms of capabilities from obviously the infra the foundation model providers to the infrastructure providers to the capabilities on top. So what is different? How should a founder think about building in this world today?

I don't think that we've had our last discovery on how you do pre-training or how you do post-training. I think there's lots of technical innovation. But if you want to start a company to do those things, I think you have to really really be careful and honest about what the business model is for that because for the most part you build an infrastructure and you basically are saying I'm going to create a platform company and platform companies are about absolute scale and the economies that come with the large absolute scale. And in the limit, the only thing that matters if you're building infrastructure for other people is, is this stuff getting more capable over time? Is it getting faster? Is it getting cheaper in a world where you've got intense competition? It's a really brutal game to go play, which is not me saying don't go play it. I think it's also a super fun game to go play, but it

It's probably hard for reasons that are different than your clever take on how to implement a particular algorithm or a piece of the technology stack. It's just got a bunch of complexity that is in many ways divorced from the technology itself. So if you want to go do that, you've got to get convinced that that's the game you want to be playing and where you can add some valuable contribution. I think everywhere else, people should be infinitely pragmatic about how they're going to solve an interesting customer problem. And I think we were talking about this a little bit earlier. There's this gigantic capability overhang that I think we have right now with these AI systems where they've already, forget about what's going to happen in a year, they're already more powerful than what people are using them for. And a lot of the reasons that people are waiting around and not solving problems is that some of the things you need to do to squeeze the capability out of these systems is just ugly looking plumbing stuff.

Or grunty product building. But I'm sure this is true for you as well.

Yeah.

It's not clever. It's grind.

Yeah. Where do you think the opportunity there is?

Correct.

I think we will see. I'm kind of excited to see progress on both dimensions. The thing that I can't really square with the open source stuff is if we still are in the domain where we get serious returns on model capability from scale, I don't know what mechanism for open source model training is better than a commercial model to drive scale. So it's a little bit different from getting Linux built open source. With Linux, you can have a whole bunch of clever people who want to spend nights and weekends making contributions to the code base, and the whole ecosystem gets better as a function of all of those contributions. But when you download and use the open-source model, you're not quite doing the same thing. You certainly aren't initiating a new round of pre-training or post-training on the model that goes back into the ecosystem and makes the weights part of the model better. And yeah, there are some contributions happening in these open source models around the ecosystem and a bunch of performance optimization work that's happening that I think is really super valuable. But again, I think that scaling thing is the thing you have to think about. But the category error I think here is thinking that it's got to be either or. I just don't see that at all. Even Microsoft uses both. We have them available in our cloud, in our own products. We're using a mixture of things. So again, going back to my earlier point, I like worlds where we're fixated on products and product outcomes, and you just leave it to the engineers to sort out what the infrastructure is going to look like. As long as the product's amazing and it's getting better and cheaper and faster and higher quality on all the dimensions that people care about, what do you care what the infrastructure is like? Whether it's a federation of open source models or your L1 cache is an open source thing running locally and your L2 is a prompt that goes to a big expensive cloud model, whatever solves the problem, right?

Yeah, and I look, I love the open source models. I was, I still am, I have infinite amounts of curiosity. So being able to tinker around with stuff, see how it works, I think it's nothing but positive.

Yeah. I mean, I think it's pretty clear if you are on or near the frontier that for pre-training at least, we're well past the point where there are enough tokens organically occurring in the world to train bigger and bigger models. So it has been the case for quite a while now that the super high capability frontier models that are trained with a ton of compute have a lot of synthetic tokens that they're doing training on. But I also think it's clearer and clearer to everyone that having experts providing feedback in the post-training part of the process is really critically important. And a lot of this, this is a thing I would encourage you all to look at to the extent that you are doing custom post trains on models at all yourself. You have to think about where advantage still can exist now in you understanding your domain and being able to identify experts that you can then use as part of your post training process to make your particular post train for your application better. So I think that's certainly a thing we're going to continue to see, and we'll see it in a whole bunch of ways. A bunch of this stuff, if you can get to scale, can happen with clever UI design inside of agents. I was on Gemini the other day. Yes, I do use my former employer's product.

Yeah. Yeah. Yeah. But the other thing that's really important when you talk about data, and this may be the most important dimension of data, is you have the data that you're using in your pre and post training part of your infrastructure building, but you also have all of the stuff we were talking about a minute ago, which is how do you hydrate memory? How do you make sure that you've plumbed through all of the data sources that you want your models or your agents or AI systems to have access to in order to actually solve problems? And so I think that maybe is the more important data, and it's the place where you're going to have the messier set of constraints, right? Because people who have proprietary data right now are on the one hand going to want to be able to use AI systems to get more value out of the proprietary data, but they're going to want to do it in a way that is not leaking a whole bunch of value out of their data.

Yeah. And so that's an interesting set of problems to go solve.

Correct. And so that's the thing I would just sort of encourage everybody, especially right now. The other thing that's happening with these AI systems is with all the coding agent work that's happening and how fast that's getting better, the cost of doing the experiments has never been cheaper.

So do the damned experiments. Try things.

Yeah.

Yeah. I mean, the other thing I would just really encourage folks is not be precious about the possibility of failure. Because one of the things, there's almost like this network effect that happens when things get easier and then we go move faster along those vectors because the thing has gotten easier. We also build up more appetite for more people to try things. So I think we have an all-time high right now in the ease of running these experiments, and I think an all-time high in people's willingness to try new stuff. It's dizzying right now. I had a buddy who is the CTO of a game company that runs in six week sprints, and he was like, 'I went into one sprint thinking that I was completely up to date on AI coding. I went heads down for six weeks to get this stuff done. I pull back up and it's like I feel like I know nothing now. The entire world has changed in six flipping weeks.' So I just think the appetite for trying all of this stuff that's changing is super high, which means it's almost this perfect network of experiments and folks who are willing to just look at anything to see whether it's worth something. I love surfing myself, and right now it's like the wave is long. It's kind of perfect, and you just get on that wave and you ride it for as long, and you just kind of not have to worry about the other people surfing the wave because if you look back then you're going to trip over yourself. You don't want to look too far ahead because you can't really predict what the wave will do. But right now it's fun and you just focus on your own form and have some fun with it, right?

I was chatting with Guido just last week about the GIL.

Well, look, I think the two things that you said are good. They need to have agency and they need to think of the thing as a tool that's there to help them tackle the things that they think are important. I think the question to ask is whether or not the AI systems are making people feel more or less empowered. For your kiddo, at 9 years old, it almost certainly has to be empowering to be able to draw a video game into existence. So in a certain sense, who cares what the mechanism is? They are accomplishing a thing and they are learning early to be fearless about how they do it. So I think kids in a certain sense are going to have the same advantages that kids always have when they're coming of age when a new technology is emerging really quickly, because they just won't be afraid of using it in ambitious ways and they won't have a whole bunch of preconceived notions constraining them. But yeah, I think a lot of it is going to be getting back to fundamentals. Do you have good taste about problem selection? Do you really understand how systems work and fit together? Are you really thinking about what you're doing in a service-oriented way? What am I doing with these tools that is of service and of value to my fellow human beings versus just screwing around inside the system for the sake of it? And I think the things that are true for kids are true for all of us. There are certain things for sure that are going to change in pretty dramatic ways, but I think everything that's changing is also presenting a set of super interesting opportunities for people because there is a positive empowerment mechanism at work here. My 17-year-old is a bio nerd, and I remember she asked me a few years ago, 'Do you think being a heart surgeon is a stable job given all the AI?' And I was like, 'Yeah, almost for certain. The population's getting older, heart disease isn't going to get cured anytime soon, and even though robotics is getting a lot better, a huge amount of medicine is about human contact and a bunch of super messy stuff that you can't solve entirely with technology, and people don't want solved entirely with technology.' So I think those are things too for kids that they ought to be looking for in terms of careers. Anything that you can look at and say, 'This is robotic and repetitious,' those are things that some technology, whether it was AI or not, was going to come get at some point because you've got the dual problem of this thing being irritating to me doing the work, and it's likely also irritating to other people.

It really is.

Sure.

Yeah.

I've always described myself as a short-term pessimist, long-term optimist. So my short-term pessimism is like everything's [ __ ]. It's just awful. And I carry myself through the day with that spirit. And I had a boss, I don't think he'll mind me saying this, Jeff Weiner, who was the CEO of LinkedIn, a dear friend, and I learned so much from him. I had a one-on-one with him one time where he looks at this grumpy [ __ ] piece of [ __ ] engineer who works for him and he's like, 'Dude, you're always unhappy. I'm going to help you reset your hedonic equilibrium.' And I was like, 'The [ __ ] are you even talking about, man?' It's like, I don't want my hedonic equilibrium set. I don't even want to be happy. That's not the first order thing. I want to do meaningful work. Meaningful work is hard and I'm not going to be happy while I'm doing it, but I will be content. And I think that's all I need to hope for in my life, is to be content doing meaningful things.

Learn to like, accept, and enjoy and appreciate the hardness. It is a privilege that you all get to solve hard problems.

Privilege.