Hi everyone, welcome to Grey Matter, the podcast from Greylock where we share stories from company builders and business leaders. I'm Heather Mack, head of editorial at Greylock. Today, Greylock general partner Reid Hoffman interviews Kevin Scott, who's the chief technology officer at Microsoft. In addition to his deep understanding of how enterprise organizations conceive, develop, and adopt new technology, Kevin is known for his dedication to figuring out how artificial intelligence can level the playing field for society. This interview took place during Greylock's Intelligent Future event, a day-long summit featuring experts and entrepreneurs in artificial intelligence. You can watch the video of this interview on our YouTube channel and you can read the transcript on our website, greylock.org.
Scale of it on all three vectors, joining you years into it, and that it's a very rare achievement. And then of course when Satya and Bill met him for the first time, they called me immediately after and said, 'Don't you think he should be the CTO of Microsoft?' And I was like, 'Yes, that's a good idea.' So now, let's actually start with the learnings around Copilot, because there may be awareness here about what's going on with Copilot, but the fact is it's a lens into the changing paradigm of how coding and engineering development is going to work. So why don't we start there? Describe a little bit of what's going on with Copilot and then describe how this is going to change how development is done.
Yeah, I think there are a couple of things to be learned from the Copilot experience that are relevant to folks who are building machine learning systems or companies around AI. So for those who don't know, GitHub Copilot is a programming assistant tool that can take natural language prompts—you can express a program you would like to exist—and it generates code for you. Shockingly, the performance of the system is improving at a pretty steady clip. When we made it generally available a couple of months ago, it was producing more than 40% of the code that its users were producing overall. Qualitatively, it's one of those tools where someone gets access to it—not everyone, there are people who don't like GitHub Copilot, which is fine—but many, many of its users are like, 'This is so valuable to me, you will get it from my cold dead hands.' Two interesting things about Copilot: one is that when we started development on it, we had evidence that large language models were actually going to be able to do this translation from natural language to code. Even when we showed people inside of Microsoft that this might be possible, we got a range of reactions from 'No, this isn't real, it's impossible, it's never going to work' to 'Maybe it'll work, but I'm highly skeptical.' A big part of what we had to do was overcome that negative bias to actually get going, because we had evidence not only that it was going to work but a really concrete plan for how we were going to make it better and better over time. I think that is a thing we're seeing across the board with these foundation models: people look at them and there's certainly a degree of hype, but there's skepticism that they actually are going to be useful for the things people want to build. The second thing, maybe the more profound thing about GitHub Copilot, is that it is just one copilot of a potential very many. What we were able to do with Copilot—automating this particular type of, not even automating, just assisting people with a particular type of cognitive work—is going to be directly applicable and replicable to a whole bunch of other domains. Any sort of repetitive cognitive work is likely going to have a copilot in the future. The model that powers GitHub Copilot, the OpenAI Codex model, really lets you think about software development in a different way. There is now a mode of software development where you can have a conversation, iteratively describing an application into existence. It's not one utterance or one prompt that generates an entire program, but you sort of say, 'Here's what I would like,' it generates something, and you say, 'Okay, that's good, but augmented in this way, change this way.' It's a multi-turn dialogue that you're having with this system. I've got dozens of these demos that we built inside of Microsoft using the API, and increasingly as people get access to the Codex API itself, lots and lots of people are seeing the power of this.
So first, one part of this: what is this going to mean for the quality of software development—error rates, security, ability to do new kinds of work? Give a little bit of the lens of what you're seeing in Copilot across these vectors and others.
Yeah, it's early days, so we will see. The thing to remember with GitHub Copilot and with Codex in general is that generating code with it doesn't absolve you of the responsibility to make sure your product is high quality, error-free, and safe. There's still a huge amount of responsibility that lies with the developer, the entrepreneur, to make sure they're building something that has integrity. The hardest part of building GitHub Copilot was not the AI bits that translate natural language into code; it was the safety layer that sits on top of the foundation model that looks at the input prompts and the outputs of the model and tries to make sure it's dealing with inappropriate biases, security, safety issues, and what happens if the model happens to parrot something that's copyrighted or under a license that would make it illegal to emit. There's a ton of work we had to do to build that layer, and there will be more work over time. The exciting thing is it does two things: one, it takes developers who are in high demand in a world with a boundless appetite for software and gives them a really interesting tool to help with productivity. But the more interesting thing is I think it opens the aperture on who gets to be a developer, which can be both good and bad. I'm on the good side of the equation, but mindful that you're going to have a whole bunch of human beings who are going to be able to build complicated things with this new capability, and we have to make sure we're dealing with all this security and safety stuff in a pretty rigorous way.
Well, one of the things that's pretty central around all these foundational large language models is the scale and how they're playing. Obviously we build these large ones—there's a set of comments I'm going to ask you about that, and then there's also going to be a whole bunch of other ones that are made as well. Just because you're building a 500-billion-parameter model, right, but there's still people who can go make and do pretty amazing things with a 6-billion-parameter model or a 20-billion-parameter model. So say a little bit about—let's start with—what are the implications of the fact that there is this massive scale set of models that a few number of companies can build, and what are the implications in the developer ecosystem and the possibilities in the world from that?
Yeah, so the models are getting—the foundation models, like the language models and increasingly what we'll see this year, multimodal models that build representations of the world across multiple different domains—they're just going to get more and more expensive to build because it's massive amounts of compute infrastructure required, and there doesn't appear to be a point of diminishing marginal return that we're approaching on scale. So you make them bigger—and I'm making this sound way easier than it actually is—but you make them bigger, they become more powerful at the tasks to which they've already been put in their smaller incarnation, and they also become broader at the same time. They can be used for a broader set of things than the previous smaller incarnations were able to do. There's plenty of incentive to go invest in bigger and bigger iterations of these models and to make sure the foundation you're building is more and more powerful over time. The way we think about it, and the way OpenAI, our partner, thinks about it, is we want to make them accessible through APIs so that you actually have a pretty rich third-party developer ecosystem building on top of the models. I don't think it's hard to imagine that any individual company, even one worth a trillion or two trillion, is going to have enough imagination and resources to build all the things that can be built that will serve the public good and humanity and produce a lot of value ourselves. It's exciting. Even I was browsing through the news yesterday and the number of excited articles about what people are doing with GPT-3, which is now a relatively ubiquitous thing two and a half years old at this point, but there's a huge amount of energy around people building things on top of this, and that's super exciting. It just gets better and more interesting over time.
One of the questions some entrepreneurs have asked is, given that there would be a relatively small number of the super-scale models provided through APIs, what are the kinds of ways they should think about distinguishing their businesses? What are some of the things you would think are ideas and lenses to keep in mind?
Yeah, look, I think to me that is the most interesting thing here. There is a bunch of interest in computer science and systems challenges like building really big models, but I don't know that there need to be 15 or 20 platforms out there for things that are going to eventually do substantially the same thing. It'll be good to have a few of them because competition is good and you want to make sure we're making things better and prices are not out of whack with reality. But the interesting thing is just the thing that everybody knows about building businesses: you're going to have a point of view on a customer need, you'll understand the customer better than the big companies probably, you will be nimble in picking these things up and just very quickly iterating on things that are highly valuable for the customer. The opportunities are huge. So think about the large models as a technology enabler, not as the product you're building.
And what are the—I mean, it's very early days, so we don't really know what the real roadmap will look like, but obviously there's a bunch of times where you say, 'Look, you should actually get the absolute best thing you can, which is the largest possible model that's trained well by a very high-elite team and a huge amount of money has gone into constructing this.' What are the cases in which going off and doing your own smaller model might make sense? What are some of the possible lenses or principles on that?
Well, I think the tricky thing about the moment right now is that our intuitions are being challenged. This is the thing we face at Microsoft: we are re-architecting ourselves internally to more and more use these foundation models and have fewer and fewer teams who are building end-to-end big things themselves. The default intuition is, 'Oh, I will be able to do better than this big general model if I have my own data, I get to choose my model architecture, and I train it and fine-tune it in this very bespoke way.' Some of the time that's going to be wrong, and some of the time that's going to be right. The thing I would look at is: start with, can I get access to this big model, and is there a way to do prompt engineering or fine-tuning on top of the big model with my data and for my use case that will make it perform? If you can do that, then you are probably architecting yourself in a way where when the foundation model gets bigger and more powerful, you can just swap that in and your application will get bigger and more powerful. You're just inheriting the improvements that are getting amortized across a whole bunch of different applications. But sometimes you're going to have to build something that's really custom. The big foundation models just don't have the thing in scope that you want to do. A good example is we are building a whole bunch of models for science right now, where the model architectures and the data that are really good for language applications don't help you much in making a molecular dynamic simulation where you're trying to get to fast, quantum-accurate models of how atoms interact with one another.
Well, let's linger on science for a bit, because it's actually among the accelerants that I think most of the people in this room have some sense of, but it's super important. Science is surprisingly amplified by tooling. Typically people think of tooling as a new telescope or a new collider or a new database, but actually part of what we're getting is the tuning through new tools, through software, and through these models. We've seen AlphaFold, we've seen Baker Fold, we've seen some of the stuff. What are some of the things that are coming in terms of the science tooling and accelerations?
