And Dario and Ilya and Sam and Greg and Elon were all, I guess, one happy family. They were really the believers in scaling laws, and that's what I believed in back in 2018, 2019. I think the biggest change now I have is I used to always think that alignment will just fall out. I no longer believe that.
Now, this show honestly is quite an emotional one for me. I started 20VC from my bedroom in London. I was 18 years old. I had no money. I had no contacts in my phone book. And today I have the chance to sit down with Satya Nadella, CEO of Microsoft, one of the greatest companies of the last three or four decades. And I will just say on Satya, the warmth that he was able to imbue in this interview in such a short space of time was almost unlike anything I've seen before. The clarity of his arguments and his ideas, his ability to go very deep, very quickly. It was almost unlike anything I've ever seen before. It was a show I'll always remember, and I really hope you enjoy it. Ready to go.
Satya, thank you so much for doing this with us. I would say, you know, thank you for coming to the studio, but you're actually hosting us here. So, really appreciate you taking the time.
Oh, like, thank you so much for having me on the show, Harry. It's been a long time coming, but I'm finally glad we were able to do it.
Now you wrote a seminal piece 10 days ago or so, and you said that we should assume a model is compromised, and I just wanted to expand on that. Does that mean that we can't or shouldn't rely on the disclosures of the model provider companies?
No, I think the challenge, the way I see it right now, is the models are becoming more and more capable, right? Scaling laws are working, except one of the assumptions I had was that you'll have alignment just fall out of scaling, and it doesn't seem to be. And there's more science challenges there that need to be tackled, right? And mechanistic understanding is also elusive. And so given that, I think the best posture to have if you want to continue to use these sophisticated models is to treat these models in a trust architecture. I mean, after all, zero trust is not something new, right? We've had zero trust architectural approaches. So the claim, at least I'm making, is let's create an engineered zero trust environment that can contain any malfunctioning agent before it does any harm. Right? So that's the point, because if you really want to have agents that are more capable and yet less aligned, then you need to have a trust system that you can rely on.
Can I ask what does that trust system look like? Who's the architect of it? Who's the arbiter of it?
It's a great point. And so that's why I kind of look at it and say you leave AI safety outside of this, right? Because that alignment, for example, at least my take is that it really mostly has to show up in training, right? In fact, maybe even in pre-training. I don't think you can even RL your way to alignment. You have to truly get alignment to be something that truly gets embedded in the weights right at the very beginning of it. And so that's the model itself, hopefully at some point will get aligned. So if you leave AI safety, then the trust architecture is going to be just, you know, what a company like Microsoft does or all the cyber folks do. So what we have done, for example, is we built these two things, right? Agent 365 and Perception. Right? What does it do? It gives every agent an identity. It then logs all of its activity, all its tool calls, in fact every model call and the CoT in it. In fact, that's where I'm also asking that all the CoT better be in English and not in new release. And then you can check using multiple models. It better be tamper-proof, right? You can't have the agent go modify its own logs. And then on that you set policy, and then you behaviorally monitor the activity of the agents to confirm to the policy. So that's the architecture.
We don't have that architecture today. Is that just symbolic of where we are in the process?
That architecture exists. So there are two, I think people conflate what I'll call an engineered security zero trust architectural system for containment and AI safety, right? Because I think people say, well, AI safety is hard and elusive and not yet achieved, so therefore there must be nothing we can do on security. And I'm not saying that. These two things have got to come together at the same time. In fact, I would just say, hey, we have to deal with the situation as it exists. So let's build a trust system outside, in fact, of the model, and then just be super aggressive in setting policy and monitoring for policy deviation.
On the monitoring side, if we think about chain of thought, how reliable is that?
It's not, right? We know that the chain of thought itself can be unfaithful, and in fact it can be made worse when the model company says, 'Well, it's unfaithful and I'm going to hide it.'
And so it becomes even more problematic, and that's where I think a little bit of the push for standardization and the CoT, at least for trusted partners, being available as is, so that it's inspectable, it's auditable, and I can check using multiple models and what have you, right? So that's going to be one of the, in fact, that's one of the things that may even drive more open weights, quite frankly, because if you can't trust... it's funny, closed models are more powerful, but if they kind of don't recognize the need to be more transparent, especially as their trusted partners use it, then we may have a problem.
Can I ask you, when we think about that rise of open source, I completely agree with you in terms of closed being more powerful. Open source is predominantly Chinese models. Is that a concern then when we think about safety being baked into pre-training, when we think about disclosures as we said there, and then juxtaposed with the majority of open source being Chinese?
Yeah, I mean, I think there are two sets of things. I do believe if you sort of step back, somehow AI safety not being there is not a good thing for China. It's not a good thing for the United States, it's not a good thing for the UK. So I think that there's a part of everyone, because this is not something that somehow an unaligned open-weight Chinese model will probably attack even China. So it's not like you can rest thinking that this will only be used as an attack vector. So setting that aside, I'm pretty pleased with some of the progress even the open-weight models in the West are making, and I'm, you know, obviously, you know, Jensen's working on it, the other folks are working on it. We will have even the MAI models that will be not open weight. We'll talk, I think of it more as licensed weights. Think of it that way. I want to turn this into just a simple, straightforward software business as opposed to a network effects intelligence business. So I think there will be different business models, and I ultimately trust in your ability to use those models and control those models is what's going to determine what gets used.
As a venture investor, I have to predict where traffic will go today between closed and open. When you think about the two different dimensions, which is like traffic and dollars, if we look forward, say, 24 months, what percent of traffic goes through open versus closed, do you think?
That's a great question. I, you know, I think it depends on what the frontier looks like, right? Which is, I think most things seem to get saturated, and when it gets saturated, open weights or even just cheaper models of even closed weights seems to work, because especially one of the things is if you can train on your own traces, then you really don't need the latest model to be able to solve that same problem if the problem has not changed. Right? So if you, you know, if you have an eval that you want to continuously keep moving forward, then you may want to pay. But even there, by the way, I'm very thrilled about this HydraFusion stuff we now have in GitHub, where, you know, I just saw this... yeah, because the cool thing there is it's a learned model. It now, it's... anyway, we are now going into these agent swarms and multiple sub-agents, so I can call a lot of sub-agents that even happen to be local models. And so I think my suspicion is that there will always be a market for frontier. There will always be some new things. Science may be the next thing that, you know, frontier is being set at. But I think predominantly token usage will be commodity token usage.
Does that mean it's open source predominantly?
It may be, it may be open weights, or as I said, it can also be cheap closed rates, right? Because there will be lots of price competition. Let me put it that way, right? Because if you look at my own usage inside Copilot, we use a lot of closed-source cheaper models along with open-weight models.
You said about kind of the importance of owning your own intelligence for some of your amazing customers that you have. You were talking about some in the UK. Is that the model environment that we have moving forward, specialized intelligence for companies with their own data?
Yeah. So it's, I think this is to me one of the most important things, right, which is what's the future of a firm in an era of intelligence, right? Because if you go back even to Hayek, his fundamental description of an economy is local intelligence, tacit knowledge is distributed throughout the economy, right? Without that, you know, there can't be only one firm, and it can't be one intelligence. And so that's sort of the foundational thing. In fact, capitalism works because of that, price discovery works because of that, or pricing mechanism works because of that. So therefore I think it'll be important for every firm to be able to accumulate knowledge at increasing scales using AI versus leak. That's why I called it the reverse information paradox, right? Which is, you know, Kenneth Arrow kind of described this paradox of information where if you're a seller and you disclose the secret without a patent or a trade secret protection or copyright protection, you leak the value. Now for the first time the buyer needs the protection, because the buyer, even, most people think of it as, oh, my data, it's even my exhaust of using it, right? Thumbs up, thumbs down signal, like who you're creating value for the model company versus capturing that value for yourself. So I think one of the things that at least I ascribe is that we need to have, for this to be stable, more of that surplus that gets created by use of intelligence to remain at the firm. And by the way, it doesn't need to be only by embedding your knowledge in weights. It can be in context. It can be in your evals, right? You may have a benchmark. In fact, one of the things that I'm very gung-ho about is we all become, every firm will have a private eval benchmark which is their core IP. And every model comes in and tries to sort of hill climb on that eval. And that to me is how a firm works.
I love this, private evals benchmarks for each company. This is for a rarified era of companies though, it would seem, your Harveys or your Loras or the kind of very forward-thinking AI companies. Will that disperse to every company in the world? You mentioned your...
Yeah, it happens always, right? Which is, you know, there was a day when I remember in the late 80s and the early 90s that the few companies that had deployed PCs and were doing forecasting, which felt magical. They were able to put it in Excel spreadsheets, send it in email and avoid the fax machine and inter-office memos, and it felt like, oh wow, when will this happen? And it took 10 years maybe, and it spread out. This time around, what is considered, oh, I'm an AI-native company and I do this, I think, you know, 5 years from now it'll be pretty passé, I would imagine. And then that's kind of what I mean, these tech cycles always start like that, right? Which is even cloud, I remember in the beginning of cloud it was like the big deal was, oh, who's cloud-native and who's not, and, you know, if you're not cloud-native, and now everybody is, right? Even Lloyd's, you know, which is what, 500 years old, is cloud-native. And so it happens.
