How Security Leaders Can Prepare for Super Intelligence

It's my pleasure. It's great to be in a room full of CISOs. Outside of, you know, whenever, every morning, every night, it's sort of, I check, what did I get from Igor or Michael, and whether I can go to sleep or do I want to get up? But it's, look, first of all, I just wanted to say a big thank you to all of you for the work you do. I know it's combat duty 24/7, and I deeply appreciate what you do for your organizations, but this community comes together, collectively, defending everything that we care deeply about. And so I appreciate all of your commitment to that.

The community around this, we wouldn't be able to sort of really make the progress we are able to make, and especially at a time like this. So I'm glad we are able to get together and spend some time and, quite frankly, learn from you all on what the real world really does need.

It comes from sort of grounding ourselves on the state of AI safety. I mean, if you think about it, right, our history here, we've been early believers in scaling laws. In fact, if you sort of go back, I still have that paper. We've just moved houses and there was this paper that, in fact, when there was one happy family where even Elon, Dario, Sam, all were in one company. There was a paper that, in fact, Dario wrote called "Scaling Laws and the Path to AGI." And that was the thing that got us to say, "Wow, there are these emergent capabilities that seem to show up with scaling, and it's like Moore's Law, it's an empirical thing. It's not really a law. But it seems to be working or can work." And that is what we in 2019 even said, "Let's go all in." The good news is it works. Here we are at Astra class models.

The bad news is alignment doesn't fall out as a capability with scaling. And that's pretty, pretty disastrous. Right, I mean, if you think about it, if you had even asked me, going all the way back, I would have said, "Oh yeah, I mean, alignment would be something that we can quote unquote in pre-training itself, fix." And when I started seeing reasoning, I thought, oh, maybe it's easy to do with sort of post-training. Right, because just basically, after all, if you can reason, you should be able to align. At least for now, you talk to all of the researchers, which I constantly across MAI, across Mustafa's folks or Ali Farhadi and Karen and, or all the folks at OpenAI, Jacob's paper, which is a good one, right? He says, "We are not building AI, we are growing AI." Right? That's what he wrote.

I think we have to now ground, I take it super seriously. It's basically at this point, my assumption is that alignment may be elusive, even while scaling works. And mechanistic understanding is also elusive. So here we are, with more capable models coming down the pike that we are going to deploy. What do we do? So that's why the metaphor even is you essentially now have quote unquote superintelligence as an insider risk. And I think that that's sort of our future, our present, even. And so, so that's sort of what I tried to say, I know your presentation was full of that, but fundamentally, what's the way we can put all of the best practices we have always had to deal with even something like an insider risk, and then harden ourselves, right? Starts with observability and identity and, but let me even say the following. In fact, it starts from even an understanding that these models are opaque and unfortunately, they're becoming even more opaque. Some because of policy and some because COT itself is not faithful. So the chain of thought, itself. So the models are— The reason why we don't have alignment is because the models are trying to fake it. And then on top of that, we have model companies trying to protect their IP and faking it even more. That is a bigger disaster. So I feel like, hey, we've got to really have standards. We have to really force and, in fact, this is where you all have the power as customers, right? Quite frankly, before regulation this, that, and the other, I just don't know how you all can attest then to anything a model does if you don't have assurance that at least I was able to inspect what it was thinking. Not in new release, but in English, and then use other models to check it and in fact, use maybe more deterministic systems, and what have you, right? So there's a, so that's kind of a lot of my blog was about what do we do? Starting with, you know, if you have nested black boxes, one, can we push back and avoid nested black boxes? So transparency is one. On top of that, let's put all of the control mechanisms, starting with sort of the identity that will allow us to observe every trace. And then I think a lot of your presentation, because basically we have to do active, active behavioral monitoring, right? If I look at the state of the art of behavioral monitoring of, let's say, a defender, it has to go way up, that score, like, I think in your thing, we will have to really get very, very good at being able to detect anomalous activity of superintelligence. I mean, that is the hardest challenge we have in front of us. And it's not even sort of, you know, some science fiction thing tomorrow. At this point, if you're deploying any of the latest models from anybody, you really need to have a ground up zero-trust architecture for essentially treating with, dealing with, I'll call it, superintelligent insider risk. That's the real issue. And, I think the more you all, it'll be good to get your feedback whether you are all already having that dialog, whether it's something that's top of mind for you. And then make sure that we are pushing in the same direction on how we ensure that there is assurance.

Yeah, so I think if you're to sort of, I think the way we are at least, you know, obviously I didn't write the paper in abstract. I kind of wrote it in the context of the principles that are guiding even what we're building in perception, right? Which is at some level, yes, we have to use AI to deal with sort of AI safety and securing AI. And so, the thing that I feel right now is if you really want to create an agent swarm that is going to then help us protect, how do you then essentially create, I'll call it using this harness in particular and harness, I'm, you know, I'm making a broader statement than a very narrow harness. But everything: the signals, the context, the models and the harness working together as essentially think of it as my governance and policy control plane/ERP, or assurance ERP, right? That's the biggest system of record, the most important system of record that needs to be, it needs to be tamper-proof, that needs to be something that I can do causal reasoning on. And that might be the other key thing, right? Which is, I want to be able to look at this stuff using non-neural network techniques. Because I think that we are going to end up with some form of symbolic logic assurance on what is happening here. And so that's what we are building towards. So when I look at Perception plus Agent 365, take these two in a narrow way, our products. But that's the goal. Our goal with these two things is to give that control. So basically, I have a governance policy. I need assurance that that governance policy is being enforced. In order to do that, you kind of need to start with identity and observability, and then the ability to do the verification and test using all the traces and at runtime, behavior monitoring, right? That's the entirety of the picture. And so I'm glad we have these products now. Now they need to be battle-tested. And that feedback cycle. And then, by the way, there's going to be significant work we will have to do together on standards. As I said, the first standard, I think— Here's another thread I want to pull. I've been thinking about right, what does liability insurance look like? It has to be priced. How does an insurance company price liability insurance today if this system doesn't exist? And so that's the other piece that's going to come, like, you know, so we will need to create even the data that insurance can use to price liability. And so we will sort of we'll have, you know, more game tape on this very quickly. Yeah. But my hope is that the products we are building are giving you that sense of, I'll call it that control plane for essentially governance and assurance, and starts pushing against the opaqueness of the models. I mean, because, believe me, I'm not like, I want to use the models, but I just want assurance that the behavior of the models are adhering to policies that I have set, because if they don't, I don't know how I can just have them operate inside the company. It'll be like saying I have an insider risk, but I don't know how to monitor for it.

Exactly. And I think it'll also be good, you know, as part of this two days to have the discussion on how you all in your procurement, in your push on standards, we can all sort of work together because I think that unless we do that, it's just waiting for regulation. What I realized is that's now it's sort of caught up in a very different circumstance, set of conditions. But for us as sort of people deploying these things, trying to get value out of them, this is a here and now. And so that's where I think this community coming together, solving this, I think, becomes even more imperative.

Yeah, in fact that's kind of been my thread, right? Which is, I mean, there are three sort of things that I've been, I mean, this is consistent with how Microsoft thinks about this, right? We are a platform provider, and a tool maker. That's kind of, even with all this talk of AGI, superintelligence, what have you, our identity still remains how do you empower people and organizations? And so to me, that means you have to build not one frontier model or two models or two products. It's about building a frontier ecosystem where there's broad spread GDP growth in the economy, that's in every country, in every sector, in every size of business. Because that's the only thing that I believe is politically stable, right? I mean, otherwise, why would the world want anything that doesn't create that? Right? If it's a general purpose technology that needs to be its attribute. So that's kind of principally where I come from. And we as a company stand for. That then led me to this thing called the network effects of intelligence are very different. It's unlike any other previous digital technology, where you are effectively paying for it twice, because it's a learning system. You pay for it and buy it, deploy it, and it sort of leaks value, because, if you remember, Kenneth Arrow's, you know, information paradox was more about patent law and everything else to protect the seller. This time around, you have to protect the buyer because you leak with your signal, like the thumbs up, thumbs down, is judgment, right? If you think about the human, the what is the firm? The firm is the tacit knowledge that gets created every day because of the human capital that comes together and works with digital systems today, right? I mean, that's what a Microsoft is and in fact, if somebody said, "What is my trade secret?" My trade secret is the people inside the company working every day with the systems and the processes and creating that tacit knowledge, that becomes trade secret. And now the trade secret is leaking because, you know, in your judgment, in fact, every time you are judging something, you're effectively letting the model know what great and good looks like inside your firm. Most people think, oh, it's my data is my asset. Really it's not your data, your asset, it's the judgment around your data. In fact, what you write down is sort of not the most useful thing. The thing that you behave or do and act around it is the most useful thing. And this thing leaks. In fact, for example, when people say they want to protect COT, they're trying to protect the training signal. It's just ironic on so many levels, I can't even tell you. But nevertheless, this thing has to be well understood. And I don't think it's well understood at all, right? Because people think of this as, yeah, this is another technology. In fact it's a little more formal driven right now. Right? Just, oh my god, if I don't have the latest model, and I, oh, by the way, the model now has the harness coupled with it. Oh, by the way, my memories are no longer with me, but they're with the harness and the model, and I'm like, looking at that and saying, "Wow, this is fascinating. This is like outsourcing your thinking." Right, I was always fascinated by the Mughal emperor who thought it was a brilliant idea to outsource tax collection to the East India Company. And, and soon after, they were ruling us. And so, you know, that is the moral equivalent of what's happening with these model companies. And so we need to really ground ourselves on what is the future of the firm? What's the theory of the firm? How does tacit knowledge get created? How do I make sure that I can retain that differentiation? And by the way, this is all full force in tech, right? What is a moat? People are saying moats exist or don't in a world where there is intelligence machines that come in. So that's, so to me, at least our body of work would be: How do you build a frontier ecosystem? How do you protect where multiple—all firms can thrive in this new frontier ecosystem with distributed knowledge. Because otherwise there won't be an economy. And so that's kind of the second. And then we now have this new thing to deal with, which is, if alignment is elusive, and that means AI safety is sort of elusive and mechanistic understanding is sort of not there, then how do we deal with the security aspect? So that's sort of, and it's all really one thread. It's not really three distinct things. And that's sort of at least what our work is.

Yeah, I think that's kind of the real question is, if I sort of thought of this as just a normal technology, the normal technology would have been, hey, get it deployed, get it to show gains. Right, even but that is a little bit of what we have been even doing up to now. Now, you have to sort of step back a little and have a real enterprise architectural approach. For both what is the knowledge in the firm and is the firm's knowledge going to actually accrue to the firm or somebody else? That's the reverse. Do you have a real crisp answer, strategically and technically to the reverse information paradox? Do you have again, strategically and technically, a crisp answer to the new insider risk? If you don't, then how do you proceed? That, I think, is fall of '26. I would say that's what I would recommend, at least when I look at even inside of Microsoft, what we are doing, that's kind of where I look at it and say, "Hey, we have got to be very principled on our understanding of these two." Have a good, I'll call it architectural and strategic approach, versus the new, shiny object approach. And I think that will be a better way to go at it. Knowing still we live in a competitive world. You have to sort of compete every day. And so that's the reality we face. But I feel like the time has come for us to be a lot more, this is not just, oh, I can rinse, repeat what I did in cloud. No. I can rinse, repeat what I did with my client defenses or whatever technology cycle. This is not the same.