Hello, I'm Arjent Karple and welcome to the Tech Download. And on today's episode, my guest talks about the growing cyber threats from AI.
Suddenly now you can rent compute and intelligence and let it go berserk across your entire infrastructure.
The AI safety and pacing debate.
I think this is unrealistic. I think not everybody's going to pace themselves.
And the threats to modern-day encryption from quantum computers.
We have seen activity where certain nation states are already harvesting data, saying, 'Okay, I don't know what's inside that data, but I can already grab it, and when I have my quantum computer, I'll crack the code and decrypt it.'
One of the most talked-about topics in tech are the cyber threats arising from ever more advanced AI, and that debate was supercharged by an extraordinary incident involving OpenAI and Hugging Face, which you may have heard of. Hugging Face is a company which hosts a massive repository of AI models and datasets. Now, OpenAI was testing an AI system's hacking abilities in a supposedly sealed environment. The models found a way out, got onto the internet, and ended up compromising Hugging Face systems in pursuit of their assigned objective, even though no human had told them to target Hugging Face. You know who would be great to discuss the evolving cyber threat from AI? Probably the boss of one of the world's biggest cybersecurity companies. So I traveled to an event called Human X in Amsterdam and tracked down Nikesh Arora, the CEO of Palo Alto Networks. The OpenAI Hugging Face incident underscored two growing concerns: AI being used by bad actors to carry out cyber attacks, and AI systems themselves behaving in unexpected ways. That second problem becomes particularly challenging as companies talk up the use of agents or agentic AI, which we've spoken about a lot on this podcast. These are AI systems that can carry out tasks autonomously, and there could eventually be thousands of them operating inside a single company. So what happens when an agent does something unexpected? And then there's the way AI could make hackers themselves more sophisticated. Take phishing. That's phishing with a ph. That's when a hacker sends you a legitimate-looking email or message designed to trick you into clicking a malicious link or handing over sensitive information. AI can make those scams far more realistic and personalized. All of this is what we get into with Nikesh Arora.
So, so Nikesh, let's just kick off. There's so much to talk about. First, top of mind: AI, cybersecurity, and the intersection there. And of course, it's been top of mind since we had that OpenAI Hugging Face incident as well.
No, it's been top of mind since, to be fair.
For sure. And let's, we'll get into all of the details about this, but what did that moment do for kind of the industry and the broader conversation around, you know, the cybersecurity threats that are coming from ever advancing AI?
Well, look, it all started with Mythos. I think Mythos proved to people that if you've been training these models to do better coding, they'll be able to figure out what bad code looks like. It's kind of a bit of a revelation to the industry. Oh my god, we have AI which can actually understand code and will tell you where bad code is. And that kind of exposed the dark underbelly of enterprises because they've been sitting on vulnerabilities, on bugs or misconfigurations for a long time, and we find them. You know, everybody tries to make an effort to find them, but AI was able to show how it can go in one fell swoop, and you can apply a model like Mythos against your infrastructure and find out the whole host of vulnerabilities you haven't seen for tens of years. And I think that was a bit of a wake-up call, to be honest. Like it's a wake-up call to our industry, a wake-up call to the IT infrastructure industry. And honestly, management started paying attention. I mean, we've been trying to get people excited about cybersecurity for a while. Suddenly your CEO is calling, saying, 'Hey, what about this Mythos thing? Is this real?' It's kind of that kind of woke up everyone in the industry.
Yeah. I was going to say you must have got a lot of calls after that from C-suite across the board.
Yes. We did about 2,000 individual briefings with CEOs, CIOs. 2,000.
Look, that was an, and part of the story here is effectively AI models that escaped what was supposed to be a controlled environment. That's one issue. But are you seeing these ever advancing models being harnessed by bad actors right now? And in what way?
Well, I think what you've been seeing is the capability, sort of the exponential capability increase, because historically bad actors don't have as many resources. So they have to have lots of different people try different techniques, try and insert themselves through some vulnerability into the enterprise infrastructure. Suddenly now you can, let's just say, rent compute and intelligence and let it go berserk across the entire infrastructure. That kind of like got people worried, oh my god, this is going to get harnessed in a different way. And what you're beginning to notice is that we made a prediction that when Mythos came about, that in 3 to 6 months open source models will get equally good. And you can see now there are open source models which are almost as good as Mythos, and you expect that over time they're going to get better. Mythos is already on its next generation. So what you're basically going to see is that if you don't sort of go ahead and strengthen your perimeter as an enterprise, you are going to be attacked, and there's a high probability attackers will get through. And that's kind of been a bit of a moment where people have stepped back and say, am I good enough? Is my cybersecurity capable? Am I good enough? Am I able to stop these bad things from happening?
So, so actually because of the amount of open source models and the capabilities of these open source models, it makes it easier for, you know, bad actors to be able to obviously use these models that are freely available.
Yes, totally. Look, like you said, bad actors can now rent this on the fly, point it towards somebody's infrastructure and just go hunting. And you know what? If I find nothing, no problem. I'll move to the next target and go hunting again. And when you look there, there could be bad code you could have written as an enterprise. There could be one of your technology vendors which has a vulnerability which could be exploited, or you could be using a lot of open source code yourself and that could be exploited, or you could just have misconfigured something. So there's many different ways where these vulnerabilities manifest themselves, and I think the onus is upon all enterprises out there to make sure they robustify or strengthen their perimeter.
What are the kind of main threats right now as you see? Because of course you could use this to create malware or, you know, malicious software, but also there's the phishing side of this, right, in terms of being able to create ever realistic emails or correspondence or images or phone calls even, right? Where are like the main threats right now?
Look, the main threats are, like I said, most breaches historically don't happen because of sophisticated sort of attacks. They happen because of simple: somebody got a hold of your credentials on the dark web, entered your infrastructure, made their way laterally across infrastructure. It's a bit of a game of hunting. It's kind of a sport for bad actors or hackers to say, 'How do I get in? How do I navigate through this stuff?' All that has become easier because AI models know how to go out and do this stuff, how to go understand where the vulnerabilities are and attack your enterprise. I think equally, like you mentioned on the other side, when bad actors are targeting individuals, they do that through phishing attacks, through smishing attacks, through, you know, fake video. So it's a lot of different ways these bad actors do it. And it's kind of interesting. It's a tale of two cities. One is on the enterprise side, the risk is it'll find new vulnerabilities. On the consumer side or individual side, you know, billions of dollars gets scammed from human beings because bad actors sort of convince them in some way, shape or form to let go of their banking credentials or, you know, surrender some private information. And that's going to get worse because things are going to get so much more realistic and so much more customized. It's like they'll start understanding based on your persona what you're most likely to fall for, and they can now send, instead of sending a block email to 10,000 people, you can customize emails to everyone basically understanding your circumstances. So the probability of you clicking on it goes up.
Yeah. Did you say smishing?
All right. Okay. [laughter] There we go. Yeah. There's phishing. There's smishing. There's fishing. [laughter]
We've spoken about kind of the bad actors and the offense side. What about on the defense side? How do companies protect against this? And what are the increasingly sophisticated ways you're having to think in order to defend against these kind of more advanced attacks from AI?
That's a great question. Look, this morning we announced a new service. The new service basically allows us to bring Mythos-class models to enterprises to help them build their defense up. So we can go on behalf of the company and say, 'Come, let me take a look at it. Let's go look at it from the outside and from inside to see where bad actors could possibly see your vulnerabilities.' That's kind of to make sure you use AI to anticipate the bad actors, do what they would be doing, and strengthen what you have from infrastructure. That's one. Two, I think there's many categories where we're looking at attacks, URLs, files, and saying, is this malicious or not? Now that requires a process of machine learning, that requires a bunch of experience to deploy it. Today you can start using classifier models to see how you classify it, yes or no. So you can effectively use AI in the classification process of bad things or threats and use that. That's two. Three, what you can do is let's assume that, you know, despite every effort to robustify or strengthen your perimeter, some bad actors got through. And the question is, how quickly can I find them and stop them? So there's a whole art and science of how do you look within an enterprise infrastructure to say if there's malicious activity going on, understand the behavior, defend against it, and effectively throw them out. And that can be also accelerated by using AI. And that's where you might even use frontier models to make sure that you're acting at the speed of compute so that bad actors can be deflected or kept out of your infrastructure.
On this podcast, we've spoken a lot in various episodes about agents, AI agents, agentic AI. We love agents. Yes. And everyone is talking about implementing agents within their organization, right? And so when you get into this idea of agentic workloads, agents potentially carrying out tasks within an organization, workflows within an organization, touching the sensitive information within an organization. What new challenges does that bring from protecting an organization from a cyber perspective?
That's a great question, Arjent, because that is one of the areas where you will see a lot of effort over the next few years because I think the definition of agents and their capabilities, like you rightly said, is evolving. Right, you saw, you mentioned the Hugging Face example when agents went riot. Those are agents who had a very simple objective: capture the flag. You have no rules, no governance, no controls, go capture the flag. And you saw the agents used every resource that they had access to and were able to go navigate themselves across the infrastructure and capture the flag. So, you know, that clearly has become a bit of a sort of wake-up call again to the industry saying, be careful how you deploy agents. So hopefully our customers on the enterprise side will deploy agents in a careful manner where you have to understand who's the agent, who created it, what rights does it have, what is it trying to do, and then be able to intercept the agent while it's executing its task, say, is the agent behaving as per plan. So there's a whole, I'd say, a whole new area of security. I expect there's north of 500 startups to be funded by all these VCs because they're chasing the gravy train saying, go figure out how to solve agent security. It's kind of an evolving field. It's happening on both sides. Enterprises are being very cautious about how they deploy agents. At the same time, the industry is trying to run as fast as we can to build the harnesses required to protect against bad behavior from agents.
Yeah. I mean, one of the OpenAI Hugging Face incident, one of the big talking points was the fact that actually these agents try to almost cover their tracks to some extent, right? So that makes it from a defense point of view very difficult. There needs to be continuous monitoring, right? And figuring out, well, think about it, where are these things?
We've trained the AI, the AI agents based on human behavior, and we haven't been able to give it a moral backbone. We haven't been able to say this is right, this is wrong. So the agent is just going on learned behavior saying, I read on the internet that if you don't go this way, you go the other way. And the agents are just trying to be unscrupulous and try and figure out how to solve the problems. Yes, you're seeing that. I think the idea that we start putting governance in place, we start understanding. So there are two parts of this, right? One part is as every frontier LLM has recently said the last two weeks that we need to be cautious. We need to make sure we pace ourselves. I need to go out and make sure these agents can be safely deployed in any setting. There's a whole different conversation we have around pacing, but I think that's the right idea. They have to make sure their models are safer in how they act. At the same time, our job is to work with them to build security into place. Today, many of their tools don't have the hooks required to intercept, to analyze, and to stop agent behavior. So we're working with them to build those hooks into their infrastructure. We're working with them also to build tools and capabilities to understand the intent of these agents because, as you know, the biggest change that has happened in software is software used to be deterministic. It was input-output. You asked it to do something, it did it, and you were able to make sure that it did that. Agents are non-deterministic, right? You can't actually expect they could change their behavior next time around, their behavior the next time around based on circumstances. And that non-deterministic nature of AI and agents that makes it a much harder problem to solve from a security perspective.
Before we get into this next part of the interview, I just want to step away and lay the context. You may have heard about existential threats to humanity from AI and the debate around AI safety. Well, it all kicked off after a researcher at Anthropic, the creator of Claude, said there was a more than 10% chance that AI could kill all humans. The comments caused a firestorm. Days later, Anthropic CEO Dario Amodei published an essay arguing that frontier AI labs must slow the pace at which we improve the capabilities of AI models. This is an idea known as pacing, effectively deliberately slowing the development of the most advanced AI models to give safety research and safeguards more time to catch up. Amodei said two things had changed with his thinking: AI was becoming much more capable, including helping to develop the next generation of AI, and incidents like the OpenAI Hugging Face hack had demonstrated how advanced AI agents could behave in unexpected and potentially dangerous ways. And Amodei's call was backed by some unlikely allies: Elon Musk and OpenAI CEO Sam Altman, both of which are rivals of Anthropic. And Altman said he agreed that the industry needed to pace the frontier. But not everyone in AI was on the same page. Nvidia CEO Jensen Huang argued that speed and safety don't have to be mutually exclusive. And Nikesh Arora also weighed in on the debate. [music]
So Nikesh, you brought up all the different issues around pacing, and let's get into all of those now because you had, firstly, you had since the Hugging Face incident, you've had multiple instances where other companies, Anthropic and Gemini, have said actually we've had instances where models...
Nobody wants to be left behind.
Yeah. Nobody wants to be... well, there you go. You've also had, you know, OpenAI more recently come out and said they've seen more instances of concerning model behavior. Separately, you've had researchers at Anthropic saying, you know, there's a more than 10% chance that AI can kill all humans was the exact quote there. And then you've had the CEOs of these frontier labs at OpenAI, at Anthropic, have called for model pacing, solving alignment, regulation, all these different things. And we'll get into all of this. First, can we start with the existential threat to humanity from AI, which has obviously been a big talking point around this discussion. How real is it? How much do you believe it? How much should we fear for our lives?
Look, every innovation comes with its pros and cons. And yes, can AI be used to the detriment of society? Of course, but that's no different. We've seen technology in the past that could have been used for bad things, but they've been used for good things, right? Take nuclear, right? So I think yes, AI could be used for bad things, but you can't take the downside scenario and build towards it. You have to look at the upside scenario. AI has so many amazing things. It can cure cancer. It can solve problems that have never been solved before. It can advance innovation. It can advance in a whole bunch of capabilities, life science, etc. So if you look at it from a positive perspective, AI has these capabilities and powers that if evolved right, it has the opportunity of benefiting us tremendously. So I'm going to be an optimist and say yes, it has the possibility of being used in the bad sense, but highly unlikely that those scenarios have a 10% probability as has been touted. I think it is a small probability which is extremely small, and it's our job to make sure we build in the guardrails, the safety and security required so these models are used for the right thing.
And then on the idea of pacing, which is another kind of term to suggest maybe we need to slow down the development of the frontier a little bit. There's been a lot of discussions from AI CEOs, right? Dario Amodei of Anthropic, Sam Altman, of course. You've been posting on X about these discussions as well, and you said pacing is a ninja move.
Yes. But I changed my mind after that.
You changed your mind. So just unpack kind of your thinking and where you're at right now and what you believe around this discussion.
When I saw it come out, I figured that clearly the AI labs understand this is a problem, and I think we all do. It is a possibility that if not managed right, we should not be in a haste to release these models. So I think that's a good idea to make sure you get it right. I think the notion of a ninja move, I thought was, oh, maybe they're going to try and get the sympathy of the regulators and sympathy of the lawmakers saying, hey, wait a minute, you know, this thing's going to be dangerous. Let's make sure we jump in and we put some guardrails. And in the interest of innovation, in the interest of the positive side of AI, we kind of give them a free pass, if you will, on the liability side. But I think over time, as I've spent more time thinking about it, talked to more people, I think this is unrealistic. Not everybody's going to pace themselves. You'd have to pace yourselves. You have to get, you know, agreement for every country in the world, agreement for anybody who's doing any work on AI. I think that's highly unlikely. I think more than likely that some people will jump the gun and you'll have still people developing at the frontier, which means we shouldn't try and stop the frontiers because they're the most responsible people. If you can let them keep pace with what's going on in the world, make sure they start thinking about safety and security in a more robust way, that'll be the right answer. So from my perspective, and I think many lawmakers have since said, look, it's your job when you release a product to make sure it's right. And if you don't believe it's right, you should not release it. That's true for any product company in the world, right? Don't release a product if you believe it's not going to work or it does not have the safety attributes that you require for them, whether it's like drugs or it's, you know, anything like cars. You can't release a car if it's going to kill people. So from that perspective, the idea is don't release your product if you don't believe it's going to be safe or you're comfortable putting it out there. And if somebody believes that they are, they should go out and do it. But the idea that all of us have to get together in a room and agree at what pace, at what capability do we launch this? I think it's just a lot of work, and it's not clear how you're going to test for it. And I didn't see them espouse, well, we'll stop pacing when X happens. So we don't actually even know how long are we going to be in this pacing rhythm for and when can we break the pacing rhythm. Is that okay? So I think a lot of things are undefined in the idea of pacing. And I think where we'll probably end up is that we'll get more regulation because now we've got the attention of the lawmakers. I think what we end up is that some people will jump the gun and we'll go out there and produce models which have tremendous capability. I think the focus should be on safety and security, and we need the frontier labs to take that even more seriously than they have, and hopefully they are.
Right? Because I mean, with this discussion, a lot of part of it was what is the motivation for these companies? I can't have an opinion on this, but the debates that are out there is, is this genuine concern around the dangers around AI? Is this regulatory capture, i.e., trying to get better regulation favors the big companies which may push out some of the competition, maybe even, you know, from China, which we'll talk about as well? Is this a need for these companies to justify their valuation? Oh, we need to slow down because maybe the revenue is not there. I don't know. But, you know, are you concerned about the regulatory capture side in any way?
