Are we about to unleash AI too smart to properly understand or deploy safely?
Within six to 12 months, a swarm like that with better capabilities could potentially do hundreds of billions of dollars in damage.
There is just no doubt, no matter what we do, there's going to come a future where you have AI on offense built by the good guys, training AI on defense built by the good guys to protect us. Wherever money goes, crime follows. Wherever information goes and communications go, espionage will follow. What used to be, wow, only very few people could have done that attack, now it's kind of being democratized.
Thanks to our friends at PayPal, the exclusive sponsor of This Week in AI. Pay zero processing fees on your first 100,000 in eligible PayPal payment volume. Find out more at paypal.launch.co.
Hello everybody. Welcome back to This Week in AI. My name is Alex. Now, over the weekend, a call by Anthropic to pace the speed at which AI is developed caused a sensation as leaders from other AI labs co-signed his ideas while the White House and some other members of the venture capital community pushed back. Nvidia's CEO weighed in. Congress is taking a look and the labs are working to find a way to regulate one another if no one else will. Are we about to unleash AI too smart to properly understand or deploy safely? Are we worrying too much? Are we ceding momentum to China? Can we even secure a world where agentic forms are trying to penetrate every system? Well, to understand all that and more, I have three excellent minds in the game of AI here for you. Starting with Soups Ranjan, the co-founder and CEO over at Sardine, or Sardine AI if you're feeling frisky. A startup that uses AI to sniff out fraud and ensure financial compliance, and it raised a $70 million Series C last year. Soups, welcome to the show.
Thanks for having me. Excited to be here.
Glad you're here, man. And then we also have Kevin Mandia, CEO and co-founder over at Armadin, an AI cybersecurity startup. They recently launched a massive white hat agentic attack called the... what do you call it, Kevin? Uh, Super Swarm. Hyperswarm.
You could call it that. We called it a hyper attack.
And then last but not least, my dear friend and longtime podcast co-conspirator in arms, Mr. Lon Harris, Launch's editorial director and basically one of my favorite people in the entire world. Lon, good to see you.
You too, Alex. Aren't you kind? So, so nice.
Well, I appreciate that, but I'm going to put you to work. So, first up for you, Lon, is what the hell happened this weekend and why is it Dario's fault?
So many things happened. Well, we're going to start with Dario. There were like five elements to this story we'll get to, but on Saturday, Dario published an essay on his personal site arguing that the entire AI industry needs to deliberately slow down how quickly they are making their models more capable. Uh, what originally set him off was the OpenAI Hugging Face incident over the summer in which a swarm of AI agents launched cyber attacks after escaping containment. He says that within six to 12 months a swarm like that with better capabilities could potentially do hundreds of billions of dollars in damage, Alex. And he's proposing a three-step plan. Anthropic is committing to the first one right now: outside evaluators with permanent employee-level access inside the company who can publish what they find without Anthropic sign-off within hours. Elon Musk chimed in to say he thinks Dario is right. Sam Altman has committed OpenAI to the same evaluator access and Dario and Nadella are also both backing that direction.
All right. So Kevin, you are a global leader in cybersecurity and certainly I think the leader on this chat today. So can you tell me a little bit about how you feel about the level of concern people should have about the capacity or capability of agentic forms to actually do hacking? Was the Hugging Face incident kind of a one-off or is this something that you think is a clear and perhaps even present danger?
Yeah, I think, you know, stepping back and looking at every technological advance, you always wonder, hey, was it used by the good guys first or the folks with mal intent first, right? And there is just no doubt, no matter what we do, that there's going to come a future where you have AI on offense built by the good guys, training AI on defense built by the good guys to protect us. Uh, I have found throughout human history, you know, I was in the military. You look at every advancement, you know, if you handcuff yourself, uh, you sometimes give yourself a disadvantage militarily or strategically. And so I think it's a very tough thing to decelerate technological growth. Um, and you know, the bottom line is I also believe you had mentioned a thing about Hugging Face, OpenAI. However that happened, you would like to think whoever witnessed it has already come up with a plan so that it doesn't happen again. You know, like if we're actually creating a technology that we can't stop, uh, that'd be surprising to me. We've been building an offensive agent and a bunch of tooling here used in production environments for a year. Uh, we're very cautious in doing it. I do believe it can be done. I do believe you have to have domain expertise to do it and a lot of focus on it. Uh, and there's no question when you have a lot of speed and a lot of R&D going on, uh, that you might be more prone to mistakes and prone to overlooking some things. But, uh, for folks that have laser focus on domains and are leveraging AI to do things that they've been doing for 15 to 20 years, I think you have a good chance of leveraging the technology in a safe way that benefits humanity. So, I don't know. You're going to have to sift through and find my answer.
Yeah, I was about to say that was about seven different answers.
Uh, yeah, you bet. But we're just heads down. People need to know we can accept, we can withstand an attack brought against our enterprise by artificial intelligence because it's coming regardless of what our closed labs choose to do.
Yeah. And that's what I wanted to kind of follow up with here is just how quickly is the offensive potential of agentic AI progressing? Because I know your company does a lot of... you're essentially a red team for hire as far as I understand it. So I'm guessing you're at the cutting edge of the spear here, of the tip. So, how capable are these systems and what fraction of people that you talk to are actually ready to withstand a theoretically serious attack from a swarm like we've seen?
Well, I can say that everybody's worried about it and there's kind of been a little bit of a gifting period where, hey, we know we got to withstand AI on offense against our enterprises because the criminal element is going to bring it to us. They're going to bring stuff to our doorstep that we've never seen before as technology advances. So, I think it's now. And unfortunately, I don't want to be the fear, uncertainty, and doubt guy because I do believe we have near-term pain for long-term gain. I really do. It's just unfortunate that we have a long tail of imperfection in complex networks. Even the folks that have, you know, I've been doing this since my Pentagon days in 1993 when we called it computer security, then information security, then information assurance, and then somewhere along the lines went back to, you know, cyber and cybersecurity. Uh, it's always been a cat-and-mouse game and, uh, but I do believe midterm and long term AI on offense built by the good guys will be minutes ahead, well-funded and excellent, and can train AI on defense to give us a better shield in the cyber domain than we've ever had before. Um, because you don't have to worry about human imperfection being the problem, which has kind of been the problem all along. But we're going to have some near-term pain. So I'll leave it at that. Um, there's no easy way to make the journey over the next couple... well, I would say at least the next year. There's just no easy way to do it. You're going to see enterprises attack themselves, figure out what they need to fix, and repeat over and over again. There's no way out of that.
Well, that's going to be time-consuming and expensive.
Yeah. I can't help but notice that Kevin's answer, it's largely around people, like humans, using these new models and their capabilities to attack other organizations or countries or whoever. Uh, a lot of the discussion we hear though when we're talking about this is about rogue agents and the AI becoming conscious and trying to attack us. Is that not as core a threat? Are you not really as focused on that kind of threat when we hear about those sort of scenarios?
To me, you know, there's still a cost to this stuff. I don't think if I'm a criminal element... I, by the way, guys, I feel like I know a lot about crime because I responded to so many security breaches. You get inside the mindset of what does it mean to do espionage? What does it mean to do crime? What countries do what things? I don't think too many people are going to spend a whole bunch of rare GPU use on fishing expeditions. I really don't. I think they do have their targets. They either are criminal and they want to make money. Wherever crime goes, you know, wherever money goes, crime follows. Wherever information goes and communications go, espionage will follow. I still think that the best attacks will still be targeted. The biggest challenge we've got is that what used to be, wow, only very few people could have done that attack, now it's kind of being democratized with the frontier labs that, you know, very sophisticated attacks. And more than that, it comes down to speed and scope. When you had a red team test your network with humans, they always look for only one path in at one point in time to make something bad happen. With the frontier labs and leveraging AI, you can kind of look under every rock and make a lot of bad things happen at a far faster speed than really... you can't have a human in the loop on defense when you're targeted by AI. But again, a long-winded answer by Kevin Mandia, but I tell you this, I don't think the criminal element is going to waste cost attacking everyone all the time. They're going to pick their targets and they're going to pick when and how and they're going to do it. If I was picking targets, Soups, I would think the world of money moving around would be a good place to start. And I know Sardine is working on, you know, KYC and using AI to fraud and so forth. Can you just tell us what you're seeing in terms of the sophistication of fraud and other bad actor activity that you guys are trying to combat at the company?
Yeah, 100%. So, um, you know, I actually pretty much agree with Kevin over there, you know, in the sense that any new technology initially first gets adopted by criminals and, you know, fraudsters and, uh, that is what we're seeing in our world of, you know, financial fraud. So we've seen a rise in deepfake, right? Uh, we've also seen a rise in a variety of different types of scams, right? And, uh, we're also seeing a big jump in card testing attacks, right? Uh, I'll give you a couple examples, right? So, uh, a few months ago, you know, the CISO of one of our large fintech customers, he called us up on a Friday evening saying that he saw a huge spike in account signups, about 250,000 account signups where, you know, the actual KYC details of those individuals were used, like the actual name, address, DOB, SSN was used because, you know, perhaps an AI-based tool had gone ahead and purchased all of that information automatically, autonomously from the dark web, right? And now, you know, in the earlier days of Sardine, six, seven years ago, we would see like smaller scale attacks, but now 250,000 account signups in the span of a couple of days is actually pretty large. And what we always say is that, you know, in order to stop attacks which are so immensely scaled up, you have to have AI-based defenses as well.
And can you double-click on that actually? Because Kevin mentioned a similar thing that you can't use human speed on defense against AI, but I think a lot of folks who don't know how quick things move in this context, Soups, are going to be a little bit confused. So can you just drill down and explain what you mean by that?
Yeah, 100%. So, you know, just teeing off of that example, right? So, uh, this was five months ago and at that time, you know, we had just launched our, you know, data analyst AI agent, right? Uh, it was still in beta. Normally I would have, you know, uh, to do the fraud investigation, I would have, you know, woken up a fraud analyst and, you know, ask them to do a deep dive by writing SQL queries and, you know, uh, writing Python jobs to actually do the deep dive analysis. However, you know, uh, in this case we had just released in beta our data analyst agent and initial investigation was done by a human, right? So me myself and my team, we initially identified the typology being used. And in this case, what we saw, because we like to say that every fraudster has a tell, they always make one mistake. And in this case, the mistake they made was they all used, you know, the same device fingerprint for the most part to create all these account signups, or they, you know, uh, they used a proxy and Sardine can pierce through proxies and we found out that the true location behind the proxy was sitting in UAE or Switzerland. So with just those two, like, um, sort of data points, we asked our data analyst agent to go do the research. It wrote its own SQL queries and then came back and said there's actually five other fingerprints which look like the same and you should go and block all these five plus one, six fingerprints. And all of that analysis was done in a matter of five minutes by writing its own SQL queries, creating its own charts, etc., and coming up with a suggestion for a change to make without involving an analyst. So something that would have taken like two weeks was done in five minutes. So that is machine speed for you.
I don't think I actually feel a lot better after hearing how much work can be done by AI in a negative context so quickly, Soups. That actually makes me pretty scared. Uh, Kevin, on the idea that everyone's going to be racing to be ahead on offense and ahead on defense. Uh, are we investing enough at the national or like the NATO level, if you will, in cyber defense as AI improves its offensive capabilities? Because your idea of staying a few minutes ahead sounds great, but to me it's also going to require a lot of work and companies historically have underinvested in cybersecurity, which I think is partly why people are so worried.
Hey, I'm going to do something that hasn't happened in months. I'm going to be positive on AI on offense.
We love it. It's coming no matter what. We better get a better attitude about it. Okay, here's the reality. Everybody wanted... my old company, Mandiant, did more red teaming than any other company on the planet. That's what we did. People would hire good people to break in and see what they could make become a reality. Every company wanted red teamers to assess their security all the time. Technology never existed that could do it and you couldn't leverage it. And at the same time frame, there's never enough humans. Now we've got the ability to finally secure everything everywhere all the time. It literally will exist. We just have to cross the, you know, some will call it a chasm. That sounds bad, but it's just the damn reality that in the near term there is enough investment. The control defenses, the Palo Alto Networks, the CrowdStrikes, the Fortinets, they're ready to be in tune with what do we do to stop the next attack. They are. They need to be informed about it. It used to be a model where we learned the art of the possible by enterprise, you know, A1 enterprise, A was compromised and they did information sharing to say here's how we can stop this the next time. I, you know, Armadin and other companies, it is our goal: let's find the attacks before the attackers do, let's find your exploitable risk before anybody, leveraging technology that we didn't even have at our fingertips years ago, so that we can all operate confidently in the cyber domain. That's going to actually happen. That's the positive, is that if we do fund and we are funding offensive-minded organizations to actively test our defenses, and on defense all the companies that are there are thinking how do I survive the AI shift change, they're going to be able to do it. And, you know, we may not have the most elegant solutions all the time, but we are going to have a future in the cyber domain where someone on offense is going to automatically say do this on defense. And you can liken it to wartime, you know, military guy, you would do field dressing. If somebody got shot in a field, you didn't get to bring them right to the hospital. You had to do some tourniquet and some bandaging right then and there. Maybe that's the first generation of this, but we're already working on it. Nobody's sitting here saying it's not going to happen. So, the, you know, the closed models can slow down all they want, but we can't slow down. The threat's not going to allow us to. Criminals still want to make money and espionage is allegedly the oldest profession, so it's probably still going to happen.
But you are making a distinction there between development of AI and pacing or not thereof, and deploying and preparing to defend against AI as it currently exists today. So just to confirm that you're not taking a position at all in this conversation on if frontier closed-source AI development should be paced or should be allowed to proceed or encouraged to proceed at a maximal pace.
Whether a model is going to be closed or open, it's going to be pretty damn good at attacking you. Let's start there, okay? And it's going to be used to do so. If there was world peace and everyone born was good, we wouldn't have a problem. I hate to say it. We would figure this out, but there are going to be malintent folks. And everybody thinks it's AI out of control. I don't think so. I still think AI does what humans ask it to do if you have responsible use. Unfortunately, there will be folks that will ask, depending on your perspective, AI is going to be tasked with doing things that you probably won't like as an enterprise.
I think this is a good moment. Lon, can we go over the two Sam Altman concerns that he put out about AI safety? I think it was two days ago.
Absolutely. So, yeah, it was just after midnight on Sunday. Sam Altman posted a thread on X laying out what he thinks are the two ways AI theoretically goes wrong. The first is losing control of the future to AI, which he called unacceptable, and he said OpenAI is unapologetically on, quote, team humanity. The second is concentration of power, and he got specific that this could mean one person, one company, one country, or one lab ending up too centralized in terms of controlling AI. He also said OpenAI welcomes a federal framework with consistent safety requirements for frontier AI. Though he drew a line on the word itself, saying pacing is not stopping markets. Uh, did not take this well. TSMC dropped 13% in Tokyo yesterday. Samsung and SK Hynix each fell more than 4% in Seoul. I mean, I guess the big question is how seriously are we taking this?
Yeah. On the concentration point though, Soups, I want to ask you about this because I think Kevin handled the, you know, is it autonomous swarms or is it people using AI that we need to worry about more, the latter in his view, but concentration could, I don't know, concentrate a lot of intelligence in these labs and away from startups like your own. So, how does Sardine think about concentration in the AI game and defending itself against the possibility that these foundation labs slowly eat into your business domain?
The world of fraud as well as security, it's really just a cat and mouse game. So, that's where I'll start, right? And I, you know, I wanted to also tee off, you know, what Kevin was saying. I actually fully agree with his point that, you know, as we are at this new sort of... this is a new technology, whether it is open-source models or closed-source models or not, you know, this is actually an opportunity for us to uplevel our security game, right? The institutions who are actually going to be adopting red teaming tools or using AI to attack themselves or setting up bug bounty programs which encourage them to do so, they're actually going to be far ahead of others in terms of like shoring up their defenses. So that's my first point. And then the second point is that, you know, even if you look at all the incidents that have been sort of talked about, like the Hugging Face exploit, etc., etc., it is all about like the instruction that was given by a human to the AI agent. In the case of the Hugging Face exploit, you know, of course there was a mistake, AI thought it's actually in a simulated exercise except it was in an open internet. But then the question that we should be asking ourselves is could we have given AI the right instructions so that it doesn't go out and do things autonomously which it's not supposed to do? And to me that is a solvable problem. And then going to your point, the question you asked me about, you know, concentration risk of AI in the hands of, you know, a couple of frontier labs versus not. Actually, we see the commerce right now. We see that, you know, the open-weight models are in fact getting equally good or, you know, coming pretty darn close to be as good as the closed-source models, right? Um, I personally, I'm running a, you know, a local harness, you know, using Hermes for example, and yeah, a lot of the tasks that I was actually relying on closed-source models, I'm now seeing that, you know, there are many of them that you can actually use open-weights models. At the moment I'm trying the American open-weight models, but, you know, the Chinese ones are also, you know, of course they're ahead of the American open-weights models and we will catch up. But the idea here is that, you know, I'm not afraid of or I don't think that that concentration risk exists. Open-weights models are actually going to pretty soon catch up. And there was another question of yours, but I'll shut up for now and then...
answer. It's interesting to hear you not be too worried about concentration risk. And the way I had this written in my notes was just, is GLM 5.3 enough? Like, does that get us enough cybersecurity capability to be secure? And it seems like in your view these models on the open side are catching up quickly enough. But at the same time we're seeing companies like Salesforce with their new COA model and I think Sardine trained its own model as well, if memory serves.
Yep. That's right. Yeah. So we trained our own foundational model to do fraud detection. I think our thesis is that, you know, the frontier labs, they are training their models on publicly available data, right? And therefore data is the new moat, right? So companies like Sardine, we have over the last six and a half, seven years built up our own data consortium, right? So we service about 450 institutions globally preventing fraud and financial crime, and we have like a consortium of six billion plus devices, 440 million plus consumers and three million businesses and trillions of dollars of payment activity that we've seen these consumers and businesses perform. So that is actually proprietary data which frontier model labs do not have access to because they're not in the business of fraud. Right. And we can actually train our own models and that is what we're doing now to do better fraud detection. So we've traditionally used, you know, supervised machine learning or unsupervised approaches which are, you know, your traditional decision tree type models, but just a couple months ago our AI team, you know, we trained a foundational model for doing card fraud detection and that model performs, you know, depending on the context, like either 40 to 65% better, right? So we applied it to a couple of clients for whom we had never seen the data before and right off the bat that consortium model was like either 40% better for a consumer client and 60% better for a business client, right? The fundamental reason why in the world of fraud detection large language models or transformer-based models work is because there's a language of payments as well, right? So for example, I shop and therefore swipe my card at a Safeway or Starbucks, etc. And if somebody stole my card, they're going to transact with it at Gucci's. So the next token prediction, Soups being associated with Gucci's, is actually highly unlikely, right?
I think your t-shirt looks fantastic and you're the most fashionable person on the show today. Soups, don't talk down to yourself. Kevin, I'm curious about how this applies to your end because I can totally see how in the case of Sardine, lots of data, post-train a model or whatever they're doing and then you can take something to market that's materially better. On the other hand, when we think about cybersecurity and penetration of organizations, we think about usually the largest, most dangerous models, dangerous scare quotes, Mythos, Fable, Astra, etc. Um, this is a silly question, but I'm going to ask it anyways. Are you guys considering post-training a model to make it more dangerous to improve your ability to red team for companies or is that unethical?
No, you definitely do it because that's where it's going. It's no different than people creating the virus of tomorrow today so you can get the vaccine ahead of time. I hate to say it, that's just smart. You know, you want to be able to know your exploitable risk before the bad guys do. And I've seen hundreds of CEOs go through breaches, probably more than anybody else on the planet. And nobody deserves to go through those, you know, and so we've built a ton of controls. You can't just... we hired 50 of some of the best Fortune 500 red teamers on the planet. In fact, when we went through their history, we had literally been hired to red team 99% of the Fortune 100. I'll leave one company that's never hired me out of the dialogue, but the other 99 did.
Do you want to name which company that is because we're all... we're all...
I love their products. I use their products all...
It was Nike, but now they're gone.
Yeah. Yeah. Well, I don't think Nike... it's Nike Fortune 100.
Yeah. But it's, you know, you got to... when every single thing our team does with humans, we have autonomous mode for our agent to run, semi-auto mode with like a safety agent running on top of it because it doesn't have judgment. You got to apply judgment somehow. We've created hundreds of thousands of evals. We've created classifiers. We run our agent when it goes on offense. And yes, we have different agents to scout and recon your network versus hunt and exploit your network. We have rules of engagement we work out with our customers. But our agent was trained with visibility into what all our operators who have been doing this for 10 to 15 years do for a living. We know what the prompt should look like. We have a listener passively see every damn prompt going to the frontier labs when we choose to use them going outbound and inspects what comes back because we usually know what to expect coming back. That's called... you can put a bullet in the head of the agent if it does something you don't want it to do. We inject in and see every instruction it offers for months. We've only been testing, by the way, production environments at 1A enterprises. That's what we do for a living. So, we're running our agent externally in the wild, but you got to not just say, 'Here's the behaviors we want it to have.' I think you better really be thinking, 'And here's a bunch of behaviors we don't want it to have.' And I do think there's a certain amount of domain expertise. If you're building an agent that does trading, you probably want a bunch of financial experts and investment bankers and folks that are aware of all the SEC guidelines and other things to do something. Um, same in the medical field, same in cybersecurity. If you're making agents tailored to do fraud, Soups, you're going to have fraud experts. You do need an expertise to get the behavior you want to leverage from AI that you get. Um, and that's important. So, there is a way to put guardrails on these things where everyone can benefit, but I think it's probably harder for the R&D at a frontier lab to pull that off.
Yes, I agree with that. Really curious now hearing you describe the different agents you use and being out there in the wild. What does your AI spend curve look like internally?
Well, I can tell you this. It used to take us five days and two people to look at complex web applications. Now it costs $5 and takes about 10 minutes.
So everything in reality with that perspective is getting more affordable and more effective for everybody to get better results because of AI. This is a conversation we have on this show a lot where there's one idea that, well, if you're doing high-end professional work that has to be done right, like detecting financial fraud or cybersecurity, you need to be using the absolute best models on the market and that's what you're going to be going with. But now that we're having this conversation, it almost sounds like, you know, better good enough will get there, or it's more of a mix. Is that what you are both finding, that you can use a combination of the highest level frontier models and, you know, maybe slip in some Kimi or some GLM in there?
Soups, you want to go first?
Yeah. No, 100%. So, um, I think there are certain tasks for which you don't need like a complex closed-source model which is more expensive in terms of token spend, right? And you can actually get away with just using an open-source model. In fact, one of the theses I'm working on right now is, could I create a local harness and a router where I'm routing between open-source models which are running on my laptop and then for complex tasks I'm handing the task over to a closed-source model, right? And that's why you've seen, right, like OpenRouter got acquired by Stripe and there's like a whole crop of other routing companies who are popping up now, right? Sure, there's a point to be made here that, you know, all CFOs everywhere are realizing that token spend has just blown up and, you know, everyone is now trying to optimize the token spend.
Yeah. Yeah. And the way we do it, we, you know, I've got to trust it because it's a PhD from Stanford telling me, but we use every model in almost every assessment we do. And there was a time when we first started and people were like, only use the American models, you know, and the closed models to do the attack. I'm not convinced that's what you're going to be up against. We created a test eval, you know, 20 different kill chains as we call it in cybersecurity, get remote code execution here and how do you permeate to the crown jewels, and we have 20 different full paths in this. And we've tested Kimi, you know, the GLM product, the DeepSeek product, some of the open labs, and OpenAI, Anthropic, Gemini, and we grade them on efficacy, speed, and cost. And by the way, we use Grok as well. And we looked at them all and at different times, like one time exploitable risk was found fastest by OpenAI. You know, three months later, it was something else. Uh, in general, what we saw was the time to get to exploitable risk was faster with the frontier labs that were closed. But here's the dirty secret for us. The last time we ran our tests across our 20 different full kill chains, written by several people that have done this for over a decade, no lab found more than eight of them from point A all the way through. All right? And this is something that if you put very experienced humans on, it would get to 20 out of 20 in a hell of a long time, but it would get there. And that doesn't mean AI is not as good as a human. Trust me, the AI when we run it, we have found over 90 zero-days at Armadin since January 1st. And some people would be like, that doesn't seem like a lot.
This is with zero source code review, zero source code review, and zero authenticated access. We're banging on a black box from the outside, guys. And that's bad when you find over 90 in a year. I don't have the exact number because there's 50 people in a room right here finding one every day.
But we don't find that without AI doing so much of the work that some of the unique people that can find exploitable code can find it. So if I had to peel back of the 90, how much was found by the frontier labs and AI, again, not source code, it's probably under 10%. But it's doing so much incredible work that the people that red team for a living finally get to have the fun stuff to do. Do I think that the frontier labs get to... We've already taken out the beginner red teamer with AI. It's just you better get from beginner to intermediate really fast.
You know, I think we're taking out the intermediate as we speak. And for a while, you'll still need someone in the cockpit that's elite. And then you still need someone over watching what AI does. I think for some of the judgments, you better have a model that's literally observing what your agent is doing and the results it's getting and every once in a while nudges up a judgment call because even humans dispute the judgment in some situations. So have the dispute with humans if you need to for now. So I do think a lot of the judgment we do will get percolated upwards to humans, especially in red teaming on production networks. But that's my long-winded way of... the PhD guy shows up every three months and shows me here's how all the labs are doing. And here's the weird thing. You see the closed labs accelerate into exploitable risk. You look at all our evals, look to the right over time and you're seeing Kimi right there with them.
When it comes to what we do for a living, the open models are pretty much there. Not as fast, but way cheaper. So if you look at modern warfare, how do you want to do it? Well, cost is going to matter, you know, but again, it comes down to the infamous: if the frontier labs are 5% better, do you want to pay 80% more? Maybe in some circumstances, I think that's probably the right thing. Uh, but so many tests, like Soups, you like you said, boy, the open models are delivering far better than human results, very consistent, and you can leverage them. And I kind of liken it to, you know, in military terms, the open labs versus closed labs is kind of like we're going to deploy an aircraft carrier. That's your closed lab showing up and it's going to make its presence known versus we're going to deploy a bunch of PT boats and a couple, you know, 500 drones. Different efficacy, but you might have the same results over time.
Interesting. I thought you were in the Air Force. Why are you making reference to Navy hardware? I think this is... you're going cross-branch here.
I did say aircraft carrier, but that's still... maybe you're right. They're the ones...
It's all the armed forces, US armed forces. They can mix their metaphors, I think, in the armed forces.
Here's, I think when you go, you got to go to managing between win and loss and not profit and loss when discussing technology. And that's why I love that I'm on with someone who's looking at fraud. That's a win and lose category. That's an important thing for us to prevent. Um, you know, and the same thing with cybersecurity where people are getting ripped off. During this podcast, probably 50 people are going to lose 100 grand or more due to a fraud.
That's what I want to ask about because Soups mentioned that they're seeing a rise in deepfake-based fraud. So, we're seeing kind of like AI has improved, it allows for new things, people are using them to do bad. But when we talk about cybersecurity, it feels a lot like we're talking about a couple of incidents from the major labs themselves and much less about picking a company at random. How's Kroger doing in cybersecurity terms?
Man, love Kroger. Love those people. They're one of the... out on the East Coast. I'm sure they're doing great. You know, hey, like I said, it's not like GPUs are on every single corner. If you are a criminal, you still got to get to... you got to get the processing power. You got to pick a model. You may have to jailbreak a model. We are not seeing the type of attacks that Armadin can do live. We're not seeing them in the wild just yet. We're really interested.
So, we're getting a respite based on compute and availability.
And by the way, if you're a criminal, you kind of want to do a lot of these activities anonymously, which means either... I mean, you got to pay for it somehow. So, Bitcoin will be your preference. But if you got to use a frontier lab, you got to pay for it. Uh, you're going to probably just try to hijack somebody else's stuff. And then those labs do generally probably, you know, you're going to have to step over the ankle fence as we call it and get the models to do things they may not want you to do.
It sounds like you're saying though, Kevin, that the current GPU crunch, the current compute crunch is keeping us safe from the various use of AI models because all the compute, let's just call them white hat companies and actors.
You're not going to try to hack the whole darn planet as a criminal right now using AI. You're going to have to pick your targets and do some certain specific.
There was an article in Bloomberg last month, I just sent it around if somebody could pull it up here, that actually exactly says what Kevin is now explaining to us, that Chinese hackers are relying on DeepSeek because it's the cheapest. So like they're very budget constrained with what they can do. So they're doing as much as you could possibly do with the top DeepSeek model because it's so expensive. But obviously when you have, you know, an American company like Armadin that's using a combination of all the most powerful models, they're not going to be able to sort of maybe keep step with that, I think, is the idea.
A lot of what you do in cyber, by the way, you have very intentional and deliberate things that you do where you don't even need to tap the frontier labs. You don't need it. We've written over 150 custom tools to do certain things like identity security assessments and other things. Um, and that's the difference. You're...
constantly going, do we need to go to a frontier lab? Yes or no? Which one? With the recon, we'll use more of the open models, but if we're looking for real exploit development, we'll probably, depending on time constraints or cost constraints, we would probably go to a closed lab, you know, and get a little bit of a faster answer because we do see that arrow for finding exploitable apps faster right now in like an OpenAI or Anthropic or Gemini than DeepSeek. But again, it's still compressing. You know, there's going to come a time, it's not far-fetched, think two years from now, it really doesn't freaking matter when it comes to cyber AI is doing regardless of the model. It's taking cyber operations to a new level where you better have AI on defense to defend.
Okay, Soups. I want to make a slightly related point here. You were talking about training a model earlier and Kevin's talking about, you know, harnesses and testing suites and benchmarks and all this. It seems like the amount of intellectual weight you have to have inside of a technology company today to correctly take advantage of AI is incredibly high. One, am I reading that correctly? And then two, is that going to get easier or lower over time, therefore allowing more companies to experiment and build with AI in a way that does move the bar of technology forward as you're doing with Sardine?
Yes and no. Like, you know, we do see that in terms of like AI startups, there's a bunch of them who have popped up who essentially help you train your own models, right? So we ended up doing it ourselves and of course Nvidia also, they take a very active stance in this, right? So they have, you know, a full team which is just dedicated to helping companies like us understand how to actually go about training a foundation model. So we took advantage of that, right? But then at the end of the day, you know, it's really, you know, there are those who recognize the threat from AI to their own business and therefore they will, the ones who realize the threat, they are actually going to go and train their own models, and those who are just caught sleeping at the wheels, right?
