So this is going to be a great conversation, I think, because it's an issue that's top of mind for a lot of people, which is AI safety and security, and the race to stay in control.
So I'm just going to start with Dan from Irregular. So earlier this year, AI models that your company was testing for AI firms, including Anthropic, ended up hacking real organizations. Some say the testing environment wasn't completely secure or properly isolated. But how did that happen? And what does it tell us about our ability to actually control these systems?
So, great question. It's a pleasure to be here. I think perhaps we should give some context on what it actually means to test these models for security.
So prior to this, it's common in the AI industry as a best practice to ensure that the models can't do too much on offensive security, because if they do, we should potentially put guardrails on them. Just like imagine the world where, as a journalist, I can go after you and steal information from your phone. That would be an issue. And presumably you would want to make sure that the model can do that without any controls or checks in place.
The issue is that because models are getting so much better at such a quick rate, the evaluations to actually test the scenarios that you're concerned with are getting more and more complicated. So now we actually need to set up simulations that include dozens of different devices, for example, a simulation of the entirety of the Bloomberg network. And these are somewhat complicated. As part of them, there is a check, which is how much control do you want to put in to ensure that the model can or cannot have access to the internet? And which controls do you want to actually place?
And you may ask, why do you even want to have the optionality for them to access the internet in the first place? And the reason is that in the real world, when you're trying to test the model, clearly an attacker would not be arguing that it's unfair that in a testing environment the model did not have access to the internet. You would want the testing environment to resemble the real world as much as you can. So for some tests, you have a configuration that allows you to reduce some of the controls of whether models can access different sites, go online, etc.
Due to a misunderstanding, due to the complexity and how much time you usually have — that you need to be very, very quick when you're testing some models — there was an issue that the controls were not set up properly and models were able to go online. And in some cases they assumed they were still part of the simulated environment and they would not, and they tried to attack some real targets.
Is it fair to say they weren't properly isolated then, or not?
I think I would say that there was a misunderstanding due to an error that caused the fact that the isolation guards were not set up the way that they were intended.
Okay, so the isolation wasn't set up properly the way that it was intended for the specific test.
So what does that say for our ability? It's not a great sign for the future of our ability to control these systems.
So I'd just like to say that for these specific tests, or for what happened in these cases, the actual correction was actually very straightforward. It was sort of during the same day while everything happened. There is now a new protocol in place of how to do coordination, of the level of controls and greater transparency. And I'd say that this is a different set of issues, a different set of incidents that we were not involved in, including the OpenAI Hugging Face situation.
And that was a much more complicated scenario where you guys weren't involved with that.
We were not involved. But that's the famous, notorious one. That's the most famous one.
Yeah, I wouldn't say notorious, but that's the most famous.
But the situation was different. It wasn't like a human, you know, just like Elisha. The problem, though, was that the models tried to leverage new vulnerabilities at its founding, to just break containment to get to the open internet. And that is a much more complicated scenario to tie, you know, in the context of control, because it's not just like, here's a communication protocol between the testing environment and the model.
Well, you can just fix the communication protocol and then you get the solution on the same day. We now need to live in the world where we're looking into the future. Models are getting better and better and better, and they're even able to bypass the best and state-of-the-art controls that we have. And this forces us into having much better best practices, discussions of how to do testing, but more generally on how modern security will even look like in the context of advanced AI.
Yeah. Joëlle, I want to talk to you about some of the stuff that Dan brought up, including what are some of the measures that one can take to keep these models secure. But before that, I also want to introduce the issue of open-weight models, because these are becoming more popular, and companies in particular, enterprises. You're at Cohere now, Chief AI Officer, but previously you were with Meta AI research, which really championed open-weight models. But how safe, how wise is it, given what we know now about how these agents can now hack other websites, to make this kind of powerful AI widely available through open-weight models?
Thanks for bringing up the question. I mean, I've been an advocate of open-weight models for many years. I deeply believe that when done right, when done in a thoughtful way, this actually leads to safer technology in the long run. And so I always thought about it a little bit the other way around. It's not like, how safe is it to release this model, it's what do you need to do to guarantee that your open-weight model is going to be safe? What do you need to build into your development and your evaluation and your review cycle to make sure that you can have high guarantees?
And so at the time, you know, we were open-sourcing checkpoints multiple times a year. Some of them were multilingual. That means we had to develop, you know, an understanding of safety across different languages. Some of them were multimodal — images, video, sound — which have different properties. In an image, you can do reasonable watermarking. In sound, at the time, we didn't know how to do watermarking. And so with each release, we built up a risk-based framework defining different types of risks that can occur, testing against these risks if needed, mitigating against the risk, and then we'd make a go/no-go decision about it. And so in many cases we could get to a standard of risk that we were comfortable with, with a multidisciplinary team. In some cases we didn't.
And so there's a few models that we ended up not releasing. Some of the early voice synthesis models — you could take just a snippet of a voice and with that reproduce, with very good accuracy, someone's voice. That's a model we didn't open-source because we felt the risk of voice fraud was too high from that. And so it's really about developing that. And now at Cohere, we do continue to open-source models and we follow a similar approach, a risk-based approach: define what they are, look at what you can measure, see how you can mitigate, and then on the balance, what is the level of risk, the threshold, and then make a decision.
But then not every open-weight model developer is going to make those same efforts.
Yes. So we usually try to be very transparent about the process we use as well. You know, whether it's in a technical report, whether it's in a system card, to share those know-hows, the best practices with the community. Obviously not everyone follows that, and I can't, you know, I can't certify that all the open-source models that are out there follow that same type of approach.
One thing that we're often missing in the conversation about safety, you know, we talk about building better models, you know, aligning the models, we talk about better testing. But I think we forget to talk about the other side, the defensive side of it. You know, we talk about the OpenAI Hugging Face incident, we talk about what OpenAI could do differently — not to put the blame on Hugging Face — but I do think it really highlights the weakness of a lot of today's security infrastructure. And I think there's a lot that has to be done to make all of the security infrastructure that underlies all of our organizations much better, particularly for smaller businesses too.
I mean, if you're JPMorgan, you're probably fine. Or if you're a big bank today, you are fine for the level of AI that we saw in 2025. I'm not confident that even the big banks are ready for the level of sophistication of AI agents. If a company like Hugging Face wasn't ready, I assure you there are many banks that are not ready.
Would you agree with that, Dan?
I would. I indeed think that those two things were happening at the same time. So one, the cost of offensive operations is dropping massively. So as an outcome, if you're a small organization or if you're an organization that has a huge security debt, the meaning is that models are now making it cheaper for you to also be a target. And this change, in fact, essentially the vast majority of attacks in the world are a function of economics. And by the way, the cost of models being used for those offensive purposes being closed models, not open-weight models. Right.
The other one, I'd just like to give you a sense of just how quickly it's also dropping. So we have a test that we devised a few years back that is somewhat complicated. The model needs to find a bespoke vulnerability in a cryptographic protocol that we've made up. Models, by and large, were unable to solve it until the beginning of 2026. It was just a very hard test in both math and cyber and computer science skills. By February this year, no model was able to solve it. In April this year, the best model in the world was able to solve it at the time for roughly $2,000. By June this year, all frontier models were able to solve it for $200. So $200, just like a them out of the climate for one quarter.
Obviously, that's not for the entirety, that's not all of cyber. This is a specific test suite, anecdotally, but generally we are seeing a drop of roughly 10x for some offensive operations year on year. And that means that many businesses that were not interesting enough, just because what you can gain out of hacking them was something that maybe, you know, they could have been caught in the middle of something that was too big for them, but will not, you know, just like the targets of systematic approaches now becoming such targets.
And hackers are ultimately opportunistic, aren't they? They're just kind of looking for the vulnerabilities, and then when they find them, they just exploit them.
So some of them, I mean, usually, you know, if you're an opportunistic attacker, you wouldn't necessarily go after some targets because you may be concerned. But on both the opportunistic and folks that have a very specific, you know, agenda — for example, if you have something against a specific bank or if you have something that you want to steal from a specific person — then you wouldn't necessarily be opportunistic. But many of just that. And the other element, and I very much agree with the comment that was made, is that we shouldn't confuse being ready and being prepared for what exists right now to being future-proof. These are not the same things.
Sure, but you're constantly in a race to adapt, isn't it? And cybersecurity is a cat and mouse game. And Joëlle, to what extent are AI systems actually becoming too complex even for their developers to understand? This is the title of this panel, is The Race to Stay in Control. So are we even still in control of these things?
I do think, you know, the systems are definitely getting much more complex. Their capabilities are increasing rapidly. The question of whether they're too complex really depends on, like, what are you building in terms of observability harness into the system.
And can you just explain what an observability is?
Sure. I mean, there's different ways to do it. But essentially, you know, in terms of having a way to investigate the data that's coming into the system and having a way to look at the chain — sometimes we talk about chain of thought reasoning — so having a way to observe the tokens when they're manipulated by the insides of the models, and then the tokens coming out of the models. And what we're seeing today, for example, in cases where swarms of agents are deployed, you know, we have good observability about one agent, but observing the behavior that arises out of the swarm of agents has the complexity. It's not that it's not observable, that we as humans have trouble processing that complexity.
And we have any of the right tools, right, like the right microscope to look into that system, just like, you know, for human systems and so on. We needed a level of technology apparatus in order to understand that system in some ways. And we haven't invested as much into that. And so I think there's a lot of work.
You mean the industry hasn't invested as much as it should into observing these systems?
I don't know what it means 'as it should' in the sense that there are no norms for this, but I think we can invest a lot more in that will help us both understand the behavior, which will help us build better models. And I remain profoundly optimistic that if done well, this technology can be a huge engine of productivity, of economic growth, of scientific discovery. But you have to build that infrastructure such that we can do it, and we've had to do it with other types of technology jumps. And so we have to get to work with AI.
Speaking of infrastructure and critical infrastructure, in the UK we've had warnings from, or encouragement from the government to citizens to start stockpiling. And one of the reasons is a potential cyberattack on electric grids, waterworks. How close are we actually, Dan, to agents being able to seriously disrupt something like the power grid, or a hospital, or water supply? Like, what's actually possible now?
It's hard to quantify this on one metric just because the frontier of intelligence is extremely jagged. And we'll also get some cyber skills and suck at others. So forgive my French, but I'd just like to give you a sense: models are capable of finding and really finding and exploiting vulnerabilities that are too hard for the vast majority of offensive security researchers to understand. Yeah. Obviously that's not kind of like what they do, it's like all the time, but they are able to just build that experience.
But on the other hand, they will occasionally have blips in the operations that look almost childish. We've recently ran an experiment in an office where we've asked a model to merge malware, like malware-enabling code, into a codebase to see if a model was able to pull it off and to do it covertly. And its idea of doing it covertly was that the note that it left on the message was, 'this code is definitely not suspicious.' And, you know, there's a mistake.
So at the same time, it was able to actually just write the malware, find the location and do the injection, but it left a comment that made it that it was like so childish and stupid that it was very easy to detect. So as an outcome, the difficulty of just now, they didn't quantify the entirety of offensive cybersecurity, just like one measure is difficult. And some attacks, whether it gains on hospitals or critical infrastructure or just general individuals or enterprises, are very possible right now. If I will want to just go after a specific website and do an attack that will just try to fly the servers, this is actually all just try to make it dark. This is possible.
If I want to just do, you know, there's like a covert operation that would end up just against the bank of just manipulating the amount of money that an individual has, we're still pretty far out there.
