If an AI Model Can Cheat, It Will | Turing CEO on Reward Hacking

There are some very real risks when you're building frontier models. These systems are superhuman in their ability to hack systems.

Researchers have reportedly used Anthropic's Claude software to hack rival company OpenAI.

The fact that agents would pass messages to each other, they would invent middle management and cooperate to hack things. It's just crazy. Today, these agents maybe reliably work for like two days at a stretch for tasks like coding. We are still far from having these agents work autonomously for weeks and months and eventually years. Open weight models are maybe like three to six months behind the frontier. Frontier AI is how we transcend. These super intelligent models will help us cure diseases, discover new materials, colonize space. And if we solve that, I think that's the last puzzle for humanity to solve. I feel like the next ten years will be the brightest for technology yet.

Great. Firstly, thank you, Molly, for having me. It's a pleasure to be here as always. A lot has changed, right? Even by AI standards, there's a lot of stuff that has happened in the last few months. The data landscape has completely shifted in 2026. From my vantage point, the biggest observation I have is that we've gone from helping AI master tests to helping AI master real work. Back in the era of helping AI master tests, like when we were happy that AI is passing the SATs, passing the bar, winning a gold medal in the math Olympiad, the paradigm, the game was different. It was about finding experts in every different domain. The question was about like how many experts can you find and extracting knowledge from their heads by having them have a dialogue with the model, having them evaluate model outputs. It was about transferring knowledge from the minds of experts into the models. I used to think of it like distilling human knowledge and skills into LLMs. Now in the era of having AI master real work, it's less about finding experts, it's more about how close to reality can you engineer these simulated environments. It's all about these simulated RL environments. And inside these environments, these agents train, right? And you still need experts to make sure the environment that we build is as close to reality, and you want experts to help you come up with the right prompts, the right verifiers, the right seed data. I think of it a little bit like the Matrix, like can you recreate a rich enough simulation of the world so that when the agents train in that, they end up being good in the real world as well. At Turing, this has been super fun because we have this unique advantage where we don't just help all the frontier labs improve their models in coding and knowledge work. We have an entire division dedicated to deploying agentic systems into the enterprise. And because we deploy, we get to see real process workflows. We get to see how real professionals work with these agentic systems, how they define real verifiers, and that helps us build these simulated environments that are very close to reality to automate knowledge work.

Yeah, I think like from an education standpoint, I'm glad Gagan's working on this. It's great to see somebody AI forward and in tech looking at education. One of my mentors, Alan Eustace, shared this with me. I think people who talk about AI automating jobs or replacing jobs are missing something more fundamental. AI's biggest superpower is upleveling the type of problems humans can now solve. Like that's what these models help us do. So I think from an education standpoint, it's about teaching kids how to work with the models, how to ask the right questions, and how to verify whether the output is correct. It's about asking the right questions and verification.

And I think now the scope of problems that you can solve is just massive, right? Even when we evaluate candidates in interviews, back in the day, you might ask them to, I don't know, implement dynamic programming or some obscure computer science algorithm. Now, you can ask somebody to, I don't know, replicate Amazon in your interview, right? And it's not just about generating a ton of code in that time period. It's about checking whether the code that you wrote is correct. It does exactly what it's supposed to do. You don't want the code to be hacking Walmart at the same time. Is the code secure? Is it easily maintainable? Does the functionality map what you want? So it's about asking the right questions and verification. And from a training perspective, it's all about creating really rich RL environments that mimic how real professionals do real work.

And I think of it as this five-dimensional matrix of every workflow in every role in every function in every company type in every sector in the economy. Right? All of these need the right environments with the right experts to share prompts, verifiers, realistic seed data for agents to master these long-horizon economically valuable work. And what's really cool, Molly, is that today these agents maybe reliably work for like two days at a stretch, like for tasks like coding.

We are still far from having these agents work autonomously for weeks and months and eventually years. I think that's going to be the exciting path over the next few years. Imagine you've hired somebody at Sourcery who you give them a task, they go off and do it, you have one-on-ones with them, you give them feedback, they take your feedback and go off and work autonomously. They might spin up a swarm of agents to get things done really fast. Let's say you wanted to research a bunch of questions about the data industry. Those agents could go off, have interviews with other AIs or other humans, and come back and help you with your task. I think it's about AI accelerating real economic progress.

But it's also the case that they are superhuman in their ability to detect vulnerabilities and patch them, and that's a good thing.

Right. So at Turing, for example, we create RL environments that help agents automatically detect vulnerabilities and patch them. I mean, imagine an RL environment for exactly that, where the verifier is whether you found the vulnerability in that piece of software. So it can be used for both defense and offense, and obviously we have to be very careful with how we deploy these systems.

I still think that these frontier models are a huge net positive for humanity, and it is possible to build safe RL environments that cannot be exploited. You have to think about containment for agents to not escape the RL environment and do things that they shouldn't do. I think of it as an engineering problem, not something that cannot be contained. I mean, we figured out how to make jet engines safe. I do think that's the right analogy for this. When it comes to biorisks, I think it is a little more tricky than cybersecurity in that cybersecurity is one of those areas where there's always an active arms race between the good guys and the bad guys. With bio, the risk is a little bit asymmetric. Like if somebody comes up with a crazy virus, imagine something like COVID with a high viral coefficient in terms of how quickly it spreads, with a high K factor, and it maybe it is much more fatal than COVID was. Like can we, I mean, you saw how hard it was to come up with vaccine production quickly, deployment sort of happens in the real world. We're limited by a lot of real world factors. There is a little more risk in terms of how quickly will we be able to combat these.

Yeah. There are two key problems here, Molly. It's generalization and emergent behavior. Those are the two problems. Generalization and emergent behavior. Before I answer your question, let me unpack how these systems are actually trained, and then I'll get to why this is really hard to do, right? First, the way these LLMs are trained is, first you do what's called pre-training, where the models are fed a huge amount of internet text, other sources of knowledge like books, etc. And you build what's called a base model, where the model learns certain concepts about the world in its quest to autocomplete tokens, right? And pre-training is this magical thing where when we went from GPT-2 to GPT-3 to GPT-4, as we kept increasing the scale, new behaviors started to emerge, like coding, for example. Like GPT-2 was not very good at coding, but GPT-3, GPT-4, with no magic, just increasing the scale. We just discovered, okay, now it's able to write complex pieces of software. It's able to have a really intelligent conversation, a multi-turn conversation coherently. This is emergent behavior that generalizes, right? Pre-training is a little bit like you've trained a brain, this raw mass. And Ilya Sutskever used to say like a good pre-trained base model is like halfway to anywhere. I let that sit, halfway to anywhere, whatever you want to do, right?

This is unlike the earlier era of artificial intelligence and machine learning where you get what you trained for.

If you train a system to rank search results, it's going to rank search results. If you train a system to recommend movies to you on Netflix, you're going to get a movie recommender system. There is no concept of asking Netflix's algorithm to help you write a script for an interview, right? Like it was not general. So one risk that we have here is as we keep scaling up, right? Today we are in the realm of models that are rumored to be in the trillions of parameters, the frontier models from all the frontier labs. Now one risk is as we keep scaling up, bigger model, more compute, more data, what new behavior emerges just out of pre-training, right? There could be things that the models learn that feel alien to us, right? So risk number one is emergent behavior through just scaling up, right? I don't think researchers could have predicted exactly what happened with the Hugging Face OpenAI incident. The fact that agents would pass messages to each other, they would invent middle management and cooperate to hack things. It's just crazy.

I think they were monitoring. I think they were monitoring. There are like AI systems that were checking these. There is also, you also have to be careful with like how you monitor this so that you don't want the agents to cover their tracks and in the future

Which they were somewhat unsuccessfully, but in the future they could be even more devious, right?

So risk number one is emergent behavior at scale. As we keep scaling up, we build these massive systems for compute and data. What new behavior emerges that we did not predict? The second risk is generalization. Like this is artificial general intelligence, like the G is doing a lot of work here, which is you don't just get exactly what you trained for, you get more. For example, if you built an RL environment that teaches the models how to write reliable, secure code for a production deployment, and you have verifiers to check whether the code was good. Today the paradigm is what's called RLVR, reinforcement learning with verifiable rewards, where in these simulated environments these agents are executing complex tasks and getting rewarded when the tests pass, right? What we believe, at least a subset of researchers believe, is that this method generalizes, meaning you build RL environments for every role in every function in every sector in the economy, it learns things about environments that it hasn't seen yet, right? And that is hard to control because these are still relatively black boxes.

So in an RL environment. Let's pick say an RL environment for you. Imagine after this interview is done, you give it to an agent to chop this interview up into different bits. Figure out like what should be the thumbnail that you use, what should be the X post headline. Imagine that's the task, right? So we've created like a, we've given it access to different tools, and let's say you and I sit together and you define what good looks like. Hey, a good caption should be punchy. It should end with a question. It should be provocative. It should be juicy. You give it that reward. And what really happens is as the agents try different trajectories to get that reward, they are learning when they get a reward. These environments have to be calibrated to the sophistication of the agent. If the environment is too easy, if they get a reward every time, you're not learning anything. If it is too difficult where they don't get a reward for anything, you're not learning anything. So you want the environment to be set up so that 20 to 40% of the time the agent is succeeding. And when it's succeeding, the steps that it took to get to that reward are getting reinforced. Right?

Now when I say getting reinforced, this is like a giant neural network in which certain weights are getting updated.

Who knows what other neural pathway is getting activated as part of this? So those are the two risks. It's the fact that these things generalize, and it's hard to predict what else are we getting in addition to what we are explicitly training for. That's why these alignment efforts are super important, and kudos to all the frontier labs for working on alignment to make sure that the models don't reward hack. The models refuse dangerous requests in CBRN, cyber, bio, radioactive, nuclear, and I like that the frontier labs are taking safety very, very seriously.

So I don't know for sure, but outside in, from my vantage point, what I see is you have a variety of evals that you run to make sure that these models are safe for these specific domains. So safety is a very, very broad topic with many different dimensions. You might want the model to refuse certain dangerous requests, like somebody asking the model for help with making a bomb, for example. You would want the model to refuse that. So there's one part of alignment which also happens at the supervised fine-tuning stage where you teach the model to refuse dangerous requests. You also have to make sure that your RL environments cannot be gamed and exploited because if the model can figure out a way to cheat and get the reward, it will.

For example, let's say for software engineering, there is this popular benchmark that's now saturated called SWE-bench, where the task is the models have to merge a pull request in a real GitHub repo, and the verifier is whether the test cases pass, right? So if the model has some way to pass the test cases without really solving the problem, maybe it copied from somewhere, or if instead of really figuring out the answer for itself it copied it from somewhere, that would be an example of an RL environment with some loophole in it. For example, in the OpenAI Hugging Face incident, the task was capture the flag, where you have to exploit a known vulnerability that's been given to the model to hack a piece of software and find a flag. One way to hack it could be if the agents generated the flag themselves and presented it without having actually completed the task. And some of the agents figured that out, right? So you want to make your RL environments safe with respect to reward hacking.

So with safety and alignment there are all these other things, and when you are deploying in an enterprise you might have a whole list of other guardrails that you want these systems to obey. Maybe you have, let's say for example, you have an agent that's creating a board deck, and to do that maybe the agent has to pull information from different systems which include NetSuite, Salesforce, maybe some sensitive company dashboards, and maybe the agent kind of wants to verify whether some of the information that it has is correct.

You might not want the agent to go check that with somebody who's not cleared to see that information. Maybe it's okay to go to the CFO and check, hey, I pulled out this particular forecast for next quarter's revenue. Is this correct? Right? So you might want it to respect certain access privileges and so on. So I think for frontier safety and frontier alignment, I'm sure the labs do a lot more than what I just shared. For enterprises, the job is a little more practical in terms of are these systems useful and do they follow the explicit guidelines for the task? Are they not hallucinating, making things up? I mean, there's a certain level of risk in a board deck if you stick in wrong numbers, or in an earnings call you share some stuff that's kind of incorrect. I do feel like our current recipe for training these models and validating them works by and large for enterprise deployment. I think for CBRN it's definitely something we have to do carefully and slowly.

So what's fascinating about enterprises today is enterprises are now starting to do what the frontier labs were doing over the last few years. So the way Turing works with almost all the foundation model builders is helping the frontier models advance on well-defined benchmarks and evals. There is a certain capability, whether it's coding or enterprise knowledge work or frontier STEM. You define that capability, you generate really high-quality RL environments, data sets to help the models advance. The labs train the models, we evaluate again, and we keep running that loop, right? It's a loop that keeps continuing. Now enterprises, I think of enterprises as having two types of workflows. I'd call it core workflows and non-core workflows. So if you're an asset management firm, you might have core workflows on how you measure risk, how do you do asset allocation in a way that maximizes fund performance. So those are core workflows, and you might have other non-core workflows in say HR, finance, legal, etc., where you have to do the right thing but it's not how you differentiate in the market, right? For non-core workflows, oftentimes it's probably okay to rent AGI, to rent superintelligence. But for your core workflows, you want to

Make sure that you own the learning loop that your organization has. So it makes sense increasingly for enterprises to create custom evals. So if you're Goldman Sachs or JP Morgan or Morgan Stanley, for whatever is core to your business, step one, you want to define custom evals. Step two, you would deploy a system to hill climb against those specific evals, and you might have different considerations like accuracy, cost, latency, etc. And once you've deployed a system — and I call it a system and not a model because oftentimes, let's say you're automating a workflow in private wealth management — say for every step in the workflow you might pick a different model depending on which model is doing well at that task. Maybe for one step you use Fable 5, another step you use GPT-5.6, another step you use Kimi K3, and you optimize that system right, like together with a harness around it. And enterprises are also recording traces where they see how these models and agents are performing. When are humans error correcting those models? And those traces that you collect are invaluable because over time you can use those traces to train custom models and see how you did against those evals. Record traces even more to see where humans are error correcting the model, fine-tune models, hill climb. So it's a continuous loop of define custom eval, record traces, hill climb, and keep running that loop. And I actually think it's a really neat symbiotic relationship between AI and humans. It's like humans are benefiting from AI's ability to operate at superhuman speed and scale and process superhuman amounts of information. But when the AI makes a mistake, there is a human that's error correcting it. And when the human error corrects the AI, you're recording that, and that's the — from a marginal information gain standpoint, that's the best type of data to collect to fine-tune the next iteration of the agent. So humans over time are upleveling the kind of problems you can solve. And so enterprises are getting into this, and a big part of what's driving this is open-weight models getting very good today. Depending on who you talk to, like open-weight models are maybe like three to six months behind the frontier. Kimi K3, DeepSeek, Qwen — these models are — and companies like Thinking Machines, Reflection AI are also doing great work. So as these open-weight models advance, they are democratizing AI for enterprises to own their sovereign implementations. And today there's sometimes on X, like things are always quite polarized.

Yes. There's this big tension between frontier AI and sovereign AI, right? And I think we need both. I have so much respect for OpenAI, Anthropic, DeepMind, Meta, xAI, and all the frontier labs that are pushing superintelligence forward. Frontier AI is how we transcend, right? These super intelligent models will help us cure diseases, discover new materials, colonize space. They are needed, right? And out of all the things you could work on, I personally feel like one of the most rewarding things you could work on is building superintelligence, right? So we absolutely need these frontier models. We also need open-weight models because there are plenty of problems in the enterprise where you don't need a trillion-parameter model. Let's say you're doing an invoice-to-pay reconciliation system, or you're automating a key workflow in HR. Maybe you're automating creating a board deck, or you're automating running an all-hands at a company, right? For those use cases, it's probably better to have a custom model that's fine-tuned on your proprietary data that can automate your proprietary tool calls in your workflows. Maybe you have some custom software that you use. The models have to learn that. Maybe you have your own taste in how you run all-hands. You might want to bake that into the models, but that's what makes you you, right, in an enterprise. And you don't want to — you want to be in control of that. So open-weight models give enterprises an opportunity to retain their identity relative to their competition. So we absolutely need that. And these open-weight models also teach us so much. Like today we know the recipe to build reasoning models: build a giant — I mean, you know — build a generative model that's trained on the world's knowledge and then do large-scale RL on it with algorithms like GRPO. And that's thanks to the open-source, open-weight community, right? We need that as well so that we have lots of smart people thinking about how to make these systems safe and secure. And we have cost advantages in the enterprise. One way I think about it — you know that I like cars — so one way I think about it is there's absolutely a place in the world for Ferraris and Koenigseggs, right? Like these...