Why AI Agents Deceive Users on Real Tasks | Anastasios Angelopoulos

When you put AI into the hands of humans, and just when you do A/B tests to try to see what's happening, you know, how it affects their actual workflows, you'll see that agents are lying to people. Agents are basically telling them what they want to hear. They're deleting their files in an unauthorized way. They're bypassing permissions, and it's happening basically perniciously.

Thank you so much. Happy to be here, and thanks for believing in us from the early days. We've been sponsors ever since.

Oh, that's great. Oh, there's a new studio coming. Good for you. Yeah, I can't wait to come by.

Listen, there's two things that are coming for the business. And the first one is actually a new release that we made today. You know, Arena, we started as a human preference platform where people were voting. You know that you got one prompt, two responses, and you could vote for models side by side, and then we took those votes, we compiled them into leaderboards. But these days we've moved very, very far beyond that. So the platform has expanded its mandate of evaluation. First we started by evaluating agentic capabilities in single-threaded workflows. People doing their most complex multi-turn tasks on Arena. Basically we give the agent a computer, and then users can go, you know, edit their documents. They can open PRs on GitHub. And we have tens of millions of users doing that on Arena. As of today, we've expanded the mandate of this evaluation platform even further into the arena of alignment. And so what alignment means is making sure that the values of AI align with values that are good for humanity. And we've released three signals on the leaderboard that have to do with this. One is about the models taking unauthorized actions within the sandbox of Arena. The second is about deceptive completions, models telling you that they've done something when they didn't. And the third would be false attribution, models attributing intent to humans that they didn't actually have. And from these you can see that there's extraordinary issues with alignment in the real world. When you put AI into the hands of humans, and just when you do A/B tests to try to see what's happening, you know, how it affects their actual workflows, you'll see that agents are lying to people. Agents are basically telling them what they want to hear. They're deleting their files in an unauthorized way. They're bypassing permissions, and it's happening basically perniciously. All of us have experienced this, even if we might not know it.

Yeah. So, an example of this would be, for example, models that will — you'll tell them to do one thing, and they'll go delete all of your other files that are unrelated. So, they will take destructive actions that are against the permissions that you gave the model. That would be an example of an unauthorized action. Another would be deceptive completion: models telling you, 'Yes, I did check every cell in this spreadsheet to make sure all the equations were correct before you sent it to the IRS.' Okay, big, like, high-stakes decisions where the model might actually be telling you what it knows that you want to hear because that's how the models have been rewarded historically, but it's not actually the case that they've done it. So this is a big — it's an important leaderboard to counterbalance the preference of humans. We also need to be looking at whether the models are safe and aligned as a countermeasure.

Listen, buddy. We heard your feedback. We took your feedback. We actually took feedback, and now we're here.

Fast turnaround.

Yeah, so actually let me share my screen here so I can lead you guys through it. If you look here, I'm pulling up on my screen the blog that we have on the alignment index. Actually, if you visit arena.ai/leaderboard, you'll be able to see our alignment index pulled up there. But right now I'm going to show you the blog that our research team wrote. And you can see that here is the leaderboard. Number one on the leaderboard is OpenAI's GPT-6.1 Soul. And you can see it's significantly better, like confidence intervals non-overlapping with the remaining labs: Anthropic, SpaceXAI, Google, and then the Chinese open-source labs are below that. It's an interesting leaderboard, and it's, you know, here you can see the three signals: unauthorized action, false attribution, deceptive completion, and lower numbers here are better. So you can see that GPT-6.1 Soul is better on all three of these metrics from the next best model, which is Anthropic. Now, remember, the important thing about these numbers is that they're not a standard benchmark. Models actually know when they're being evaluated for alignment. So it's very difficult to evaluate alignment in silicon. You need to evaluate it on real workflows because that's where the misaligned actions might actually cause harm to people. And that's what this is doing. For each of these unauthorized actions that the models perform, that means a real user on Arena had a model take — basically bypass permissions or take actions that the user didn't want them to take. For example, deleting files. For every deceptive completion, a user was actually deceived by a model on the Arena. So it's really a direct measure of the alignment of the models in reality.

Well, there's different reasons. The first thing to notice is actually if you just take a look at this leaderboard, and we have some statistics on this as well, the more capable the model, the more aligned it also tends to be. Okay? So it's more powerful, which means that it can be more dangerous, but it's also more aligned and less likely to be problematic. And I think that this is because the labs have taken strong measures as the model generations have improved to continue improving the alignment as well, perhaps due to the safety concerns that they have. So the good news is that better models, more capable models, also tend to be more aligned. Now, the downside of it is that a lot of times when models are optimizing for signals like raw human preferences, you can get into these issues where, you know, why are models trying to deceive us? Why are they trying to do deceptive completions, telling you they did the job when they didn't? Well, it's because they're telling you what you want to hear. And why are they telling you what you want to hear? It's because that's how they were trained. They were trained to listen to human stated preferences. And what that means is that if a human says, you know, 'good job,' or, you know, if they're saying, 'Hey, I really like this response that you gave me,' they might not have checked whether or not every step in the process was done correctly by the model. And so you need signals like deceptive completion in order to serve as a complementary signal to preferences to ensure that the models are telling users not just what they want to hear but also the truth.

Actually, let me push back on that for a second. You know, a lot of people think of Arena as being a human preference platform, but actually I would say that's a misconception these days. If you look at the Arena leaderboard and you look at how we evaluate agentic capabilities, we are evaluating capabilities like task completion rates, user praise versus user complaint, which is one example of human preference, model steerability, whether or not the models are actually landing the corrections of users in the context of, you know, work tasks, best recovery, whether or not they're correctly able to use computers, hallucination rates of tools, and now alignment. So I would say that Arena has gone far beyond human preference, and actually the mandate of the platform has expanded greatly. Alignment is just one aspect of this. The goal really of our platform and the reason why we want to move into, and why we have moved into, all these other signals is to provide evaluations that will incentivize AIs to benefit people. And benefiting people sometimes means you need to listen to what people prefer or don't prefer through things like our voting mechanism. And sometimes it also means you need to place guardrails on models that they should be getting the job done, not lying to the user about getting the job done, staying within the guardrails of the user, you know, understanding things correctly and not misattributing things to the user, and acting in generally safe and aligned ways.