More likely to be messy. In some sense, they are — they involve working with other people. They involve working in much larger code bases. It is all sort of more open-ended problems, maybe with something even adversarial going on in the software engineering context — that might be that someone's trying to make a change to the parts of the code base that you're currently working on, and you need to work around that. And we do tend to see that the AIs are less capable of working on these more messy problems. I don't want to overstate that. You know, it's not an enormous effect. But, you know, that's one thing that gets in the way of these productivity increases. You know, and then I do think that there's something to the reliability question, right? Where, you know, if it was true that for a certain type of task you only had, you know, 80% reliability, then every time you're going to need to go back and verify the work of these areas, and not only verify the work of these areas, but without the context of how they implemented the solution relative to it. If you went about the task yourself, you'd already have that in your head. And so this verification step, quote unquote, would take less time. You know, I don't expect these frictions to be sort of so fundamental in some sense, or I imagine they go up levels of abstraction. I think not only is the underlying technical progress real, but I think the productivity improvements are also going to show up increasingly. But yeah, there are these versions.
Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal. And I'm Tracy Alloway. Tracy, one thing about AI is that lots of lines that go up.
Yes. Famously, famously, there is perhaps one line that has captured the attention more than others when it comes to lines going up.
Yeah, but yes, but we're recording this, April 7th. Did you see the Anthropic revenue chart, by the way?
Oh, it's just extreme. Yeah. Okay.
On the number of lines going up. I mean, there are some really... Let me caveat that up until recently, there was one chart of a line going up exponentially that became, I think it's fair to say, the most viral chart in AI, right?
Yes, I would absolutely agree with that.
So one of the many lines that go up, and there are various lines that sort of capture this, is essentially just measures of AI progress or what they could do, what the models are capable of, and so forth. And, you know, there's all different benchmarks out there and hobbyist benchmark creators, etc. All kinds of benchmarks out there. An organization called METR, based out in San Francisco, and they measure how well AI models are doing at various sort of engineering tasks, etc. And they have these charts showing how long, you know, certain tasks — how long it would take a human to do them, and then whether AI could do them. And yes, the lines just almost, almost vertical. I think there was someone like one of the ones that came out maybe very early this year or late last year, showing the latest Claude model. And yes, this is crazy. So that one was really interesting because when I look at these charts, they're called Time Horizon charts. Yeah. When I look at them, I like intuitively, I kind of understand what they're saying. And you can kind of see the leap in progress between some of the previous models and Claude. Right. The latest Claude model. Yeah. And that's what got everyone excited, was you had this big exponential shift up in the capability of that particular AI model. But then when I start like diving into what it actually says on their website about what these charts represent, I start getting really confused.
I'm kind of the same. And like, I know everyone wants to get excited about AI and charts going up in general, but I think there's a lot of nuance here, and we should probably talk about it. Because the other thing going on with METR right now is they've become sort of the industry standard benchmark. And so a lot of investment decisions are being based on these charts. And if you oversimplify them as just like, okay, lines going up. Yeah. And then suddenly it goes up even more, obviously people are going to start to get like maybe a little overexcited.
Can I say one other thing too, that I'm very curious about? Like, I'm really glad that there are people designing various benchmarks for measuring AI progress. Seems like an important thing to get a handle on. But like, if I were like, say, like talented or smart enough to be like doing these things, I would go work for one of the labs and make $10 million a year or something like that. And so I'm actually curious to like nonprofits, etc. It's like, do you really like want to be like working at the cutting edge of AI in a nonprofit? I mean, I guess OpenAI is owned by a nonprofit, weirdly enough. But you know what I'm saying? Like, I would want the money. We should talk about it with our guests who are currently sitting right here. That's exactly right. I'm very excited to say we have the two perfect guests to talk about the most viral and maybe important chart in AI right now. We're going to be speaking with Joel Becker. He is a member of the technical staff at METR. Also going to be speaking with Chris Painter, the president of METR. So, Joel and Chris, thank you so much for coming on Odd Lots.
Yeah, really excited to chat with both of you. Why don't we just start? Chris, since you're the president, I'll start with you. Like, what is METR, or how long has it been around? What is this organization? What's it all — just give us the sort of 60-second synopsis of METR.
Yeah, totally. I can try and, you know, sometimes I give it a long version. I can try and do a short version here. So METR is a research nonprofit based in the Bay Area, like you said, dedicated to advancing the science of measuring whether and when AI systems might pose catastrophic risks to humanity as a whole. Focus specifically on threats that come from AI autonomy or AI systems themselves. So when you talk about this kind of this whole field in AI of dangerous capability evaluations, people seeing, can this AI system assist with a chemical or biological weapon attack? Can it advance kind of like bad actors' ability to execute cyber attacks on a really large scale? METR is sort of specialized in specifically assessing how autonomous are AI systems. What is the scale and like length and difficulty of tasks that they're able to do by themselves, partially because we think it sets the stakes for conversations about AI misalignment. So we sort of see ourselves as being on the hook for, at any given point in time, giving humanity the bits of evidence that are most informative for establishing the stakes of: are we reliant on AI systems as a society in a way that could make it really bad if they are misaligned?
I'm going to let Joe ask the question about why you're both working in a nonprofit instead of one of the labs later, but one question I do have is when I think of METR, you guys always come up in the context of these time horizon charts, and I don't mean this as an insult or anything, but I hardly ever hear anyone talk about the actual like safety aspect of your mission. Why do you think that is?
Yeah. So I think there's some distinction between our motive for assessing time horizons and the kind of how it gets used then by the rest of the world, or kind of like what the origin of the rest of the world's interest in it for METR. I think the reason that we work on things like the time horizon charts is because if we're trying to establish the stakes for talking about, you know, could AI systems go rogue, or one day could they like try to take over and subvert human control? The three years ago, if you went back to around when METR started — about four years ago — and it was started by Beth Barnes, Paul Christiano. And this was kind of the initial motive. Is if you went back then and you said, why don't I think that AI systems are going to go rogue and like take over or overthrow humanity today? The kind of most intuitive — you know, you can come up with a lot of abstract reasons, debates about the goals AI systems might or might not eventually have, but the kind of most damning in the moment reason is the AI system just can't do much, right? It doesn't make sense to talk about a question-answering system that like can't even reliably answer programming questions saying like, is it going to hack my systems or like backdoor me in some way? It just doesn't make any sense to talk. No. It's going to write you a poem that you asked for, right? Or won't. Even at the time, they couldn't do anything with themselves. And so if you're like kind of being able to subvert human control depends on agency. And so we wanted to come up with a measure that kind of tracks agency over time to kind of say, when would this argument no longer apply? When are AI systems now able to kind of do long, complex enough actions by themselves that the argument kind of the goalposts almost move somewhere else to, like, well, we would catch the AIs, or the — I don't want to subvert human control. And so I agree that there is a distinction between like how I think partially the exercise of trying to come up with these measures throws off things that are very like grounded and intuitive measures of AI progress that might be more intuitive than just benchmarks, right? So if you — a lot of people are in the game of making just benchmarks where you say, like, here's my home bench or something, that I get 70%, that's much less of a kind of grounded or long-lasting metric. Like, it's hard to say what that means or how that generalizes. But the idea with time horizon is like, maybe it's more intuitive. And I think that helps both for safety and for like business understanding.
So let's talk about what this chart is. I got the main chart here of METR right on the front page. It's, you know, it's this time horizon chart, and it shows Opus 4.6 as of February 2026 able to complete a task length in 11 hours and 59 minutes with a 50% success rate. I have to admit, the first time I saw this chart or versions of this chart, what I assumed, and I suspect others assume, is that it was able to go off and work on a task for 11 hours and 59 minutes and then come back with an answer. But apparently it's not that. What do you — walk us through, like what's really being measured here? By the way, the previous high was always GPT 5.3 Codex. That was five hours and 50 minutes. So I guess part of the reason this chart blew people's minds, because literally that's basically a double. But why don't you talk to us about what's really being measured here?
Yeah. So fundamentally, you know, in simpler terms, we are plotting the difficulty of tasks that the AIs are able to complete over time. And, you know, the particular way that we measure the difficulty of tasks is in how long it takes humans to complete those same tasks that we're asking the AI to do. So in this case, you know, we're talking about Opus 4.6, something like tasks that take humans 12 hours to do. We predict that it will succeed at those tasks around 50% of the time. And yeah, you know, it turns out that when you plot using this particular difficulty measure, how performances are relative to how long it takes humans to complete these tasks, we see an exponential increase in capabilities for AIs. And you know what? That's what that ends up meaning, is that you keep on having these doublings of capabilities every, let's say, four months, it seems, on recent trends, where, you know, the next model is not mainly going to have necessarily, you know, an hour longer time horizon, but perhaps be having some multiple of the time horizon of the previous model that does come out.
So then explain how that number, that 12 hours, is established. So there is some engineering task and you say, okay, this is a task that would require 12 hours. But humans have all different types of talent capabilities. How do you establish that? Okay, this was a 12-hour task. This was a six-hour task. This whatever it is.
Yeah. So the simple answer is literally we get humans to sit down and complete the tasks that we give to AIs under as close to identical conditions as possible. So first we come up with the tasks, and that's, you know, that's a whole other kettle of fish. We can talk about exactly how we do that. And then using essentially the same tools that we're about to give the AIs, we take talented humans, you know, not people who have seen this particular type of task before, but people who have relevant expertise. So if it's a software engineering task, you know, they have software engineering expertise; machine learning task, they have machine learning expertise. And then we time them. We see how long it takes for them to complete those tasks successfully. And then roughly, we call the difficulty of the task as measured in human time to complete as the average time it took these humans to complete the task. Then we'll run the AIs on this same set of tasks. You know, typically today for the very easiest tasks that they're more or less always going to succeed, there's some mid-range of tasks where, you know, perhaps they succeed 50% of the time, or perhaps for some tasks in that range they succeed 0% of the time and for others 100% of the time. And so they're getting 50% on average, let's say. And then for the much harder tasks, perhaps they're getting closer to 0%. And then the point at which we predict, you know, in the middle of all of these zero percents and 100% spikes by task, the point at which we predict that they'd have a 50% chance of succeeding — that is either a 50% chance at succeeding on some task or 50% of the tasks of that difficulty that we think they would succeed on — that's what we're going to call the time horizon of these models.
I think one thing also that could be good to explain here is the task distribution. I mean, this is not all activities that humans do. We are specifically interested in — or there's some question in what tasks are, you know, like Joe mentioned, we're having people come into our office do the tasks to get a sense of how long it takes. We're not having them come in and like, you know, paint paintings or write novels. You know, we're focused here specifically on things that are in the distribution of work that an engineer at a — we like to think of it as like a frontier AI lab — the tasks that they might be doing. So this is things like software engineering, it's fine-tuning AI models. I don't know if you want to give other examples, but it is like software, machine learning, that kind of task.
Wait, can I just ask, why did you decide to focus on engineering? Because you could have widened it out to, you know, if we're talking about AI being capable of, you know, taking over the world, there are all sorts of substantive tasks that would fall under that category. So why just do engineering?
So for one thing, I think maybe other people on the team, or maybe Joel has thoughts about this, but might have other answers. I think my particular motive in being interested in the time horizon on software tasks is that it is kind of, first of all, it's the thing that the industry is very like already, even before we started working on this, is very focused on. So it's one of the capabilities that you should expect to come along for the ride earliest. It's the thing that like a lot of optimization pressure is being exerted on. And then I think that it is kind of the like thing that you would expect as an early warning kind of sign of this AI R&D automation. So to some extent, METR thinks of itself as trying to build, you know, science that can advance science that can say, when are we getting to the point that AI systems could improve themselves or speed up the pace of AI development? When will AI research kind of feed on itself? And the kind of core capability for that might be software engineering and machine learning research ability. There are other skills that could be relevant to taking over the world.
I think other people have done time horizons on like cybersecurity since.
Yeah, yeah. But I suppose it is true. Like the Basilisk isn't going to paint its way into like power or something like that.
Okay. The other — it might deceive you. It might be like, you know, you could — it might be very convincing or cunning in some way and fair hand over the keys and might. I might say, if your mental models, you know, we don't have perfect evidence of this whatsoever, but my rough sense, sort of colloquially, or, you know, my probably before evidence comes in, is that if we did study tasks on these very different distributions, you know, not machine learning, not software engineering — I'm not sure about painting exactly, but, you know, perhaps or other kinds of task distributions that we could — and you might — that basically we would see this similarly shaped exponential progress over time where every — I'm not sure exactly, but let's say, you know, four months, six months, something like that — the level of capabilities as measured in time horizon would be doubling at something like that pace, maybe from a much lower level. So, you know, one example that we do have better evidence of is that the AIs today are much less performant at, you know, anything that requires vision capabilities, seeing what's on a screen, clicking around at a computer. But they're getting, you know, tremendously better at that sort of thing over time.
I just want to mention quickly, we did actually do a very kind of brief investigation of this on other task distributions. That's on our website somewhere, like cross-domain time horizons. I think we looked at data from the Teslas shared on self-driving. I'm forgetting the other one. There's like OSWorld. Maybe some of these are like somewhat similar, still kind of in the distribution of software tasks, but trying to get further afield into things like vision.
Yeah. So the other question I had is how big is the sample size on the humans who are actually doing work? And also, is it getting harder getting like human engineers into the room to compete with like Opus 4.6 versus say, you know, if I was a mediocre engineer — and I'm not, I'm a non-existent engineer — but if I was a mediocre one, I would like maybe I would feel good about going up against like GPT-3 or something, and maybe I would feel a lot worse about myself going up against like Claude.
Yeah. You know, on these tasks, I mean, I'm in a pretty similar position myself to you. So we have approximately three, although it varies quite a lot across tasks, human baselines per task. So, you know, typically we're averaging over something like three. I think the final numbers — it's my impression that they're not going to be so sensitive to the particular baselines that we choose. Aren't the longer tasks we more weakly baseline? I'm thinking of Nathan Witkin. He's in my year. Yeah. So indeed I think it will get a lot harder to baseline these tasks as the length of task the AIs are able to successfully complete gets longer and longer. You know, you might think at some point that the length of tasks that they can complete is longer than the doubling time. You know, in four months' time, they're going to be able to complete tasks of more than four months. And then it's, you know, kind of becomes perhaps close to impossible to get these four-month-long baselines. Of course, we're not at that point yet, but, you know, definitely it has become more difficult to get these baselines as time has gone on. At the moment, not impossible, but very challenging.
Joe, these are the future jobs for displaced engineers. Right. It's competing against the codes for benchmarks, for benchmark evaluation. We found the jobs.
Oh, here's an all right. Here's a question. So we mentioned at the beginning the most viral chart in AI is this chart that you have on the front of your website. Website defaults to this. And it shows, you know, this doubling. So if we actually go back to November and say November 2025, Gemini 3 Pro three hours and 44 minutes, Claude Opus 4.6 12 hours. This is the 50% success benchmark. If we go to the 80% benchmark, which the website doesn't default to, the pace of improvement looks a little less impressive to me. So okay, now it's like it just doesn't — it does not have the same gap. Pretty clearly now 80% is still not 100%. And I know that this is your — METR's goal is about like, you know, human safety and all this stuff. But when we think about people look at this and they use it as a stand-in for how performant are these models, even 80%, you know, certainly for like any business application — I understand you're not like serving business here per se, but probably businesses care about this. Even 80% may not be very good enough, and it does not look as crazy when you look at the 80% chart as it does at the 50% chart. Why the focus on the 50% chart? And given like, why not? Like why not look at the chart that just does not look as impressive?
Yeah, maybe two central things to say. You know, one, to my eye, is the 80% chart basically does look as impressive. For the doubling time is about to say this is cope on my part. Usually it's the same. It's the same. It's the same as an offset of. It's the same pace of progress. You know, something like five times. Five times smaller than the 50%, okay. Than the 50% number. But you know, that only takes you two doublings. And if each doubling takes around four months, that means that in eight months' time, you're going to have the same 80% success rate roughly as you do 50% success rate today. That's one thing to say. Maybe a second thing to say is, you know, remember at the beginning I said, essentially what we're doing is plotting the difficulty of tasks that these AIs can complete over time, just with this particular measure that ends up showing this clean exponential trend. And we've picked a particular number as our difficulty number. And, you know, that is this 50% reliability threshold. We could have picked a different one. I think there are reasons for picking the 50%. One in particular, it's the one that's statistically we're better able to measure for some technical reasons. It's the one that shows up in previous literature. There's a couple of other reasons why we can go for 50% rather than 80%. Maybe a final thing to say is that this 50% number is sort of equivocating between these tasks it's able to complete 50% of the time and 50% of the tasks it's able to complete 100% of the time and 50% it's able to complete 0% of the time. And actually, I think the situation is it's somewhere in between, but it's a little bit closer to the latter, where there are some tasks that it's completing with near-perfect reliability and some tasks in that range that it's completing with very low reliability. And, you know, for downstream economic applications or for applications inside of these major AI companies or something, you know, you might think that that's more favorable in some sense, that there are some of these tasks where we're getting 100% reliability, even for very challenging tasks.
I think two other things maybe it could be useful to just explain when you said that there are technical reasons why it's easiest to measure at 50%. One, like it is just the case that 50% is the point at which it is like least sensitive to — like the distribution is kind of thickest, right? I mean, correct me if this is wrong, but my — I mean, there are like to resolve something like 95%, you would need way more samples because then you need to have some that are like — yeah, you need way more samples to be able to resolve that level of precision in my.
I think there are some caveats to that picture, but let's say even more extreme, you know, let's say that we cared about, you know, 99%. In that case, if we had 1% label noise, quote unquote. Yeah, it's, you know, if sometimes we were accidentally grading some of the failing tasks as passing, some of the passing tasks as failing, then we'd just never be able to estimate that reliably. Right? Okay. And at 50%, this comes a little bit closer to washing out. And I think one other intuitive thing here, one intuition, is that if you give me a task and you give me the model, all you tell me is the length of task that it takes a human to do the task. The 50% time horizon is the like point at which I think it is more likely that the model will be able to do the task than that it can't. And I just find that intuitive. Yeah.
How much interest do you get on these charts from potential investors specifically? And the reason I ask is because I was just messing around and like googling some stuff. And when the Opus chart, the latest Opus chart came up, someone posted it on Reddit and I think like the second comment on it was someone going, how do I invest in OpenAI? And like people were — they were trying to club together to like invest in these companies. So clearly there are people out there who are using these charts as investment tools.
I would say, you know, we don't get an enormous amount of inbound from investment firms. I mean, sometimes, you know, VCs or whatever we're based in the Bay Area will reach out to us. I think that there's some kind of principle of our goal is to inform the public and give them the best evidence that we can about when we might get to this point of kind of, you know, AI being, you know, fully autonomous or able to improve itself. And there's some principle at play here of like, I kind of want to enable people to do whatever they will do with that information. And I think that, yeah, we don't engage a ton in kind of the like business side or investment implication of the work. I think that if I knew — one kind of thought experiment I sometimes do like say to myself, as if I do believe that at some point we're going to get this AI that's improving itself, and we're like AI research is automated, and you have all these fears about a singularity. Would I rather that like all of Wall Street didn't like falsely didn't think that was coming when I believed it was coming? Or would I want them all to know that it was coming, given that I believe it's coming? Maybe this is more a personal view, but I think if this is possible that we will automate AI research, I think all of humanity being aware of it, being aware of where we're heading, is sort of a precondition for us all being able to figure out what to do about it. And so I don't kind of want like certain people or one side or one team to kind of like selectively be in the dark because they might invest on the basis of this or something like that. But we don't, you know, it's not where we put our time. We're focused on informing the public. The public includes some investors.
