Good morning, we have an exciting one for you today. We started this 12-day event 12 days ago with the launch of o1, our first reasoning model. It's been amazing to see what people are doing with that and very gratifying to hear how much people like it. We view this as sort of the beginning of the next phase of AI where you can use these models to do increasingly complex tasks that require a lot of reasoning. And so for the last day of this event, we thought it would be fun to go from one frontier model to our next frontier model. Today we're going to talk about that next frontier model, which you would think logically maybe should be called o2, but out of respect to our friends at Telica and in the grand tradition of OpenAI being really truly bad at names, it's going to be called o3. Actually, we're going to launch, not launch, we're going to announce two models today: o3 and o3 mini. o3 is a very, very smart model. o3 mini is an incredibly smart model but still a really good performance and cost. So to get the bad news out of the way first, we're not going to publicly launch these today. The good news is we're going to make them available for public safety testing starting today. You can apply and we'll talk about that later. We've taken safety testing seriously as our models get more and more capable, and at this new level of capability, we want to try adding a new part of our safety testing procedure, which is to allow public access for researchers that want to help us test. We'll talk more at the end about when we expect to make these models generally available, but we're so excited to show you what they can do, to talk about their performance. Got a little surprise, we'll show you some demos. And without further ado, I'll hand it over to Mark to talk about it.
Cool, thank you so much, Sam. So my name is Mark, I lead research at OpenAI, and I want to talk a little bit about o3's capabilities. Now o3 is a really strong model at very hard technical benchmarks, and I want to start with coding benchmarks. If you can bring those up. So on software style benchmarks, we have SWE-bench Verified, which is a benchmark consisting of real-world software tasks. We're seeing that o3 performs at about 71.7% accuracy, which is over 20% better than our o1 models. Now this really signifies that we're really climbing the frontier of utility as well. On competition code, we see that o1 achieves an ELO on this contest coding site called Codeforces about 1891. At our most aggressive high test-time compute settings, we're able to achieve almost like a 2727 ELO here.
So Mark was a competitive programmer, actually still coaches competitive programming. Very, very good. What is your, I think my best at a comparable site was about 2500.
That's tough. Well, I will say, you know, our chief scientist, this is also better than our chief scientist Ilya's score. I think there's one guy at OpenAI who's still like a 3,000 something.
Yeah, a few more months to enjoy there.
Great. That's, I mean, this model is incredible at programming. Yeah, and not just programming, but also mathematics. So we see that on competition math benchmarks, just like competitive programming, we achieve very, very strong scores. So o3 gets about 96.7% accuracy versus an o1 performance of 83.3% on the AIME.
What's your best AIME score?
I did get a perfect score once, so I'm safe. But yeah, really what this signifies is that o3 often just misses one question whenever we tested on this very hard feeder exam for the USA Mathematical Olympiad. There's another very tough benchmark which is called GPQA Diamond, and this measures the model's performance on PhD-level science questions. Here we get another state-of-the-art number: 87.7%, which is about 10% better than our o1 performance which was at 78%. Just to put this in perspective, if you take an expert PhD, they typically get about 70% in kind of their field of strength here. So one thing that you might notice from some of these benchmarks is that we're reaching saturation for a lot of them or nearing saturation. So the last year has really highlighted the need for really harder benchmarks to accurately assess where our frontier models lie. And I think a couple have emerged as fairly promising over the last months. One in particular I want to call out is Epoch AI's Frontier Math benchmark. Now you can see the scores look a lot lower than they did for the previous benchmarks we showed, and this is because this is considered today the toughest mathematical benchmark out there. This is a data set that consists of novel, unpublished, and also very hard to extremely hard problems. Even turns out, you know, it would take professional mathematicians hours or even days to solve one of these problems. And today all offerings out there have less than 2% accuracy on this benchmark. And we're seeing with o3 in aggressive test-time settings, we're able to get over 25%.
In addition to Epoch AI's Frontier Math benchmark, we have one more surprise for you guys. So I want to talk about the ARC benchmark at this point, but I would love to invite one of our friends, Greg, who is the president of the ARC Prize Foundation, on to talk about this benchmark.
Wonderful. Sam and Mark, thank you very much for having us today. Of course. Hello everybody, my name is Greg Kamradt and I'm the president of the ARC Prize Foundation. Now ARC Prize is a nonprofit with the mission of being a North Star towards AGI through and during benchmarks. So our first benchmark, ARC-AGI, was developed in 2019 by François Chollet in his paper on the measure of intelligence. However, it has been unbeaten for 5 years now. In AI world, that feels like centuries. So the system that beats ARC-AGI is going to be an important milestone towards general intelligence. But I'm excited to say today that we have a new state-of-the-art score to announce. Before I get into that though, I want to talk about what ARC-AGI is. So I would love to show you an example here. ARC-AGI is all about having input examples and output examples. Well, they're good, they're good, okay. Input examples and output examples. Now the goal is you want to understand the rule of the transformation and guess it on the output. So Sam, what do you think is happening in here?
Probably putting a dark blue square in the empty space.
Yes, that is exactly it. Now that is really, it's easy for humans to intuit and guess what that is. It's actually surprisingly hard for AI to understand what's going on. So I want to show one more hard example here. Now Mark, I'm going to put you on the spot. What do you think is going on in this task?
Okay, so you take each of these yellow squares, you count the number of colored kind of squares there, and you create a border of that with that.
That is exactly, and that's much quicker than most people. So congratulations on that. What's interesting though is AI has not been able to get this problem thus far, and even though that we verified that a panel of humans could actually do it. Now the unique part about ARC-AGI is every task requires distinct skills. And what I mean by that is there won't be another task that you need to fill in the corners with blue squares. But we do that on purpose, and the reason why we do that is because we want to test the model's ability to learn new skills on the fly. We don't just want it to repeat what it's already memorized. That's the whole point here. Now ARC-AGI version 1 took 5 years to go from 0% to 5% with leading frontier models. However, today I'm very excited to say that o3 has scored a new state-of-the-art score that we have verified on low compute. For o3, it has scored 75.7% on ARC-AGI's semi-private holdout set. Now this is extremely impressive because this is within the compute requirement that we have for our public leaderboard, and this is the new number one entry on ARC-AGI Pub. So congratulations to that.
Yeah. Now as a capabilities demonstration, when we ask o3 to think longer and we actually ramp up to high compute, o3 was able to score 87.5% on the same hidden holdout set. This is especially important because human performance is comparable at 85% threshold. So being above this is a major milestone, and we have never tested a system that has done this or any model that has done this beforehand. So this is new territory in the ARC-AGI world. Congratulations with that.
Congratulations for making such a great benchmark. Yeah, when I look at these scores, I realize I need to switch my worldview a little bit. I need to fix my AI intuitions about what AI can actually do and what it's capable of, especially in this o3 world. But the work also is not over yet, and these are still the early days of AI. So we need more enduring benchmarks like ARC-AGI to help measure and guide progress. And I am excited to accelerate that progress, and I'm excited to partner with OpenAI next year to develop our next frontier benchmark.
Amazing. You know, it's also a benchmark that we've been targeting and been on our mind for a very long time. So excited to work with you in the future. Worth mentioning that we didn't, we target and we think it's an awesome benchmark, we didn't go do specific... you the general, but yeah, really appreciate the partnership. This was a fun one to do.
Absolutely. And even though this has done so well, ARC Prize will continue in 2025, and anybody can find out more at arcprize.org.
Great, thank you so much. Absolutely. Okay, so next up we're going to talk about o3 mini. o3 mini is a thing that we're really, really excited about, and Hongyu, who trained the model, will come out and join us. Hey.
Hey. Hi everyone. I'm Hongyu, I'm an OpenAI researcher working on reasoning. So this September we released o1 mini, which is an efficient reasoning model in the o1 family that's really capable of math and coding, probably among the best in the world given the low cost. So now together with o3, I'm very happy to tell you more about o3 mini, which is a brand new model in the o3 family that truly defines a new cost-efficient reasoning frontier.
Yeah, though it's not available to our users today, we are opening access to the model to our safety and security researchers to test the model out. With the release of adaptive thinking time in the API a couple days ago, o3 mini will support three different options: low, medium, and high reasoning effort. So the users can freely adjust the thinking time based on their different use cases. So for example, for some we may want the model to think longer for more complicated problems and think shorter with simpler ones. With that, I'm happy to show the first set of evals of o3 mini. So on the left-hand side we show the coding evals, so it's like Codeforces ELO which measures how good a programmer is, and higher is better. So as we can see on the plot, with more thinking time, o3 mini is able to have increasing ELO, all performing o1 mini, and with median thinking time is able to measure even better than o1.
Yeah, so it's like for an order of magnitude more speed and cost, we can deliver the same code performance on this, or even better insurance, right? So although it's like the ultra min high is still like a couple hundred points away from Mark, it's not far.
That's better than me, probably.
But just an incredible sort of cost-to-performance gain over being able to offer with o1, and we think people will really love this.
