Every time a new trillion-dollar industry emerges, there's a need for this independent testing group. When Meta released Llama 4 on our held-out private benchmarks, the model is actually underperforming. But on all of the major public benchmarks, it was showing incredible capabilities.
What's the limit of what you can achieve? And then within that, how are you going about it?
In an ideal world, you'd take a frontier model and have it train the next version of itself, but obviously that's very expensive and slow. And so what we're doing is forming a set of proxies for every part of the process it takes to build the next version of the models. Evaluations, as they become more complex, have a smaller sample size but a larger set of criteria or expectations of them.
Where do you see the gap that's happening today?
The government kind of has an inclination of what it's afraid of, be it biohacking, cyber hacking. But then there becomes the question of: can the model do it, and then can you get the model to do it?
What do you think the landscape will look like?
So I'll start with a question. When Val got started in 2024, after a team discovered that all the public benchmarks are just not sufficient enough to measure model progress, and there needs to be a new methodology and approach coming to keep us on the frontier and help model labs continue to hill-climb. Maybe just take us back to the inception of Val, and what do you see was missing in the market then?
Yeah. I mean, so I had a background doing research and, in particular, building benchmarks and evaluations. And so what was very clear to me was the very tight relationship between what it takes to build new systems for generation and actually new mechanisms for evaluation. In fact, in order to get one, you often need to get better at the other. And actually, one of the biggest drivers for model capability is having a new legible way to evaluate models. And so around early 2024, what we were seeing is there were actually many interesting models coming to market. They weren't all coming from OpenAI. And then also, in particular, it was harder than ever to actually ascertain what was newly capable with the new models. And so in kind of a first-principles way, what we realized is that there would need to be some third-party company that solely existed to build really high-quality evaluations and benchmarks to be able to discern what was newly possible with these models. And so we released our first benchmarks in 2024, and now over the last couple of years that has kind of been realized by many different parts of the industry.
And why do you think — I guess one of the obvious questions is why do you think the labs can't do this by themselves? Because they know best where the models are hill-climbing on and what is missing capability-wise. Why can't they be the benchmarking stores?
Yeah, I mean, internally they do build a lot of great benchmarks and that's what drives model progress. But I think there's an issue when we speak about model capabilities in a way that's self-reported. And so one of the early indications of that you saw was when Meta released Llama 4. That was a bit of a disaster. And interestingly, what we saw is that on our held-out private benchmarks the model is actually underperforming, but on all of the major public benchmarks where the questions and rubrics are actually open source, it was showing incredible capability. So there's a huge disconnect between what was self-reported based on these open benchmarks and then what we were actually finding with our higher-quality, higher-signal benchmarks. But I think it speaks to a broader concept which the labs, I think, understand: that they would like to see a rational buying market. You know, they would like to see that when they invest billions of dollars to build a new model, there are actually substantive ways they can point to evidence and say we're advancing in these ways. And it's not just entirely self-reported to justify that investment. And so you also see instances of Demis and others in the industry calling for an ecosystem of third-party evaluators.
And what's the historical analog that you have in mind here? There are rating agencies, audit firms. What's the right comparable?
Yeah, I think there's honestly lessons to learn across the board. And every time a new trillion-dollar industry emerges, there's a need for this independent testing group. And I think the fact that it's moved so quickly in AI has caused a necessity for a lot of these parallels to be borne out. You know, we think about ourselves as trying to sit on both sides of the market. So there are mechanisms by which labs need to prove that new models are very capable, but there are also parallels where enterprises need to figure out what adoption strategy is going to amount to the greatest ROI for them.
Yeah, I want to shift to take us through sort of the six-hour pre-release window before the model drops. Obviously you need to run tens of billions of tokens without delaying the launch. Which part is you doing an all-nighter versus it being automated? Take us through that.
It's honestly been a journey. And I think the real goal, the North Star we think about, is we never want to be kind of a lagging indicator or a delay to a model release. And so that means we have to move really, really quickly and extract the most possible signal with the rate limits or capacity that we have. And so early on, what this looked like was my co-founder Lynx and I pulling an all-nighter to try and get as much done as possible and get results out the door. Now we've built up a team, but we've also really invested heavily in infrastructure. And so we're able to run evaluations in a massively distributed way, running effectively the maximum possible rate limits with every model we get access to. And we also have this internal system called Steve — Steve, the economic valves employee. And so that's been a mechanism by which we're able to actually take more of the human work over time and put it into Steve.
And how do you deal with the kind of issue that it's a little bit of an AI-complete problem in that, you know, we still aren't really good at evaluating humans, or we haven't agreed on it? Like there are things like IQ tests, there's EQ, there's, you know, the Big Five personality and so forth, but there's not really an agreed-upon framework for which we do it. And people have issues with things like the SAT and this and that and the third. And then, you know, of course models are really good at hacking the benchmark, as on the floor proved. And so how do you think about, you know, that issue, and what's the limit of what you can achieve, and then, you know, within that, how are you going about it?
Yeah, I think the honest answer is that it's forcing a lot of the more fuzzy or distributed forms of eval to be made explicit — like what is really the distinction between an associate and a partner at a law firm. And there isn't a clear test or an eval for that in the human world, and so we have to first establish a lot of that in these different enterprise or real-world workflows for us to be able to test models in the same way. And I think long-term that will actually be the biggest bottleneck: our ability to take companies and their emails and make them legible. Because that's how we'll figure out what signal we'll climb on and where we actually adopt.
Got it. Very interesting. Actually, maybe one question for you, Ben, on just like how the industry has formed before intelligence came through. Like we're now measuring something that's very fluid versus before, like when we're talking about enterprise software, like, you know, there's Gartner rating on like 70 different metrics, like you can sort of stack-rank on a quadrant to say this company has these features covered, these features not. But now it's like, you know, very jagged frontier that's very hard to measure given industries. Like what do you see — you know, one is the analogy to the past, the lessons we can borrow, and what do you see that's really going to be the challenge and the missing pieces going forward?
Yeah, it's a little bit reminiscent of the MPAA, right? Where it's like, what's art, what's porn, where's the line, when is it R, when is it X? And by the way, the definition of that has changed over time, I think. Things that used to be X are now R and so forth. And, you know, then what's PG? What's PG-13? You know, all that kind of thing. And there's no — and the famous line is, well, I know it when I see it. And I think that this one, I just think it's going to be necessarily fuzzy, but there will develop norms over time. And, you know, like if enough kind of people who run companies or run finance or run whatever it is kind of agree, yeah, no, that's a norm, then I think it is, as opposed to kind of what we have a lot now in the open benchmarks, whereas if you can solve this specific problem then you're at this level and so forth. I think that's, you know, one, it's hackable, and then it's too narrow.
I think when you also look to some historic analogies, there's a lot of lessons that you can take from them as well on what's gone wrong and what we need to avoid. And I think, for instance, for Val, one very early decision we made was the decision to never sell training data to labs. It's often a place that we're pushed when we start working with a new lab to actually source and sell for them a bunch of training data.
Yeah, that's a lucrative business.
Yeah. And actually a lot of that industry has now built these gimmick-style benchmarks as a mechanism to sell their data. And so that's become kind of their go-to-market as well. But, you know, I think if you look at auditing as an industry, you end up with issues like Enron, where if you have the same group who's responsible for doing the audit as well as also consulting and supporting the company, you have a mixed incentive structure, and then it just becomes pay-to-pass-the-audit, or in this case, pay-to-win-the-benchmark. And that's really not what the market benefits from and what we're trying to do.
And Ryan, today you have already a pretty extensive catalog of different types of benchmarks. Some of them are more focused on, you know, specific industries. Some of them are more like consumer mental-health related. Maybe first just talk through like what are the benchmarks that are most popular and most read-upon, and we'd love to dive into one of them as well.
Yeah, well, we've done a lot of work in kind of the economically interesting applications of models. Our finance agent benchmark is used by a bunch of the big financial institutions to get a sense of how models are improving. We also have a lot of good work in coding. So our VCodeBench measures how well models can take a natural-language prompt and build a full-stack web application. And so that's been a keen way to track model improvements over the last nine months. Yeah, we're also doing a lot more experimental work. So one benchmark we released recently I'm very excited by is our Recursive Self-Improvement Index. It's a topic which a lot of big labs have been talking about and starting to report on in their model cards. But there isn't a shared language to talk about the RSI potential of models. And so we created this as an apples-to-apples way to actually benchmark across the models.
Yeah, I thought that one is a very cool benchmark. It's sort of all the rage in the research community of how do you measure like progress you can make through having more frontier models that you can just compound on the capabilities. How do you actually go about like building this RSI benchmark?
Yeah, I think, you know, in an ideal world what you want to do is actually take a frontier model and have it train the next version of itself and see where the delta comes from. But obviously that's very expensive and slow. And so what we're doing is forming a set of proxies for every part of the process it takes to build the next version of the model. So there's some work around pre-training, post-training, harness-level engineering. And then seeing in which mechanisms and behaviors the models are able to do very good research work and build something new and where they're struggling.
Very cool. There's also cases where you have deprecated indexes and benchmarks. It's funny that I always watch this benchmark industry — people like, just like early diffusion model days, like you cherry-pick whichever image shows up the best and the most perfect. Like benchmarks as well, like you pick something that's, you know, very popular but maybe already saturated and you rank very well on that or score very high. But you took a very different approach in like if these benchmarks are saturated, you'll deprecate it. Like maybe talk us through the thinking around that too.
A necessity, and this is kind of the infinite game we're in. I mean, you're wearing our shirt, and so we have this unofficial motto: always a higher peak. So, insofar as foundation model labs are hill-climbing, they're searching for the next peaks to summit. It is our job to perpetually construct these next mountains for them to summit. And I think that's also how the economy has naturally functioned over time. As, you know, agriculture becomes less important for our labor market, there are new forms of labor less required out of our population. And so in the same way, we should expect our benchmarks to keep up with the new frontier for what we want models to do. There's another component of retiring benchmarks which I think is underappreciated, which is that benchmarks should also be reflective of the current state of the world. So in the same way, if you're a lawyer you have to, you know, retake the bar exam and get certified, or if you're an architect you have to get your certification, or, you know, a doctor — we should also expect models to be tested on the current state of the world and what we know in medicine or what we have as our set of laws. And so in the instance of case law updating to something like a legal research benchmark, that's a desire to create a benchmark more reflective of the current state of the world and also push the models in places we want to see them go.
And how has — I guess one, like it used to be like we're doing these like multi-answer or multi-step questions to like just evaluate prompt answers. Now there's like a lot more agentic work that's happening, whether it's on finance or legal or coding especially, like there's a lot of, you know, async background agents that can just complete tasks. How has that changed sort of how you build infrastructure? How to think of evaluating the capabilities of not just the models and agents themselves. And there's also like a lot more dimensions that people care about. It's not just like capability, it's cost, it's latency, it's like, you know, whether this model is flexible enough to address like broader domains and tasks and so on. So how do you think about the additional parameters to what you evaluate?
Oh yeah, there's — I mean, there's a lot that goes into that. I think on the infrastructure level you have a whole new set of problems. I mean, for instance, now we're testing models in their ability to run over hours, days, sometimes weeks, and so the infrastructure needs to be very stable to support evaluation over time. And if there is a failed request, we should be able to retry from that one and not redo the whole trajectory. So there's some simple mechanisms in the infrastructure we have to think about. But I think in general what we've seen is evaluations, as they become more complex, have a smaller sample size but a larger set of criteria or expectations of them. And so what I mean by that is a benchmark is largely some kind of input space of things you're trying to query a model to do and a set of requirements or rubrics that you see in expectations of the output. And so early on you have things like ImageNet which have millions of images you're trying to see a basic categorization for. So it's a one-to-one mapping between an image input and a text label output. Now what we have is a far smaller set of tasks — you know, generate me 50 full-stack web applications — but a much larger complex mechanism for evaluating the output produced. And I think that trend is going to continue as we see more complex workflows evaluated with models.
And do you think that it will become kind of a real-time kind of mechanism? Like so for something like OpenRouter, which Stripe just bought, would OpenRouter look to Val and say, okay, where should this next request go, or is this going to be kind of strictly for like picking a model in an enterprise for a task?
Yeah, I mean, I think, you know, OpenRouter is a bit of a misnomer in that most of their usage comes from being a model gateway, and so it's actually up to their users to decide which models they want to use when. And that's because really the hardest part of routing is building the evals and trying to determine in what places a set of intelligences should be used for a particular application. And so, you know, our effort in supporting enterprises in building evals has actually supported a lot of them in also adopting routers.
And say more about why it's not only important to labs but also existential for enterprise, and maybe just say more about how you guys work with enterprise.
Yeah, of course. I mean, I think that the lab side of this is very clear. Like, you know, if you're raising lots of money, investing heavily in building models, it's essential for you to show why your model is getting better and then why those customers should pay a premium for them. But what I think is still underappreciated is on the enterprise side this is turning to be existential as well. You know, I have a small anecdote related to this actually. I was meeting with a company in the Fortune 10, and the way that they've adopted Claude Code has been with a roughly $100-a-day budget for their engineers. And so what I was hearing is that this has actually fundamentally changed how work gets done in this company. In that there's a rate limit which resets at 4 p.m., and so the most productive hours of work are actually now 4 to 6 p.m. when the rate limits reset. But then there's this dead period in the afternoon when people go on walks or, you know, get a coffee because they just don't have the rate limits. And so I think what's really illustrative there is that you see that there is a misvaluing of intelligence happening at every layer of the stack. And so by that I mean you have engineers who have $100 worth of usage limits and they don't really know how to apportion that to the greatest productivity for them. You also have this Fortune 10 company which has kind of arbitrarily said they're going to allow $100 per employee — they've actually recently increased it to $300 per employee, so almost an employee's worth of salary in tokens for them to use. And this is actually pretty arbitrary because it's hard to quantify what the right usage limit should be. But then also Anthropic is running on pretty narrow margins to support this, and they have, you know, massive costs to serve these models. And so I think we're in this world where it is still very unclear what ROI looks like and how to value this intelligence that's being used. And so as we talk about the existential concern for enterprises, I think it is this kind of direction we're shifting in where token spend may start to eclipse salary spend. And so if this is such a meaningful line item in your costs, you actually have to justify the ROI much more keenly than you've seen over the last six months. And over time, as we were talking about, I think a firm really is just its eval. So the ability for a company to make its eval legible in order to solve this ROI calculus is going to be the reason why that company wins out over the competitors in the long term.
