Episode 90: The State of AI

Yeah, thanks for the opportunity.

Yeah, as you mentioned, I've been doing this for 10 years, and originally the idea of doing an annual report was so that there was enough time that could elapse such that we could find interesting things to write about. And now it's like every morning I was waking up and reading the news and checking my WhatsApps and reading research and like having to make edits to multiple slides. And so it feels like 12 months is a crazy long period of time today to write a report like this. So, I'm very relieved to be able to talk about it now that it's properly in print and we'll all change. The biggest things I changed my mind about was just pace of change. You know, everybody talks about this, but it really is the case that, you know, if you use one of the frontier model applications today and you compare it to just where it was in like January, it's like materially better. There's very few people who really pushing the envelope of what like models can do and like daring to put in like ambitious work into these agents and try to coax them into producing good responses and now they really really do. I'd say that the second thing is the acceleration of frontier AI across different areas, most particularly embodied AI or physical AI or basically AI for robots, if you will. We had a lot of slides this year into the various improvements that have come to bear in that domain and we can dig into that as well. And then perhaps the third bit I'll highlight is just the very clear cybersecurity risk surface that has been exposed across the internet as a result of models that are getting increasingly good at math and coding. So those are like the three broad areas and because my day job is investing in startups, I see a lot of the progress on a day-to-day basis. So the main thing that's also changed my mind is just seeing insane progress across 11 labs, Perplexity, Inflection, Wave, you know, whether it's like image, video, self-driving cars, biology that's coming to bear and I wanted to really try to express some of that in a way that people could understand through the report this year.

Yeah, there's a slide in the industry section that looks at just basically how many weeks does a single model vendor have a model in the top five? The other issue, and I get into it in the research section, which is like the benchmarks measure like static tasks or static evaluations, like a test that you give to a student in a university and as we all know, in real life, somebody can perform incredibly well on a test and just suck at real life work. Maybe not suck, but it's not the same thing. And as you'll see in the research slides, most of the benchmarks that were really ambitious 12 months ago when we published the report have now saturated, which means that the model has reached close enough to 100% that any material changes in performance cannot be measured. And also, you can't really tell one model from another at that point. So, we're in this tricky situation now where the industry is incredibly good at taking a metric and hill climbing on it, as we say, which is just improving the score on that benchmark. And now, we're lacking realistic, ambitious, difficult benchmarks that can differentiate models at the frontier. And importantly, not just differentiate them on a benchmark, but also recapitulate what real work looks like. So, at the end of the day, that's why a lot of the industry still refers to the testing models as trying to get their vibe, which is like, how does it feel in your hands? If it feels good in your hands, that's sort of like all that matters, really, outside of it scoring well on the benchmark sheet.

It means that you quasi give up on building frontier models yourself, and especially by doing pre-training yourself, and you pick up open weight models and provide most likely a fine-tuning service to customers by leveraging all the compute capacity that you built up with the original intention of using it to pre-train, but now you have it and the inference demand has grown larger than what the pre-training demand is. So, you're in a good position to repurpose that compute towards serving open models and fine-tuning services for your customers who increasingly want to find other solutions than using a frontier model for every task. And as they go through their own work and realize, you know, what workflows are repeatable and sort of don't need a frontier model, they can go towards an open weights and do some fine-tuning and use one of these services.

Yeah, I think for the general-purpose AI to do your job market or your white-collar job market, there I would say three fields, right? And we can look in other industries, there generally are like two or three or maybe four big companies per large category. So, as an equilibrium state, it sort of makes sense. And anyway, most enterprises would want to do RFPs with more than one vendor, and so structurally we're sort of built to have a bit of competition at the top. In other modalities, I do think you can have winners, particularly in the sciences like in biology, drug discovery. I think there's some nuances with types of data and what information your model should have to make it useful on predicting drug structures or editing drug structures, etc. Same goes for physical AI, I think you can have independent winners there. But I think at the sort of desktop for like AI to do your white-collar job, I think at this point it's really Anthropic or OpenAI and maybe Google. Although there might be a bit of differentiation if, you know, you use just the pure model through an API or you use the actual application or quote-unquote harness.

Yeah, yeah, I think it's a hard one to call, but I would say the direction of travel has been we had this powerful model in GPT-3, and it was just sort of a raw model. You input some text and it outputs some text, and it's not connected to any useful data. It doesn't have some guardrails or some ability to store memory or turns. It doesn't have all the sticky tape around it to make it useful. And like the harness, just the scaffolding that engineers have built to make the model, to control the model and like guide it in a way that can take on increasingly like long horizon work. So, managing state, like knowing what tools to call, I mean, the ability to talk to APIs, etc. And so, then you've had this like parallel development of model labs improving their capabilities, and then developers in the open or outside those labs that are developing harnesses to sort of like stand in for certain core capabilities that the core model had. So, you can always imagine it as like crutches to somebody who's recovering from a broken leg. At some point, the person might recover from their injury and no longer need the crutches. So, in the same way a model might recover from those issues that a harness was previously supporting it for and no longer need that aspect in the harness. It's a bit of this competition approach where long-term, I would bet that the model subsumes the majority of the kind of capabilities that a harness helps it for cuz it'll learn it or it'll just be trained directly into the model. So, I wouldn't independently be like long on harnesses cuz I think the models will solve it. Although, in the short term, like what you show in the report, you can take the exact same model, improve the harness, and then improve performance on a set of tasks. You can also get a model to optimize its own harness because the harness is basically code, and the model can do that perfectly cuz again it has a very objective evaluation. And the other part I'll add is like what you mentioned around usage traces, and those are incredibly important because labs will use them or model developers will use them to really introspect whether a model's taking the right appropriate tasks and called the right tools over a long horizon. So, if you're still in the business of building models, you need access to those traces. But if your application is generating traces that are useful to labs, there's a big data business there as well, which we get into a bit in the report. Yeah, so I'd say like still long model and then harness is just the short to short and medium term like fix until the model is subsumed it.

I sort of always believe that companies will use the best model to start, really try to experiment with what it can do, and find applications or workflows in their business that are amenable to being solved with a model. And then the next logical step is to price out how expensive that model would be relative to the cost of the task and the value of the task, and then compute some kind of ROI. And then perhaps there is ROI with the frontier model, but perhaps you can go with a smaller model and do some fine-tuning, etc., post-training. But I think big labs are certainly coming out with cheaper models, with distilled versions of their big ones. There's a fight there. I think Jeff was like the best example of this in the last couple of weeks of take a really, really focused research path and try to outperform by making compromises in other directions that that particular task doesn't need. And so in the Jeff case, it was basically a classification model that, you know, doesn't require any feature engineering and runs really fast and super cheap. And adoption of that went through the roof because there are lots of these applications in receipt like categorization or accounting or fraud detection things like that. But I'd say like even just in my own work, I feel like anytime I'm doing something where I'm really sensitive on the quality of the output, I'm just willing to pay to use the best model because I don't want to risk there being a better version of the work that could pop out if I just move the toggle from like high to extra high. So, I think it's all contextual and what's the value of the task that you're using the model for and are you willing to compromise or potentially compromise assuming you have good evals for the task?

Yeah, well it's clearly the frontier labs are using some form of routing because when you use like the most frontier system on high or on extra high, it's spawning sub agents and the sub agents aren't necessarily the exact same model than the one that you originally instructed the task to because again like maybe the model figures out it doesn't need the frontier model for that particular sub agent. So, I think routing's already getting baked into the harness pretty much for the frontier models. There are like, yeah, a couple of routing businesses that have popped up. One of them got acquired. I think I'm not actually sure where they end up. My diagnosis of it is that it's useful when you have heterogeneous work that is unstable or not fixed because if your work is quite fixed, I think if you're a big business, you have some workflows that you just repeat all the time. Once you've used a router to find the best configuration, why do you still need to use the router? You should just use that as like sort of in the robotic process automation era, like you would use process mining to figure out what is the step-by-step path my employees are taking to do this task and then once I figure out the best step or the most optimal step layout, I just implement a bot for that. I think in a similar way here, once I figure out the best optimal models, I should just go and implement that. I'll probably be able to get even cheaper. I don't need to pay a routing business for it. And then I think the other confounding variable with traction in routing businesses, they often offer cheaper tokens for the exact same model. So, just through promos, it's very clear we're living in this sort of whoever has cheaper tokens gets the volume. So, I don't know if that's a good enough read into the actual value of the router. And maybe lastly, like if the set of models that one uses is power law distributed, then again, like why do I necessarily need a router if I'm going to pick from like the top five anyway? That's my current view. Like we'll see how it evolves. Yeah, yeah, I might change view, but...

My good feel is that it'll happen some point in the next year for a couple of reasons. The couple of months before that particular fact came out about the 26% of model R&D is guided by Claude. Anthropic also published the results of an experiment where they tested out different model generations where a human would undertake a research task and then at some point along that workflow their work would fork where on one fork they would continue pushing it with ideas of their own and then on the other fork it would be a model and this would be, you know, the older one all the way to like Mythos preview. And then at the end of the research trajectories on both forks they would have a human evaluate whether the human directions were better, more tasteful, more interesting or less than the model rollouts. I think it was 64% of cases the humans judged Mythos's ideas to be better than the human and that was compared to, I can't remember the exact stat but it was significantly lower for like Sonnet or something like this. And then the second thing is any time you see more and more benchmarks getting created for a particular task you can usually assume that a lot of effort will go into saturating those benchmarks. And then the third bit was this whole idea of RSI and auto research got extremely hot at the start of this year and is a major focus of incumbent labs but also neo labs and I expect it to happen in the code arena first because it's just generally verifiable and works a lot faster than for example in like the biology wet lab. And I have a prediction in the report where it would be like you would have like a team of models versus a team of humans with the exact same budget which would be a time compute however we can organize that to make it fair on like an open-ended research problem and then you know over a large span of time I don't know several weeks or something you have a committee like properly assess the contributions at the end of it and you would have to conclude that like the models were like vastly better than humans. And if you got something like that you would have a good sense that models have taste because it's non-trivial to sort of evaluate this and as is always the case with scientific work there's debate over is this a new idea or is this a re-composition of existing ideas or is this drawing dots between disparate fields? What's sufficient to be an invention? But all that notwithstanding, like I expect we'll get there in the 12 months.

I mean my sense is like we don't really have a choice. I don't see what you're supposed to do to stop this kind of work. I mean the Pandora left the box many years ago. I think why I am a, why I think a lot of AI folks are like passionate about RSI is it's the ultimate step of human discovery or just discovery in general. Many of us were inspired by DeepMind's work with Atari and AlphaGo many years ago, which in a gaming scenario showed that the solutions that we've devised today are the result of human ingenuity and human work and got us so far, but there are far better solutions out there which just haven't access because of time, resources, ingenuity. And so how much better could we get on important problems across like biology, physics, maths, industry, etc. if we had a tool that could, that was basically only rate limited by how much money we were willing to spend on the problem. And so that's probably one of the rate limiters. This will probably get very expensive and there'll be only so many groups that are willing to spend a certain amount for a given task. And so I think the physics of reality comes into play there. It's like you're willing to spend a million dollars on RSI for a task that's not that expensive to solve or doesn't have that much value on the other end of the rainbow. And then I think maybe the other part that could like rate limit progress obviously just like the physical build out of compute versus the demand of running existing systems on inference and then just trading off how much capacity you have to push really ambitious experiments in RSI, which will be extremely compute hungry.

I'm inclined to side with the camp of we have the right ingredients today. That doesn't mean that it's the most efficient way of getting there, but I would say we generally have the ingredients.