Dylan Patel: Is the "Low Hanging Fruit" in AI Gone?

Thank you. Thank you Lambda. We appreciate it.

So NeurIPS comprises of like two things, right? One is like parties which are actually more fun than you would expect. I think people are like, 'Oh, AI researcher parties.' It's much better than San Francisco AI researcher parties. Then the other side of things is just like walking around poster sessions. And so last year in Vancouver, the main thing I was doing, I was just walking around looking for people with verifiable RL, anything RL and anything verifiable. Right? Like I remember the first NeurIPS I went to, it was like '21 or '22, and I was like I would walk around, I'd stop at only posters that said the word transformer, like and anything that didn't say transformer I just didn't read it. And then like you know that's evolved and last year like you know obviously I stopped at a lot of random posters as I've learned more but last year was the first time I like stopped at a bunch of like RL stuff and verifiable, you know, test time, you know, sort of scaling stuff that you know, verifiable RL stuff, all that kind of stuff. This year I have no clue what to stop at.

Well, the whole reason to stop at the posters is not like that's the poster you want to talk about. It's this is what they worked on six months ago.

So what did you work on then and what are you working on now? is like, you know, that's the interesting thing to talk to them about because they're obviously on top of their field, whether it's like, you know, RL or, you know, whatever they're doing, they're on top of it. So, like, it's more about like stopping at a poster with interesting keywords and then not talking about the poster at all.

Wait, am I giving away too much alpha?

Back to the research thing where we don't know what to look at is actually somewhat interesting cuz I feel like the path of progress has been so obvious for a while and like academics could do stuff but it's like we are in the age of research. I know it's like is this maybe a little bit true? Is that kind of bearish for AI if it's like or is it good for AI that is just like...

Everything is like you know like there's no incremental thing that takes up the majority of the head space anymore I guess is one way to look at it. The other way to look at it is like well actually just every layer is innovating hugely right like there's tons of gains in pre-training there's tons of gains in scaling there's tons of gains in quantization and systems but there's also tons of gains in RL stuff and both the like you know infra side of things and non-infra side of things right so it's...

No, but like it's, you know, when you talk to someone at a lab, they say there's a ton of low hanging fruit, right? Like...

Like well the lore is like I made the SFT data set for reasoning in like a weekend cuz we had like unlimited Azure credits for a certain period of time and I was just bombarding like DeepSeek points but I was like we haven't made our thinking SFT set. So I was like let's just get as many tokens as possible.

I'm trolling. Sorry. We just distill from OpenAI directly. We do it for research purposes. Our models are intended for research purposes and click to see our responsible use policy, but they're still Apache 2 with no clickthrough and no terms of service, but there is low hanging fruit and it's like it would take a meaningful amount of GPU hours on our scale to like do inference on DeepSeek and a lot of the tooling to do batch inference on a model that needs to be served multi-node is just not that good. And it's like we also used Frontier a bunch which is like the US AMD supercomputer and we did a whole bunch of like Qwen 32B because it works on a one node 8 by like MI whatever rack and we use millions of GPU hours on synthetic data because we have this frontier lying around. It's like that's very much AI2's data vibe and there's a lot more.

I think there's just like I think part of AI2's balance especially in post training is that we like bring in the best students from UW who are all like super excited top few PhD students in the world and they have like their things that they want to work on a lot of it is just like building a platform where they can run with it. And like there's like one student like Scott and Victoria are two people that are doing all the DPO stuff and he had this one DPO paper and it's like we just got massive gains from doing DPO and this is this guy that's just like I have my chance to prove my paper on the almost scale and just goes like crazy hours and he does it and I was like that's not really that like it's not like I'm giving an engineering plan to some full-time employee. It's just like doing it between the hours of 10 and 2 a.m. for the week training the 32B model. And I'm like, 'Okay, I've got to go to bed.' Like, 'Good luck. It's not broken right now.' Right.

We pay them a little bit. They're student collaborators.

They need to be paid more. They're underpaid. I agree. They are underpaid. The students are fun to work with, but it's a very different dynamic of everybody.

Probably.

Probably not as much as they should because AI2 is out in this like Seattle bubble and people kind of ignore it. But I'm always one of the most prominent worriers about poaching with like with a small team, it's just like you lose three people and you're back a few months realistically and it's just so hard. There's like a couple labs at Stanford where I know just got pillaged by, you know, even Meta most recently, right? So, it's like, you know, they'll just like, well, we need people. Where should we go? Okay, here's the like active players, but what if we want to pay people only a million or two, right? Instead of paying them 10 million, well, let me go find the PhD students and offer them that, right?