Like how long can an AI do a task? Is it like 7 seconds and it loses its train of thought? Is it 14 seconds? And it seems to be doubling every 7 months. So before they can maybe write you a little function, now they can write you a whole package of code, a whole file of code. Now they can maybe write you many different lines of code and keep engaging with you over a session as you say, you know, I want to add this feature or something. So if that keeps doubling then you have this really exponential process where you have this discontinuous impact. Going from like 30 minutes to an hour is a big deal. Going from an hour to 2 hours is a big deal and going from like a day to 2 days is like an incredibly big deal, right? The amount of increased scope you can have as a result will increase exponentially. So I expect these kinds of discontinuities. I am still confused about why these models have no traction whatsoever on things which are like short horizon tasks like it doesn't take you that long to let's say edit a video or book a flight or start playing a new video game which you've never seen in your training data and there they have very little traction whatsoever or organize a logistics for like a happy hour or something, right? So the big question, it seems like pre-training is exhausted. The reasoning paradigm, the big question is how far that gets you. And so we had o1, o2, I guess we skipped o2 and now we went to o3. The next question is like, okay look, they've conclusively shown that this gets you a ton of traction on math and code. What are the other kinds of RL environments you can spin up where you get this thing where you're now doing tasks that are like a human's day-long worth of input? I think in 3 months we will know if we have models which can reliably do computer use, any sort of an RL environment for that. If not, then I think we're not going to have some huge economic impact from AI that we haven't seen so far in the next 18 months because fundamentally there's not that much in the economy that's just chat-based. But if you do have that then I think you can start seeing tens of billions of dollars of revenue per AI company and other kinds of big impacts.
Hello. Hello. It is Ashley Vance back here with y'all for another episode of the Core Memory Podcast. In this episode, the headline is that the podcaster becomes the podcastee. I sat down for about an hour with Dwarkesh Patel who has a massive YouTube show and podcast where he gets into deep, he goes deep on science, history, you name it, tech, everything. And he's a great, great chat. We went to Dwarkesh's studio in San Francisco so I really turned the tables on him properly. And there's more headline news which is that he has a book coming out. It's an oral history of artificial intelligence technology. Obviously, we get into that at length in the show. We get into his background, everything you could ever want from a podcast on Dwarkesh. And per usual, do us a favor. Give us a review, give us a like, wherever you get your podcast. I promise I'll stop harassing you about this once we have asserted our position as one of the giant podcasts of the universe. But until then, we need your help. Anyway, come check this out. Come hang out with us in Dwarkesh's studio. Here we go.
Dwarkesh, how are you?
I'm good, Ashley. Good to see you. The tables have turned.
Mhm. You are the interviewing, but I'm in your lair. I'm on the home turf. We should have actually turned the tables because this is usually the side I sit on. And I actually should have gone on the other side, but I was wondering which seat I was in, but it's better if you can see what's going on. I'm so excited to talk to you. There's a whole bunch of things I wanted to do. One we'll start with in a sec is you have a book coming out, which is fantastic. I feel like you and I have known each other a bit, but there's not a ton about you on the internet biographically, I guess, because partly you're so young and you've been on this rocket ship. And so I thought this was a good chance for people to get to know you a little bit and then obviously I've got some AI and other questions.
Excellent. Well, as an author, I wanted to plug your book first while everybody's listening for sure, just because these things are hard to do. And so it's called The Scaling Era, an oral history of AI, and it comes out on March 25th.
March 25th. And is it digital at first?
It's digital at first and then July we'll have the print version.
And over these last few years I've been doing these interviews of people in AI, the CEOs of the big labs, researchers, people doing everything from low-level research to you know the hypers like Zuckerbergs and Satya Nadellas. And there's all kinds of different insights that feed into understanding of what's going to happen in AI from just basically like how much does Satya want to invest to what do primatologists think about what happened between chimps to humans. And so there's insights from anthropology, there's insights from computer science and everything in between that I've gotten a chance to talk to people about over the last few years. And with Stripe Press, I got a chance to put all the best parts from all those conversations into one book. Added with commentary and footnotes and other insights and useful charts. And even for me, seeing all of that together was actually just super helpful because you forget about how all these different ideas connect together. And just having this sort of elevated collection of the different interviews, yeah, I'm super pleased with how it turned out.
No, it's awesome. I got a sneak peek. I only got the sneak peek a couple days ago, so I have read a lot of it, but and you know, it's an oral history, so I'd heard a lot of the podcast and then as someone who's done some oral histories, it's hard to like stitch these together. So I have questions about that, but just so people know, it's amazing. You had this brilliant, I give you all the credit in the world for seizing on this moment, this fascinating moment in time in technology quite likely in human history. And you call it about the scaling era. I mean, but you really capture this birth of this moment through many of the most important figures who are doing it. So it's really cool to see how...
Thank you. Thank you. I mean, it's one of those things where I don't think afterwards you can get the sense of how people were thinking about it at the time, especially if AI is as big a deal as I expect it to be, maybe you expect it to be. We'll talk about what you expect it to be. Like how were people thinking about it in this crucial period where we're getting this early progress, but it wasn't clear where the next step was. And yeah, anyways, it's super cool to see and I'm excited for people to see it as well.
Did you... Okay, so my favorite oral history ever is West of Eden by Jill Stein. I don't know if you've ever...
So yeah, East... No. This is West.
She does an oral history of the birth and rise of Los Angeles told through kind of like a movie family, rancher, you know, as LA is changing from this pastoral unknown place to this booming city. And it's fascinating. She's brilliant. It's an art form though to shape these. I mean, you had I'm sure it's probably like millions of words that then you have to figure out how you're going to arrange it. Just as a writer, I was curious about what the process was like, like how you found it, if you discovered anything magical about putting together an oral history. I'd be curious.
Yeah, let's see. And first of all, I should really give credit to my co-author Gavin and our editor Rebecca for helping us put together these different segments. I think what was just really cool is so the first chapter is about scaling, right? And you can talk to Dario and you can ask him like why does scaling work, right? What is it about the universe that if you throw bigger blobs of compute at training with more data, you just get something emergent on the other side. And then you can ask somebody who's studied how the brain works and what circuits in the brain map on to how attention works and then he has a perspective on why scaling works. And then you talk to somebody who's been studying like what exactly changes between primate brains to human brains and he has a perspective on why scaling works. And you just do that from chapter after chapter from economics to interpretability to whatever. So just seeing all these snippets next to each other was a really fun experience and compiling that together was, yeah, it was a fun time.
Yeah. No, it's a fascinating read. All right. So we've done the plug. Plug the book. I feel like it's important. And well, you know, I was so curious. I don't think I was following you from the very earliest days, but I heard about the Lunar Society and things early on. Everyone's like, oh, there's this really smart kid. He's doing a podcast. He's interviewing interesting things. When I got to know you a little bit and I think it comes through on the podcast a bit. I don't know. You have this old soul vibe, you know, I don't know if it's just the beard or... Are you still 23 or 24 now?
24. Okay. So, time flies by. But both you and I would say even Leopold who's in the book and who you talk to a lot. I don't know. You guys exude this kind of wise old soul in a young person's body. I don't even know if you agree with that. I was curious if you've always, if you do agree with that, if you've always felt like that a little bit.
Yeah. Yeah. I mean, the beard gives me an extra 10 years or something.
I saw these videos of you doing debates in high school. Oh, yeah. You got this chiseled chin and I don't know, man. I think you should go back to that at some point.
That's right. Yeah. I should do like some sort of fundraiser where I raise a million dollars. I'll shave off my beard for something.
Have I always been old? Well, you know, you have a certain, you've got like a, not to like blow smoke or anything, but you've got like a gravitas. You kind of come at, obviously you do a lot of history podcasts, but you kind of come at it from this... Yeah, I would say this like wise, with this unfair amount of wisdom for a 24-year-old sometimes to have.
Yeah. I mean I will say that I have the good opportunity to ask the questions and I can just talk to people who have spent a lifetime studying the subject. So I'm under no illusions of having some deep understanding of every single topic I've interviewed people about. But the good thing and I'm so glad that I got into this career. I mean I studied computer science and so this was the thing I was doing on the side. But I'm really glad that the way I got into it was with podcasting because it's, I mean talk about AI, right? These models you make them better first by imitating somebody else and then you do RL on top of them. Similarly, I feel like I got a similar experience where I interview people, you don't really need to understand the whole topic in order to interview somebody. You can just be curious about it. You can read the book. You can have questions about it. It's a great way to give yourself an excuse to read more to build up your domain knowledge and so forth. And honestly, it was in retrospect such good training to be able to do that. And then yeah, I guess I'm 24, but also I've been doing this for 5 years. And so over time, as I've interviewed different people, you just pick up a lot of different perspectives. You pick up a lot of different context. And the other thing is I really believe in the hive mind of like once you get into this community and as I'm sure this has been your experience as well, you keep meeting all these people who teach you more things. And honestly, what I learn offline as a result of producing good podcasts, which leads to me meeting more interesting people who teach me more things, which leads to a better podcast, that is the main feedback loop. The show growing on YouTube or whatever, that is not the main feedback loop. The exponential growth happens on the back end.
Yeah. I mean, it's like the favorite part about my job is just you're constantly learning from very fascinating people and being immersed in it. You know, by the way, just as a side note, we were, I don't know if this is working chat house rules or something. If it is, we can cut it out. But we were at a conference together and people were doing like three-word intros and then yours was 'journalists allowed here.' But I feel like that's been part of your edge, right? That you just know the top people in technology who can teach you what's happening. It's a weird thing because there's always this debate about like access journalism and things like that and everyone's always like, oh, you're optimistic about tech, which I think I am in my heart. I don't know, at least in our documentaries and our TV show and in my writing, you know, I try to tell both sides or if there are two sides, there aren't always on some things. But I just try to do enough reporting where people can make up their own mind. I think if I have an advantage or so, it's just I'm not a super judgmental person. I'm interested in interesting people living interesting lives and like you can decide if you think they're... it's like if I'm doing a documentary on Brian Johnson, it's like you can make up your own mind about this guy but this is what it looks like, you know. So but you remember that you do have like a, your memory is, you seem to have like an encyclopedia type memory. I was curious, you know, like when you do the history podcast in particular, I was always impressed that you're a conversant person on quite in-depth with these people and on the subject matter. And I know you do tons of research going into these podcasts. So I was wondering if this comes from you being extremely well read and widely read or if it's these podcasts as you're heading into an interview that you deep dive on. I know before this podcast I was texting you about a couple of my favorite people like Emma Goldman and Mikovsky and you were like oh I don't know those two particular people. So yeah I was just kind of curious growing up were you reading tons of fiction and non-fiction? Were you reading tons of history or is this something you've come to a bit later as you're doing these podcasts?
I think mostly it's come through the podcast and look, I mean I started the podcast when I was 19. And so over time, I started the podcast, I was mostly interested in economics and history and my first guest was Brian Kaplan and after that Tyler Cowen and so that was the main thing I was interested in for a while and AI I almost came to not begrudgingly but quite late in the career of the podcast because I just thought look this is important so I should cover it. Now what I've realized is AI is one of those fields where in order to think about it, you need all these different lenses. If you want to think about what will the world look like in 2040, it's not just enough to say I expect the benchmarks on AI to saturate in this particular way. I debate with you, you had a debate with the economist about like will there be explosive growth? Will there be 30% growth? And then Tyler Cowen has all these different arguments that we had this debate at the progress studies conference that I published on my podcast. Great. But that's like an economics debate, right? That's not even just an economics, that's a history debate like what exactly happened in the industrial revolution, how do we best understand that. And so the thing about AI has been that like having done these different kinds of interviews where you talk to different kinds of people you can always ask them about AI and you can always learn something about them from, ultimately we're trying to predict what this future society will look like and there's no field that is not relevant to it. So when I interview like David who's a geneticist of ancient DNA I can ask him about like you know what exactly happened to the Denisovans, why were they able to take over so fast and what does that teach us. But yeah, I mean that's why it's been fun to talk about AI.
Well that's a great example because that particular episode was fascinating but so that's something where you've read, you've done your research for that interview but you were not, because your knowledge on some of that was impressive, but you were not doing like super deep on biology in high school and college. That's something really where you gathered a lot of that knowledge for the particular show just in compilation. I was curious.
Yeah, maybe. I really think that it's one, at least a level, look, I understand I'm not like going in the lab and myself doing research, right? Like I know there's different levels to this, but for the level of knowledge where you can explain the main arguments, you can explain the evidence for it, you can understand how to evaluate that evidence, I think people would just underestimate themselves like this is something just a normal person can do. And is a more interesting way to engage with intellectual content. So, yeah, for that one, I mean, I don't know, as far as my process goes, I have gotten in the habit ever since I interviewed Andy Matuschak of making space repetition cards for everything. I have other things in my process, like I fundamentally not that complicated. I have a big Google doc where I write up my notes and questions as I'm sure you do as well. I do a bit...
I was, I mean, I've seen you write about your process. I actually think it's fascinating. Because I've been doing this a long time and I oscillate between doing tons of research and then sometimes it's actually beneficial to go in a little bit blind. So that's funny. I mean there's different ways at this depending on who you're talking to.
Okay, so you were born in India, right? And about 8 years old you moved to the States and your dad's a doctor.
Did your mom work? I didn't even know this was on the internet. I found it. Deep research gets too far. The question was about your mom.
Yeah. She's a tax preparer. And she works during tax season. But yeah, mainly it's my dad who's the breadwinner.
Did you get shunted off into like rural America? I mean you were in like North Dakota, Maryland. And it sounded like your dad was getting hired at either different hospitals or like I don't know what kind of doctor.
So he was on an H1B and so you have to show that you can't get domestic help at the same place. So that's why you know North Dakota they can say oh we're having trouble hiring here and that's why we're hiring. Or similarly then we went to West Virginia for 3 years after 3 years in North Dakota then Maryland then West Texas where they've been. And so the selection of like we can't hire somebody to want to live here selects for these rural areas which is the H1B program. And yeah so then I just like I guess it was a bit of a culture shock from India to North Dakota.
But then we moved around a bunch and you know I guess in the grand scheme of things I don't really know because yeah I mean how did it impact me or what was it like? I think it was, I mean one thing I realized is just like a lot of places are quite similar. Like there's not that much of a difference between North Dakota and West Virginia growing up there. Yeah I'm not sure. I don't know if I have any takeaways from that period other than the fact that we just moved around a lot.
Yeah. I mean I moved around a lot too. I mean, yeah, it has like a bit of an influence on you, I think. Where did you go? I mean, I was born in South Africa and then I did Connecticut, Houston, Midland, Texas, which could not be too far. You were in San Angelo, I think.
Not, I mean, Texas is big, so kind of far, but not too far. We had debate tournaments in Midland. I was going to ask you about that. And then I went to New Jersey for a while, then back then to Houston. So yeah, just a bit, that was like a 10-year span.
Yeah. So, and my mom's Australian. So, we went...
And was your dad's work or mom's work?
Yeah, it was my dad. My dad was, he, this is the truth, people can judge it as they will. He was a lawyer for Exxon doing kind of land deals. So we got moved around a lot. And he had met my mom in Australia. There's quite a scandalous story how they ended up in South Africa. They were both married before and kind of ran away together and then... Yeah. So I always traveled a lot as a kid.
Yeah. And but that part of Texas is, I don't know, is San Angelo kind of like, I mean it's middle of...
Yeah. I mean it is like West Texas, that part is its own country. It's its own culture. Middle of nowhere, tumbleweeds. Rodeo.
Yeah. Yeah. I mean, it sounds like you already had your culture shocks by the time you got to high school, but that's like its own thing.
Yeah. I mean, I will say there is an element of sort of universal culture where I do feel like high school everywhere is kind of similar. You know, it has more to do with generation than with where you are. And I think young people are kind of similar everywhere and older people are kind of similar in a bunch of places. Maybe urban, rural is a notable divide, but other than that, I think...
Yeah, because I mean like Maryland to West Texas was just like not that different. Same with like West Virginia versus North Dakota. Were you doing Friday night lights or you were doing Friday night debate prep then?
More the debate prep. Yeah.
So, I saw these videos, man. I mean, I don't know what to make of some of the comments on there, but it seemed like you did well at debate and I saw people, there was one where it's like he had 30 minutes to prepare for this. That was it. And you delivered, I don't know if it was 30 minutes to prepare or it was a 30-minute speech. I'm trying to remember now. You delivered this soliloquy that was, you know, just spot on and so concise and I saw the people in the comments marveling. And so were you kind of a debate star?
I was okay at it. The that event is like extemp where you get 30 minutes to prepare a speech on a topic. You get chosen randomly. I was actually, I watched it like a year or two ago again. And I was like thinking I've actually gotten worse at public speaking. Like when I do these podcasts I'm like stuttering all the time and using filler words and that speech I wasn't. So in some ways I've degraded from then. But yeah, I would in retrospect there's like all these influences you kind of forget about and I never think about debate that much. But you know looking back on it yeah really grateful to my debate coach Julie Schneers for giving us that basic training pre-training.
Was that the main thing you did outside of school? Or I guess through school but like your main...
Yeah, I think so. Yeah. And then there was YouTube and Netflix and whatever but...
Okay. Just consuming not producing yet.
Okay. And then you're a computer science major at University of Texas and in the middle of it or near at some point in it, the co-part, you start the podcast but did you and I know I've seen you talk that you thought you might do a startup or something at some point but like in the bigger picture I mean whether it's high school or as you're getting into college did you have a thought of what you were going to do?
I mean genuinely like all startup. I got hooked up with this roommate my freshman or sophomore year who was super into Paul Graham essays and he introduced me to them and then I got super into this and we would talk about it all the time and we're trying out these ridiculous startup ideas like he had one where we would buy your, instead of you throwing out your old furniture we would buy it off you and then store it somewhere and resell it or something and we spent like a couple weekends I was just helping him out we would U-Haul this furniture around. And then I actually had this, I made this website where I was really into David Deutsch's ideas. And he's into this idea of criticism and reputation as this is how knowledge grows and he's into this philosopher Karl Popper who laid this out initially. So I made this website. I'm like, okay, this will be a cool thing to do. Like there should be a website where the content has these primitives of like coming up with questions and then refuting them and blah blah blah. So let me make a forum that does this. And I said, well, you know, it's influenced by the ideas of Karl Popper, so I'm going to call it popperplay.com. I didn't know what poppers were back then.
And I would like all these people. I mean, at the time, I didn't know anybody, so I would just like cold DM people who are in this community like Ben Landry and the...
Yeah. I mean, I think poppers is like what, you know, like loosens up your sphincter, right? So it's like used for...
Okay. It's probably many kinds of threat. So anyways, I would invite, I would cold DM people who are in this community like, hey, I made this website. I'm inspired by your ideas. It's called Popper Play. Can you...
