AlphaGo never stopped improving. Like you could have, it became super intelligent, and you could have sunk 10x or 100x more resources in it and get an even more super intelligent AlphaGo. And so in principle, these systems never stop learning. It's just a matter of how many resources you want to sink into them. Now, with language models in RL, we're still, it's still early days. So I don't think that we've discovered the sort of maximally scalable blueprint. But there's a foothold.
Welcome to Manifold. My guest today is Misha Laskin. Misha has a background in theoretical physics. He has since transitioned to AI research and to founding his own AI companies. And so I think this interview will be especially interesting for people with a physics or academic science background, but also interesting for people who want to understand the current state of AI. So Misha, welcome to the podcast.
Thanks for having me, Steven.
So I understand that you were an undergrad at Yale in physics and that you actually did your, you finished a PhD at the University of Chicago. And maybe just tell us what the 20-something, early 20s Misha thought he was going to do with his life, why you were attracted to physics. Just give us a slice of life for you at that stage.
Yeah, I think that when I, when I moved to the States, I'm Russian-Israeli, and when I moved to the States, I got really interested as a teenager in two things, and it was physics and literature, basically. Literature is basically because I didn't read very well. And so I kind of, you know, it's sort of like eating your vegetables until you like them. And so at first it was very painful, but then I actually started liking it. And physics, we had this, you know, group of, like, the Feynman lectures in my parents' library. I had a lot of time on my hands and just got, was just something aesthetically, I would say, really beautiful about it, of understanding pretty interesting implications that are non-obvious about how the world works from some set of first principles, ability to explain things very clearly. I just really enjoyed reading it. And so that's kind of, you know, my initial inclination into physics began there in high school, and then I basically wanted to do things like that.
I wanted to work on impactful science. I think that was the, that was the thing, work on impactful science, and I thought that physics would be the place for me. And so I got very interested in theoretical physics. Did my undergrad in physics and double majored in literature, but then professionally really wanted to go into the scientific realm. And did my PhD at UChicago in theoretical many-body physics, many-body quantum physics. And so it was actually a really wonderful time. I think it's one of the, I look back really fondly. It had this sort of almost physics I imagined 100 years ago, because theoretical physics you can do, at least that, I don't know what it's like now, but then you could still do a lot of it on chalkboard. And so it was a lot of sessions with my advisor or some colleagues where we're doing something interesting on the chalkboard. Yeah, it was a really fun time in my life. And, yeah.
I think right after you finished your PhD, did you go into Y Combinator and start an AI company? Do I have the details correct?
Yeah. So towards the end of my PhD, I kind of sort of had a, I would say, a change of heart, not because I think that the stuff that I was learning was very interesting, but I felt that I became an expert in this very narrow sliver of science, and while it was aesthetically pretty beautiful and very interesting, it was hard for me to see, to imagine the kind of impact that I would have, even if successful, in decades from now. I think that, yes, you know, some people go into physics and I think that are able to see kind of maybe, I mean, I don't know, at a young age or older, depending on how patient they are, a great amount of impact, and there were scientists like that around me, but maybe I was just impatient. It was hard for me to imagine waiting decades to know whether the things I was working on would bear fruit or not. And so I did this kind of, I would say, I think that my personal confidence went down then because it was sort of a bet that I took. I had a lot of conviction and put in almost a decade of my life into it and then didn't really know, you know, what it is that I should do. But I wanted to try kind of doing sort of almost the most practical thing I could imagine. Somehow entering the job force or the workforce didn't seem that appealing. And frankly, I think to a physicist who's like just trained very theoretically, that is also a bit intimidating. You're kind of, you know, I hadn't studied, you know, CS formally, and so it was all very foreign to me. So I taught myself to code, or this was happening towards the end of my physics PhD anyways, where I had to pick up coding for some of the projects I was working on. And I decided to do almost what I'd call a random walk through startup ideas without having, like, any, I would say, internal conviction at the time around what it is that I should build, as kind of just a random walk, is this useful or not to, you know, to someone else. And ended up converging on a company that was effectively building inventory prediction systems for retailers. So, you know, how many items of clothing or, you know, should you make for your next season or something like this? And I learned a lot about startups at that point in time. Mostly about, I learned a lot about what I didn't want to do, maybe during that process. But it was also interesting building useful stuff. It's just that there's some, I think, fundamental principles around startups that maybe I should have read some of the Paul Graham essays before, but I was reading them as I was building a startup. And I definitely think some of them were true. And, you know, one of them is a simple notion of having deep empathy for your customer and sort of loving your customer. And the reality of it is that I didn't have very deep empathy for, you know, for the people who I was trying to help, right, on the retail side. I didn't understand them that way. And so it kind of converged on this consulting business that was generating revenue, but didn't really have a product around it. And I wasn't, you know, particularly fulfilled working on it either. But at the same time I saw deep learning taking off, and in particular, I remember seeing AlphaGo, and that kind of changed something in me in terms of, I kind of think of that part of my life as a wandering through the desert kind of part and trying to kind of put the pieces back together and find, like, what is deeply internally motivating to me. And when I saw that come out, that to me seemed like this is the impactful science of, you know, the time that I live in that I really want to work in. And so I basically dropped everything I was doing and went into a cave where I learned deep, you know, deep learning and reinforcement learning fundamentals. And that was kind of my first foray into AI.
And what made you, so at that stage, you might have jumped immediately maybe to a company and start working on AI, but you actually went to Berkeley for an academic postdoc. So what was the thought process there?
I think at that time, AI as a useful piece of technology was, it was still not obvious that that was the case. It was still very much, this was a number of years before language models took off, and it was still, I would say, there's both a lot of industry research happening, but it was more academic in nature, I would say. And I was considering a few things. I was in the process at OpenAI, they had this fellows program for people who are coming out of, from a different field to sort of ramp up into AI. And at the same time got introduced to Peter Abbeel, who is an AI researcher and a professor at Berkeley, who's done a lot of foundational work over the last decade. And my decision at that point in time was, do I think I'll learn most in the shortest period of time? And I thought that as a postdoc, I'd basically be able to iterate through a lot of different ideas and have a lot more learning events. Which in retrospect, I mean, I don't think that that's necessarily, I think both options were great. But at the time, it was just not clear that AI as an industry, like as a commercial industry, was the place that would have the most impact. And it kind of seemed like the place where I'd learn most of the time was still in an academic setting.
Now, at this point in time, were transformers a big deal already or just starting to be?
They were, the, you know, some of the foundational transformers papers had come out, but I remember this was right before GPT-2. So it's not obvious, you know, when I joined the lab, again, since it was inspired by AlphaGo, I really wanted to do reinforcement learning, decision making, kind of solving the problem of autonomy. And there we were just using RNNs and LSTMs for anything that required memory, and oftentimes just MLPs or convnets, so it was very rare to use a transformer in reinforcement learning at all. And they were taking off in NLP, but at that time, NLP was not, you know, language understanding was one of the several, like, sub-areas of AI. It wasn't, you know, there was computer vision, there was, you know, language models, there was reinforcement learning. And I would say that the thing that was probably most top of mind at the time was reinforcement learning, because it was just coming from, you know, the AlphaGo breakthroughs. And it wasn't, when AlphaGo happened, it wasn't just AlphaGo, it was a series of papers of increasing, I would say, both they got progressively more beautiful in their simplicity and how many assumptions they removed, and kind of power. And so these were AlphaGo, AlphaGo Zero, which then learned the game of Go without any human demonstrations. Then AlphaZero, which generalized beyond the game of Go to other games. And finally MuZero, which was an algorithm that sort of learned the rules of the game as well, instead of being given them. And so that was like the thing that was top of mind to, I think, the AI community at large. I think reinforcement learning was very top of mind. And even though transformers were definitely taking off in NLP, it was kind of one of the few things that was happening. It was not clear that that would be the big thing.
Do you feel, this is jumping ahead a little bit, do you feel a little bit like the field has come full circle now where maybe RL is like, because of the recent reasoning models, like, or the ability of RL to condition these reasoning models, it's sort of now back at center stage a little bit?
I think it's, I think it's getting there. It's been a pretty interesting turnaround because after, I'd say, GPT-3, and certainly after ChatGPT, I think the whole field of reinforcement learning probably took a bit of a backseat. I wouldn't say it became irrelevant, since the workhorse algorithm that powers alignment of these models is RLHF. RLHF is a pretty weak form of reinforcement learning, and a lot of people questioned to what extent it was even necessary as opposed to just really high-quality curation of instruction tuning data. So RL definitely took, I'd say, a backseat. And so after I worked on Gemini, which was Google's large language model, I kind of realized that I thought that the ingredients might be on the table where you have these really general objects that are language models, and there's nothing fundamentally wrong with reinforcement learning. It's not like there's something that we learned that was wrong. It's just that you need a good reward signal to optimize against. It's basically the thing. You need a good, you need a task distribution to learn from, and you need a good reward, like, way to verify that those things are being solved. And so I kind of thought that, you know, after we launched Gemini 1.5, that it was probably the time to start looking into scaling up reinforcement learning on top of language models. I think that's now come full circle in terms of reasoning models, which is also rather, I think, like, one of those non-obvious things. I think a year ago, maybe, yeah, let's say a year and a half ago, reasoning models were one of the many things that an AI lab pursues. It was not clear that they would be as powerful as they're starting to get today. So I think we're kind of seeing this sort of, I mean, this is kind of maybe a normal part of these AI waves is that the work starts, you know, obviously kind of earlier before it's clear to other people, but when it's happening, it's actually not clear that this is going to be the thing that wins, except for, you know, for me, for a small subset of people who have a lot of conviction right now and are seeing something that other people aren't.
Right. So coming back to your bio, because I skipped ahead and so the audience doesn't really know what happened to you. So you did your postdoc for a couple of years at Berkeley, and then you transitioned to Google DeepMind, if I'm not mistaken, and that's where you worked on Gemini.
That's right. Yeah, I joined DeepMind to, I mean, continue basically scaling up research in reinforcement learning. And again, it was not clear to me that it was, not really to work on language models at the time. It was, I joined a team that was called the General Agents team. So, you know, it was really to solve the problem of agency and autonomy with reinforcement learning. This is a team led by Volodymyr Mnih, who was the first author of the DQN paper that started basically the deep reinforcement learning era in 2013, I think. Then what happened is that, I just remember this very vividly, I was at NeurIPS in New Orleans and ChatGPT came out, and that afternoon I was giving a talk and I had some kind of dissociative moment of, you know, why am I saying these words? Like, the thing that matters is clearly, you know, I don't know, somehow it clicked with me that it's so obvious, like, what the thing that matters is now. And so why am I at a conference, you know, talking about something that is not this thing? So as soon as I got back from NeurIPS, I basically dropped everything I was doing, started working on language models and entered a project with the, at the time it was a small group of people that became the Reinforcement Learning and RLHF team for Gemini.
So just to clarify for the audience, so you're giving your talk at NeurIPS on, you know, some research, some research you've really poured your heart and soul into. But in the back of your mind, are you thinking, scaling transformers as language models is the thing that really I should be focused on? And is that explicitly what you were thinking or...
It was something similar. I was thinking the following. So the problem with reinforcement learning before language models was that we had developed these extremely powerful algorithms that worked in very narrow domains. So you had a super intelligent Go player that was hard to, if you want, it didn't really generalize to anything. You had to, if it did generalize, you had to, it did so in the sense that you had to retrain the entire model for a different domain. And the problem is that most domains of interest, it's just impractical to collect the amount of data and have, you know, the verification signal that you need in order to get something useful working there. And so there was this big, like, existential, I would say, generalization problem of we have really powerful systems, we have no idea how to make them general. And when ChatGPT came out, and I played with it that day, and it was very clear that the system is very general. It might not be very capable yet. It's not autonomous. It's, you know, at that time it was a pretty weak chatbot, but it was very general. You could ask it about almost anything, and still can, right? And it will answer it, and sometimes very capably. I remember at the time they shipped a feature that formatted basically code as almost like blog posts about code. So you can ask it something about code and then it writes you a blog post. That was pretty magical. So what I realized is that we were, or at least I was, let's say, spinning my wheels trying to solve the generality problem when it had already been, like, it was solved for us already, right? But these language models are very general, and so it was just a different way to approach, like, the problem. And that's what I thought was really interesting. You know, when I was very young, before I actually learned any physics, I read books like Gödel, Escher, Bach, and so I was actually quite interested in AI before I knew any physics, and I always wondered about this problem of how would you instantiate knowledge about the broad world in your AI? And at the time there was some very huge project at MIT, I think, where they were just like, just literally typing true sentences into a database in the hopes that eventually it would reach some critical threshold and know about the world. And so the big thing they accomplished, which surprised me, was like, okay, transformer trained by next token prediction on trillions of tokens. Wow. You get a world model that actually knows a lot about the human world, you know, as observed by human writing. And yeah, that was a zero-to-one moment that was just shocking. And I think people are, people are still probably underappreciating that zero-to-one moment. Like, historians will look back and go, yeah, that was a, that was a discontinuity in this whole thing.
Yeah, it's very non-obvious. Still, I mean, I think it's still not obvious to me why it works. It's magical that it does, but yeah, because the internet is such a big, messy data set. And so when you pre-train a model on them, I mean, most of us don't actually get to, most consumers certainly never play around with like the actual pre-trained model because it's very user-unfriendly. But I remember getting, you know, access to some pre-trained checkpoints and playing around with them. And if you kind of poke at them the right way, you get to elicit some very interesting answers. And the fact that they have these powerful world models that are then, I would say, very steerable, like instruction tuning and reinforcement learning. You don't need to do it for that many steps before you go from your pre-trained checkpoint to something that is usable to people. And that was really interesting as well. So it solved, there was this whole field in AI before then, and maybe, I mean, still now, still is now, but I think that this kind of was an answer to that of meta-learning, which is the notion of how do I learn very quickly from a very small number of examples. And there are all sorts of sophisticated algorithms for how to do this. And it turned out that the best meta-learner, the best meta-learning algorithm, was just next token prediction on the internet. And from a meta-learning perspective, few-shot prompting is basically learning very quickly from a small number of examples. And that was really surprising. Yeah, I think there's just some magic in, you know, in the idea that, you know, big neural net, which is general enough, and you force it to do next token or get good at next token prediction, it builds structures within itself through that automated process that reflect, you know, things about the world. And I think a priori I had no idea that was going to happen, but somehow they stumbled on the right way to, or a way to do it.
Definitely. Yeah, I think that it's, it is really magical and something that was in the back of my mind, and I think certainly there is a bit in which it's magical, where the magic fades in some sense, in that I thought there's only so much information you can extract from the internet. There's only so much you can compress from it because it's sort of a fixed body of knowledge that's very noisy. And at some point I thought we'd hit a point where you're getting diminishing returns from how much you can imagine, you know, infinitely large brain that's soaking up everything on the internet. That's sort of the max of how well you can do. And it was just not clear at what point will we get there? At what point will we basically get brains, I'd say the neural brains, right? These neural networks that have sufficient capacity where you've kind of extracted almost everything there is to extract from the internet.