EP 15: Kevin Weil (OpenAI)

Thank you so much for having me.

So when I came to college, I thought I was going to be a math major. I wanted to study math. I wanted to be a math professor. I had grown up, my dad was an engineer at Microsoft for many, many years. And so I'd grown up going to work with him and been like programming as a hobby and things like that, but had just totally fallen in love with math. Then I took this class taught by a guy named Howard Georgi, who's known to be an incredible science communicator, you know, almost won a Nobel Prize, like incredible physicist, but also great science communicator and teacher. And it was just like, wait, this is like math except it also tells you how the universe works. And so just fell in love. And so from then on I wanted to be a physicist. Came out to grad school, wanted to be a physicist. And you know, I think at some point in there I was like, oh man, I actually, if you're, I was doing theoretical physics, theoretical particle physics, and if you're lucky in theoretical particle physics, you make a contribution over the course of your career. At least, you know, obviously some people are incredibly prolific. I was like, oh man, am I, you know, what if I'm that person that works for 40 years and doesn't actually change anything about the world? We don't discover something or my theories are interesting, but they're wrong. And the timeline to discovery, to validation in the real world is very long in particle physics, right? You're waiting for the world to build a particle collider, which are literally the biggest science experiments that humanity has ever created. You know, like the LHC is a 100-kilometer ring beneath the city of Geneva that was like a 190-country, many thousands of people collaboration over 20 years to make it exist. So you're waiting for one of those to learn if your stuff is right. And so anyways, I ended up meeting my now wife who at Stanford who had kind of gone the opposite direction through school. She spent her entire time working at startups and in the venture industry. Basically paid her way through college working at these startup companies. I had no idea these things existed. I just had like math and physics blinders on. But she kind of opened my eyes to it and I was like, oh man, you mean I could ship something, you know, write some code, ship something to a bunch of people today and touch, you know, a million people's lives tomorrow? That feels so much more tangible. And so I ended up dropping out of my PhD, going and working at startups, and you know, I've missed physics and math. I've always tried to stay close to it. So that's why this role is such a fun and, you know, what a privilege to be able to kind of come full circle back to it. But yeah, that's how I left physics.

I don't know. I'm a very pragmatic person. I like feeling the impact that I have in the world. I've been really lucky to work on big products like Twitter and Instagram that billions of people use. There's something really powerful about feeling like you are hopefully changing people's lives in a positive way. It's one of the reasons I came to OpenAI as well. So that's just, it's a harder thing to do in academia. So I don't know whether I had the right temperament for it at the end of the day, but I've stayed a physics and math nerd. I still try and read a bunch of papers, now thankfully with the help of ChatGPT because it's a lot easier when I can say, hey, what does this mean, how does that work. But I don't know, and you asked about the, like physics in particular seems to be a source for a lot of AI people. I think there is, you go into physics generally because you're very pragmatic, you know, it's using a lot of other sciences but you're trying to describe the world and physics is willing, you know, sort of you use whatever mathematics you need to use, you use whatever other fields you need to use to try and figure out the world in physics. And I think it just draws people who are interested in figuring things out. And it turns out there's lots of things in the world to go figure out and right now AI is one of the most interesting things to figure out and so you see a lot of physics people going, huh, I wonder if I could contribute to that.

Probably more it's the kind of brain. But physics teaches you valuable skills. It teaches you how to think and how to solve problems. It also teaches, you know, there are things that I use that even though I don't do physics at all day-to-day, I mean a little bit more now than say a year ago when I was in my old role, but it's not like I'm solving physics problems as part of my daily job. But it's still, physics teaches you, for example, when you have a solution to a problem, how do you check whether your answer is right. There are probably three or four different ways that you can quickly get a gut check for whether you're on the right track. You know, you take limiting cases of your answer. You take some parameters to zero. Does it still make sense? Check your units, things like that. And that general tactic of, okay, we're trying to do something that we haven't done before, which is kind of every day at OpenAI. And we have some ideas. Now, how do we know if those things are right? How do we know if we're working on the most important problems? Physics also teaches you to like simplify equations and solve the most important parts of equations rather than trying to solve the complete equation in every case. And like those kind of instincts tend to be really valuable in business and in my work at OpenAI even though it's, you know, not directly a physics application. They're just things that you learn as a physicist to do.

Yeah. But hard skills are nice. I think hard, going quantitative.

Yeah. Planet was one of the most fascinating places that I've ever worked. One of the things people don't typically think about, San Francisco is outside of LA where SpaceX is building tons and tons of communication satellites for Starlink. Outside of that one example, San Francisco is the satellite building capital of the world because of Planet. Planet operates, you know, 200 plus satellites that they built in the basement of their building in San Francisco. And they collectively image the entire planet once a day. They've been doing it for eight plus years. So there's like 2,500 data points over every single point on Earth showing daily change. It's such a cool data set. And you gave a bunch of great examples, huge numbers of different, you know, economic validators, not to mention national security implications, economic climate implications, things like that. Just one of the most interesting data sets in the world, by the way, extremely amenable to AI as well. Because one of the things that's happening, and I think this is true of maybe open data in general, is there's so much data being produced all over the world. You've got more than humans know what to do with, have the bandwidth to process themselves. But if you have AI, especially if you have a regular data set of, you know, everyday daily imagery, then what you want is to understand anomalies, what's happening in the world that you don't expect to be happening for a variety of reasons. And you know, again, theme with AI, this doesn't put humans out of a job. It helps humans do a more interesting, higher leverage job, which is not to just be, you know, imagery analysts trying to understand where change is. It's more about explaining the change. What does this change mean? Why did it happen? And what should we do about it? And so it empowers people, which I think is a general theme of AI today.

Yeah. I mean, one of the interesting things about building a consumer business is you sort of daily have to prove your value. You have to build something that matters to people because people are very busy. They've got their lives to lead and if you're building a consumer product and it's not directly relevant to them, they're not going to use it. And so you learn very quickly how to find problems and try to, you know, put yourself in the shoes of a user. Understand what matters to them and figure out how you can build a product that solves problems for them and solves not just like problems they might have or they have occasionally but problems that they have on a daily basis. That's a skill that is relevant to building anything that matters. We're definitely, as we think about how we build products for scientists with OpenAI for Science, bringing that lens. It's not just, you know, hey, we'll build a really good science model and it'll give them superpowers in all these different ways. That's all true, but it's also how do you then integrate that amazing scientific AI model into the workflows of scientists on a day-to-day basis. And so when you think about it from a science perspective, you still need to be solving problems. You're talking about building products for scientists. You need to be solving the problems that they encounter on a daily basis. So it isn't just building an incredible AI model for scientists. It's also that, that's obviously a huge component of what we can do and how we can accelerate scientific discovery around the world. But I think it's also how you build that model into the workflows of scientists on a day-to-day basis. So we built this product for example called Prism that is a product that brings AI into the scientific writing and collaboration flow. So when you're writing up one of your ideas, you're going to publish, you know, a paper, for example, you're collaborating with other scientists. You're writing this language called LaTeX that ultimately gets compiled into a PDF, but anytime you see a PDF that has tons of formulas and other things that originally came from this source language called LaTeX, AI can help you do that faster, better, more completely. And so we took this amazing science model that we have and we built it into that workflow. And there's an opportunity, I think, to do that in more places. Because in general, scientific tools are not super well supported. It's not a massive industry. And we're really excited to try and bring great scientific tools to scientists.

Yeah, it's an exciting thing because we have a variety of different skill sets coming to play. First and foremost, we're an AI research team and we're helping to make the AI models better. One of the ways that we, and better for scientists in particular. One of the ways that we do that though, I mean, the models now are incredible. We've gone from three years ago when my mind was blown because GPT-4 could get a 700 on the math SAT to models that are regularly solving open problems in math and physics and other scientific fields, problems that humans have not solved before. You see an AI model beginning to solve them. So it's not just really good at understanding, you know, all the things that humans understand. It's actually, models are now, our OpenAI models are beginning to push past the frontier and solve new open problems. One of the ways that we do that is actually by hiring practicing mathematicians, physicists, biologists who can help teach the models to be even stronger at these fields. It takes, you know, I am not even a physicist, let alone a biologist, right? You know, I have more background in physics than most people, but I can't make the models do what one of the physicists on our team can make the models do when it comes to trying to solve hard physics problems. And so when you're at the frontier, you need people who know how to operate at the frontier to help the models get better in these areas. So we also have been hiring subject matter experts, you know, prominent physicists, mathematicians, etc. to the team. And all of that is in service to making the models really incredible collaborators for scientists all over the world. Our mission is not to win a Nobel Prize ourselves. It's to see 100 scientists win 100 Nobel Prizes using our technology. Now in the fields that we think about, math, physics, biology, material science, we also will probably take kind of moonshots of our own because it will actually help us, you know, we can potentially move the world forward. If we can really solve something important and impactful, that's awesome. But also, it will teach us, since we're using our own product the way that other scientists use our product, it'll teach us how to build a better product for them. Or a better model really. So that's the work that we do on the model side and then I talked a little bit about Prism and the work that we do on the product side and it's really all of those coming together. It's not just an amazing model for scientists, it's also integrating that model into the way that they do their work on a daily basis. Both of those things will help accelerate scientists doing what they do every day.

Yeah.

I don't, I would challenge you to come up with a domain of science that isn't really ripe for being accelerated by AI. So I was talking to a guy named Pratyusha, who's a linguist at Berkeley. He studies sperm whale language, right? Turns out...

Yeah. Okay. For two reasons. One, because what he does is so cool. He studies sperm whale language and it turns out sperm whales have a regular language that they speak to each other. It has some grammatical structure. In particular, it has vowels and he uses AI to understand and decode the speech. Which is amazing. It's so cool. And you know, if we can do it for sperm whales, why can't we do it for dogs? Second though, he said something that's really stuck with me. He said, AI is a metal detector for hypotheses. So, you've got more ideas than you know what to do with, certainly more than you can experiment with. AI, if you have this collaborator in a frontier AI model like GPT-5, it has read substantially every paper across every field of science that have been published in the last, you know, N decades. It's infinitely patient. It's there whenever you want it. So, you get inspiration at 2 a.m. You can talk to the model. It's also, you can just have it explore any particular idea that you have. Write pros and cons. Think for an hour about your idea and explore every possible subtlety. You can do that with 10 ideas in parallel, right? You can't do those things with a normal collaborator. And taken together, you end up being able to evaluate ideas across way more axes than you could, you know, as a human alone. So it's a metal detector for hypotheses. He's a linguist which is not math, physics, chemistry, computer science, you know, but I mean you could be studying the evolutionary biology of snails and the idea that AI is a metal detector for hypotheses still holds true.

Yeah. I mean, we don't teach the models Spanish or Chinese or English, right? We don't, it stands, they learn them. It stands to reason that models could understand other languages from other species as well.

Yeah, I assume. Not to mention AI is superhuman at a bunch of more basic research things like literature search. You have an idea, you want to understand who has studied this before. Sometimes you by default you would do a normal search using language, but what if they wrote in a different language? What if they use different wording for the concept? And AI has an ability to search in sort of concept space across languages that is by itself superhuman. So there are all these other ways that scientists are already using AI today.

Yes. The idea of AI being like a metal detector for hypotheses, I can imagine there's a certain point of diminishing returns where volume of experiments or volume of hypotheses doesn't really matter. It's about knowing what are the right ones to run. Obviously, if you're screening a million different molecules, the volume is a necessary element of this, but how can you talk a little bit more about how we actually hone in on the right things to be doing or the right flashlight to be shining to find the way forward? Because there's this meme where when AI is superhuman at everything, the one human skill that matters is taste. And so the difference between AI slop for science versus science that is meaningful and profound and advancing the human condition that is done by AI will require some sort of human-level taste in the scientific driver's seat. What does that actually mean or how are you guys thinking about that?

Yeah, it's because when you look at a scientist actually going back and forth with AI and trying to solve some hard problem, it is really a back and forth. It's not just typing the right question into the box and the AI does all the work for you and out comes this thing that you just yolo into a paper and publish without looking. It's a back and forth. You're exploring a new concept together. Sometimes AI can solve it in one go. But if that's true, then that just is an opening for you to go try harder problems.

And so when you're really at the frontier, it's a lot of back and forth. And if you take, I have a colleague named Alex Lubsa who's a physicist, a black hole physicist. And I have enough of a background to be able to have conversations with him, but I don't have nearly enough background to do the kind of research that he can do. So me with GPT-5.4 cannot do anywhere near the kind of physics research that Alex with GPT-5.4 can do. Expertise still matters because at the end of the day, this is you going back and forth with a model, understanding the right questions to ask, pushing back on some of the answers. 'No, that doesn't seem quite right, but there's something interesting in what you said here. Let's explore that. Go deeper here.' So that is actually the process of going back and forth with these models. And it's why it does require taste and an understanding of what the right questions are to ask. And if the model can just do stuff for you, then that just means you can go deeper and do more.

It's a really good question. I think, I mean, it's a tool. It's a tool that gives you superpowers. It is probably different in character than most other tools. You can't exactly compare it to a calculator or even to, you know, I used a lot of Mathematica when I was in grad school and it's an incredible tool, but having something that can think for itself opens new vistas.

One of the things that we've seen over the course of the last few months, there's this class of problems called the Erdős problems in mathematics. Paul Erdős was this incredibly productive combinatorist, number theorist, and he left behind like 1,200 open problems.

I mean, you know, fingers crossed. I'm pretty sure that's right. But, so he left behind 1,200 open problems of massively varying difficulties. Some are, like 400 of them or 500 of them are solved now. Others are sort of unsolved but maybe they're just a little bit beyond the frontier of what people have done before. Others are very important unresolved problems that would be major news were someone to solve them. So we put out this paper four or five months ago trying to basically just benchmark for people what AI could do for science at this point. So it gave a bunch of different examples with 10 outside authors from a bunch of different fields showing how they used AI solving some open problems as an existence proof. You really can solve open problems with AI. One of them was one of these Erdős problems. And in the space of the next, I don't know, 30 days there were like 10 or 15 more Erdős problems solved by a variety of different people. And one of them was an account on Twitter. He's solved a bunch of them. He was very prolific in this subculture of solving Erdős problems. And he was sort of an anonymous Twitter account. Didn't have a name and his profile picture was like an animal. And so I reached out to him. I was like, you know, I wanted to get to know him a little bit. Turns out he's a 20-year-old kid. This college student solving a bunch of open math problems with AI.

And it blew my mind especially as I talked to him more and I realized the sheer amount of mathematics that this kid knows and then you get into it deeper and he turns out he knows a bunch of AI too.

And I was blown away. I was just like, how did you learn so much? How do you, like as a 20-year-old I was happy that I was taking topology or something and thinking I was advanced. This kid is doing research-level mathematics, knows more about AI than I do. I mean, it's just... And he said, 'Look, I've spent the last three or four years of my life having access to this incredible AI model that is a personalized teacher for anything that I want to learn.' And so, I've been able to learn way faster on my own than I could have learned any other way. So, that was pretty eye-opening for me.

Yeah, just wait for the summer.

What you said is exactly right. Science is a massive ecosystem. The surface area is huge. It's the reason that our first and most fundamental goal with OpenAI for Science is to empower scientists all over the world with amazing AI models and tools because we will never do even a fraction of all the science ourselves. We need to get in people's hands and help accelerate their process. As far as what, I think there's a lot more available than people even realize. I mean, you can start using ChatGPT for free and for $20 a month, you have access to our incredibly advanced reasoning models that are able to go off and do a bunch of science like everything that we've been talking about so far. There's a huge gap today between the people that are really leaning into it, looking at this as an opportunity, getting excited, and saying, 'Wow, I could do what I'm trying to do. I can discover more. I can discover more quickly. I can educate myself about adjacencies that I may have said, you know, that's not exactly my area of specialty. I'll wait for someone else to solve that.' Now, you don't need to wait for somebody else because you have a collaborator in GPT-5.4 that can help you kind of go into any adjacency that you want and do new things.

And then you have others who are more hesitant or cautious or afraid and aren't taking advantage of these tools. So honestly, I think the number one thing is just get in and try these things. Just start using AI models in your research and I think people will very quickly realize that they are an accelerator to discovery, that it's not something to be afraid of. It's actually something to be really excited about because they help you do what you're trying to do whether you're at an FFRDC or whether you're in industry or whether you're faculty or a postdoc or a grad student. They just help you do more and get more done and learn more things. But adoption is still uneven and that is the number one thing I wish I could change.

Yeah. Well, honestly, I think a lot of it is just jumping in and trying, you know, like at 20 bucks a month to put in enough effort to show yourself that this stuff really works, everybody can do it. But then when you really start using it, we have, if you're using our most advanced model that you can sort of use commercially, which is our Pro model. So, you're using GPT-5.4 Pro, it will if you give it a really hard question it'll think for maybe 90 minutes sometimes. That uses a lot of compute which is not cheap and that's why in order to use that you're on a subscription that charges that cost $200 a month that is defraying that cost.

I know, I'm not doing science. I think that maybe, yeah.

When you have models that think for longer they are able to do more. They're able to discover more. They're able to have ideas that they couldn't have in a shorter period of time. So you really want models that can go beyond 90 minutes of thinking and do 90 hours of thinking or 90 days of thinking in the future. And that will require a huge amount of compute. The people that we want to have that compute are the scientists who are going to do the most amazing things with it. But those are sometimes the people who have the least ability to pay for that kind of compute. And so that I think is a problem that we're going to want to solve as a community, as the US. Like we're going to want our scientists in order to accelerate discovery. We're going to want our scientists to have access to huge amounts of compute and that won't be cheap. That's a problem we need to figure out.

It's awesome. I love this project. Yeah.

I think the Genesis mission is one of the most exciting projects happening right now. It's got a lot of different focus areas, but one of them is saying look, the national labs, there are I think 17 of them. Many of them have operated for decades and decades doing fascinating science, important science. They have a huge amount of scientific data that is currently mostly unused. And if you could teach AI models that science, it would make them better at science, which would confer advantage on the US and US industry in general. That would be an amazing thing. So, how can we take this data, which is probably mostly sitting inert, and actually use it to make AI models even better at science? What a cool idea.

So one area that I'm really excited about is fusion. So look at Lawrence Livermore, which was, they run the fusion instrument. They were one of the first, I think the first in the world to demonstrate positive energy fusion. This is like what, 2022 or something. Fusion is a massively multidisciplinary thing. You've got plasma physics, you have material science, you have you name it. This is a part of building a successful fusion device. And you've also got simulation that's heavily involved. So imagine that we could take all of that data that has come off of the individual fusion runs that they've done, which by the way are individually massively expensive. So that's not a thing that OpenAI is ever going to be able to do itself. But there's this data that tells us a lot about how the world works at hundreds of millions of degrees that we could potentially build into AI models. And if we were better, if AI models got better at understanding these processes, you could imagine a world not unlike what you were talking about with Ginkgo, with the work we did with Ginkgo Biosciences where you have a robotic lab and you have an AI thinking about experimental setups and then rolling those out to the robotic lab which does the experiments in the real world, puts the results back to the AI model which thinks some more and you have this loop out through the real world, scientific experimentation. Imagine that you have an AI model that understands fusion very deeply and again cross-disciplinary, understands the engineering behind it, the physics, the plasma physics, and also knows how to run all of the fusion simulations that Lawrence Livermore and others have built over the years and so it can think about the process, test a lot of the parameters that it may want to run fusion with in a simulation, get feedback from the simulation and iterate a bunch there using heavy amounts of compute but still all in silico. Then you have the parameters that you want to try out in real life and you run one of these very heavy fusion experiments, very costly fusion experiments, then you take that data and roll it back in and you have a process that allows you to iterate far faster and hopefully again metal detector for hypothesis, you're hopefully zooming in on the actual ultimate parameters for fusion that you need and you're able to do it faster because you're combining the best of what the national labs have to offer with what frontier AI has to offer.

I always knew Dario was a genius.

Well, first this is one of the reasons why in order to train really good science models, we are hiring professional scientists. Bringing them into the model training process because they're able to bring their understanding of the field at the frontier and help make sure that the model as it trains is getting better, getting more factual, able to solve harder problems truthfully. And then the other big part of this is at the end of the day, this is a human and AI doing science together and the importance of the scientific process is undiminished. The importance of a human validating these results, doing experimentation in a lab if necessary, the importance of peer review, things like that are all undiminished. These are still critical parts of the process whether you use AI or not. And so that at the end of the day is how we're going to trust these results. The AI is hopefully going to help you think, examine more hypotheses. It will push your thinking, metal detector for hypothesis, will help you experiment faster in the case of things like robotic experimentation. But at the end of the day, scientific validation and peer review deeply matter.

Some of these processes are human processes that humans have put in place and humans fully control. So humans will decide whether they change or not at the end of the day. But I think that AI can play a major role in helping with things like peer review. Doesn't mean that it's not important to have a human also read the output. If I were writing a paper right now, the first thing I would do is have AI do a review of it. We have a council of editors in our project instructions to validate whatever we're doing from all these different perspectives and I imagine that being similar in a scientific way. And so you can have more confidence in anything that you submit than you have before. People also worry a lot about AI slop and is AI leading people to publish a bunch of bogus scientific work and you can have AI again review any of these incoming things that have been posted and AI is pretty good at filtering that kind of thing out these days. So it's kind of like when we introduced email suddenly there was a problem of email spam which didn't exist before. And then we built systems over time that were able to identify it and weed it out. And now, you know, mostly don't think about email spam a whole lot. I think the same will be true of this AI slop. We're going to ultimately use AI to help filter it out. And you'll use AI, legitimate, good science will use AI to help tighten and examine and push on their paper before they submit it. And peer review will still be a thing and will be important because it's part of doing good science.