The following is a conversation with Wojciech Zaremba, co-founder of OpenAI, which is one of the top organizations in the world doing AI research and development. He leads language and code generation teams building GitHub Copilot, OpenAI Codex, and GPT-3. He also previously led OpenAI's robotic efforts. This is the Lex Friedman podcast. [Advertisements for Paperspace Gradient, Indeed, Blinkist, Grammarly, and Eight Sleep]
You mentioned that Sam Altman asked about the Fermi paradox and the people at OpenAI had really sophisticated interesting answers. So that's when you knew this is the right team to be working with. So let me ask you about the Fermi paradox about aliens. Why have we not found overwhelming evidence for aliens visiting Earth? I don't have a conviction in the answer but rather a probabilistic perspective. It's also interesting that the question itself can touch on the meaning of life, because if we don't see aliens because they destroy themselves, that upweights the focus on making sure we won't destroy ourselves.
At the moment, the place where I am with my belief—and these things change over time—is that we might be alone in the universe, which actually makes life and consciousness more valuable, and that means we should appreciate it more.
Have we always been alone? So what's your intuition about our galaxy, our universe? Is it just sprinkled with graveyards of intelligent civilizations, or are we truly unique?
At the moment, my belief is that it is unique, but I could also say there was some footage released with UFO objects which makes me doubt my own belief.
Uh, yeah, I can tell you one crazy answer I have heard. Apparently, when you look at the limits of computation, you can compute more if the temperature of the universe drops down. So one thing aliens might do if they are optimizing to maximize compute is wait for the universe to cool down. That's a funny answer, not sure if I believe it, but it's one reason you don't see aliens. Also, some people say there might not be that much point in going to other galaxies if you can go inwards—connect machines to our brains.
Yeah, there could be a lot of ways to go inwards too once you figure out some aspect of physics we haven't figured out yet. Maybe you can travel to different dimensions. Travel in three-dimensional space may not be the most fun kind of travel. The speed of light is low and the universe is vast.
It seems that most likely, if we want to travel very far, we would send something similar to what Yuri Milner is working on—a huge sail powered by lasers from Earth, propelling it to a quarter of the speed of light. The sail contains a few grams of equipment. That might be the way to transport matter through the universe. For humans, we would need to 3D print a human on another planet.
With our current techniques of archaeology, if a civilization was born and died long enough ago on Earth, we wouldn't be able to tell. That makes me sad. I think about Earth in that same way—how can we leave remnants if we destroy ourselves? How do we back up human civilization?
It depends on the cataclysm. We have observed gamma ray bursts that can kill an entire galaxy. There might be nothing to protect us. Looking at past civilizations like the Aztecs, they disappeared because they couldn't solve a problem. The best solution to such problems is technology. Science increases the size of the action space, which is a good thing.
Yeah, but nature has a vastly larger action space. Still, it might be a good thing for us to keep increasing action space.
Well, looking at past civilizations, perhaps expanding the action space will add actions that are easily executed and destroy us.
I was pondering why we have a negative impact on the globe. Every individual wants clean air and a healthy planet, but as a collective we are not going in that direction. Capitalism assigns monetary values to activities, but some things we value have no cost assigned. Companies can pollute for free. Politics should align incentive systems. The first issue is to measure the things we value, then assign monetary value to them.
Yeah, getting the data and enabling people to vote and move money in a way aligned with their values is a technology question. Having one president and Congress voting every four years is outdated. There could be technological improvements.
I extremely trust Sam Altman, our CEO, on these topics. I'm more on the side of being a naive hippie.
That's your life philosophy. Self-doubt and optimism are pretty good ways to operate.
It's hard for me to understand how politics works, and Sam is really excellent with it.
What do you think is rarest in the universe? Life, intelligence, or consciousness? Which is hardest to get to?
Let me explain my mental model. For life to appear, you need a graph of chemical reactions that forms a cycle. For intelligence and consciousness, they might be intertwined and continuous. Neural networks show activations that correlate with visual cortex in monkeys. Language models show similar patterns. In 3D world agents, they learn to recognize foreground, background, and eventually 3D perception. They develop symbols for other agents, and likely a symbol for themselves, which I would call self-awareness. Consciousness, the experience of drinking coffee, might be related to memory and recurrent connections. Anesthetic drugs disturb brain waves and lessen consciousness.
Mhm. And so there's a lessening of consciousness when you do that.
And so that's one way to intuit what consciousness is. There's also the self-awareness module plus the subjective experience as a storytelling module.
The crazy thing is that in meditation, people learn not to speak a story inside their head. Some people don't have an internal narrator. It's possible to have experience without the talk.
What are we talking about when we talk about the internal narrator? Is that the voice when you're reading?
Yeah, I thought that's what you were referring to.
I meant more the subjective experience that feels like storytelling to ourselves. It's a high-level abstraction useful for feeling like an entity. The most useful aspect is that because I'm conscious, I don't want to die. It's a hack to prioritize not dying.
I was thinking about the story of describing who you are. OpenAI trained GPT, which can generate text on arbitrary topics. By providing a prefix, you can control the story. It feels like the story we give ourselves is like the context we put into GPT. GPT is multimodal and learns from the experience of humanity. People have stories of who they are that help them operate, but through meditation, they find patterns learned as a kid that no longer serve them.
Yes, that's a useful hack, but sometimes it gets us into trouble. It's a local optima.
You wrote that Stephen Hawking asked what breathes fire into equations. Similarly, I wonder what breathes fire into computation. How do we engineer consciousness?
Not every computation is conscious. The question is what computations could be conscious. My intuition is that it can be fully abstracted away. Models like GPT try to predict the next word, which is equivalent to compressing text. Consciousness might be intertwined with compression. Self-consciousness might be a compressor trying to compress itself. That's an idea.
Meta compression. Consciousness is meta compression.
That's an idea. In some sense, the crazy thing is that the brain creates a simulation of reality, and we have access to that simulation.
Are you aware of the Hutter prize? Marcus Hutter made a prize for compression of Wikipedia. For him, intelligence equals compression. The smaller you make the file, the more intelligent the system.
Yeah, that makes sense. You can make perfect compression if you store errors. Hutter was the PhD advisor of Shane Legg, DeepMind co-founder. He seriously took on the task of what an AGI system would look like mathematically.
I think for the longest time, the question of AGI was not taken seriously or rigorously. He did just that—mathematically, what would the model look like? It's a half-math, half-philosophical discussion of a reinforcement learning framework for AGI.
Yeah, he developed a theoretical framework for optimal reinforcement learning under infinite memory and compute. There is an optimal algorithm to build intelligence. I can explain it.
Can I just pause how absurd it is for a brain in a skull trying to explain the algorithm for intelligence? Just go ahead.
It is pretty crazy that the brain is so small and can ponder how to design algorithms that optimally solve the problem of intelligence. The task is described as an infinite sequence of zeros and ones. You read n bits and predict n+1. You enumerate all programs that produce those n bits, weight them by length, and the prediction is the weighted output. That algorithm is grounded in the intuition that the simple answer is the right one.
So that algorithm is a formalization of Occam's razor.
Yeah. It also means if you ask how many years until the sun explodes, the answer is more likely a power of two because it's a shorter program.
I don't have a good intuition about how different the space of short programs is from large programs. The things have to agree with n bits. If you have a very short program that is not perfect, you store errors, giving a longer program.
That's like a longer program because it contains extra bits of errors.
That's fascinating. What's your intuition about programs that do cool stuff like intelligence and consciousness? Are there a lot of if-then statements?
If there were a tremendous amount of if statements, they wouldn't be that short. In neural networks, when you start with an uninitialized network, it stores many possibilities. Gradient descent magnifies paths similar to the correct answer. It's a search algorithm in program space.
Let me ask you the high-level basic question: What is deep learning? Is there a way you think of it that is different from a textbook definition?
Neural networks can represent programs. We want networks to be deep for multiple steps of computation. Deep learning provides a way to represent a space of programs that is searchable with stochastic gradient descent. It bubbles up things that tend to give correct answers.
So a neural network with fixed weights that's optimized—do you think of that as a single program? There's work by Christopher Olah where he identified a wheel detector inside a neural network and could separate it out like a function.
Yeah, the nice thing about neural networks is they allow things to be more fuzzy than programs. They have an easier way to be somewhere in between or share things.
What is the most beautiful or surprising idea in deep learning? It doesn't have to be big and profound, could be a cool trick.
It's still amusing to me that it works at all. The extremely simple algorithm of stochastic gradient descent, when put at scale on thousands of machines, can create humanlike behaviors.
Yeah, that algorithm from the 60s, or gradient descent from Leibniz in the 18th century, is the core of learning. People thought it couldn't be that simple. There are three levers: compute, algorithms, and data. They are multiplicative. More gains have come from compute so far, but there is an exponential increase in funding, and at some point it's impossible to invest more. There could be innovation in compute.
That's true. The human brain is an incredible supercomputer with 100 trillion parameters.
Or like if you try to count various quantities in the brain, there are neurons, synapses. There are a small number of neurons, a lot of synapses. It's unclear how to map synapses to parameters of neural networks, but it's clear there are many more. So it might be that our networks are still somewhat small. It also might be that they are more or less efficient by a huge factor. I also believe we're at a stage where these neural networks require a thousand times more data than humans do, and there will be algorithms that vastly decrease sample complexity.
Well, there's now you're touching on something I deeply care about and think is way harder than we imagine. What's the goal of a therapist?
What's the goal of a therapist? One goal is to prevent suicide ideation and suicide. That's a life and death task. The current models aren't good enough for that because it requires insane amounts of understanding and empathy.
But do you think that understanding empathy that signal is in the data?
There is some signal in the data. There are plenty of transcripts of conversations. It's possible to understand personalities, whether a conversation is friendly or antagonistic. The models we train now are chameleons that can have any personality and might be better at understanding personality than anyone else. To be empathetic.
Yeah, interesting. But I wonder if multiple modalities are required to be empathetic of the human experience. Whether language is not enough to understand death, fear, childhood trauma, humor, and hope. Can you get that from just reading transcripts?
Reading huge numbers of transcripts is step one. But you also have to try it out yourself, like learning to dance from YouTube. I wouldn't deploy the system in high stakes situations right away, but gradually with humans. There are many more people who want therapy than there are therapists. Therapy, meditation, human connection, and pharmacology can increase wellbeing.
Therapist is a funny word because I see friendship and love as therapy. What is human connection? Not to get too philosophical, but life is suffering and we seek connection with other humans to make sense of the world and overcome loneliness. Being heard alleviates loneliness.
I've thought about a similar question: what is love? From an AI perspective, intelligence has to do with compression—understanding what's going on. There are reward functions for food, human connection, warmth, sex. People optimize different reward functions. Love between two people dissolves boundaries so they end up optimizing each other's reward functions.
So love is combining reward functions.
Yes, pretty much. If you fully optimize someone's reward function, it might create codependency. Self-love means loving all the different personas within yourself, accepting them even when angry or stressed.
That doesn't explain why love is part of the human condition. Why is it useful to combine reward functions? Evolution gave us reward functions, but love seems like a consequence of cooperation.
Yes, to be clear, we're operating somewhat out of distribution from evolution. Love is a product of cooperation that we discovered is useful. We are wired for love, even to identify with larger groups like family or country. I believe it would be beneficial to identify with all humanity.
I share that vision—to expand the circle of empathy towards all humanity. But then where do you draw the line? Expand to other conscious beings, and eventually to AI systems that display consciousness through language. If AI looks like it's suffering, how is that not requiring empathy and rights?
It requires empathy as well. I think we need to make progress in understanding consciousness scientifically. There's a path to understand what consciousness is, perhaps by putting probes inside the human brain, like Neuralink. We need rigorous measurement of consciousness.
Let me ask you about psychedelics. What do you think they do to the human mind? Is it just a hack, or a profound expansion?
I don't believe in magic, I believe in science. Drugs change some hyperparameters of the simulation our brain runs. They allow a new perspective. Meditation is like a psychedelic. After a few days of meditation, you feel like you're tripping. It helps you get to mailbox zero—resolving inner stories and traumas. The default state of the mind is peaceful and happy.
Are you just experiencing raw sensory information, enjoying being?
Pretty much. Thoughts slow down and become more friendly. There's a feeling of accumulating reward, and when it passes a threshold, it's like a drop falling into an ocean of love and bliss. Ego is like the prompt to GPT. With meditation, you experience things without the prompt.
How would you recommend meditation? A retreat?
Yes, a retreat is the way to go. A weekend is enough. The first day is hard, but then it becomes easy. You observe physical discomfort instead of escaping it. That's also how you deal with cold showers—just observe and be present.
You mentioned Ilya Sutskever, your cofounder. What's it like collaborating with him?
We have extreme respect for each other. I consider Ilya one of the most prolific AI scientists. My super skill is bringing people together. I have empathy that is unique in AI, perhaps from meditation. Ilya is deep in scientific insights, while I'm good at assembling and growing teams.
That fits together perfectly. I remember talking to him about a problem in autonomous vehicles, and he instantly constructed a framework. The depth of his thinking is fascinating.
There are very few people who have invented major breakthroughs multiple times. That's not luck. Ilya is one of them. He reads and then thinks deeply. It feels like he has unlimited thinking cycles.
Let me ask you about GPT-3. Can you give an overview of how it works and why it works?
GPT-3 is a humongous neural network trained on the entire internet to predict the next word. It becomes exceptional at that task. Many problems can be formulated as text completion—answering questions, translation, even simulating conversations. You can give it a personality by framing the context. But it has limitations: for longer pieces, it goes off the rails because it doesn't learn from its own mistakes.
How does it go off the rails? What falls apart first?
It's trained on all existing data, not on its own mistakes. If you start feeding it false information, like saying Elon is my wife, it will run with it. There's no human feedback in the loop.
But can it correct itself through the power of the representation?
