In this conversation with Wojciech Zaremba, co-founder of OpenAI, you'll learn how they transformed early stage artificial intelligence technology into something that changed the world. This conversation was originally conducted in Polish, Wojciech Zaremba's native language. The following is an English version with a voiceover.
Do you remember the day in November 2022 when you released ChatGPT to the world?
Yes I remember it even happened on my birthday. I think this is the wrong way of thinking about this technology. It is AGI; it is a class of technology. It is just like a computer, so we will never have a single point where we say "Oh, there it is, OpenAI has released AGI."
Yes, you'll get an inside look at his work on AI and see the huge impact he's had on the development of this technology. Artificial intelligence will solve many of the problems we have today, but it will also create new problems that we haven't had before.
What will the world of technology look like? Where will artificial intelligence take us? What impact will it have on humanity? These are the questions Wojciech Zaremba addresses in this conversation.
Have you ever thought about the fact that in November 2022, you kind of opened a bit of a monster?
Thanks for this meeting in San Francisco. We appreciate your time. Absolutely, I'm glad to be here.
Do you agree that AI is still just statistics?
No. There are a few things; it actually depends on the interpretation of what "only statistics" means. It is known at this point that if you look at, for example, problems that require reasoning which the model has never seen, it is capable of solving these problems. It's not like this model memorized something exactly the same elsewhere. You can see in many, many cases that these models, so-called, generalize, meaning they were trained on certain data and are able to behave intelligently on very different data. So I think it's not just statistics. It really depends on how you define it. You could say the human brain is statistics, but if that is statistics, it's really unbelievable; it's magic.
Magic. Well, I could say it's interesting. What's interesting in the case of these neural networks is that we have some empirical understanding that if we provide this amount of data, these will be the results, although there is no very deep theoretical understanding of why this happens. Many models tend to memorize this data, and when slightly different data is used as input, these models essentially stop working.
I can say imagine the following situation. Let's assume, for simplicity, that you have training data. Let's assume they are just digits, the background is black, and now you have trained a model on such data. Now you want to test them on data where you only change one pixel in the corner. Instead of a black pixel, you put a white pixel. It turns out that this model works well in such a case. In fact, despite seeing a black pixel in this corner on all the data, you change it to white, and this neural network model still works properly. It will be able to recognize the numbers even though you change that pixel. However, from a theoretical perspective, we do not have an explanation for why this happens. The thing is, at the moment you change that pixel, the data is completely different from the data you trained on.
What is the reason that the data you are testing on can be so different from the data we trained on, and it still works? Are you trying to say that some kind of interpretation process is happening there?
I can say it's like we know many algorithms such as decision trees or nearest neighbors, where it's very easy if you stray a bit from the training data, they fall apart. In the case of the transformer, this does not happen. People have an intuition about it, and we have seen this repeatedly in various empirical data experiments. We know the trends of this, but it is not so deeply understood; there is no theoretical explanation of where it comes from.
Let's talk a bit about borders in general. The first boundary is that we perceive the world spatially in three dimensions. We understand this fourth dimension; we feel it a little, but for years we have fed the data exclusively to models only with image data, and we reached the point where models do not understand context. We are in San Francisco, where a few months ago an autonomous Cruise taxi was set on fire in Chinatown. This happened not because the car harmed anyone participating in the procession, since the model knows very well it must stop at an appropriate distance from people, but because it violated the sacred space by getting too close. When I'm driving and see a Corpus Christi procession, I know I cannot drive right up to the last person as that would violate the sacred space. However, models which are trained only on visual data do not understand such cultural contexts. Isn't this what distinguishes us from AI today?
When we talk about AI, there are many AIs and they are trained on different training data. In the case of self-driving cars, there is indeed a huge amount of data on which they are trained, and this data comes from the driving of other cars or road image annotations. Actually, their main role is to avoid collisions; this is their way of understanding the world. There are also many models trained on the diversity of all data, including images, text, even video generation, sound generation, and understanding voice. It seems to me that if the model is aware of data coming from truly multimodal sources, it will have a much deeper understanding. One can also imagine that there might be a situation where it doesn't understand something because it's not recorded in any way in the existing data, but there is some degree of triangulation. It's not like it has to see every single case to understand it. You can even take a picture of the procession and upload it to one of the models, to ChatGPT or to Gemini, send the photo and say, "In the case of a self-driving car, where would you stop?" and you'll see whether it answers you properly or not.
But this is still not context in my opinion. It is still not contextual understanding because context also means understanding a moment with all the subjectivity of that moment and the analysis of data we gather through our senses, which models do not have today. Don't you have the feeling that we are creating something different, some completely different quality that relies on data acquired in a completely different way, and that by using the word "awareness," that it is conscious intelligence, we are misleading?
I think there are a few differences, and I can say what the major differences are in my opinion at this moment. Currently, the training method involves gathering a huge amount of training data, training on the largest cluster available to someone, and then there is the separate process of so-called inference or testing, where the model is evaluated. So this model receives an image or text as input and provides its interpretation. One could say that currently, in the case of computers, the training and testing process are two separate processes, whereas in the case of the human brain, it is a single process. And this can be a significant difference. It is also the case, I would say for humans, technically it's called on-policy or off-policy, which means in the case of humans, we train based on data from our own experience. So we operate in the world, and sometimes good things happen to us, sometimes bad things; we overhear something, we read something; we are training on our own data. However, the computer is training on data from another person. It's not like this computer has walked around the house, fallen over, hurt its leg, and will now be careful on the stairs only because someone recorded that moment and shared it with them. If you look at the cube, at that hand that was solving the Rubik's cube, or at the game like Dota or StarCraft, the model is exactly trained there with reinforcement learning, meaning based on its own experience.
The analogy could be roughly as follows: if this car, based on the fact that it was burned there, learned next time to avoid that procession, it would literally learn from that data. What is most likely to happen at this moment is that a number of people will look at this type of data, or they will generate a number of data cases similar to this procession, and once again it will be put into such a large training set, and then the subsequent generations, or after a new model deployment following training, will this model actually avoid this procession at an appropriate distance? While there will still be some other issues it will have, and if they are not proactively addressed—for example if data is not collected around them—it will keep making mistakes.
Don't you feel that the limit of AI development is physics, our knowledge of the world around us, because we are not able to feed models with data about things we don't understand and don't know? For example, we don't know what the source of gravity is, we don't know what connects the classical and quantum worlds, we don't know how human consciousness arises. As a result, we cannot capture these processes with data. We have these puzzles, but they are not yet complete. How can we create an accurate model of reality if we don't know how that reality works?
We have two main processes for how we can input knowledge into models. One of them is called pre-training, where we collect a large amount of data, for example human data from the Internet, where we train the model to predict the next word. In this case, the model is really learning to some extent to grasp what we have already understood so far. While, as you say, it would be hard to get to this point for the model to understand something that we do not understand—although it is also not certain, and I will tell you right away why it is not necessarily certain that such a thing is impossible. Another process we have in training is the so-called reinforcement learning, where we give the model a reward for appropriate behavior. In the case of reinforcement learning in limited domains such as Go, we were able to train models that came up with moves in Go that people playing this game for thousands of years were not able to invent. But they invented moves based on principles that it knew. I'm talking about the moment when the model was not fed the rules of this game, because here we fed it rules, but we don't know the rules of how the world works.
At this moment at OpenAI, we are considering the following classification of what we see as the levels of AI or AGI development, and I will clarify a bit when I think the model will be able to even come up with something we don't know. The situation is such that we recognize five levels.
Level number one is such that the model can have a conversation with you. It is really at the level where the model passes the Turing test. The Turing test is a test in which a person cannot distinguish whether they are talking to another human or talking to a computer. And now it turns out that in the case of current language models, they are actually already at such a level that it becomes quite difficult for a person to tell whether they are talking to a person or a computer. Perhaps a person can determine whether they are speaking with a human or a computer based on the response delay, even if it's not directly related to the content.
Now the second level, that we believe will come very soon, is thinking about such models that are capable of solving problems that require 10 minutes of reasoning. For example, there is a math problem that I would need to think about because I cannot immediately say what the answer is. This is more or less another level, which really turns out to be significantly different from the first level. When you think about the first level, it's a model you tell it something and it immediately responds to you. In the second case, in terms of reasoning, you have to consider different paths; you need to deeply understand what these words mean in this task, what the problem really is. It will be such that we will have models that in various fields like mathematics, physics, biology, computer science, are able to solve tasks that are non-trivial. It's not always about confirming and searching for evidence to support claims, but it's about solving non-trivial tasks. I could say even the evidence of small theorems too.
Now the third level we are considering. This is the level where we will have models called agents that are capable of performing longer tasks in the world. So for example, you tell the model, "Listen, I would like you to make a website for me." And now it starts buying some domain, starts writing in Heroku, has some Heroku server reserved, starts writing code, starts uploading the code. At the very end, in the middle of this work, they start sending you some mockups and ask you which one you want, maybe even sending you an email. You know, two hours later, it happened after you asked that. The model processing all the time processes, also performs actions in the world. It is a significant difference that in the case of ChatGPT, if you tried even at this moment to make it perform actions in the world—and more than one person tried to implement something like this—it would get lost quite quickly and would not be able to move forward. In the case of agents, it seems to us that we will reach models that are able to solve tasks that take hours or days. These are still tasks at a human level. You were asking about something beyond the human level. This is still human level. It involves browsing the internet, gathering information, writing code, creating some visualizations, putting it all together, and doing it very specifically.
Now the fourth level we consider is the scientist. It is such that a scientist spends months thinking about topics that other scientists have worked on for even decades, looking from different perspectives. Sometimes it turns out that our assumptions were wrong. As a scientist, you have a lot of considerations that are based on certain assumptions. Just like in the case of Einstein, it was like he realized that maybe the assumption that time is constant, that time is continuous and shared on the internet as only one timeline, turned out to be... it was an assumption that no one expected could be an assumption. Surely we have some assumptions that even block us from new discoveries. And that will be level four, really.
And the fifth, the fifth level is when AI is competent enough to manage entire organizations. So for example, you have a company that employs, for instance, 1,000 people. Regarding artificial general intelligence, we realize that people think about such a point that when it is reached, we can say "This is artificial general intelligence." I think this is the wrong way of thinking about this technology. It is AGI. It is a class of technology. It is just like a computer. You have different kinds of computers: small, large, in watches, in phones. And that's it. We will never have AGI. So we will never have a single point where we say "Oh, there it is, OpenAI has released AGI." Yes, for example, even when looking at ChatGPT, it is such that at this moment in ChatGPT, it passed the Turing test. The Turing test was historically considered a test that indicates that machines are intelligent. It turned out that ChatGPT passed the Turing test, and no one even noticed. There was no big announcement about it. And it really depends somewhat on the definition of what you consider AGI.
We are now in a situation where, for example, you have the task for the model to write a poem where every first letter in each line is 'A'. For me, it would be very difficult to write such a poem. I can say this problem becomes truly superhuman. It's like it's very hard to write such high-quality poems. At this point, models are easily able to do something like this, and in some way it's superhuman. But the thing is that you have a lot of these types of competencies, and now these models increase competence in all fields. In some areas, the human level will be achieved faster than in others. However, once a human level capability is achieved in all fields, there will be areas where that level was superhuman a long, long time ago.
When will it happen? When will we reach level five?
It's hard to say. Actually, many things depend a bit. I could say from my perspective, 99% that it will be shorter than 10 years. Also, even when looking at the whole stage of evolution, even of organisms, these single-celled organisms existed for a billion years, then multi-celled organisms existed for a bit shorter. So the time when only they existed, that time of evolution between multi-celled organisms was shortening. Even when you look at the history of human development, we are in a similar situation. Homo sapiens has existed for 200,000 years. The first cities appeared 20,000 years ago. Industrialization appeared 300 years ago. Computers appeared 60 years ago. The Internet appeared 30 years ago. So each of these stages shortens, and it seems to me that something similar will happen here too.
Sure. One needs to consider that among the elements that might still influence how long it takes are, for example, regulations and how it integrates with society.
I can say that this will be a non-trivial integration of this technology with society.
We'll talk about this later, but I would like to present you with a certain idea. Let's imagine we are in Africa on the savannah and there's a wild cat walking around. We take it from this savannah, put it in the laboratory, and we create an artificial breed of cat by combining it with a domestic cat. So what are we doing? We are copying a fragment of the theory of evolution, of the principles of natural selection. We have created a breed that has no genetic defects. We have adhered to all principles of precision and biology, including molecular biology. And we want to release this cat onto the savannah. We are releasing it, and for the most part, these synthetic breeds of cats would not be able to survive there. They wouldn't be, because we don't know what environmental factors influenced the emergence of this particular cat in this particular environment during the evolutionary process. Isn't it true that today we create models by copying only certain fragments of factors that influence what we call consciousness or what we call humanity? Although I don't like that word because no one knows what it is. Consciousness too, no one knows, but a little more. Don't you have the feeling that it's a bit like creating artificial animal breeds? At this moment, we have models that can replicate some human values. Within consciousness, it's hard to say to what extent these models are conscious or not, and it's truly a separate philosophical question, or at some point it will be a technical question. No one even knows how to approach this technically.
We'll talk about consciousness soon, but I mean something completely different. I just mean that it's flawed, that what we create is flawed. A perfect example is also that the human brain needs 20 watts, while a language model only needs 10. That no matter how much we try, we can't match this biology, that it's so very flawed.
It's a bit like when you compare birds with an airplane. A bird is very light, it also flaps its wings, and it can be acrobatic. It's even incredible how a bird can fly through a tree. An airplane is heavy, it can weigh many tons, and has some components that are shared with the bird, but at the same time it is different. It's so different that you can fly 400 people across the Atlantic with this plane. So it seems to me that we'll be in a similar situation as humans. There are some differences, like how economical the brain is, how little energy it can use, how little data it can process. While on the other hand, we might be able to make these models more like the human brain. The human brain is efficient because our DNA contains a lot of information about how to efficiently utilize reality. You could say that this DNA contains training data which is the result of this DNA being trained on billions of people.
So you mean to say that we can reach a similar point but by a different route than evolution?
Historically, even at OpenAI, people considered building artificial intelligence as something more akin to evolution. And if you look at the amount of computational power that evolution utilized, it is incredibly powerful, much more than what is used now. Evolution used a great deal of computational power over billions of years, and the computer is the entire Earth. Thanks to evolution, millions of different species emerged, and intelligence arose as a result of such powerful optimization. Intelligence was discovered multiple times across different species. With these neural networks, there is a training phase. It doesn't even matter that it requires a large amount of data or computational power. As a result of this, you get a model with the incredible ability to learn fairly quickly from a single conversation, even based on its own mistakes. My nephew was explaining Polish grammar to an even older model, and the model was able to grasp it within a single conversation. There is one stage where a very large amount of data is needed, and then we move on to the point where the model can catch on very quickly and learn many things within a single conversation. Ultimately, we would like to have a model that if given a new problem—it will solve it. For example, the problem might be global warming; there is no solution to it, but the model will start to think and it will be able to use a small amount of data to solve the problem.
How can we talk about AI consciousness when we don't know how consciousness emerges from matter? So we can't create a mathematical formula that describes the process of the birth of consciousness and an algorithm that describes it.
Maybe I'll start with a definition, so we can talk about the same thing. In terms of how we perceive reality, how light enters our eyes, whether touch is transmitted through our nerves—all the information goes to the brain as bits. Now the fact that this brain is in our head, really from the brain's perspective, it wouldn't be distinguishable if it were sitting in a jar in the basement and the same bits were coming in through a cable. The interesting thing is that this brain is never able to touch reality. It only sees the bits that enter the brain, and the brain must create a simulation of reality.
An immersive image of the world, exactly.
And consciousness is our experience of this simulation. And a philosophical question is why a person has such an experience of this simulation. One can imagine a philosophical experiment called a philosophical zombie, where there would be a person who behaves just like any other human, except they don't have that internal cinema, that internal simulation. You ask me how one could check if artificial intelligence had something like that. I have two main ideas that I can share with you, and I can say what might be an important element for this awareness to arise.
We trained models in a 3D world some time ago, models that collect apples or some points in a computer game. In the case of models, we can even understand or visualize what the model sees. Now it turns out that initially, when you start training this model, it can distinguish very simple things: it can tell the difference between the sky and the ground. After some time, it starts being able to tell where the apples are that it's running to. The interesting thing is that we can even ask it what it thinks it will see if it turns its head. It first starts imagining what it sees. It's quite interesting that at the very beginning, even if you ask it what it sees when it turns around 360 degrees, it thinks it'll see something different. But at some point, it begins to realize that when it turns around 360 degrees, it'll see the same thing. This means it starts to understand 3D reality better. One interesting thing is that at some point in such a simulation, this model must begin to simulate its own existence because it itself participates in changing reality. Just like at the very beginning, this model understands that there is a sky, earth, it understands where the apples are because they are crucial for survival. At a certain point, something clicks and it starts to understand the physical reality, and then again, something clicks.
What is that clicking? Is it a data range?
In the case of neural networks, their training leads to a representation where they can solve various types of tasks. This representation initially considers simple elements that become more complex, and at some point, it even begins to consider the existence of the agent itself within the simulation. Initially, this agent was not even in this simulation. Its understanding of reality was so small that it was not even aware of it. And at some point, it appeared in its own simulation. And I could say that this might be a moment of self-awareness, although it might differ slightly from it.
And what if we assume that consciousness is a quantum effect? If it were understood what consciousness is, then it would be possible to build something like that. If we are not able to understand consciousness, then it is hard to build it. Penrose strongly suggested that it is due to a quantum effect.
Two experiments come to my mind that could indicate whether models have consciousness. One experiment would be something like: first, we take all the model's training data and eliminate any mention of consciousness. We don't talk about it at all. We are training the model on such data. So now the question is whether this model will suddenly be able to—you know, when you have a conversation with it on this topic—to say, "Yes, well I noticed that something is in such a style... No, no, I was wondering about this topic. I didn't know how to talk about it. It's kind of strange to feel this way." Well, if it were the case that there was nothing like this in the training data and suddenly the model started mentioning that they had this type of experience, that would be a hint that they might actually have consciousness. That's one possibility.
Another possibility that comes to my mind is something like if AI were connected to the brain and a person would have the experience that their consciousness expands as a result of it. However, the second experiment has some drawbacks. For instance, in the case of psychedelics, a psychedelic could be administered to a person. The psychedelic itself is not conscious, and the person says, "Oh my consciousness has expanded." So it's not necessarily true that if a neural network didn't have consciousness, it could be that the neural network doesn't have consciousness while connecting it to the brain gives the experience of increasing that consciousness.
In my opinion, the standard way of training these models looks like this. First, we train it on all the data from the Internet so that in every article it predicts the next word. It turns out that this is a way to instill a large amount of knowledge into the model; there are even fundamental reasons why. Now the second stage is the so-called post-training stage, where a person looks at various model responses and says, "I like this answer more than that one." In the case of post-training, the model is trained to provide more answers that people like.
In the case of hallucinations, the problem is that the model may speak confidently about things it doesn't know. So where does this come from? If we look at how this model is trained with humans, the human will reward the model for explaining in a way that the human likes. Rarely will he be given a reward for saying "I don't know." It may be that he gave several answers, and in one of those answers, he guessed, and the person chose, "Oh I like this answer; this answer is actually correct." And the model guessed. And now the model is being trained to always provide an answer even when it doesn't know. If he doesn't know, then let him guess, because when a person gave preferences, they preferred an answer when the model guessed rather than when the model said it didn't know.
So could the cause of hallucinations be this boundary between knowledge and ignorance? When a person trains a model, it might be the case that the person doesn't know what the model knows or doesn't know. He says, "I always like it when you give me an answer." And we train the model so that it always provides an answer, regardless of whether it knows or not. So during the current training, as this feedback is given by the person at the last stage, this feedback leads to the model responding regardless of whether it has this knowledge or not.
So how are you going to deal with this?
