The following is a conversation with Greg Brockman. He's the co-founder and CTO of OpenAI, a world-class research organization developing ideas in AI with a goal of eventually creating a safe and friendly artificial general intelligence. One that benefits and empowers humanity. OpenAI is not only a source of publications, algorithms, tools, and data sets. Their mission is a catalyst for an important public discourse about our future with both narrow and general intelligence systems. This conversation is part of the artificial intelligence podcast at MIT and beyond. If you enjoy it, subscribe on YouTube, iTunes, or simply connect with me on Twitter at Lex Fridman, spelled F-R-I-D. And now, here's my conversation with Greg Brockman.
So in high school and right after you wrote a draft of a chemistry textbook, I saw that that covers everything from basic structure of the atom to quantum mechanics. So it's clear you have an intuition and a passion for both the physical world with chemistry and now robotics to the digital world with AI, deep learning, reinforcement learning, and so on. Do you see the physical world and the digital world as different and what do you think is the gap?
A lot of it actually boils down to iteration speed. A lot of what really motivates me is building things. Think about mathematics, for example, where you think really hard about a problem, you understand it, you write it down in this very obscure form that they call a proof, but then this is in humanity's library — it's there forever. This is some truth that we've discovered, and maybe only five people in your field will ever read it, but somehow you've kind of moved humanity forward. I actually used to really think that I was going to be a mathematician, and then I actually started writing this chemistry textbook. One of my friends told me you'll never publish it because you don't have a PhD, so instead I decided to build a website and tried to promote my ideas that way. And then I discovered programming — in programming you think hard about a problem, you understand it, you write it down in a very obscure form that we call a program, but then once again it's in humanity's library, and anyone could get the benefit from it, and the scalability is massive. The thing that really appeals to me about the digital world is that you can have this insane leverage — a single individual with an idea is able to affect the entire planet. And that's something I think is really hard to do if you're moving around physical atoms.
But you said mathematics. So if you look at the wet thing over here, our mind, do you ultimately see it as just math, as just information processing, or is there some other magic as you've seen through biology and chemistry and so on?
I think it's really interesting to think about humans as just information processing systems, and it seems like it's actually a pretty good way of describing a lot of how the world works or a lot of what we're capable of. If you just look at technological innovations over time, in some ways the most transformative innovation that we've had has been the computer, in some ways the internet. What has the internet done? The internet is not about these physical cables — it's about the fact that I am suddenly able to instantly communicate with any other human on the planet. I'm able to retrieve any piece of knowledge that in some ways the human race has ever had, and those are these insane transformations.
Do you see our society as a whole, the collective, as another extension of the intelligence of the human being? So if you look at the human being as an information processing system, you mentioned the internet, the networking — do you see us all together as a civilization as a kind of intelligence system?
I think this is actually a really interesting perspective to take, to think about that you sort of have this collective intelligence of all of society. The economy itself is this superhuman machine that is optimizing something. In some ways a company has a will of its own — you have all these individuals who are all pursuing their own individual goals and thinking really hard and thinking about the right things to do, but somehow the company does something that is this emergent thing, and it's a really useful abstraction. In some ways, we think of ourselves as the most intelligent things on the planet and the most powerful things on the planet, but there are things that are bigger than us that are these systems that we all contribute to.
If you've read Isaac Asimov's Foundation, there's this concept of psychohistory in there, which is effectively this — that if you have trillions or quadrillions of beings, then maybe you could actually predict what that huge macro being will do, almost independent of what the individuals want. I actually have a second angle on this that I think is interesting, which is thinking about technological determinism. One thing that I actually think a lot about with OpenAI is that we're coming onto this insanely transformational technology of general intelligence that will happen at some point. And there's a question of how can you take actions that will actually steer it to go better rather than worse. One question you need to ask is as a scientist, as an inventor, as a creator, what impact can you have in general? You look at things like the telephone invented by two people on the same day — what does that mean about the shape of innovation? Everyone's building on the shoulders of the same giants, and so you can't really hope to create something no one else ever would. If Einstein wasn't born, someone else would have come up with relativity — he changed the timeline a bit, maybe it would have taken another 20 years, but it wouldn't be that fundamentally humanity would never discover these fundamental truths.
So there's some kind of invisible momentum that some people like Einstein or OpenAI is plugging into that anybody else can also plug into, and ultimately that wave takes us into a certain direction. That's what you mean by...
That's right. That's right. This kind of seems to play out in a bunch of different ways — that there's some exponential that is being ridden, and the exponential itself changes. Think about Moore's Law — an entire industry set its clock to it for 50 years. How can that be? How is that possible? And yet somehow it happened. You can't hope to ever invent something that no one else will — maybe you can change the timeline a little bit — but if you really want to make a difference, the only real degree of freedom you have is to set the initial conditions under which a technology is born.
Think about the internet — there were lots of other competitors trying to build similar things and the internet won, and the initial conditions were that it was created by this group that really valued anyone being able to plug in, this very academic mindset of being open and connected. The internet for the next 40 years really played out that way. Maybe today things are starting to shift in a different direction, but those initial conditions were really important to determine the next 40 years worth of progress.
That's really beautifully put. Another example of that I think about — I recently looked at the formation of Wikipedia and I wonder what the internet would be like if Wikipedia had ads.
There's an interesting argument about why they chose not to put advertisements on Wikipedia. I think Wikipedia is one of the greatest resources we have on the internet. It's extremely surprising how well it works and how well it was able to aggregate all this good information. Essentially the creator of Wikipedia — there's probably some debates there — set the initial conditions and now it carried itself forward.
That's really interesting. So the way you're thinking about AGI or artificial intelligence is you're focused on setting the initial conditions for the progress.
That's powerful. Looking into the future, if you create an AGI system, like one that can ace the Turing test, natural language, what do you think would be the interactions you would have with it? What do you think are the questions you would ask? What would be the first question you would ask it — her, him?
At that point, if you've really built a powerful system that is capable of shaping the future of humanity, the first question that you really should ask is: how do we make sure that this plays out well? That's actually the first question that I would ask a powerful AGI system.
So you wouldn't ask your colleague, you wouldn't ask like Ilya, you would ask the AGI system.
Oh, we've already had the conversation with Ilya, and everyone here. You want as many perspectives and a piece of wisdom as you can for answering this question. So I don't think you necessarily defer to whatever your powerful system tells you, but you use it as one input to try to figure out what to do. Fundamentally what it really comes down to is if you built something really powerful — think about the creation of nuclear weapons, right — the most important question in the world was what's the world order going to be like, how do we set ourselves up in a place where we're going to be able to survive as a species. With AGI I think the question is slightly different — there is a question of how do we make sure that we don't get the negative effects, but there's also the positive side.
Imagine what will AGI be like, what will it be capable of? I think one of the core reasons that an AGI can be powerful and transformative is actually due to technological development. If you have something that's as capable as a human and that is much more scalable, you absolutely want that thing to go read the whole scientific literature and think about how to create cures for all the diseases. You want it to think about how to build technologies to help us create material abundance and to figure out societal problems that we have trouble with — how are we supposed to clean up the environment. Maybe you want this to go and invent a bunch of little robots that will go out and be biodegradable and turn ocean debris into harmless molecules. That positive side is something that I think people miss sometimes when thinking about what an AGI will be like. If you have a system that's capable of all of that, you absolutely want its advice about how do we make sure that we're using your capabilities in a positive way for humanity.
So what do you think about that psychology that looks at all the different possible trajectories of an AGI system, many of which, perhaps the majority, are positive, and nevertheless focuses on the negative trajectories? You get to interact with folks, you get to think about this within yourself as well. You look at Sam Harris and so on. It seems to be, sorry to put it this way, but almost more fun to think about the negative possibilities, whatever that is deep in our psychology. What do you think about that? And how do we deal with it? Because we want AI to help us.
I think there's kind of two problems entailed in that question. The first is more of the question of how can you even picture what a world with a new technology will be like. Now imagine we're in 1950 and I'm trying to describe Uber to someone.
Yeah, I mean that's going to be extremely complicated, but it's imaginable, right? But now imagine being in 1950 and predicting Uber — you need to describe the internet, you need to describe GPS, you need to describe the fact that everyone's going to have this phone in their pocket. The first truth is that it is hard to picture how a transformative technology will play out in the world. We've seen that before with technologies that are far less transformative than AGI will be. It's just even hard to imagine and to really put yourself in a world where you can predict what that positive vision would be like.
The second thing is that it is always easier to support the negative side than the positive side. It's always easier to destroy than create — less in a physical sense and more in an intellectual sense. With creating something you need to get a bunch of things right, and to destroy you just need to get one thing wrong. What that means is that a lot of people's thinking dead-ends as soon as they see the negative story. But I actually have some hope — the positive vision is something that I think we can talk about. Simply saying the fact of there's positives, there's negatives, everyone likes to dwell on the negative — people actually respond well to that message and say, 'Huh, you're right. There's a part of this that we're not talking about, not thinking about.' That's actually something that's been a key part of how we think about AGI at OpenAI. You can look at it as: OpenAI talks about the fact that there are risks and yet they're trying to build this system.
Like how do you square those two facts? So do you share the intuition that some people have — from Sam Harris to even Elon Musk himself — that it's tricky as you develop AGI to keep it from slipping into existential threats, into the negative? What's your intuition about how hard is it to keep AI development on the positive track?
To answer that question, you can really look at how we structure OpenAI. We really have three main arms. We have capabilities, which is actually doing the technical work and pushing forward what these systems can do. There's safety, which is working on technical mechanisms to ensure that the systems we build are aligned with human values. And then there's policy, which is making sure that we have governance mechanisms, answering that question of whose values. The technical safety one is the one that people talk about the most — think about all of the dystopic AI movies, a lot of that is about not having good technical safety in place. What we've been finding is that actually a lot of people look at the technical safety problem and think it's just intractable — this question of what do humans want, how am I supposed to write that down? Can I even write down what I want?
And then they stop there. But the thing is, we've already built systems that are able to learn things that humans can't specify. Even the rules for how to recognize if there's a cat or a dog in an image — turns out it's intractable to write that down. And yet, we're able to learn it. What we're seeing with systems we build at OpenAI, and they're still in early proof-of-concept stage, is that you are able to learn human preferences. You're able to learn what humans want from data. That's kind of the core focus for our technical safety team, and I think we've had some pretty encouraging updates in terms of what we've been able to make work.
So you have an intuition and a hope that from data, looking at the value alignment problem, from data we can build systems that align with the collective better angels of our nature, to align with the ethics and the morals of human beings. To even say this in a different way — think about how do we align humans, right? Think about a human baby can grow up to be an evil person or a great person. And a lot of that is from learning from data, right? You have some feedback as a child is growing up. They get to see positive examples.
The only example we have of a general intelligence that is able to learn from data to align with human values and to learn values — I think we shouldn't be surprised that we can do the same sorts of techniques, or whether the same sort of techniques end up being how we solve value alignment for AGIs.
So let's go even higher. I don't know if you've read the book Sapiens, but there's an idea that as a collective, as us human beings, we kind of develop together ideas that we hold. There's no, in that context, objective truth — we just kind of all agree to certain ideas and hold them as a collective. So do you have a sense that there is, in the world of good and evil, do you have a sense that to the first approximation there are some things that are good and that we could teach systems to behave to be good?
I think this actually blends into our third team, which is the policy team. This is the aspect I think people really talk about way less than they should. Imagine that we build super powerful systems that we've managed to figure out all the mechanisms for these things to do whatever the operator wants. The most important question becomes: who's the operator? What do they want? And how is that going to affect everyone else? This question of what is good, what are those values — I think you don't even have to go to those very grand existential places to start to realize how hard this problem is. You just look at different countries and cultures across the world, and there's a very different conception of how the world works and what kinds of ways that society wants to operate.
The really core question is actually very concrete, and I think it's not a question that we have ready answers to: how do you have a world where all the different countries that we have — the United States, China, Russia, and the hundreds of other countries — are able to continue to not just operate in the way that they see fit, but in a way where the world that emerges, where you have these very powerful systems operating alongside humans, ends up being something that empowers humans more, that makes human existence be a more meaningful thing, and that people are happier and wealthier and able to live more fulfilling lives. It's not an obvious thing for how to design that world once you have that very powerful system.
If we take a little step back — we're having a fascinating conversation — OpenAI in many ways a tech leader in the world, and yet we're thinking about these big existential questions, which is fascinating, really important. I think you're a leader in that space and that's a really important space of just thinking how AI affects society in a big picture view. Oscar Wilde said, 'We're all in the gutter but some of us are looking at the stars,' and I think OpenAI has a charter that looks to the stars, I would say — to create intelligence, to create general intelligence, make it beneficial, safe, and collaborative. So can you tell me how that came about? How a mission like that and the path to creating a mission like that at OpenAI was founded?
I think that in some ways it really boils down to taking a look at the landscape. If you think about the history of AI, for the past 60 or 70 years, people have thought about this goal of what could happen if you could automate human intellectual labor. Imagine you can build a computer system that could do that — what becomes possible? We have a lot of sci-fi that tells stories of various dystopias, and increasingly you have movies like Her that tell you a little bit about maybe more of a utopic vision. You think about the impact that we've seen from having bicycles for our minds in computers — the impact of computers and the internet has just far outstripped what anyone really could have predicted. So I think that it's very clear that if you can build an AGI, it will be the most transformative technology that humans will ever create.
What it boils down to then is a question of: well, is there a path? Is there hope? Is there a way to build such a system? For 60 or 70 years people got excited and ended up not being able to deliver on the hopes that people had pinned on them. After two winters of AI development, people kind of almost stopped daring to dream — really talking about AGI or thinking about AGI became almost this taboo in the community. But I actually think that people took the wrong lesson from AI history.
If you look back starting in 1959, that's when the perceptron was released. This is basically one of the earliest neural networks. It was released to what was perceived as this massive overhype. In the New York Times in 1959 you have this article saying that the perceptron will one day recognize people, call out their names, instantly translate speech between languages. People at the time looked at this and said this system can't do any of that, and basically spent 10 years trying to discredit the whole perceptron direction and succeeded. All the funding dried up and people went in other directions.
In the 80s there was this resurgence, and I'd always heard that the resurgence in the 80s was due to the invention of backpropagation and these algorithms that got people excited, but actually the causality was due to people building larger computers. You can find these articles from the 80s saying that the democratization of computing power suddenly meant that you could run these larger neural networks and then people started to do all these amazing things. The backpropagation algorithm was invented and the neural nets people were running were these tiny little like 20-neuron neural nets.
Right, like what are you supposed to learn with 20 neurons?
Yeah. So of course they weren't able to get great results. It really wasn't until 2012 that this approach — that's almost the most simple natural approach that people had come up with in the 50s, in some ways even in the 40s before there were computers with the McCulloch-Pitts neuron — suddenly this became the best way of solving problems.
I think there are three core properties that deep learning has that are very worth paying attention to. The first is generality — we have a very small number of deep learning tools: SGD, deep neural nets, maybe some RL, and it solves this huge variety of problems: speech recognition, machine translation, game playing, all these problems. Small set of tools. There's a second piece which is competence — you want to solve any of those problems, throw out 40 years worth of normal computer vision research, replace it with a deep neural net. It's going to work better. And there's a third piece which is scalability — one thing that has been shown time and time again is that if you have a larger neural network, throw more compute, more data at it, it will work better. Those three properties together feel like essential parts of building a general intelligence.
Now it doesn't just mean that if we scale up what we have that we will have an AGI — there are clearly missing pieces, there are missing ideas, we need to have answers for reasoning — but I think the core here is that for the first time it feels that we have a paradigm that gives us hope that general intelligence can be achievable. And so as soon as you believe that, everything else comes into focus. If you imagine that you may be able to — and the timeline I think remains uncertain, but I think that certainly within our lifetimes and possibly within a much shorter period of time than people would expect — if you can really build the most transformative technology that will ever exist, you stop thinking about yourself so much. You start thinking about how do you have a world where this goes well, and you need to think about the practicalities of how do you build an organization and get together a bunch of people and resources, and to make sure that people feel motivated and ready to do it.
But then you start thinking about what if we succeed, and how do we make sure that when we succeed that the world is actually the place that we want ourselves to exist then — and almost in the Rawlsian sense of the word. So that's kind of the broader landscape, and OpenAI was really formed in 2015 with that high-level picture of AGI might be possible sooner than people think and that we need to try to do our best to make sure it's going to go well. Then we spent the next couple years really trying to figure out what does that mean, how do we do it. Typically with a company, you start out very small — it's you and a co-founder, and you build a product, you get some users, you get product-market fit, then at some point you raise some money, you hire people, you scale, and then down the road the big companies realize you exist and try to kill you. For OpenAI it was basically everything in exactly the opposite order.
Let me just pause for a second. You said a lot of things and let me just admire the jarring aspect of what OpenAI stands for, which is daring to dream. You said it, it's pretty powerful. It caught me off guard because I think that's very true — the step of just daring to dream about the possibilities of creating intelligence in a positive, in a safe way. But just even creating intelligence is a much-needed, refreshing catalyst for the AI community. So that's the starting point. Okay, so then the formation of OpenAI, what's...
I would just say that when we were starting OpenAI, the first question that we had is: is it too late to start a lab with a bunch of the best people? Is that even possible? That was an actual question, that was really the core question. We had this dinner in July of 2015 and that was really what we spent the whole time talking about. You think about where AI was — it had transitioned from being an academic pursuit to an industrial pursuit and so a lot of the best people were in these big research labs and we wanted to start our own one. No matter how much resources we could accumulate, it would pale in comparison to the big tech companies, and there's a question of are we going to be actually able to get this thing off the ground? You need critical mass — you can't just do you and a co-founder build a product. You really need to have a group of five to 10 people. And we kind of concluded it wasn't obviously impossible, so it seemed worth trying.
Well, you're also dreamers, so who knows, right?
Okay, so speaking of competing with the big players, let's talk about some of the tricky things as you think through this process of growing, of seeing how you can develop these systems at a scale that competes. You recently formed OpenAI LP, a new capped-profit company that now carries the name OpenAI. So OpenAI is now this official company. The original nonprofit company still exists and carries the OpenAI Nonprofit name. So can you explain what this company is, what the purpose of its creation is, and how did you arrive at the decision to create it?
OpenAI, the whole entity, and OpenAI LP as a vehicle is trying to accomplish the mission of ensuring that artificial general intelligence benefits everyone. And the main way that we're trying to do that is by actually trying to build general intelligence ourselves and make sure the benefits are distributed to the world. That's the primary way. We're also fine if someone else does this — it doesn't have to be us. If someone else is going to build an AGI and make sure that the benefits don't get locked up in one company or one set of people, we're actually fine with that. Those ideas are baked into our charter, which is kind of the foundational document that describes our values and how we operate. But it's also really baked into the structure of OpenAI LP.
The way that we've set up OpenAI LP is that in the case where we succeed — if we actually build what we're trying to build — then investors are able to get a return, but that return is something that is capped. If you think of AGI in terms of the value that you could really create, you're talking about the most transformative technology ever created. It's going to create orders of magnitude more value than any existing company. And that all of that value will be owned by the world, legally titled to the nonprofit to fulfill that mission. So that's the structure.
So the mission is a powerful one and it's one that I think most people would agree with — it's how we would hope AI progresses. And so how do you tie yourself to that mission? How do you make sure you do not deviate from that mission, that other incentives that are profit-driven don't interfere with the mission?
This was actually a really core question for us for the past couple years. The way that our history went was that for the first year we were getting off the ground — we had this high-level picture but we didn't know exactly how we wanted to accomplish it. Really two years ago is when we first started realizing in order to build AGI we're just going to need to raise way more money than we can as a nonprofit — we're talking many billions of dollars. So the first question is how are you supposed to do that and stay true to this mission. We looked at every legal structure out there and concluded none of them were quite right for what we wanted to do. And I guess it shouldn't be too surprising if you're going to do some crazy unprecedented technology that you're going to have to come up with some crazy unprecedented structure to do it in.
A lot of our conversation was with people at OpenAI — the people who really joined because they believe so much in this mission — thinking about how do we actually raise the resources to do it and also stay true to what we stand for. The place you got to start is to really align on what is it that we stand for, right? What are those values? What's really important to us? We spent about a year really compiling the OpenAI charter and that determines — if you even look at the first line item in there, it says that look, we expect we're going to have to marshal huge amounts of resources, but we're going to make sure that we minimize conflicts of interest with the mission. Aligning on all of those pieces was the most important step towards figuring out how do we structure a company that can actually raise the resources to do what we need to do.
I imagine OpenAI — the decision to create OpenAI LP was a really difficult one, and there was a lot of discussions as you mentioned for a year. There were different ideas perhaps, detractors within OpenAI, different paths that you could have taken. What were those concerns, what were the different paths considered, what was that process of making that decision like?
If you look actually at the OpenAI charter, there's almost two paths embedded within it. There is: we are primarily trying to build AGI ourselves, but we're also okay if someone else does it. And this is a weird thing for a company.
That's really interesting actually. There is an element of competition that you do want to be the one that does it, but at the same time you're okay if somebody else does it. And we'll talk about that a little bit — that tradeoff, that dance, that's really interesting.
I think this was the core tension as we were designing OpenAI LP, and really the OpenAI strategy is how do you make it so that both you have a shot at being a primary actor — which really requires building an organization, raising massive resources, and really having the will to go and execute on some really really hard vision. You need to really sign up for a long period to go and take on a lot of pain and a lot of risk. To do that normally you just import the startup mindset — how do we outexecute everyone, you have this very competitive angle — but you also have this second angle of saying that the true mission isn't for OpenAI to build AGI, the true mission is for AGI to go well for humanity.
How do you take all of those first actions and make sure you don't close the door on outcomes that would actually be positive and fulfill the mission? It's a very delicate balance. Going 100% one direction or the other is clearly not the correct answer. Even in terms of just how we talk about OpenAI and think about it, one thing that's always in the back of my mind is to make sure that we're not just saying OpenAI's goal is to build AGI — it's actually much broader than that. First of all, it's not just AGI, it's safe AGI. That's very important. But secondly, our goal isn't to be the ones to build it. Our goal is to make sure it goes well for the world. Figuring out how do you balance all of those and to get people to really come to the table and compile a single document that encompasses all of that wasn't trivial.
So part of the challenge here is your mission is I would say beautiful, empowering, and a beacon of hope for people in the research community and just people thinking about AI. So your decisions are scrutinized more than I think a regular profit-driven company. Do you feel the burden of this in the creation of the charter and just in the way you operate?
So why do you lean into the burden by creating such a charter? Why not keep it quiet?
It just boils down to the mission. I'm here and everyone else is here because we think this is the most important mission. Dare to dream, right?
Do you think you can be good for the world or create an AGI system that's good when you're a for-profit company? From my perspective, I don't understand why profit interferes with positive impact on society. I don't understand why Google that makes most of its money from ads can't also do good for the world, or other companies — Facebook, anything. I don't understand why those have to interfere. You can — profit isn't the thing in my view that affects the impact of a company. What affects the impact of the company is the charter, is the culture, the people inside, and profit is the thing that just fuels those people. So what are your views there?
I think that's a really good question and there's some real long-standing debates in human society that are wrapped up in it. The way that I think about it is just think about what are the most impactful nonprofits in the world. What are the most impactful for-profits in the world? It's much easier to list the for-profits. I think there's some real truth here that the system that we set up, the system for how today's world is organized, is one that really allows for huge impact. Part of that is that for-profits are self-sustaining and able to build on their own momentum. And I think that's a really powerful thing. It's something that when it turns out that we haven't set the guardrails correctly causes problems — think about logging companies that go and deforest the rainforest. That's really bad. We don't want that.
It's actually really interesting to me that this question of how do you get positive benefits out of a for-profit company is actually very similar to how do you get positive benefits out of an AGI. You have this very powerful system. It's more powerful than any human and is kind of autonomous in some ways — it's superhuman in a lot of axes — and somehow you have to set the guardrails to get good things to happen, but when you do, the benefits are massive. When I think about nonprofit versus for-profit, I think just not enough happens in nonprofits. They're very pure, but it's just hard to do things there. In for-profits, in some ways, too much happens. But if shaped in the right way, it can actually be very positive. So with OpenAI LP, we're picking a road in between.
Now the thing I think is really important to recognize is that the way that we think about OpenAI LP is that in the world where AGI actually happens — in a world where we are successful, we build the most transformative technology ever — the amount of value we're going to create will be astronomical. Then in that case, the cap that we have will be a small fraction of the value we create. The amount of value that goes back to investors and employees looks pretty similar to what would happen in a pretty successful startup. And that's really the case that we're optimizing for — thinking about in this success case, making sure that the value we create doesn't get locked up. I expect that in other for-profit companies it's possible to do something like that. It's not obvious how to do it. As a for-profit company, you have a lot of fiduciary duty to your shareholders and there are certain decisions that you just cannot make. In our structure, we've set it up so that we have a fiduciary duty to the charter — we always get to make the decision that is right for the charter, even if it comes at the expense of our own stakeholders.
When I think about what's really important, it's not really about nonprofit versus for-profit. It's really a question of if you build AGI and humanity is now in this new age, who benefits? Whose lives are better? And I think that what's really important is to have an answer that is everyone.
Yeah, which is one of the core aspects of the charter. So one concern people have, not just with OpenAI but with Google, Facebook, Amazon, anybody really that's creating impact at scale, is how do we avoid — as your charter says — avoid enabling the use of AI or AGI to unduly concentrate power? Why would not a company like OpenAI keep all the power of an AGI system to itself?
The charter. So how does the charter actionalize itself in day-to-day?
I think first to zoom out — the way that we structure the company is so that the power for dictating the actions that OpenAI takes ultimately rests with the board of the nonprofit. The board is set up in certain ways with certain restrictions that you can read about in the OpenAI LP blog post. Effectively the board is the governing body for OpenAI LP, and the board has a duty to fulfill the mission of the nonprofit. That's how we tie all these things together.
Now there's the question of day-to-day — how do the individuals who in some ways are the most empowered ones, right? The board gets to call the shots at the high level, but the people who are actually executing are the employees — the people here on a day-to-day basis who have the keys to the technical kingdom. The answer looks a lot like how does any company's values get actualized. A lot of that comes down to you need people who are here because they really believe in that mission and they believe in the charter, and that they are willing to take actions that maybe are worse for them but are better for the charter. That's something that's really baked into the culture, and honestly I think that's one of the things that we really have to work to preserve as time goes on. That's a really important part of how we think about hiring people and bringing people into OpenAI.
So there are people here who could speak up and say, like, 'Hold on a second, this is totally against what we stand for'?
Yeah, for sure. I think that's actually a pretty important part of how we operate and how we have, even again with designing the charter and designing OpenAI LP in the first place. There has been a lot of conversation with employees here and a lot of times where employees said, 'Wait a second, this seems like it's going in the wrong direction and let's talk about it.' One thing that I think is very unique about us as a small company is that if you're at a massive tech giant, it's a little bit hard for someone who's a line employee to go and talk to the CEO and say, 'I think that we're doing this wrong.' You look at companies like Google that have had some collective action from employees to make ethical change around things like Maven. Maybe there are mechanisms at other companies that work, but here, super easy for anyone to pull me aside, to pull Sam aside, and people do it all the time.
One of the interesting things in the charter is this idea — it'd be great if you could try to describe or untangle — of switching from competition to collaboration in late-stage AGI development. This dance between competition and collaboration. How do you think about that?
Assuming that you can actually do the technical side of AGI development, I think there's going to be two key problems with figuring out how do you actually deploy it, make it go well. The first one of these is the run-up to building the first AGI. You look at how self-driving cars are being developed and it's a competitive race. The thing that always happens in a competitive race is that you have huge amounts of pressure to get rid of safety. That's one thing we're very concerned about — that multiple teams figure out we can actually get there, but if we took the slower path that is more guaranteed to be safe, we will lose. And so we're going to take the fast path. The more that we can both ourselves be in a position where we don't generate that competitive race, where we say if the race is being run and someone else is further ahead than we are, we're not going to try to leapfrog — we're going to actually work with them. We will help them succeed. As long as what they're trying to do is to fulfill our mission, then we're good. We don't have to build AGI ourselves. And I think that's a really important commitment from us. But it can't just be unilateral — it's really important that other players who are serious about building AGI make similar commitments. To the extent that everyone believes that AGI should be something to benefit everyone, then it actually really shouldn't matter which company builds it. And we should all be concerned about the case where we just race so hard to get there that something goes wrong.
So what role do you think government, our favorite entity, has in setting policy and rules about this domain from research to the development to early...
Stage to late stage of AI and AGI development.
So I think that first of all, it's really important the government's in there in some way, shape, or form. You know, at the end of the day, we're talking about building technology that will shape how the world operates and that there needs to be government as part of that answer. And so that's why we've done a number of different congressional testimonies. We interact with a number of different lawmakers and right now a lot of our message to them is that it's not the time for regulation, it is the time for measurement. Our main policy recommendation is that people, and the government does this all the time with bodies like NIST, spend time trying to figure out just where the technology is, how fast it's moving, and can really become literate and up to speed with respect to what to expect. So I think that today the answer really is about measurement. And I think that there will be a time and place where that will change and I think it's a little bit hard to predict exactly what that trajectory should look like.
So there will be a point at which regulation, federal in the United States, the government steps in and helps be the—I don't want to say the adult in the room—to make sure that there is strict rules, maybe conservative rules that nobody can cross.
Well, I think there's kind of maybe two angles to it. So today with narrow AI applications, I think there are already existing bodies that are responsible and should be responsible for regulation. You think about for example with self-driving cars that you want the national highway to be regulating that. That makes sense, right? Basically what we're saying is that we're going to have these technological systems that are going to be performing applications that humans already do. Great, we already have ways of thinking about standards and safety for those. So I think actually empowering those regulators today is also pretty important. And then I think for AGI, you know, that there's going to be a point where we'll have better answers. And I think that maybe a similar approach of first measurement and start thinking about what the rules should be. I think it's really important that we don't prematurely squash progress. Like I think it's very easy to kind of smother a budding field and I think that's something to really avoid. But I don't think the right way of doing it is to say let's just try to blaze ahead and not involve all these other stakeholders.
So you recently released a paper on GPT-2 language modeling.
But did not release the full model because you had concerns about the possible negative effects of the availability of such model. It's outside of just that decision, it's super interesting because of the discussion at a societal level, the discourse it creates. So it's fascinating in that aspect. But if you think about the specifics here first, what are some negative effects that you envisioned, and of course what are some of the positive effects?
