Greg Brockman: OpenAI and AGI

So first of all, it's really important the government's in there in some way, shape, or form. We're talking about building technology that will shape how the world operates, and there needs to be government as part of that answer. That's why we've done a number of congressional testimonies and interact with lawmakers. A lot of our message right now is that it's not the time for regulation, it is the time for measurement. Our main policy recommendation is that the government, which does this with bodies like NIST, spend time figuring out where the technology is, how fast it's moving, and become literate about what to expect. So today the answer is about measurement. I think there will be a time and place where that changes, but it's hard to predict exactly what that trajectory should look like.

I think there's kind of two angles to it. Today with narrow AI applications, there are already existing bodies that should be responsible for regulation. Think about self-driving cars: you want the national highway authority regulating that. We're going to have technological systems performing applications that humans already do, and we already have ways of thinking about standards and safety for those. Empowering those regulators today is pretty important. And then for AGI, there will be a point where we have better answers, starting with measurement and then thinking about rules. It's really important that we don't prematurely squash progress. It's very easy to smother a budding field. But I don't think the right way is to just blaze ahead and not involve other stakeholders.

Mhm.

To zoom out, with language modeling we're clearly on a trajectory where we scale up models and get qualitatively better performance. GPT-2 was actually just a scale-up of a model we released the previous June, run at much larger scale, and we got results where it suddenly started writing coherent prose, which we hadn't seen before. Now we're going to scale GPT-2 by 10x, 100x, 1000x, and we don't know what we'll get. The model we released last June was a good academic toy, not something with serious negative implications. But fast-forward to GPT-20 and the capabilities will be substantive. There needs to be a point in between where we draw the line and start thinking about safety. For GPT-2, we could have gone either way. When we announced we decided not to release it, some people said it was obvious we should have, others said it was obvious we shouldn't. That almost definitionally means holding it back was the correct decision. If it's not obvious whether something is beneficial, you should default to caution. For future models, and possibly sooner than you'd expect because scaling doesn't take that long, those you definitely won't want to release into the wild. We view this as a test case: how do you go from having no concept of responsible disclosure to a world where you have a powerful model and go through some process? The security community took a long time to design responsible disclosure. In AI, the response to GPT-2 proves we don't have any concept of this yet. As for the substance of GPT-2, it's been trained on the internet with biased data and offensive content. You can ask it to generate content on basically any topic, and it writes like internet content, even saying 'advertisement' in the middle of generations. Think about the possibilities for fake news or abusive content. People have tried to generate Facebook messages in my own style, or generate fake politician content. There are things where you have to think: is this good for the world? There's the flip side too. There are awesome applications we want to see, like creative applications. If sci-fi authors can work with this tool and come up with cool ideas, that seems awesome. We've had people write to us asking to use it for creative applications.

Have you ever seen the popular meme of a robot with a physical arm clicking the 'I'm not a robot' button?

I think the truth is that trying to distinguish between robot and human is a losing battle.

I think it's a losing battle ultimately, in terms of the content and the actions you can take. Think about how CAPTCHAs have gone. They used to be simple: an image with artifacts that humans could read but OCR couldn't. I can barely do CAPTCHAs anymore. I think CAPTCHAs were a moment in time. As AI systems become more powerful, there being human capabilities measurable in an easy automated way that AIs can't match, that's just an increasingly hard technical battle. But not all hope is lost. Think about how we already authenticate ourselves: social security numbers, ways of identifying individual people. Having real-world identity tied to digital identity seems like a step toward authenticating the source of content rather than the content itself. Building out good reputation networks may be one solution. This question is not obvious. Maybe sooner than we think we'll be in a world where I can't tell if a tweet was written by a real human. Look at the FCC comments on net neutrality, where millions were autogenerated. What do you do when statistical techniques can't tell the difference? The most persuasive arguments could be written by AI. It's not sci-fi anymore. GPT-2 can make a great argument for why recycling is bad, and you'd read it and think 'huh, you're right.'

I think the biggest gap between us and AI systems right now is physical manipulation. So maybe that's the last frontier. But here's another question: why is solving this problem important? What aspects are really important to us? I think we'll hone in on what we really want from knowing if we're talking to a human. It comes down to identity. I expect the internet of the future to have lots of agents interacting with you. But the question of whether it's a real flesh-and-blood human or an automated system may actually just be less important.

This is a really interesting question. One thing that's important is honesty. If you have AIs pretending to be humans and deceiving you, that feels like a bad thing. It's really important that we feel in control of our environment, that we understand who we're interacting with. But the flip side is: can I have as meaningful an interaction with an AI as with a human? I think you can turn to sci-fi here. Her is a great example. It starts by asking how meaningful are human-virtual relationships, and then you have a human with a relationship with an AI, and all your emotional buttons get triggered the same way as if there was a real human. I think we can have meaningful interactions, and if there's a funny joke, in some sense it doesn't matter if it was written by a human or an AI. But what you don't want, and where I think we should draw hard lines, is deception. The reason we build AI systems is to enhance human lives and make humans feel more fulfilled. If we can build AI systems that do that, sign me up.

I think the Turing test in its real form isn't just about language; it's about reasoning too. To really pass the Turing test, I should be able to teach calculus to whoever's on the other side and have it really understand and solve new calculus problems. We need more than what we see with language models; we need some way of plugging in reasoning. How different will that be from what we already do? That's an open question. It might require radical new ideas, or we might just need to shape existing systems slightly differently. Language modeling has already gone way further than many expected. There are interesting angles to explore, like how much does GPT-2 understand the physical world. You read a little about fire underwater in GPT-2, so maybe it doesn't quite understand. But you also see things like smoke coming from flame. GPT-2 has no body, no physical experience, it's just statically read data. We don't know yet, but we're starting to be able to ask these questions of real systems, and that's very exciting.

I think it's unlikely that if we just scale GPT-2 we'll have reasoning in the full-fledged way. The type signature is a little bit wrong. There's something we do called thinking, where we spend a variable amount of compute to get to better answers. That type signature isn't quite encoded in a GPT. A GPT spent a long time in its evolutionary history baking in information, getting very good at the predictive process, and then at runtime just does one forward pass and generates stuff. There might be small tweaks to get the type signature right. It's not really one forward pass, you generate symbol by symbol, so maybe you generate a whole sequence of thoughts and only keep the last bit. But I would expect you'd need to make changes like that.

Yep. And there's another interesting piece, which is out-of-distribution generalization. Thinking somehow lets us do that. We haven't experienced a thing, yet somehow we keep refining our mental model of it. This feels tied to whatever reasoning is. Maybe it's a small tweak to what we do. Maybe it's many ideas and will take us many decades.

Mhm.

Right. How many new story ideas have we come up with since Shakespeare?

Yep, I have.

One thing that was really interesting about the reaction to that blog post was that a lot of people read it as saying compute is all that matters, and that's a very threatening idea. I don't think it's true either. It's clear that algorithmic ideas have been very important for progress. To really build AGI, you want to push as far as you can on computational scale and on human ingenuity. You need both. But if you can find a scalable idea, you pour more compute into it, more data into it, and it gets better. That's the real holy grail. So the answer is yes. Part of why we're excited about deep learning and the potential for AGI is that the most successful AI systems, when you scale them up, they work better. That scalability gives us hope for building transformative systems.

There's a space of possible progress, a space of ideas and systems that will work. And there's a portion of that space, to some extent an increasingly significant portion, that does require massive compute resources. Part of why we have the structure we do is because we think it's important to push scale and build large clusters. But there's another portion that isn't about large-scale compute. These are ideas that, if scaled up, would work way better than at small scale, but you can discover them without massive computational resources. If you look at recent developments, things like the GAN or the VAE, people came up with those without having massive computational resources.

This reminds me of a blog post from one of my former professors at Harvard, Matt Welsh, a systems professor. I remember sitting in his tenure talk. He had just gotten tenure, went to Google for the summer, and decided he wasn't going back to academia. He made the point that as a systems researcher, he comes up with cool system ideas, builds a proof of concept, and the best thing he can hope for is that people at Google implement it and make it work at scale. That's the dream: build the little thing and they turn it into the big thing. He said he was done with that; he wanted to be the person actually building and deploying. There's a similar dichotomy here. Some people find real value in being the person who produces those ideas and builds the proof of concept. You don't get to generate the coolest possible GAN images, but you invented the GAN. There's a real trade-off, and it's a very personal choice, but there's value in both sides.

Take a look at the June 2018 model that we released and scaled up to turn into GPT-2. At small scale, it set some records and had some cool generations. They weren't nearly as stunning as GPT-2, but it was promising and interesting. With a lot of these ideas, you see promise at small scale. But there's a very big asterisk: sometimes we see behaviors that emerge which are qualitatively different from anything at small scale, and the original inventor looks at it and says 'I didn't think it could do that.' This is what we saw in Dota. PPO was created by John Schulman, a researcher here. With Dota, we ran PPO at massive scale and were able to get long-term planning and behaviors that played out on a time scale we thought was not possible. John looked at it and said 'I didn't think it could do that.' That's what happens when you're at three orders of magnitude more scale than you test at.

Well, to that point with Dota and PPO, here's something very surprising about Dota that people don't pay much attention to: the degree of generalization out of distribution. You have this AI that's trained against other bots for its entire existence.

Yeah, so with Dota...

Dota is a complex video game. We started trying to solve it because we felt it was a step towards the real world relative to games like chess or Go. Those are very cerebral games with a board and discrete moves. Dota has continuous time, a huge variety of different actions, a 45-minute game with different units, and a lot of messiness that hasn't been captured by previous games. All the hard-coded bots for Dota were terrible because it's so complex. This seemed like a great place to push the state-of-the-art in reinforcement learning. We started with one-versus-one, solved that, and beat the world champions. The learning curve was this crazy exponential. It was really interesting to see how the human iteration loop yielded very steady exponential progress.

The approach we used is self-play. You have two agents that don't know anything. They battle each other, discover something a little bit good, and now they both know it. They just get better and better without bound. That's a really powerful idea. We went from one-versus-one to five-versus-five, like a team sport where you need coordination. We pushed the same self-play idea to get to the professional level at the full five-versus-five version. What's really interesting is that these agents are almost like insect-like intelligence. There's a lot in common with how an insect is trained. An insect lives in this environment for a long time and a lot of experience gets baked in. It's not smart in the human sense, not able to learn calculus, but it can navigate its environment extremely well and handle unexpected things. We see the same thing with our Dota bots. They can play against humans, which never existed in their evolutionary environment, with totally different play styles, and yet they handle it extremely well. That was very surprising to us. It doesn't emerge from what we've seen with PPO at smaller scale. We're running at something like 100,000 CPU cores with hundreds of GPUs, hundreds of years of experience going into this bot every single real day. At that massive scale, we start to see very different kinds of behaviors out of the algorithms we all know and love.

Yeah.

One thing that's interesting is that we lose all the time because we play here. The Dota team plays the bot against better players all the time. The first time we lost publicly was at the International against some of the best teams in the world. We lost both games, but we gave them a run for their money. Both games were 25-30 minutes and went back and forth. I think that shows we're at the professional level. The coin could have gone a different direction, which was actually very encouraging. The International was at a fixed time, so we pushed as far and fast as we could. Two weeks later, we had a bot with an 80% win rate versus the one that played at TI. You should think of the march of progress as a snapshot rather than an end state. We'll be announcing our finals pretty soon, playing against the world champions. For us, it's really the final competitive milestone for the project. Our goal isn't about beating humans at Dota. Our goal is to push the state-of-the-art in reinforcement learning, and we've done that. We've learned a lot and have exciting next steps.

Massive scale.

I'll put an asterisk on that and just say, I think it's about ideas plus scale. You need both.

We really have a lifecycle of projects. We start with a few people working on a small-scale idea. Language is a great example: it was really one person pushing on language for a long time. Then you get signs of life. With the original GPT, we had something interesting and said it's time to scale. We put more people on it, more computational resources, and kept pushing. The end state is something like Dota or robotics, where you have a large team of 10 or 15 people running things at very large scale, with material engineering and machine learning science coming together to produce results that would have been impossible otherwise. We do this whole lifecycle. End to end, it's probably about two years. One team we're just starting: Illya and I are kicking off a new reasoning team, to really try to tackle how you get neural networks to reason. We think this will be a long-term project that we're very excited about.

Theorem proving.

I think so. There are other problems dual to theorem proving: programming, even security analysis of code. They all capture the same sorts of core reasoning and out-of-distribution generalization.

It would be very, very exciting, especially if it turns out that P does equal NP. That'll be interesting too.

Yep.

I think reasoning is an important one, and it's going to be hard to get good results in 2019. The lifecycle takes time. For 2019, language modeling seems to be on that ramp. It's at the point where we have a technique that works and we want to scale 100x, 1000x, and see what happens.

I think it's hard to have a real opinion about it. I separate out things that can yield materially different predictions about the world from ones that are fun to speculate about. I kind of view simulation more like whether there's a flying teapot between Mars and Jupiter. Maybe, but it's hard to know what that would mean for my life.

I feel like there are two separate questions. Can we use simulation for real-world problems? Take a look at our robotic system Dactyl. It was trained in simulation using the Dota system, and it transfers to a physical robot. Everyone looks at our Dota system and says 'it's just a game, how do you escape to the real world?' The answer is, we did it with a physical robot that no one can program. Simulation goes a lot further than you think if you apply the right techniques. Now, there's a question of whether the beings in that simulation are going to wake up and have consciousness. That seems much harder to reason about. You really should think about where exactly does human consciousness come from. Is it just that once you have a complicated enough neural net, you have to worry about the agents feeling pain? There's interesting speculation there, but it's hard to know for sure.

I'll stick to the non-grand answer first. Just look at what we're already making work. A lot of people would have said that to get GPT-2's results, you need real-world experience, a body, grounding. How are you supposed to reason about smoke and fire if you've never experienced them? GPT-2 shows you can go way further than that kind of reasoning would predict. So do we need consciousness or a body? The answer is probably not. We could continue to push the systems we have. They already feel general. They're not as competent or as quick to learn as AGI would be, but they're at least proto-AGI, and they don't need any of those things. Now, the grand answer: are neural nets conscious already? Would we ever know? When we think about animals, there's some continuum of consciousness. My cat is conscious in some way, not as conscious as a human. You could imagine a little consciousness meter. Point at a cat, it gives a little reading. Point at a human, a much bigger reading. What would happen if you pointed one at a Dota neural net? If you're training in this massive simulation, do the neural nets feel pain? It becomes pretty hard to know the answer is no. It's very possible that the reason humans have consciousness is because it's a convenient computational shortcut. If you have a being that wants to avoid pain and eat food, maybe the best way of implementing that is to have a conscious being. If that's true, then maybe we should expect really competent reinforcement learning agents will also have consciousness. But that's a big if, and there are many other arguments you can make in other directions.

I hope so. If there's any better way to end, it is on love.

Thank you for having me.