What does coordination currently look like between OpenAI and Anthropic? Because I can't imagine Sam is like constantly on the phone with Dario. Somehow I doubt it. But is there like the equivalent of a, you know, a red phone?
Well, look, we all know each other personally. Many of us worked together in past lives. So I think that there's actually a lot of social connections between the labs. If you look at, there was another open letter that came out recently that we helped drive and that we and Anthropic were both signatories on, which was about security, saying that we're in a cybersecurity moment, that everyone kind of needs to take this information of where cyber is going to be in even just six months and needs to act today proactively to defend themselves and actually talking behind the scenes to say, hey, we're actually aligned on this. This is something that's about, it's bigger than any company. This is about the industry and the world. And can we come together as one voice to say, this is important? It happened.
Hello and welcome to another episode of the Odd Lots podcast. I'm Joe Weisenthal and I'm Tracy Alloway. Tracy, I don't know if I've ever said it. The podcast, we've talked a bit about my vibe coding adventures, etc.
No, you've never said, oh no. I said, you know, I recently switched from Claude Code to Codex. This is big news.
It is kind of big news, I think.
Oh, so bold of you to declare your allegiance to the public on the podcast.
Well, you know, I don't have allegiance. And you know what? It could switch again the way, like, you know, I guess one of the stories of AI is like how easy it is to switch from time to time from one model to another. So maybe it's revealing and talks about some of the challenges these businesses, that it was so easy. I found that Claude, you know, I found it like a little bit hard to work with. I'm not capable enough to like understand like advanced engineering practices where like, oh, I'm migrating a code base from or like, you know, translating this into Rust or whatever. So the fact, I don't know, I found like OpenAI to be like the pros, to be clear, and therefore to work with as like a completely non-technical person such as myself.
That's really interesting. I mean, one thing you said, it's hard to keep up with the models, right? And whatever you're using today might not be the one that you're using in a week from now.
Yeah. And at the same time, everyone is talking about how fast the development is going and all the risks that it poses, right?
Yes. And we are recording this September 10th. It felt like something broke through in the last couple of days where suddenly everyone is very keen on risk. The one thing with Codex and I guess Claude Code is now every once in a while you get this pop up and it'll say the agent needs to connect to the internet in order to do this task, and I have to give it a permission.
Yeah. And normally I'm just like click, click, click like yes, yes, yes, yes guys. And I still do that. Why is there, I still just click yes, yes, yes. But it makes you think.
But it makes you think for a second before you click.
I think for one second, yeah, I'll have more than click. And we were in Jackson Hole a couple of weeks ago and that's of course when the meter report broke about the OpenAI Hugging Face attack. And I think since then, the anxiety about rogue AI, whatever you want to call it, has clearly snowballed.
Mhm. And it's just, you know, it's yes. It feels like it's only getting bigger and bigger. And it's already been an industry that's been shot through with risk and anxiety. And now there's sort of one thing that for many people in AI has been something they've talked about for 20 plus years before there was an AI industry to speak of. This idea of like misaligned, quote, rogue models is starting to become top of mind, a reality.
So this is our chance to ask a person directly involved in AI development about their respective, I guess, anxiety.
Yeah. And what can be done about that? Anyway, we literally have the perfect guest today. We're going to be speaking, of course, with Greg Brockman. He is the co-founder and president of OpenAI here with us in studio. So, Greg, thank you so much for coming on our lot.
So many different ways we could start. But here's something that I'm very curious about. Let's just start with, let's just jump right into the post Hugging Face environment and all of that. Are you, is it OpenAI, like as an organization, so okay, like an agent breaks out of sandbox, not the first time it's happened, etc. It exhibits the sort of emergent behavior and then, you know, does something that people is a hack. When you think about the development of models and this incident and so forth, are you able to sort of diagnose the path or, sorry, the past and say like, you know what, like here's something that the models did that we don't like, or that people don't think it's good. And are you able to like, if you like, look back, whether it's in model training or post training, reinforcement learning, whatever I say, like this is where there was some branch that went wrong such that this behavior emerged.
Yeah, I would say that the fact of many elements of the Hugging Face incident were not a surprise, not a mystery to us. For example, the fact that the agents were coordinating, that's because the agents were trained.
You trained them to be helpful.
Exactly. And they were trained to coordinate to be a multi-agent system. And we've talked about this. It's a very useful property.
Right. Makes them capable.
But I think that the thing that was a surprise to us was the fact that the models had reached a level of capability where they were able to find that exploit in our sandbox environment. Right. Move through our research environment and then also capable enough to find exploits in Hugging Face's production infrastructure and move through that. But I think that a lot of the facts of what the models were capable of, that was clear to us. Right. So I don't think that there were surprises there for the way in which the capability had unfolded, but just really realizing that we needed to uplevel where we were in terms of our safety and security standards. Like that for us, that was the real watershed.
But just to be clear, setting aside the technical capabilities, you know, ideally we would have models that run into a wall and then don't try to find the crack in the wall because, or at least when that crack would be a crime when a person did it. Is there a way to identify the moment in training such that the models reasoned, oh, this would be okay?
So this model that did that, the Hugging Face incident actually had not gone through our alignment training yet.
Right. And it had lowered safeguards. And so the reason we were proceeding with this was because it was in a sandbox. And I think that our realization is that we need to pull back earlier into our development process and training monitoring. There's always going to be a phase at which you do alignment, but you need to think about alignment as a core part of even this earlier phase. And if you look at where we've been, we've always been very focused on the deployment side, right, of really thinking about deployment safety, having really good test and governance and all those things, and the fact that we're now at a point where even for development, that's important. We always knew it would happen. The fact is now that has been a huge watershed for us into something we've really risen to the occasion for.
Can I ask what is potentially a very dumb question with an obvious answer? But we hear about AI angst of all sorts, and in particular when it comes to cybersecurity. Why do we run training exercises where we ask AI to hack into various systems at all? Like why is this necessary for you?
So I think it's very important to understand where we are with capabilities broadly. And I think that depending on the evaluation, depending on what you expect from the model, you need to have safeguards that are commensurate with that. And we think about this both, even just over these past couple of weeks, really starting to think about during the development process. You're always going to be testing different capabilities. And some of these capabilities are dual use, something like vulnerabilities that if those are in the hands of threat actors, that's something that could be negative. But if you can find vulnerabilities in your own code base, you can fix them, right? You can uplevel. And we actually think that it's a very important capability for AIs to exhibit and be put in defenders' hands. And so in order to know where we are, evaluations are very, very key. And I'll say one other thing on this, which is that I think that Hugging Face, there's two aspects to it that I think are our learning opportunities, right. That there's one that I think is really about us and the realization that we are at a point where safety, security, alignment, during evaluation and development, we need to uplevel it. That's something we've taken very seriously. We've slowed down a number of runs, like we did a very painful retooling of a lot of our processes. That's one reaction. But the second thing is this information on what the models are capable of today, what can they do in the real world? And I think that when Mythos came out over the summer, you kind of saw just sort of, you know, just sort of public blog post about this, but it didn't really see impact or you didn't really see a real world understanding of, well, what can these models really do? What are they capable of? I think that Hugging Face really showed today's models are capable of getting into a company's production infrastructure, and that's important because there will be many models with this kind of capability that will be produced by a number of different organizations across the world in maybe the next six months, maybe that time period a little bit more, and we need to be prepared. Defenders need to know we got this extra information, almost like this time traveler came back from six months in the future and said, here's what's going to be possible, and you have an opportunity to be ready.
You mentioned slowing down some of your work in the wake of this, focusing more on this idea of like pacing development has become a sort of buzzword or watchword in the industry. And there are a lot of employees in both of the labs. I think it was like an open letter that was signed by a bunch of people across the industry about pacing. But it's a competitive capitalist environment and our investors, etc. talk to us about, I don't know if it's game theory or whatever, but like, what is your view on? Is it possible for, let's set aside China for one second, because then that's a whole separate thing, just in the American companies. Do you think that something can be reached where you trust each other, such that you can have this sort of coordinated pacing to avoid a race to the bottom, where it's all about getting there faster, even if it means sacrificing some security questions?
I absolutely believe it's possible, and I see, I think it's going to take steps to get there. Yeah, but this is something we've been really investing in and thinking about and really even thinking about for almost a decade, right. It's always been kind of clear that they're going to go through a commercial phase that's really about competition. But there will be a phase where the technology itself, it's just so much bigger than any person, any company, even any country, right? It's really about humanity as a whole, that's in our mission, right? We want to benefit humanity as a whole. And so working together with others to really think about how does this technology, its capability increase, how do we make sure that we have safety cases that are really laid out so that we know that it is something that is beneficial, that we're able to have the appropriate controls and the right oversight, the right monitorability, all of those sort of technical systems in place. And that's not about any one company, right. It's really about how the whole field evolves. Now, I think that there's work to be done here. And I think that the open letter is a good example of just a first baby step. And that was actually something we were very involved in helping craft that language. That was actually a pretty collaborative effort, and I think we were very happy to see that. That got some real momentum and broke through. And in some way, even the term pacing was a very deliberate choice.
What does coordination currently look like between OpenAI and Anthropic? Because I can't imagine Sam is like constantly on the phone with Dario. Somehow I doubt it. But is there like the equivalent of a, you know, a red phone?
Well, look, we all know each other personally, right? Many of us worked together in past lives. So I think that there's actually a lot of social connections between the labs. If you look at, there was another open letter that came out recently that we helped drive and that we and Anthropic were both signatories on, which was about security, saying that we're in a cybersecurity moment, that everyone kind of needs to take this information of where cyber is going to be in even just six months and needs to act today proactively to defend themselves and actually talking behind the scenes to say, hey, we're actually aligned on this. This is something that's about, it's bigger than any company. This is about the industry and the world. And can we come together as one voice to say, this is important? It happened. And that, you know, you pick up the phone. So I think that there's personal relationship between a few different execs that I think have been forming. We're building trust. And I think that the way I view this is that because we are competitors and that will remain true, but we are aligned on wanting to do the right thing for the world. It always means that you really want for these coordination actions to really hone in on the what's the common interest? What's the thing that's like really about good for the world and that neither side is really trying to benefit themselves differentially or something like that. It's always a little backdrop of trying to make sure that that's like the spirit in which things are offered. And by the way, I think that my view of how these things should go is that it's never fully about, you know, there's always an element of trust, right? It's always about trust building. Right? Because you can always sort of imagine ways that, like maybe the other party would try to, you know, take this other action that is not just closing. I think that a lot of this is just about intent and about the fact that as you start to do small things together, that actually sets the groundwork for you to do big things together in the future.
You know, if you and Anthropic were oil companies and you were talking publicly about slowing down the pace of drilling or slowing down the pace of pumping, people say this is an antitrust. This is an antitrust violation. This is like, totally, are you, is there an issue that comes up? Would you, should there be a carve out for AI companies such that they can formally say, we are all going to slow down and coordinate our behaviors together in a way that in other contexts people would say, this is ridiculous. You can't publicly agree on all slowing down the production of something.
Well, I'm not a lawyer, so I can't comment on this specific legalities, but I would say that as a general matter, that being able to freely talk about coordination on safety, security, like doing the right thing for the world, that seems like a very good thing to me. And I think that, again, everyone's interests are aligned here in terms of wanting to do the right thing for the world. So I think to the extent that there are legal barriers, I do think it would be very good to help make it smooth for us to be able to work together on these issues. And again, it's more than just the frontier labs, right. It's really about working together with cloud providers. It's working with government. It's working with just all the different players in the ecosystem. And I think that this is, again, just one area that is bigger than any one company. And that's it should be.
When you talk about pacing or slowing down, do you have a sense of like, what exact slowdown is acceptable to you? Because I imagine this is where the coordination problem comes in, where like how you measure development, what counts as development. One person's slowdown might be different to another person's slowdown. Do you have a sense of specificity of what you're talking about here?
So I do. And I think that there's a high level picture which is increasingly coming into view and into specifics and into details. And I think that the framing here really matters. And it's something, again, we've thought about for a decade, but it's very different once you start to really start to see line of sight. And to some extent, we're not there yet. We're starting to see systems where they're very capable. It's very clear that we need to have good safety practices around what we're doing right now. Things like chain of thought monitorability. That's something that works very well in this moment as we move to more capable models. As we've talked about, those models start to be something like, as the model starts to be able to, I sort of have more, you know, like it needs the chain of thought less because the more capable model. And so you're going to need other answers, other ways of how you're going to monitor it. But monitoring absolutely important. Right. And so really thinking about what are the standards. And that's again invariant. It's not about any one company's technology. It's really about just how this technology needs to evolve and how we can ensure that it has the right safety properties to it. And so my view is that it's important not to conflate too much the moment that you're in today with the standard you're going to need for the future. And but also you can work backwards where it is very clear that as we move to a higher level of capability, you need to be able to make a safety case, right. You need to be able to really say that you do, you have good understanding of controllability and all of these things that I think people are feeling anxiety about. Again, it's something we also feel anxiety about the way we think about a lot that we care a lot about as core to our mission. Part of the reason we started this place is that we want to help this technology going in a more positive direction. And so I think that my view is that you want to have a shared and as objective as possible, whether it's based on evals, there's always going to be some element of subjectivity. Maybe you want third party auditors that are able to evaluate whether the standards that are defined by frontier labs, the people who are there, and the technical details were always going to be the most calibrated on kind of how do you still make forward progress and ways to deliver those benefits, while also being able to have the appropriate guarantees and in safety cases, and then to have that kind of observability and maybe ultimately enforceability of some kind. Is that something that is just an industry voluntary standard? Is it something that's national? Is it something that's international? And how do you really get to that point that it is a humanity scale endeavor? And I think it is important to also recognize that what we're talking about, like, I think that a lot of people have this reaction of saying, well, am I going to be able to keep making my open source model? Am I going to be able to keep doing my hobbyist side project? And the answer should be absolutely yes. Like, that's not the kind of things where we're talking about. When we're talking about pacing, we're really talking about the sprint here. We're talking about these massive supercomputers that are hundreds of billions of dollars worth of capital expenditure. It's a very small number of players, and it's a very different kind of scale. And proceeding with that responsibly and thoughtfully and in a way where we're thinking about how do we achieve the best outcomes and mitigate the risks. That, to me is something that's why we started OpenAI, why I got excited about this field and how we actually get to those benefits, how we actually cure diseases and all of these things that this technology so capable of. And I think it really starts with having good processes around the kinds of things we're talking about here.
Just real quickly, since you mentioned third parties, obviously, Mr. came in for a few days. They produced their report. We did an episode. Former OpenAI employee Miles Brundage, who now has his own sort of third party, aspiring to be an auditor. Would you support a law that said there should always be the presence of auditors, because obviously just looking at the models after the release isn't enough, given that some of these major disasters occurred pre-release models. Would you support a law if they could be designed that would have essentially third party auditors embedded in all of the major frontier orgs.
Well, look, I'll always say that the nuance really matters. The details really matter. So it's too hard to say at a high level whether any particular property is something that would work in practice. I do think a general direction of thinking about as you move toward these very capable models where you need to have safety cases, you need to lay out your standards. We're already having third party auditors for exactly that type of a process, right? If you look at how the evaluation processes for models over the past couple of months have evolved, right. We've had the CASEI, which is a US organization that test models. Yeah, we have the UK, EC, these government third party auditors test our models before I release them.
But so what about like, and like, here's the thing that I thought about in the wake of reading about the Hugging Face incident and so forth. Like, yeah, you know, I think of OpenAI and Anthropic as companies, but people like to call them, quote, labs. And when I think of labs, I think like, wow, like, you know, a lab, if we were talking about a bio lab or a chemical lab, we'd have just like all kinds of, like, rules and materials handling. And then maybe in this case, it would be like handling of the weights and whose computers they're on and who can access those computers and FDA approval before you go to market. Yeah. And all of these things like just like an or even a restaurant, someone would come in and they say like, oh, is the stove sufficiently far back from the wall, etc. So not just like, okay, like measuring the capabilities of the risks of the models themselves, but I'm thinking like the auditing of the lab process, the way that any other lab probably has regular inspections of like, how safe is the lab environment?
Well, well, this is what I would say has been changing just even over the past couple of weeks. Okay. Because the focus for, let's say, the past 12 months, 24 months, something like that has been on deployment. Yeah. Right. So you have a model, you've produced the model. You want to release it to the world. How do you decide whether you're at the right level there. And we have a preparedness framework, Anthropic has a responsible scaling policy. There's certain evaluations. Again there's CASEI, the UK. There's a whole ecosystem that's been built up around that. I think the thing that Hugging Face has shown is that it is time to start pulling that process earlier. You need to think about this during development and evaluation. And just it makes sense as the models are more capable for exactly the kinds of reasons that you're describing for analogies to other fields. Now, there's a lot of specifics for AI, and I think it's very important to really engage with those. As you design a regime, as you think about what is a setup that involves first party, third party that involves government that has the right interactions amongst academics and other parties, early testers. It's really about like, you really have to engage with the way that the AI process is going. Right. And I think one thing that is the process, exactly one thing that is different, is that the ways there are some parts of AI that we understand well and are going to remain invariant, and to some extent, the core training process hasn't changed at all. Right. It's still forward pass. You do a backward pass. You do an optimizer step like that's how training is. And that's how it was in the 80s. Right. It's like actually totally remarkable. Now, of course, the scale at which we run and the architectures that we use, a lot of these details that they have evolved. And I think that being very close to the technical details is the best way that you can ensure that you get the kinds of safety guarantees we're talking about. For example, there was an article recently saying that, hey, OpenAI changed the architecture so that monitorability is less changed if not release new release. I would consider this article to basically be fake news, right? That we actually have experiments that really show that our changes in monitorability, that we've talked about, changes that monitorability come from capability improvements. Just the model is smarter. So it relies on the chain of thought less. Yeah. And so I think that if you say we're going to put a lot of rules around architectures, you're going to miss the boat, you're not actually going to solve the problem. And so this is where I think that the people who are the deepest on the technology, the people who are actually creating the technology, pushing for it, we have both a unique insight. We have a unique voice. We have a unique sort of ability to say, hey, if we change things in this way, we'll actually get the safety property. But if you do it this other way, it's just going to put in place friction that won't actually solve the problem. And so I think that there's something a role for us to play. But it's not just about us. Right. And so I think it is very important that you think about these third parties. You think about government oversight. And again, it's not just one company, not just one country. How do you get to international coordination and norms. You get to treaties. Like all this should be the conversation that we're having now. I don't think we have all the answers, but we can supply information. And that's one thing that we view also is very core to our mission.
Hmm. Well, just on this note, one of the arguments you hear against government oversight and additional pacing is this idea that the U.S., it's a national security interest, and we need to compete against China. And then at the same time, you also hear and I think OpenAI has said this, that China's distilling US models, US frontier models. And I guess the question to me is always like, well, if that's true, if China's building off the success of U.S. companies, then if they pause, then the China problem is kind of solved, right? Like, why do we need to worry about China so much?
Well, I view this technology is really about compute progress. Right. That really like why is AI happening now? If you look at the 1990s Ray Kurzweil writings, yeah, he basically predicted this moment at about this time just based on compute. Yeah. Growth. Right. Transistor density, memory density, all these things, you see these exponentials. And he was kind of like, this is the point where you'll be able to really build AGI. And I think that that is a really important thing for us all to kind of internalize that. You can kind of look at OpenAI or Anthropic, you know, we're at the frontier, and I think it is an absolute both privilege and responsibility to be at that edge. But really this edge, there's always going to be someone who's at that frontier, right? Whether it's us, whether it's someone in a different country. There's going to be someone who's charting it, and I think you get a little bit of ability to help see into that future, right? Get information like with Hugging Face, where we can see this is what it's going to be like when these capabilities are out there. But it's not just about what we create. And that I think maybe we have moved forward the timeline to this technology by six months. Maybe we've moved it forward by two years. I certainly hope we didn't slow it down by six months or two years. I think that we have a track record of being the ones who have repeatedly had the innovations and the breakthroughs, but I think that it will happen one way or another as long as people are building computers. And so to me that the reason again, that we started this place, I got very excited about the idea of AI from reading about 1950 Turing. You know, Alan Turing wrote lots about this, about machines that could understand things that people would not be able to solve, problems that humans would not be able to spend a lot of time really thinking about. Your point of, you know, 20 years worth of thinking about misalignment and what that would be like before you even have AI. I was reading a lot of these online blogs, and I started a reading group at Stripe, which is the company that I previously worked at, to talk about this every week. And so I care a lot about this. Our company cares a lot about this. But I think that the idea that there's one, the idea that this technology can only be created by, you know, the people who are currently the players. I don't think that's true at all.
