It's great to have you at the San Diego Alignment Workshop, Yoshua, and you've been one of our, I think, most frequent flyers at this event. So, always good to have you back again. Now, one thing I'm really curious about is you won the Turing Award for pioneering deep learning, who really had a huge impact on the development of this technology that's now powering everything from ChatGPT to Claude. But you're also one of the sort of most vocal, I guess not exactly skeptics or critics, but warning of the risks of this technology. So what caused you to sort of shift your perspective on that and really raise the alarm on this technology you had such a pivotal role in developing?
Yeah, so around January 2023, after playing with ChatGPT for a few months, it just exploded in my face that we were on a dangerous path. And I've talked a lot about why we're building very powerful machines. We don't really know how to make sure they will do the things we want. It could have lots of catastrophic consequences, either by misuse or by machines competing with humans and having greater intellectual abilities.
Now, I think an interesting question about this is why didn't I think about this before? In fact, I was exposed to a lot of the arguments. One of my students was very active in safety and I read the relevant books. But somehow, I think like many machine learning researchers, I looked the other way. The arguments, I heard them and I didn't really pay emotional attention to them because we all want to feel good about our work. And it's only when I started thinking at an emotional level, what would it mean for my children, that I really was forced to confront that we're taking crazy risks.
Yeah, I can definitely relate to that. I started working in AI and in AI safety back in 2017, but I had deep uncertainty about whether the risk was real. I sort of had to get dragged kicking and screaming into the field. I think if I hadn't been surrounded by some friends who wouldn't stop talking about it to me, I would have just ignored it. And when I first started working on it, it did feel like, why am I worrying about this? I can't get my half cheetah in this simulated robotics environment to run forward. Like AI is going to take over? That's crazy. But then, and I wish I was wrong, but the pace of capabilities progress has been pretty phenomenal. A lot of the risks people have been warning about for a decade or more in some cases, like sort of self-preservation and instrumental convergent goals, we're now seeing with coding agents learning to cheat.
In the last year, it's been a succession of empirical evidence confirming a lot of the worst fears. Now, it's still on a small scale and capabilities aren't really at the level where humanity is in danger, but clearly the signs are very obvious.
Yeah. So, you're not just warning people about this, you're also actively doing research to try and put us on a safer path.
I couldn't do anything else. Yeah.
And so you recently founded Law Zero, which is a nonprofit with, I believe, over $30 million in philanthropic capital, which is very impressive to have raised, dedicated to research and development of safe-by-design AI systems. So could you share a bit about what is a safe-by-design AI system? Why isn't every AI system safe by design? I mean, that sounds great. Why would you not design a system safely? What are the key missing pieces that you're working to solve?
Well, because it's not obvious. Because we've been trying to just make AIs more capable broadly, I mean as a field, and me included. And we also, I think, made the bet that intelligence would be built in a way that's similar to human intelligence. But humans can do bad things, and if you create something like a human but more powerful, then that sounds very dangerous.
So after I realized that we were on a very dangerous path, I decided I need to better understand this and do whatever I could, both on the policy side and the technology side and research side. But Law Zero is really focusing on how do we design AI that will not harm people. And that means when we think about the design, even before we build anything, we really have a strong theory of why we think this particular approach would avoid the issues that we're already seeing.
And in the case of the scientist AI, the bet that I'm making is that first we can disentangle the ability to understand how the world works and make predictions about it, like the laws of physics for example, and the preference for states of the world, agency, and achieving goals, which in humans are completely together. But I think we can disentangle these things so that we can eventually have full control over the intention part, in case we continue building machines that are smarter and smarter and have more capability. So that's really the game. If we can build machines that have no intention, can we use that as a building block for safer or even very powerful AI systems? And by the way, I'm now, because of the work I've been doing and the theory behind it that I've been working on, I'm now more and more convinced that it is feasible. There's still a lot of open problems, but I'm in a way more optimistic than I was two and a half years ago when I started realizing things are bad.
Well, that's heartening to hear. So I often, when I talk about safe-by-design AI with people, especially sort of deep learning researchers or people who are building these models, I get a lot of skepticism. And I'll admit I'm not completely convinced myself. So I think the argument I most often hear is, okay, maybe it's possible in some abstract sense, but we've gone quite far down this technology tree of large pre-trained foundation models trained on all the internet's data. We do some post-training, we've got something that works pretty well. To sort of go several steps back in the technology tree, try and solve several open problems and then kind of rebuild back up to that level, maybe it's possible, but you're just going to be kind of several years behind and that's not really, right now months count for a huge amount of competitive advantage. So, what would you say to that cynical take? Do you think this technology can't overtake other approaches? Do we need some kind of policy pressure to make sure that people build systems safe by design? They can't just cut corners on safety.
Yeah. So there are two aspects of this. One the technical and the other the policy. So on the technical side I am convinced that we can build machines very quickly, like in a matter of a few years, that will be safe and will be capable. One way I like to think about this is more like we have this strange historical path where all the companies are essentially doing the same thing and there's this huge toolbox of machine learning knowledge and expertise and algorithms, but everyone is circling around a little quarter of surface because they're all worried about competition in the next few weeks and staying on the leaderboard. But really the richness of what has been developed in machine learning is completely underexploited and there are pieces of this which don't need to be completely reinvented that can be brought together in order to find solutions.
Now whatever we do there will be what people call a safety tax. For example if you want an AI to be safe, that means there are things it cannot do. So there's a cost for that. And so we also need policy. We need incentives. We need to know what are the right ways to do it, might be different in different countries. I like market-based solutions like insurance for example, liability insurance. But there is no way we end up solving these problems without international coordination as well.
