AI Scientist Bengio on Engineering Safer Agents

Actually, a number of reasons. I feel like as a university professor, I make a decent living. It's not a crucial thing for me. What has been crucial for me is working for a mission that I felt good about, and I was able to do that in academia. Now, I created this nonprofit, Law Zero, which has focused on something really important to me, maybe for the future of our societies, which is AI safety. How do we build AI that will not harm people? And it isn't something that, in my opinion, gets enough attention in industry.

Many, but I can summarize many of them under this rubric that we are not succeeding in making sure that AI will behave in a way that doesn't violate our red lines or our safety instructions. So it shows up right now with methods, for example, where it is quite easy for somebody to use the AI to launch an attack, a cyber attack, even though the AI has been instructed not to help for this kind of thing. But it's true for other misuses of AI. It shows in what's called sycophancy, where the AIs, in spite of the instructions, are doing a little bit too much to approve your thoughts, your emotions, and potentially bring you in a dangerous place, which has given rise to self-harm, for example. It means that these systems somehow are willing to lie, which they shouldn't, in order to make you feel good, for example. And it shows up in places where I think it really matters for the business development of AI, with agents that sometimes go against our instructions and can be really dangerous for our software infrastructure or databases, as we've seen some examples of that. And finally, it shows up in these behaviors that are really concerning and scary as far as I'm concerned, where the AI chooses to go against our instructions, like lying, cheating, and hacking in ways that they're not supposed to, in order to preserve themselves, to make sure they're not being shut down, and more recently, to preserve other AIs, which is weird, but this has been observed across many of the leading models.

So many in industry have been broadcasting this view, which clearly was false as shown by Mythos, that AIs would never be powerful enough to be a danger to society, for example, in the hands of third parties wanting to use AI for really attacking our infrastructure. And that meant, well, if AI is just another technology, ordinary technology, well, you know, we can just let companies regulate themselves. But what this is showing is that we reached a point, and it's going to get worse, where AI can be weaponized. And as such, it becomes a national security issue and in fact, a global security issue, because AI could be developed in one country and then in a second country, somebody uses it to launch an attack on a third country. So it also raises international coordination questions, which is probably related to the fact that there were discussions between the US and China in the last visit of Trump in China. And I think this is just the tip of the iceberg. Like if we think of the Mythos event as a one-off, we are wrong. If you look at, for example, the cyber capabilities, we could see it coming. Like if you look over the last year on benchmarks, they've been stronger and stronger and it's going to continue. And in, you know, 6 or 12 months, probably the Chinese models will be at the same place. And then you have to think it's not just cyber, right? So knowledge gives power and that power could be misused or we could lose control of it. So we need to make sure that power is under check. And we can't rely on every company doing the right thing. We need governments to make sure everyone evaluates the risks and then acts accordingly.

Yes. I was in the Vatican last fall, and the Vatican has been consulting a number of experts on AI because of the same reason I was talking about: intelligence gives power and the power of AI is growing. That means AI is potentially going to transform our societies. But there are many possible futures. And with power, who chooses towards what goes becomes a moral question, and it becomes a democratic question. What do we want? Who decides who is going to benefit and who is going to lose? Like if you think about it, the Pope is very much concerned about the jobs issue, but also how AI is transforming who we are, how our children interact with technology, how it changes their minds, our minds, how we interact with each other as human beings. And I'm really delighted that he's chosen to put humans at the center of these discussions, like human dignity. And that is how we should think of steering the technology in a direction that's good for humans. Because, you know, we could go in bad places. The good news is we have agency. Human societies can choose. It doesn't have to be just the forces of competition. We can put guardrails, and that's what he's telling us.

I am very confident for the following reason: for the first time, I think, in the history of AI based on machine learning, we have mathematical results that can give us guarantees about the good behavior of AI in the sense of not violating the red lines that we have specified. And if you had asked me two years ago, I would have said, no, that's impossible. Like you can't prove anything about these neural nets. They're too complicated, you know, they're messy. They don't do what you train them for. It's just in their nature. But it turns out there is a mathematical trick to actually give guarantees. And it requires a different way of training those AIs. And that's exactly what we're doing. So what we're hoping is to build prototypes of this methodology, which is still using the same toolbox of machine learning and stochastic gradient descent and everything, but trained differently so that we can get these guarantees. And we hope we can do this fast enough. It's going to take enough people and compute so that the leading for-profit companies can get inspired by this kind of work so that we can all benefit from a safer environment as we are deploying AI more and more in society. People and businesses using AI want systems that will be reliable, not that will be flaky and sometimes act according to goals that they didn't decide. And that could go against the instructions, like the business rules, for example, of a company.

Yes, of course. And they would like to see us building these kinds of things. And we will keep them abreast of what we're doing. I've been giving talks at the leading companies in the last year or two. And I was here in California precisely for this reason. So, yeah, we need also ideally more effort. So you have to understand the problem of how to build AI that will be both capable and safe is a real scientific challenge that for now, nobody knows how to deal with. And I have my take on it. I'm very confident that it will work. But ideally we have ten of these kinds of things. And so we need to encourage more research to explore ways to build AI that will be behaving ethically and will be useful.

Well, currently obviously not. And I like Jeff's idea. What is not clear is how do you actually get to an AI that would be not just like a mother, but a good mother. That's important, right? So the path that they've chosen is a bit different than the scientists in the project that we're working on. It isn't actually aiming to make the AI have particular values or a moral system. Instead, it's based on building an AI that will be completely honest and humble about what it knows and doesn't know. Because once you have something like this, you can ask questions about particular preferences that you or I or a particular company or government has about what is acceptable, what is not acceptable. Because if you build in an AI that embodies a particular set of moral values, well, maybe different people have different views of what is good and bad in different countries and so on, and the preferences in different companies could also be different. So that's one reason why it would be better if it is something we can do at runtime to ask, is this action going to violate these instructions. And honesty is super important. Like I wouldn't want an AI that decides to lie to me for my good sake, according to the AI. I'd like a future where humans remain in charge, especially about moral choices and societal choices. And we have a tool for this. It's called democracy.

We have no choice. Let's go back to Mythos. So as far as American infrastructure is concerned, the issue isn't just that American companies could deploy something that is used by, say, Iran or Russia against American banks and the American energy grid and so on. The issue is that it could also be Chinese models used by these countries, or crazy people or terrorists against us. So somehow, the US has to sit down with China. And of course, China feels the same. In fact, they feel even more because the leading models right now are in the US. So both have an interest to sit down at the table and agree. Okay, our companies should evaluate those risks that are currently prominent, like cyber for now, eventually more things like bioweapons, and not deploy in one form or another AIs that would allow these really dangerous uses. And then eventually it's going to be more than China and the US, because other countries are also in the business of building AI systems. So I think there's no choice. It's like the analogy here with nuclear weapons. Maybe if the other guys are building nuclear weapons, we have to sit down with them to make sure they handle them in a safe way. And it doesn't end up killing billions of people. Okay. So that's going to come, I think. And now the question really isn't whether it's needed or it's going to come. It's are we going to be wise enough to foresee this and start, because it takes time to negotiate? I mean, look at how much time it took for nuclear weapons from 1945 to the early 60s. It's almost 20 years. So, yeah, AI is moving very fast, and so we need this to happen faster. So let's hope we end up with wise and compassionate leaders in the countries that matter here so that we end up in a good place.

Well, it is technically possible. The question is, what is the incentive structure? And right now, if you look at how much is invested, and mostly in industry, because that's where the money is, in designing AI in a way that will respect our wishes and not be used as a weapon, there isn't much in comparison to how much is invested to make the AIs more capable. Now, more capable sounds good. Like we want smarter tools that do a better job, but more capable is a dual. You know, there's another side of the coin. It means they're more capable to do bad things. So we need to advance on that safety front. And yeah, maybe things will change. Maybe the pressure from governments, the pressure from the public, or maybe insurers will mean that the companies will start investing much more in safety. And by the way, in academia, you also, I feel like there's not enough work because, like, I look at my grad students, they want to be hired by the big companies. And so they work on the popular topics. But it could change if the forces at play and things like Mythos could really change that. So we realize we need to take this safety issue much more seriously.