So Yoshua, I know for a very long time you've been very focused on the risks of AI, but we've spoken a couple of times before.
Actually, it's fairly recent. Only after ChatGPT came out that really focused on that.
It feels like a long time.
It has been, lots have changed.
But I know that your priorities have changed a bit when it comes to risks and what you're thinking about at the moment. What is concerning you at the moment? What's kind of keeping you up at night? Where do you feel like we should be focused?
We are on track to build machines that are already smarter than us in some ways and could become more broadly intelligent, and yet we don't know how to control them. We don't know how to make sure that they will behave well. We don't know how to make sure that third parties couldn't use them in very dangerous ways for all of us. So I feel that we're not paying enough attention to this. I mean, I think it should be mission number one for humanity to figure out how we continue moving forward with this transformative technology in a way that will be safe for our democracies, for our future.
Yeah. And in terms of the models themselves, how have the technical risks changed? We were talking earlier about OpenAI's new model. What are you kind of seeing in terms of how these models are behaving?
So what's concerning is that as they get smarter, they get to reason better, to strategize better, to be more autonomous, they also seem to get better at deception, at cheating. And experiments for the last year and a half have been revealing self-preservation drives, which would not be surprising for a person, but you don't want your tool to want to avoid being shut down. Now, what makes me hopeful, however, is that many of the issues we're seeing, so in addition to these things like sycophancy that we all experience, actually have a common root, the same problem that people have been seeing for many years of bias and discrimination of these models. It's called misalignment. It's really because these systems have goals, preferences that we didn't choose explicitly. And if there is a common root, there is also probably a way to fix all of these problems in one stroke.
Yeah, I know you'll say you're feeling a bit more positive at the moment than you have been previously. Can you tell us a bit about why that is?
Yes. So, I've been thinking about how we could address this alignment problem for the last couple of years. And last year, I created an organization, a nonprofit, to figure it out and actually implement and prototype this methodology that we call Scientist AI. So the organization is called Law Zero, and we just put out a paper that shows mathematically how you can get guarantees for the good behavior of an AI if you train it in a particular way.
And it is something really I did not expect if you had asked me two, three years ago. I thought, well, neural nets, they're like a black box. How can we ever know how they're going to be behaving? But actually you can get probabilistic guarantees, very strong ones.
And perhaps for those in the audience that aren't too familiar with Law Zero, you can give us a little bit of history. I think you're one year old now or is it longer than that?
And you know why you founded this, what you're trying to do.
Figure out technically how we can build AI that will behave ethically. That will not be lying, that will be honest all the time, that will have what I and others call epistemic humility. So that will not be claiming things that are false because the morally right thing to do and the hardest thing to do when you're not sure that something is true is to say I don't know. But currently the AIs are unfortunately overclaiming, and this could have disastrous consequences. So maybe one analogy is if you have a doctor who 90% of the time gives you the right advice, but that 10% of the time not only gives the bad advice but is very, very confident that it is the right thing. You'd rather have another doctor, maybe slightly less accurate, but when it doesn't know it says I don't know, so you're not going to enter into a very dangerous path.
I know you published a recent paper on this that you feel like has been a breakthrough. I don't know if you can walk us through that a bit.
Yes. Just a few days ago we put out on arXiv this paper, and it's very mathematical, but it states a set of sufficient requirements for how we can train neural nets and arrange their reasoning so that we will get these guarantees. And that is, in my opinion, an incredible opportunity for us. So I'm super excited, but at the same time we don't know what is the timeline for very powerful AIs, even more powerful than the ones we've seen, that could create catastrophic outcomes on this planet. So I feel a bit of urgency for us to develop this and work with as many others as possible to make these kinds of safe-by-design AI solutions actually happen and be deployed by the companies that are building AI.
And the ultimate aim is to get AI labs and potentially governments as well to adopt this framework in their training of the models.
I mean if you had the choice between two AIs, one being much more reliable than the other, you would take the latter.
In some industries, if you think about it, whether it's healthcare or it's your electrical grids or your banking system, you don't want to have a bad surprise when you wake up next morning and all your files have been erased or some really bad advice that leads to dangerous outcomes for your population have happened. So reliability isn't just something about ethics, like we think about safety as a public good, but it's also something that industry needs in many critical cases.
Mhm. And in terms of getting tech companies to sort of adopt this framework when it seems very much the focus now is building more powerful models and more autonomous capabilities. That's where the commercial pressure is. How do you go about making that case that this is what they need to be doing?
Well, we have to build a prototype that shows that by using this kind of methodology, not only we get these guarantees, but we don't lose in capability. So, you could have something very safe but doesn't do anything useful and then nobody will adopt it. So, the only way something like this is going to work is if it's both safe and capable. And that is exactly the trajectory that we are aiming for.
