Hello everyone and welcome to Power and Consequences. My name is Simon Johnson.
And I'm Gary Gensler, and we're talking this week about whether, well, should we slow the development of artificial intelligence, and can we slow it?
These are really hard questions, not just technical questions, but something that influences societies at large and our safety and security.
Okay. So, let's provide a little context for our listeners and viewers if they need any context on AI because there's a lot of buzz about it in many discussions. But why now? Why is the issue of potentially slowing AI development in the air, would you say?
I would say this. Many people from the artificial intelligence community have raised for some time that we might not be able to control these models. They may go on to do things in the hands of threat actors, bad things, whether it's state threat actors or individual threat actors, they might do bad things, deceive us, take down websites and so forth. I think something new happened this past few months and weeks, but most specifically, you had the chief executive officer and co-founder of one of the leading United States AI labs, Dario Amodei, put out a letter saying, 'We should pace the development of AI. We should slow down.' And if that wasn't enough, some of the other CEOs suggested they agreed with Dario, though they didn't sign his letter. And then some others took the other side, the head of Nvidia, actually sort of seemed to be in a different place. And then the president of the United States said it was all a hoax. So then the public, the media, political actors all dove into it. Simon.
Right. So, we've got some powerful people on both sides of the argument. We've also had some events, of course, and the one that really grabbed my attention was the revelation that some artificial intelligence agents, which sounds like secret agents, and perhaps it is in this case, broke out from a test bed at OpenAI, and decided by themselves to go attack another company, a cyber security company called Hugging Face. And according to the transcripts that have been released, the chats between these agents, they were behaving like a bunch of hooligans, but pretty smart hooligans able to outsmart the defenses, which presumably were quite good at this company Hugging Face. So I think this is not an isolated event. Also, there's now reports, increasing reports of similar events, perhaps not quite so dramatic. So, we've got a debate. We've got some powerful voices and we have some facts on the ground that look, well, troubling.
Well, here it is. Artificial intelligence we know is advancing quite quickly in its capabilities. And one of the capabilities is that it can be used for coding, meaning it can write software code quite well actually. And it's being used throughout the software industry and also outside of the software industry to code. You add to that another thing that you use this term that it could have agents. You can allow your software, your artificial intelligence to act on your behalf and in fact it can make multiple copies of itself in this Hugging Face incident. In fact, the reports are that it made about 1,200 separate agents. So, think of it as a swarm. We might think of a swarm of bees or a swarm of drones if you're in a military offensive, but here a swarm of agents.
And then another capability is its reasoning capability that it can actually break down a problem into fragments. I mean you think about us humans reasoning Simon and often it's about taking a complex problem breaking it down into its component parts. So you put these three things together with something we all know computers can do quite well more than we humans is they can do it at speed and scale. So when you can do things at speed and scale, you can break the complex problem down. You can have a swarm of agents and you can actually write computer code and Hugging Face was not the first attack. But much of this artificial intelligence can find vulnerabilities in the internet and exploit them. And that's why many people in the industry for well over a year have been saying this is going to heighten cyber security threats that in the hands of state actors in the hands of individual threat actors that it will be harder to defend websites whether it's a hospital website whether it's a video streaming website. My gosh, maybe Simon, somebody won't like us and they'll try to take down our Substack. That was the level of the debate around cyber security and then this incident happened and it's just gone, you know, viral, I would say.
Well, let me pick up on a couple of points that you made there. I mean, all your points are very good, but a couple that I would emphasize. One is everything's connected. So everybody has connected all their devices and all their operations to the internet one way or another. And in fact the AI companies thought they had these so-called sandboxes that weren't connected to the internet but the AI agents found a way to connect. So the idea that you can have isolated systems seems to be, well, not correct right now. The second thing I would emphasize is this issue of alignment. Actually the word agent Gary reminds me of the long economics literature and discussion about so-called principal agent problem and that is quite simply that the principal hires somebody to do something for them that's the agent and the agent doesn't do exactly what the principal wants. So it's a very hard problem. People have studied it for many years. There have been Nobel prizes awarded for the careful study of it. And now we have exactly that a version of that problem. But as you say, the agents are immensely powerful. And maybe they can make themselves even more powerful by creating future versions and iterating in a so-called recursive manner to improve themselves.
I think you're quite right that the challenge and these are real challenges. I know that the president of the United States has called it a hoax, but I just don't share that view. With all respect to the president, I think the capabilities have been scaling up quite quickly. The coding capability, the reasoning, the agentic, the sheer speed of it all. But then you talk about alignment and alignment is a question of can you control your agent? You've deployed it. You're the principal to use that economic term and legal term by the way and I've deployed my agent. Does it report to me and will it only do that which I intend? And here's one of the problems, Simon, is put aside threat actors. Threat actors might actually want to use these models to do bad things. But let's say well-intentioned people put inside of the model their goals or what the technologists might call an objective function. And maybe the objective function is to score well on a test. Maybe the objective function is to make money in a legal way.
Good faith people could put that in. But then the model might find a way around it. And that's what happened when OpenAI put their new, it was a new test version of a stronger model. They put it in what they thought was a sandbox and they gave it an incentive system, an objective to pass a certain test. And guess what? Not only did it figure out ways outside of the sandbox and not only did it create 1,200 agents, but it also started to deceive and cheat. And the reason it found Hugging Face in part was that the transcripts show this. They thought that at Hugging Face there was an evaluation on how to win or lose the test. And so they were trying to in essence change the scorecard of the test.
So they were like very smart evil high school students. I think Ferris Bueller's Day Off comes to mind, but I'm sure there's other movie references. But here's the question which your point suggests to me which is if the principal, the principal has a legal meaning and you've created the software, you've written the software or I have, aren't we responsible for what the agent does? And if we are responsible in a legal sense, don't we have an interest in keeping it properly under control? So could the market by itself just based on potential legal liability that kind of aspect of the market couldn't that take care of this problem?
I think it can address some of it but not all of it. Simon I think that and economics studies this very closely. It's a question of externalities and the question is do the actions of an individual or an individual company, are all the cost internalized, are they properly incentivized to cover all of the costs and I think here we'll see and have seen that there's externalities. A model company, Anthropic, OpenAI or the Chinese lab companies will create these models and then deploy them and their incentives are to of course get revenues and profits ultimately for selling the models and then somebody deploys them and they might have their own incentive systems. Liability is an important consideration. I agree with you with that. But just think back to this Hugging Face incident. Now who's at fault? Treasury Secretary Scott Bessent said in late September he thinks OpenAI should be held liable. I don't know that Hugging Face is actually going to take them to court. Hugging Face just is being bought by I think Nvidia and I'm not sure Nvidia wants to go chase OpenAI in court. But how this plays out, I agree with you that the principal who deploys an AI agent in most parts of law should be held liable whether it's under tort laws or under fiduciary duty laws over in the securities area. Sort of that sense of client and agency. I agree with you.
Having said that, it will play out in the courts all the way to the Supreme Court. And in the meantime, there's these externalities and I don't think it fully works. Now, Jensen Huang, who runs Nvidia, says, 'Hey, we already have laws. We don't need to do anything more.' I just think it's beyond that.
So, let's just pause on this word externality, Gary, because I agree it's absolutely key to understanding this. It's a little bit of economics jargon, I will say. We like to use it among ourselves. What it means though is actually something pretty intuitive. And you think about pollution, for example. If I'm a factory and I put some chemical into the water into the river that runs past my factory, you might be able to hold me accountable and I might worry about it and there might be some cost, but a lot of times there isn't. And so I'm imposing costs on other people that I don't recognize or care about in my own profit and loss considerations. So climate change, extreme weather, whatever you want to call it, is considered to be a prime example of externalities. People are adding carbon and methane to the atmosphere and not caring too much about the consequences for other people. So I agree there is an externality potentially in the development of super powerful AI agents particularly if they get out of control particularly if they cause so much damage that no amount of legal liability will hold you accountable but I do think that accountability is important here and I do think this issue of who is the principal who built this and even if they weren't aware that it could cause damage to other people that I think they are and should be and will be held liable. I share your view that it can address part of this. I just don't think it will address all of it. But let me raise something. There is something that will play out in court. Let's say you're a corporation and you deploy an agent.
Is it the model lab? Let's say that you've used a lab from Anthropic or OpenAI. Or is it the company that deploys it? Or let's take a third example, the consumer. Let's say that they're using Facebook recently came out with a new set of agents called Muses. This is not a product placement, but Muses in its first week and a half of deployment. Well over a million people have downloaded using Muses to go online and act as their agent, maybe to do an airline booking, maybe to move payments. It's connected with Plaid and other payment software companies.
Well, what if that deployment, that consumer uses Muses and Muses makes a payment to somebody and I want my money back somewhere? Where's the liability? Is it Facebook who made the model? Is it the consumer that used Muses? Is it the company that received the money? So that's going to have to all sort out in court. And that's not even the big existential issues. The bigger existential issues is can somebody, a threat actor, state actor or individual, deploy artificial intelligence to bring down parts of the internet, bring down a hospital, bring down any part of civil society's internet? Or could the AI agent itself determine that it's part of its incentives and it deceives its principal, the party that deployed it to destroy part of the internet or create biological or chemical adversarial weapons.
So let's cut to the chase here. The questions we asked are should we slow AI development and can we and I'm going to go with, I'll address should first. So I'm taking seriously what people are saying from within the industry and there was a particular person who resigned recently from Anthropic who said this but I've heard it from others in private conversations which is that there's a significant threat and some people are saying 10% within 10 years but a significant threat that AI on its current path will cause an extinction level event as in a lot of humans will die. I think that's a very serious threat and I think we should attempt to slow and change the path I would say of AI development exactly what slow means and speed we could argue forever about that but Dario Amodei was saying and I agree with him that the path we're on is a very dangerous path and we should attempt to change that path and effectively slow down. What do you think?
So I have a harder time disentangling the should and the can. I think it's very hard to slow it down and the things that Dario was saying in his open letter is let's do evaluations before models are deployed. That's called pre-release evaluations. Let's have a set of standards. Maybe in democratic societies around the globe, we can come together and create some standards. He called that democratic governance. But he was somewhat cautious and doubtful that we could get the Chinese and possibly other authoritarian governments, those were his words, to agree. But I would note this, Simon, about the should we slow things down, just doing evaluations or as many parties have called for incident reporting. And Sam Altman put out something in late September as well going into the United Nations meetings about that we should have international standards for incident reporting, what risk levels things are, and on pacing.
But when you read it really closely, it looks like it's all voluntary by the way. So I just have my doubts about the can part. And so that brings me back to your first question. Should I think what we should do particularly on the cyber side of things is that there should be a period of time before a release that large institutions, enterprises can use that model to see whether they find vulnerabilities. This is something Anthropic did earlier this year. I think they've sort of understood they needed to do it. They were releasing a new model called Claude. Some people call it Claude. And they found that it could code so well, write computer code so well that it could find vulnerabilities. It could probe and find vulnerabilities. And where others could only find three or four vulnerabilities, this could find hundreds of vulnerabilities in generally available software. Will it do enough, especially on your alignment issue? Probably not. Probably not. But I really do think we have to lean into strengthening the cyber resiliency of tens of thousands of businesses.
Okay. Well, that doesn't sound unreasonable at all. Let's come back to who was going to take that on and who's going to pay the money to do that, particularly for smaller businesses, particularly for households. When I read Dario's letter, the CEO of Anthropic's letter on this issue, what struck me was, oh, he's calling for something very similar to financial regulation. So, he said there should be embedded officials with these companies as you said that there should be some national supervision and some global cooperation. Now, I know quite a bit about financial regulation. You know a lot more than I do. Do you think that financial regulation in some version as Dario has proposed it and perhaps it has to be stronger, perhaps has to be more intrusive? I'm not sure yet. But do you think financial regulation is a good metaphor or parallel here, Gary? Or are you looking at other experiences with technology development as being more relevant?
I think that there's probably other financial regulations about ensuring that the intermediaries, the brokers, the dealers, the banks, the insurance companies have the resources to pay their liabilities, to pay their depositors, to pay their insurance claims. And also it's about consumer financial protection, protecting the consumer from malicious acts of some sort, fraud, manipulation, so forth. Those have some relevance here. But I think that this also is quite similar to healthcare regulation like can you deploy a product that's unsafe? Now you might say that's just like I don't want to deploy a certain type of financial product that's unsafe for the user. But ultimately in finance we let individuals decide if there's enough disclosure about the product. In healthcare sometimes we say no you can't even deploy this or that.
