Let me bring in Gary Marcus, cognitive scientist, and he's joining me from Vancouver, Canada. Gary on X says that he is the original OG Gen AI skeptic. So, Dr. Marcus, welcome to this broadcast. You have said, Dr. Marcus, that it is preposterous that humans will be extinct in 5 years and equally preposterous that everything's going to be fine. The two most dominant narratives are entirely wrong. Those are your words. If both extremes are wrong, what exactly do you place... I should say, where do you place Jacob Coxon's warning, that the Anthropic researcher who quit saying that labs are in fact gambling with our lives?
I think the specifics of what he has said are preposterous and he has kind of doubled down in a series of interviews. So he is very much on the side of like we might all die in 5 years and I think that's ridiculous. But I think that there are serious warnings. People in the field talk about P doom, the probability of extinction. I think that's pretty close to zero. You know, if the robots come for us, we will fight back. We're geographically diverse. We're genetically diverse. AI is not going to wipe out the entire population. But there's also what I call P catastrophe, the chance that something serious might happen. And if something serious means, for example, electrical grids being knocked out by cyber crime, maybe by malicious individuals using these technologies, or from disinformation causing accidental wars. I think that that's very serious. I think that we probably will have some very serious side effects. We're not going to literally have extinction. People don't need to, I think, have nightmares about that. I mean eventually, you know, the sun will explode or something in the very long term but in the near term we don't have to worry about those things. Um, but we do have to worry, excuse me my voice is giving out because I've done so many interviews. We do have to worry about the consequences of insecure AI. Basically another way I would put it is like a lot of AI is fine, like GPS navigation systems are not going to kill anybody. You know, people basically know how to make that software but we have this new kind of software called AI agents where you take large language models that are not reliable, you hook up a lot of them and you connect them to the internet and sometimes they do bad things. The OpenAI Hugging Face incident was an example of that where a set of AI agents couldn't follow basic rules like don't hack into other people's systems and that inability to follow rules is where the danger really comes from. It's not because AI is too smart. It's because current AI is not that smart. It can't follow rules. That means it can be exploited by bad actors and it means that it might just misunderstand an instruction and do something bad. Not extinction level, but serious. So like Jacob Coxon, I think, is way overstating the case, but there is something to be worried about. We can't just sit here and have no regulation and hope for the best as some world leaders may have suggested.
Okay. Dr. Marcus, I'm going to move away from what Jacob said to BRICS Summit here in New Delhi. Prime Minister Modi told BRICS that weaponization of technology can hinder shared progress, calling for inclusive access. Is the danger you see the same one Prime Minister Modi is naming, nations using tech access as leverage, or is the more urgent weaponization happening elsewhere, perhaps in how AI capability itself is being marketed?
I mean there's a lot of different things going on. It's hard to say like which is most important. Um, there is though, it's not really what I was talking about, issues about equitable access to these technologies, about how the countries other than US and China are going to have access and what that's going to look like. Um, so there's a whole set of issues about economic distribution and things like that. There's another set of issues which is really ones that I'm talking about which is that current AI is not very well controlled or people use the technical term aligned and that can mean it does things we don't want it to do or that bad actors can exploit it to do bad things and that's a different set of risks than the economic set of risks. Um, and there's still other risks like non-consensual deep fake porn is yet a third category of risks. Um, that is, you know, starting to cause a lot of problems in high schools and so forth. Um, you know, the thing about AI is it's not one thing but many. Um, there's many different underlying technologies with different capabilities. There's all kinds of social and political things that are factoring in. So, we can't just say like this is the one thing you should worry about. The truth is it's like a hydra with many different heads. There are also many advantages but there are many disadvantages we need to worry and to rein in.
Then are you worried more about disadvantages than advantages given the pace of AI and the concerns being raised now by the big, you know, the tech giants themselves?
I'm pretty worried at this point. I think I'm long-term positive in the sense that I think that there's a possible path to an AI that is really good for society, but short term I'm worried. I think the current technology is unreliable. And I think that unreliability causes problems. And I think that the companies that are running it are mostly pretty greedy and not really mindful of the consequences. And I think the governments are mostly letting those greedy companies do what they want at the expense of humanity. So right now I think AI is probably a net negative for society. I don't think it has to be that way but I think that's kind of where we are right now and we need to up our regulation game and we need to have better technical tools as well.
So what can be these regulation and technical tools? What's responsible AI then, or should I say how do you make AI responsible?
On the governance and regulatory side? I think the number one thing that we need is that every country needs to do something like what the US does for drugs with the FDA, which is to say we require a cost-benefit analysis. You have to show that the value of your drug offsets the consequences. Um, the risks, you know, lots of drugs have side effects. That doesn't mean we don't put them on the market, but you have to make the case that the benefits outweigh the risks. And you have to have independent scientists weigh in on that. You have to make an argument that what you have is good and they may send you back to the drawing board or ask you to do more experiments or come up with a way of dealing with something. We need that for AI. That's the number one thing that we need on the regulatory side is some kind of screening procedure. It needs to be transparent. It needs to have independent scientists in the loop. Secondly, on the technical side, we need AI that can follow instructions. That might sound trivial, but it's actually pretty hard given the starting point we have now. So the starting point that we have right now is these large language models that try to predict essentially what a person would say in a given context. And it does that by looking at the statistics of how people talk across a whole lot of different kinds of text and so forth. That system is not built from the ground up to follow instructions. It often follows instructions, but it's not guaranteed to do so. And that's a real problem. And I think we need to go back to the drawing board for better technology.
So, Professor Marcus, China here is pushing open-source AI models to the global south and BRICS itself has just unveiled an open-source AI ecosystem framed as tech equity. Is there a genuine, or do you see this as genuine democratization or a subtler form of the same weaponization that Prime Minister Modi has warned against?
