I've become fascinated and somewhat terrified by the risks of AI and talked about it at length on recent shows. Now, those risks that we've discussed have included job losses and misinformation, but also concerns that sound more like sci-fi, including AIs taking control. Now, one pushback I've got from you guys, sort of in the comments and by email, is to say that I've radically overestimated the power of the kind of large language models that power applications like Claude and ChatGPT. And one name more than any other has come up as a skeptic I should read: Gary Marcus. Now, Gary Marcus is professor emeritus of psychology and neural science at New York University. In 2014, he founded Geometric Intelligence, a machine learning company later acquired by Uber, and he came to international prominence when he was called to give evidence on artificial intelligence to the US Senate in 2023 alongside his co-witness Sam Altman. Now, in that hearing, Gary Marcus highlighted many of the dangers of AI and the need for regulation. But what he's most famous for, perhaps, in AI debates is his skepticism about the potential progress of large language models. As early as 2022, Gary Marcus was arguing that deep learning, which is the method for training LLMs, was hitting a wall. I mean, his very popular and influential Substack contains plenty of articles explaining why he thinks many of the claims made about AI are all hype. He's even become a verb. When Apple released an influential paper last month claiming to show that LLMs only provide the illusion of thinking, the venture capitalist Josh Wolf said, 'Apple just Gary Marcus'd LLM reasoning ability.' And last year, Gary Marcus published the book Taming Silicon Valley: How We Can Ensure That AI Works for Us. And I'm delighted to say he joins me now live. Gary, thank you so much for joining us. I suppose, can you start by explaining your position in the AI debate and the arguments that have made you so prominent in this space?
Well, I guess my position is that AGI is possible, that AI could be a good thing, but also that LLMs themselves are wildly overrated, are not going to bring us to AGI, have serious problems in reasoning and comprehension, and that we need to have some fundamental innovation.
So you think that AGI, artificial general intelligence, sort of this idea where AI becomes more intelligent than us in various spheres or potentially all spheres, you think that's plausible, but it's not going to be via the LLM route. What do you think would be needed? What's your alternative?
So I think what we're most fundamentally missing are the tools of symbolic AI, of classical AI, where you write things in code, you have databases and what we call knowledge graphs and so forth. In large language models, they're a second-class citizen. People are increasingly trying to farm things out to symbolic tools like calculators and so forth. But LLMs themselves are not very good at reasoning over abstract knowledge. And so what we need is what I've long called neurosymbolic AI. We're starting to see some movement towards that, but pure LLMs are not going to cut it.
I suppose where I stand here is I don't have your expertise. I don't have the expertise of Geoffrey Hinton. I find it very difficult on an intellectual level to say whether or not I think LLMs could bring us AGI or whether it needs to be a different form. I suppose one thing sort of I look at, you know, I use my ChatGPT and I know something that's sometimes put to you is that you've sort of said that a current AI system or a current LLM is unable to do something and then potentially it gets overtaken by events. And obviously you're not alone in this. I want to show a quote from Yann LeCun. So he's the chief AI scientist at Meta. He's one of the three godfathers of AI and he said in 2022, 'I take an object, I put it on the table and I push the table. It's completely obvious to you that the object will be pushed with the table. There's no text in the world, I believe, that explains this. If you train a machine as powerful as could be, your GPT-5000, it's never going to learn about this.' Now, anyone at home can put that scenario into the free version of ChatGPT now and it can perfectly answer that question. And I just want to go to one example of you as well. Now, obviously, this isn't a gotcha because I'm not in a position to really debate AI with you, but it's to sort of try and draw out the argument being made. So, this is from an article you wrote in 2020. You were sort of calling out the ability of GPT for doing physical reasoning. And so you said, 'You're having a small dinner party. You want to serve dinner in the living room. The dining room table is wider than the doorway. So to get it into the living room, you will have to remove the door. You have a table saw so you can cut the door in half and remove the top half.' So that's obviously AI getting it completely wrong. You explain why it's got it completely wrong. So you say, 'This is one confusion after another. The natural solutions here would either be to tip the table on its side, often sufficient depending on the specifics of the geometry, or to take the legs off the table if they are detachable.' I won't read all of this. Actually, you're sort of saying how ridiculous that answer is and you're absolutely right. But then if you put that into the latest version of ChatGPT, so I put that scenario in and it gives you just the perfect answer: to get the dining room table into the living room, you will have to rotate or tilt it to fit it through the doorway at an angle, likely turning it on its side or diagonally. Then it gives you a perfect step-by-step which suggests, from reading it, that it kind of understands spatial reasoning, you know, it understands how to get the table through the door when it definitely didn't four years ago. And I suppose, how do you, if someone's looking and they're saying, well, the people who've critiqued LLMs as being able to sort of make serious advancement, if their predictions keep getting overtaken by the next model, how seriously should we take them? And I suppose, how do you respond to that? I know this is not an original question I'm putting to you, but I think it's an interesting one.
Yeah, I've been asked that question many times. And what you're missing is the context that any specific example that someone prominent such as myself writes about gets trained on, as far as we can tell, by OpenAI. So I'm sure that that whole paper has been trained on. They've probably paid people, maybe from Scale AI or who knows, to do variations on that problem. But what we see, Ethan Mollick calls it the jagged frontier. I used to call it the pointillistic nature of these models, is they'll get specific examples but you never know which examples they're going to get right and which they're going to get wrong. So if you go back to that 2020 paper, we talked about a core set of problems: physical reasoning, causal reasoning and so forth. And there are still many problems that those systems have trouble with. It's just not those specific problems because the systems have been trained on essentially every text that's out there, including, you know, things that I have written before and so forth. And I don't know if there's a compression problem. And so these systems get trained on these new examples, or sorry, on the older examples, but people still find new examples all the time. There's a paper just published called something like Pumpkin Reasoning Problems showing lots of inconsistencies in how these systems reason up to o3-mini and so forth. So you see the same kinds of problems, you don't see the specific problems because the specific problems are trained on by the large language model manufacturers to some extent. They are never candid or transparent about what training they're actually doing. So from the outside, we can never know how much the model is actually improved as opposed to how much it's trained on the specific examples, but we always see a kind of fragility where they'll get something right and then they'll get a variation wrong. So here's an example. They were trained on a lot of these river crossing problems. So a man and a goat have to get across the river. There's a boat, there's some cabbage, the goat can't eat the cabbage, etc. And so the systems will now answer problems like that because they're in the training set. But then you change them slightly. So now you have a man and a woman in a boat. Now this one's been done and they'll also be trained on. But when somebody tried this a few months ago, a man and a woman are in a boat and need to get across the river, the system missed the obvious solution. If you put the man and the woman, you go across the river, and instead says, 'Well, the man goes across the river, leaves the boat there, swims across to the other side.' I forget what it does next, and then blah blah blah, and takes like 17 steps. And my daughter, I think she was 11 at the time, said, 'Why don't you just put the man and the woman in the boat and cross to the other side?' And so these systems will often get the things they were trained on, which now often I think includes particular examples that I've written about, but they don't generalize well.
People are saying now, because I mean even if you look at the current GPT, you can put in sort of the old version and the new version. And I've put in a bunch of questions where the old version gets it wrong, the new version gets it right. And I don't think that can just be because of training because presumably they're trained the same. But if you get it from the reasoning model, it does quite well. And I know that it's sort of said that they're quite good at frontier maths now. So really hard maths problems that a PhD student would struggle with. And there were mathematicians that put in a bunch of these that it couldn't possibly be trained on. And some of the new reasoning models did quite well. But you're still skeptical.
I'm still skeptical. For example, someone just did the US Math Olympiad six hours after the problems were released and performance was at 5%. Right? So on some set of tests where there was advanced knowledge, clearly they were doing what we call data augmentation. They're building problems that were similar to the problems in the test set and systems did well. So somebody tried to control for what we call data contamination when there was only six hours. So they couldn't do that. The systems were at 5%. This was published, I guess, about a month ago. So you always have to worry. This is kind of my first answer, my previous answer. You always have to worry about data contamination. You always have to worry about the extent to which they can actually generalize if they haven't seen those problems or at least very similar problems in the training set. And because the manufacturers, the developers won't release what they've trained on, you don't know. Well, the other thing we're finding is they do better on math problems where you can create augmented data, which is to say synthetic data rather than naturally observed data. You can do that pretty well in math because you know what the answers are. You can use basically classical symbolic systems to calculate the answer and then you do the training. That works better than in other domains. So they're getting best performance in math where they can do this kind of workaround, but they're not finding that it works across the board.
Let's look at, I'm sure you've seen this clip many times as well. Gary Marcus, I want to show a clip of you in that Senate hearing from 2023 and you kind of break protocol and put a question to Sam Altman instead of just answering questions from the Congress people. 'When we get to AGI, artificial general intelligence, maybe let's say it's 50 years, that really is going to have, I think, profound effects on labor. And there's just no way around that. And last, I don't know if I'm allowed to do this, but I will note that Sam's worst fear I do not think is employment. And he never told us what his worst fear actually is. And I think it'd be good to find out.' 'Thank you. I'm going to ask Mr. Altman if he cares to respond.' 'Yeah. Look, we have tried to be very clear about the magnitude of the risks here. I think jobs and employment and what we're all going to do with our time really matters. I agree that when we get to very powerful systems, the landscape will change. I think I'm just more optimistic that we are incredibly creative and we find new things to do with better tools and that will keep happening. My worst fears are that we cause significant, we the field, the technology, the industry, cause significant harm to the world. I think that could happen in a lot of different ways. It's why we started the company. It's a big part of why I'm here today and why we've been here in the past and we've been able to spend some time with you. I think if this technology goes wrong, it can go quite wrong. And we want to be vocal about that. We want to work with the government to prevent that from happening. But we try to be very clear-eyed about what the downside cases are and the work that we have to do to mitigate that.' So Sam Altman to the Senate saying he started OpenAI because he was just so worried about AI risk. Did you believe him then and do you believe him now?
I did believe him then and I don't believe him now. I think that he told the Senate what it wanted to hear. I mean, on that particular point, I don't think he wanted to tell the Senate anything. And it's because I asked the question and asked Senator Blumenthal to reassert the question at all about his worst fear, which was significant harm to humanity. But I think in general, he was not a candid speaker at the Senate. He told the Senate, for example, that he supported licensing for art or some kind of payment for artists and writers, and he doesn't anymore. He told the Senate that he supports regulation for AI, and he doesn't really anymore. And the last time he was at the Senate, which was not then two years ago, but just a few weeks ago, he basically implied his worst fear was like China would get ahead of us or something like that. And he didn't really acknowledge then the significant harm. So, you know, some of what he said in the Senate was true and some of it was partial truths.
Tell us what you're really concerned about when it comes to AI because I know that you're sort of, AI 2027, the report which we've talked about on this show before, where, you know, there's sort of total global catastrophe and AI could kill us all by the year 2030, you're sort of skeptical of, but you are still very concerned about AI. You've written a book about it. What are your big fears?
Well, I think there's two sets of fears. One is about what happens with LLMs, which are already causing a lot of harm. There was a piece by Kashmir Hill in the New York Times about how they're causing people who didn't have psychiatric histories to have delusions. They're obviously being used in misinformation. They're going to be used, or they are being used, in cyber crime. They've made some major cyber crime incidents. There are a lot of immediate, pressing risks from these systems. They're discriminating in employment, etc. But we're using these systems. They're not actually that smart, and putting a not-smart system in charge of things is not itself a smart thing to do. So I'm pretty worried about that. I think there's a lot of that. We're seeing more and more of it every single day of these kind of inappropriate uses of these tools. In the long term, if systems do get smarter, we have to worry about how we're going to make them align with human values. LLMs are not well aligned with human values and we need to make sure that the next generation of AI is better aligned.
