The entire economy is hinging on overattribution of intelligence to these machines, right? You have people betting trillions of dollars that these machines are intelligent in ways that they aren't actually. And then we have like government policies built around these things, or you know, considered around these things. The entire world is overattributing intelligence to LLMs. It's not that LLMs can't do anything, but their intelligence is still limited, and we probably need a completely different approach.
Welcome to Prof G Markets. We've spent a lot of time talking about AI lately. From the Trump administration's export restrictions on Anthropic's models to the ongoing questions surrounding the economics of companies like OpenAI and Anthropic. Taken together, these stories point to two fundamental questions. One, is the AI boom financially sustainable? And two, are we moving too quickly with a technology that we don't fully understand? Few people have been asking those questions longer than our next guest. Long before concerns about AI safety, regulation, and business models entered the mainstream, he was warning about the technology's limitations and challenging some of the industry's most ambitious claims. He has testified before the Senate on the risks posed by AI. He's founded a machine learning company that was acquired by Uber and is now one of the field's most prominent skeptical voices. So, here is our conversation with Gary Marcus, AI skeptic, author, and professor at NYU Stern. Gary, thank you so much for joining me on the show today.
You are one of the original critics of AI, and that's quite interesting because you're also, you work in AI. You started a machine learning company, which I think you could say is an AI company. You've done a lot of AI research. You are sort of part of the AI world, but you have issues with it. Let's just start broad. What are your concerns?
Well, my concerns are we're all in on a particular technology that I think is inelegant, harmful, not where we should wind up, and being abused by the people that are using it. So, I want AI to succeed, but I think we wound up down this really dangerous path. So, you know, you think about the Star Trek computer. You ask it a question, it gives you an answer that you can count on. Presumably, it's not done to sort of wreck society. It's done to help people. Well, what we actually have is everybody running around with LLMs, which are inherently unreliable. They're unpredictable. They can't be aligned to human values, and they're being run by companies that don't seem to really give a [__] about the consequences for what they're building for society. It's like a nightmare for those of us who have worked in AI to suddenly see what we're building be used in so many bad ways and with people really not caring. You know, we should want a more reliable technology that we can really count on that is compatible with humanity. You know, five years ago, that didn't seem out of the question. Five years ago, the field was healthy. It was considering lots of different things. It wasn't driven so much by money, but by intellectual curiosity. How do you make a machine that's intelligent? And everything changed when people started to realize that there might be money to be made. Still not clear that there actually is money to be made, by the way. Right. And I'm sure we'll get into that because we have very similar views about that. But the thought, the scent of money, possibly misguided sense of money, really changed how the field grew. And it's also, you know, a technical thing. Transformers are interesting and people got into them. But fundamentally, I think it's the scent of money really changed how people built AI, how they thought about it, what they wanted to do with it, who was running it. I think a lot of grifters came in that don't even necessarily have a technical understanding of the questions and do a lot of lying and hyping about what their things might actually do. And it's just really been unpleasant for the last several years being honest about it. And it's not because I don't want AI to succeed. I still think that there's a chance that AI could help a lot in medicine, that it could help with all kinds of technologies. Like I would still like to see AI succeed, but not on the path that we are right now. This is just not a good path.
You recently wrote, you said, quote, 'Generative AI has been inherently unreliable from the start. None of the problems that I warned about over the last half decade have been properly solved.' There's the financial question, which we will get into, but then there's also the question of the technology itself. What do you see as the problem? What makes Gen AI inherently unreliable? What is the path that you are worried about this technology going down?
The technical problem is that large language models, pure large language models, are basically next-token predictors. That's what they do. That is literally how they are built, is to predict in a sequence of words or other kinds of tokens what might come next. And that's an interesting thing to do. It's part of what humans do, is we do some prediction, but it's not all of what cognition is, right? Cognition, right? Intelligence is about cognition, about understanding things and so forth. There's many different components to it, and they're just not really built into LLMs. And so LLMs basically fake everything else. And we can talk about some complications. People are building in harnesses, and we can go there, but let's just talk about pure LLM. What they do is predict the next token. And if you train them on the entire internet, which is what people in fact do, they can make a pretty good approximation of human beings and how they talk and so forth. But that approximation is very superficial. It's very data dependent. And when you push them outside of the regime in which they've been trained, they will do really stupid things. So, like a couple years ago, there were all these examples of so-called river crossing problems. Like you have a man and a goat and a woman, and they have to go across the river, and these systems would say the most absurd things in response to those problems. It got so embarrassing that Anthropic built in river crossing problems into their system prompts to try to keep the systems from making these embarrassing errors. And what the embarrassing errors revealed is the systems are not really reasoning about things like a man or a river or a boat or what it means to go across the other side. They're just trying to kind of glom the words together that they have seen. I mean, you know, the technical details are a little bit complicated, but to a first approximation, what they are doing is just stringing these words together. There are other ways to think about building intelligence. So you might start, for example, with a database: who did what to whom, when, and where. If you actually did that, if you started with that, you would not have all these crazy hallucinations. And so here we are, you know, in 2026. I started writing about LLMs in 2019. And I said, they don't have stable models of the world. You can't trust on them. And everybody said, 'Gary, Gary, Gary, we're just going to add more data. All these problems are going to go away. Hallucinations are going to go away.' Mustafa Suleyman, who's the CEO of AI or whatever his title is at Microsoft, said, you know, they're going to go away in a few months. This was in 2023, I think. I offered him a bet, and he kind of walked back what it was. Reid Hoffman said he would bet any amount of money that hallucinations would go away in a few months. This was 2023. I said, 'I'm over here. How about $100,000?' He never got back to me. But here we are in 2026, and hallucinations have not gone away. It's because the core of next-token prediction does not allow you to address that problem. You have to add something else, and the something else, you know, rarely works all that well. It sort of works a little bit. You know, I just saw a study yesterday showing there's a new benchmarking, I think it's called Hard, Hallucination Hard, and all the systems are still making errors on this. So, none of this has gone away. It's hard for people who are not trained in cognitive science and artificial intelligence to understand when they play with these systems that they don't think like human beings, that they're really operating over different principles, because they are built to mimic human beings, and human beings are not built to distinguish AI systems that work differently from themselves from actual humans. Like we have a lot of evolutionary machinery to find fast things that are moving that might be snakes or bugs or lions. We have nothing built into our brain to really help us think about the nature of intelligence. And so people are very easily fooled. We've actually known that for a long time. We've known it for 60 years, right? That ELIZA was the first example of an AI system that could fool an average person into thinking it was much more intelligent than it was. ELIZA behaved as a psychiatrist, and it just did simple keyword matching. So you say, you know, relationship, and it asks you to tell you more about that relationship or whatever, just by matching keywords, not understanding anything. So Weizenbaum wrote about this in the '60s, how we are vulnerable to overattributing, is the technical term, intelligence to machines. And the, you know, that was a curiosity, I guess, when he wrote about that in the '60s, but now that is the whole world, right? The entire economy, and this is, you know, where our shared interest is, I suppose. The entire economy is based right now, is hinging on overattribution of intelligence to these machines, right? You have people betting trillions of dollars that these machines are intelligent in ways that they aren't actually, because those people placing the trillion-dollar bets don't have enough cognitive science background to know the right test in order to evaluate intelligence. And then we have like government policies built around these things, or you know, considered around these things. The entire world is overattributing intelligence to LLMs. It's not that LLMs can't do anything, like they're great for autocomplete, for the purposes of computer coding, and they're great for certain kinds of brainstorming and so forth, but their intelligence is still limited, and we probably need a completely different approach. You know, I like to have a metaphor of climbing mountains, right? And you could get to the peak of one mountain and think, well, I must be close to the top, but actually you might not be, right? If it's a mountain range and there's a whole bunch of different peaks, you might be at the peak of one. In order to get to the tallest peak, you might have to actually go back down the valley. And that's what we need to do. We actually need to give up some of the progress that we've made in order to come up with new ideas. But everybody's obsessed with one idea. You know, they're obsessed with the large language model that actually has these problems of, and we didn't even get into, but bias and unreliability, etc. But people are so addicted to the one thing that they're, you know, they're all in on that. And, you know, we'll get to the economics soon, I suppose. But part of the economic problem hinges from that, that everybody is using the same solution. If you had a healthy ecosystem, you might have a hundred different companies trying different approaches, and you could say, 'Let the best one win.' But we have basically a hundred companies, maybe not a hundred, but a dozen companies doing exactly the same thing. And if it's not the right thing, that's a problem. And even if it is the right thing, it's a problem. You know, I'm pretty sure it's not, but it's still a problem if everybody's doing the same thing. That means making profits is really hard. So, you know, we'll talk about the economics and why nobody is making profit. But the underlying reason nobody's making profit is they're all doing the same thing. If we all have the same toothpaste, nobody's going to pay that much for it. You can't charge $100 for a tube of toothpaste if you have, you know, nine competitors building basically the same thing for less.
Basically, we're all using one of two models essentially. You're probably using OpenAI's model, you're probably using Anthropic's model, and then a lot of these companies are building wrappers and building all of these gadgets and gizmos on top of those models. But to your point, you're basically just putting wrappers on top of the same fundamental thing.
Well, and those two are actually basically the same, and they're basically the same as what...
It's the same cognitive architecture.
Yeah, they're small differences, but they don't persist, right? I mean, that's another thing that we have seen over the last few years. I wrote this tweet, I think in 2024, describing what I said is going to be a new regime where basically LLMs are going to run out of headroom. Everybody's going to wind up building essentially the same thing. There's going to be no moat between them. And that's going to lead to price wars, and it's going to lead to, you know, no huge difference between them. And that's what we've seen. There's a lead that goes back and forth, right? Somebody's ahead for a week, and they pay like, you know, a hundred billion dollars or whatever, you know, the numbers aren't public, but you know, or the ten billion dollars, I guess, would be more plausible, in order to get that lead that lasts like three weeks. Like that's insane.
Exactly. And then depending on which engineer you talk to, some would say Codex is better, and then some say Claude is better, Claude with slop, and then people say that OpenAI is coming out with slop, etc. I think one of the main points that you put forward, which I think is true by the way, is that these AI models are actually dumber than many of us would like to believe, especially those of us who are financially incentivized to believe that these things are the greatest and smartest things ever. And I think a lot of people would recognize that we see this in our daily interactions with LLMs. We do see a lot of hallucination. I've said to my team outright, you're not allowed to write with AI because I can immediately tell, and it sounds kind of dumb, and it's formulaic, and it doesn't present as very good or smart critical thinking. And there's even this guy who's gone viral on social media recently, and maybe we can pull up a clip in the post of this podcast, who kind of exposes how dumb ChatGPT can be. Like he'll tell ChatGPT, you know, 'I want you to count to 20 with no interruptions. And if you do that, some guy's going to give me $1,000. So make sure you do it.' And it can never accomplish it.
I'll start counting right now. One, two, three. And I'll keep going until we get to 100. Here we go.
Okay. You stopped just after three. I need you to get to 100.
Ah, got it. No problem. I'll keep going all the way. So, continuing four, five, six, seven, and I'll just keep counting steadily until we reach 100.
Yeah, but you're not actually counting. You're just saying you're going to keep counting. He wants to hear you say the full 100 count.
Ah, I see what you mean. You're right. I'll actually go through it properly. So, here we go. Starting at one. One, two, three, four, five. And I'll keep counting all the way up until we reach 100.
Having said this, and just to be clear, I'm with you on this. One belief is AI is dumber than we think. Another belief is AI is very dangerous and perhaps could be a lot smarter than we think, and therefore we need to regulate it. Both of those arguments are somewhat anti-AI, and I see them conflated a lot of the time. And I guess my question to you is, if it's not as powerful, then why are we worried about this? What's the problem?
The example you just gave actually is a really nice illustration of it, right? Which is they are dumb in the way that we can't count on them to follow instructions, right? Let me put some nuance around the dumb. I mean, they do some things that you might count as smart. And what people in cognitive science, which is my native discipline, will tell you if they know what they're doing is that intelligence is a multi-dimensional thing. So, you know, they have the intelligence to play chess really, really well, better than I can. You know, I got beat by a chess computer in like, when was it, in 1999 or I don't know, a long time ago. I can't even remember. Well, I mean, I guess Kasparov got beat by the best one in '97.
Yeah, don't, I wouldn't beat yourself up about it.
I played Kasparov once, by the way, and he annihilated me while playing too many other people. Anyway, I'm not a great chess player, but the point is it was probably even earlier that I got beat. But, you know, AI can play chess really well. It can play Go really well. A GPS navigation system, a different kind of AI that can do navigation really well. But LLMs can't do a lot of things. So, LLMs actually are not good chess players. As it turns out, they make illegal moves. They can't even follow the rules. And so, their stupidity about rule following, and you just gave a beautiful example of this. That's what you need to worry about, right? You know, the reason that we need to regulate them is because they don't reliably follow instructions. It's not that they can't do anything that you might characterize as intelligent. You could argue about your definitions of intelligence. So, one definition would be that you can do essentially any kind of problem given enough, you know, resources, that you're adaptive and so forth. They're not very adaptive. But there's another definition of intelligence which is like, you know, can you play chess? Then sure they can, right? Well, LLMs can't, but other kinds of AI systems do. Asterisk here, by the way, there are different forms of AI. You know, my beef is with generative AI, and that's mostly what we're talking about. Generative AI cannot follow instructions. Chess computers, you know, purpose-built chess computers actually do follow the rules of chess, and I have in some ways less concern about them. LLMs are terrible rule followers. That is one of their weakest points as an intelligence. You know, another rule would be don't make stuff up. Like, you know, you can tell an intern like, don't, you know, write something if you can't fact check it. Like just don't, please.
And if you do, I will fire you or I will sue you.
Exactly. You're going to get fired if you... Right. I mean, that's the other crazy thing about what's going on is like calculators never make mistakes. Right.
There was a scandal when the, what was it called? The Pentium 4, I think, made very, very rare mathematical errors. Huge scandal. They had to recall the chips and stuff like that. Somehow the standards have fallen. Like everybody knows LLMs make mistakes all the time, and they're perfectly happy with it. I'm like, I wouldn't want an intern who does that.
We'll be right back after the break. And if you're enjoying the show so far, send it to a friend and please follow us on YouTube and Spotify or wherever you get your podcasts.
Support for the show comes from LinkedIn Ads. There's no worse feeling than making a major investment in something only to realize it didn't exactly live up to the hype, such as buying a nice piece of tech that ends up in storage collecting dust, or taking a business workshop where your main takeaway was little more than a few motivational words. If you work in marketing, this can happen with ads. You optimize for the numbers that look great, impressions, reach, and reactions. But when they don't show revenue, well, that can turn into an unfun conversation with the CFO. LinkedIn has a word for that: bull spend. Reach the right buyers with LinkedIn Ads and invest in what looks good to your CFO. According to the 2026 Dream Data Benchmark report, LinkedIn Ads generated the highest ROAS of all major ad networks. It's 121%. You can target by company, industry, job title, and more. It's time to cut the bull spend. Advertise on LinkedIn, the network that works for you. Spend $250 in your first campaign on LinkedIn Ads and get a $250 credit for the next one. Just go to linkedin.com/scott. That's linkedin.com/scott. Terms and conditions apply.
Support for the show comes from Odoo. Running a business is hard enough, and you don't need to make it harder with a dozen different apps that don't speak to each other. One for sales, another for inventory, a separate one for accounting. Before you know it, you find yourself drowning in software and processes instead of focusing on what matters, growing your business. That's where Odoo comes in. It's the only business software you'll ever need. Odoo is an all-in-one, fully integrated platform that handles everything. That means CRM, accounting, inventory, e-commerce, HR, and more. No more app overload, no more juggling logins, just one seamless system that makes work easier. And the best part is Odoo replaces multiple expensive platforms for a fraction of the cost. It's built to grow with your business, whether you're just starting out or you're already scaling up. Plus, it's easy to use, customizable, and designed to streamline every process. It's time to put the clutter aside and focus on what really matters, running your business. Thousands of businesses have made the switch, so why not you? Try Odoo for free at odoo.com. That's odoo.com.
We're back with Prof G. OpenAI was just subpoenaed by a group of attorneys general to investigate their models. And some of the things that they said that they are investigating here: one, how they handle consumer data, also health data, also deep learning models. But my favorite is that they are investigating model sycophancy. And it seems as though that the concern from regulators is, to your point, like it's not just that these AI models are dumb and making mistakes. It's like that is something that we need to actually punish. We can't have the largest, I guess, information provider, one of them in the world, going out and putting false information out into the ether with no culpability or no accountability. And I do think that that is an important thing for these AI companies to contend with because at a certain point, if enough of us just decide, I can't trust these models anymore. They lie too much, they make things up, they say things that don't make sense, then eventually we're just going to stop using them. How do you think that plays out?
I mean, we might or we might not stop using them. There should be consequences, right? Sycophancy, by the way, is when they kiss your ass, right? Yes. Right. So, that's a separate problem from lying, although it's a form of lying perhaps. Like when they tell you that your idea is the greatest idea ever and it's not actually that.
That's what I mean. I'll say, 'Is this right?' They go, 'Yeah, you're right. You're right.' Even if the thing is totally wrong, and then I go to Google and learn that I'm wrong, but the model will tell me, 'No, you're right. You're the best.'
So, we're in this, I don't know, awkward space where they do some things that feel magical, right? So, brainstorming for some people, I don't get that much out of them, but like I'm working in disciplines I know well. If you're working in an area that you don't know very well, it'll give you a few things to get started. It feels magical. All right, so there are some things that are good about them, even though I'm not rattling them off personally. And they're clearly good at writing code and so forth, but they come with these consequences too, right? They come with consequences that they make stuff up, that they are so sycophantic that they lead people into delusions. This has been documented a number of times and so forth. And so, society has to make a decision. And the initial decision was, well, we'll just let it all ride. There's so much fun to play with that who cares what the consequences are. And what the subpoena, which was I think filed by New York State, but is part of a larger, I think 46 states or something like that are involved, is a statement that no, we're not going to let all this ride. Like, you know, there are different theories about how to proceed. One would be if you cause all of these problems, you should be held responsible for all those problems. There should be financial penalties. There should be warnings, etc., etc. Another is like maybe you shouldn't distribute the product until you can fix these, right? There are different ways to address it. But the initial reaction was to just completely give a free ride to companies like OpenAI and say, 'Hey, these are great.' And now society is waking up and saying, 'Hey, there are a lot of consequences. We have, you know, suicides that seem to be tied to these things, and we have the delusions. You know, we're destroying the educational system because the students are using these things, and we're ruining critical thinking skills and so forth.'
You know what the companies want to do is, there's a famous phrase, I actually tried to find the origins but it's so old I couldn't find, but is to privatize the gains and socialize the costs, right? They want to make whole society accept the costs while they get rich.
