NYU’s Gary Marcus on Building AI We Can Trust

I mean, it could be the next industrial revolution, but the industrial revolution raised productivity, and so far AI has not really done that. There's been a lot of promises. You know, when ChatGPT came out, there were stories about how it was going to fix everything. There were going to be a lot of billion-dollar companies with one employee. It hasn't panned out. There was yet another study yesterday showing that it's hard to find a measurable impact of AI. That was not true in the Industrial Revolution. Although you could ask how long did it play out? We may eventually have AI doing amazing things, but I don't think we will with the current technology.

I think one thing outsiders don't understand is AI is actually a whole grab bag of different technologies. Most of the work right now is on large language models, which I joked the other day might soon be called large liability modules because they're not very well constrained. They don't really follow instructions. And for most business applications, if you have an intern that doesn't follow instructions, you fire them. LLMs don't really follow instructions. All the so-called rogue AI things are examples of that, where they're not really following instructions, they're following loosely. It'd be like if you told an employee, hey, do really well on this exam, and the employee took it on their own initiative to break into the Educational Testing Service in order to find the answers of the exam. You know, you could say, well, you followed my instruction? Obviously, you're missing something here, it is not cool for you to have done that. That's what happened with OpenAI, Hugging Face attack, and 10,000 others.

I don't think we'll get it with LLMs per se. LLMs might be part of the answer. You can think of LLMs as a tool, and that tool might be part of the final answer for artificial general intelligence or superintelligence or what have you. But on its own, it's a very flawed tool. What it does is it approximates what people have said in certain contexts. It has a lot of contexts to draw on, but its understanding of what's going on in the world is superficial, which is why it can't actually follow instructions. In the early days of this stuff, I called it autocomplete on steroids. Now we have autocomplete on steroids with another trick, which is we have verification. So like all these math things that are so impressive, they're still using kind of autocomplete on steroids and then use a separate technology. Not all of them. Actually, formal symbolic AI that has roots going back to the 1950s that everybody kind of dismissed prematurely. And those tools verify them. So it's like you have monkeys, their limbs are sort of like just a little bit like monkeys and typewriters, and, you know, someone else going through the solution saying, yeah, this one makes sense and this one doesn't.

I mean, they really have been solved. That doesn't mean math has been solved. So these systems are very good at solving particular kinds of problems. That doesn't mean, for example, that they can write a proof that someone else understands. It doesn't mean that they've developed any new mathematical principles the way that, you know, Newton or Euler or Gauss did and so forth. But it means that they're pretty good at churning out solutions for known problems, and that's useful. Although some mathematicians are upset about it. We could talk about why they really can do this, but it's because these particular problems can be formally verified. You can do a bunch of logic to make sure your answer works. And we've seen this kind of movie before. So when Deep Blue beat Garry Kasparov in 1997, a lot of people thought the world was going to end, employment would end, and none of that happened, right? It was what we call a domain-specific solution. It could do a particular thing. We saw that with AlphaGo. People made a big fuss over it. But, you know, there's some ideas that were borrowed from it, but it's not like it immediately changed the world. The fact that these systems can solve math problems doesn't mean that they can solve other things in the real world. There are two things. The current AI is really good at coding and math, and they both rely on the same thing, which is external systems, if you want to call them that, doing verification, saying, does this answer actually work? And the problem is lots of things, like the things we've talked about in this conference, you can't do that. You can't verify formally that your solution to what you're going to do about, I don't know, middle powers being treated in a different way is formally correct. You just can't do it. And that makes those systems much less helpful in the other kinds of problems.

I heard that yesterday. I don't think that's a good thing. I mean, you want more code that is good, solid code that is maintainable in the service of a product that you can use. I've seen a bunch of studies showing, like, there are three times as many apps submitted to the Apple App Store because anybody can write an app now, but nobody's using those apps. Right. And so, and then like sort of code slop, a lot of it is code slop. It's a question of, you know, somebody writes this code, but can anybody fix it three years later? So, you know, the real job of that software is, again, because they don't know how it was made in the first place. You don't necessarily know how it's made in the first place. We're killing the pipeline of people who do learn how it works in the first place. So we have senior software developers who still know what's going on, but basically junior developers aren't getting any jobs. The senior ones will eventually age out. We may actually be in trouble.

The thing that they can do well is they can make candidates for drugs. The thing that they cannot do is to tell you if those candidates really work. So human biology is incredibly complicated, you know, 20,000 genes and, you know, hundreds of millions of molecules running around. What you would like to do is to simulate and say, if I have this new candidate drug, will it actually work? But we can't actually do that. The complexity is just way too much. And so you're still left having to do clinical trials for every candidate. So there are more candidates than before. So far, I don't think any cancer drug has come out of AI that's actually curing cancer. I mean, we keep hearing them talk about curing cancer in five years. It's not going to happen. Not going to happen. I'm sorry. You know, both my parents had cancer, and I would love for that to be the case, but it's just not. Because even when you have a candidate, you have a molecule, maybe this will work, you have to do the clinical trials, and they still take five years. So when Dario actually gets on television and says, you know, we might cure cancer in five years, I'm like, first of all, do you understand that there are many kinds of cancers? Like, how can you run this company and not understand that? And second of all, do you understand the process by which we evaluate these things? You do clinical trials. You have to have a number of people that have to last a number of years. It's just absurd. I wrote a whole piece. You know, my Substack title starts... No, well, I think it's like, no, Dario Amodei, AI is not going to solve cancer in five years. And then it's just filled with quotes of all kinds of scientists, even pretty positive ones, and medical people like Eric Topol, who's a pretty big fan of AI in medicine, all ripping Dario to shreds. Like, I didn't have to do it myself. I just cut and paste because it's just so obviously wrong.

I mean, that's a dream. That is a dream. I share that dream. You know, there aren't enough good teachers in the world, and we pay them so poorly that people who might become good teachers do other things. I mean, there are still some good teachers. I don't want to overstate that. But it would be great to have, you know, personalized tutors. And you can use these things a little bit. What happens with most people right now is they kind of zone out the way that, like when you're driving, you're not really paying attention to where you're going. So most people, studies seem to show, don't do better as a function of using AI. Like they might do better on the exam that day, but they don't retain anything. They don't have long-term development of their critical thinking skills. There are some people who treat the AI with the appropriate skepticism, realize that it's not giving you the answer, it's giving you the information that you can use to get to the answer. And there's at least one study that shows that that's actually helpful for some people. Again, you have to contrast where we are now with where we're going. I honestly believe that eventually AI will be really, really good for education, but when you build it on LLMs that hallucinate all the time, don't have deep conceptual understanding, and throw them at people who are inherently lazy, it winds up being a mess. So we need a different sort of AI.

The original concept, you take a transformer, there's this idea of scale is all you need, that all you would need to do is add more data and they would get better and better. They get exponentially better. That's already basically known in the industry to not be true. The industry doesn't want to talk about that because it's a great pitch to venture capital. Just give me more money, bigger data centers, it'll be great. But what they've realized they need to do other things like use classical symbolic AI that they always had ridiculed, but they actually need to do it. So Claude Code is not a pure large language model anymore. It has lots of code, symbolic code that works together with it. So it's a little bit more complicated now. The way things get talked about in the press, it was just AI. But the reality is they're different systems. They have different properties. Some are good for certain kinds of problems. Some are good for other kinds of problems. I think we need a fundamental rethink of what we're doing. One way to think about it is what we have gone all in on is black boxes that nobody understands internally, and I think that was a fundamental mistake.

I think there's almost zero chance of literally extinguishing the species, which is what people talk about. And people like somebody wrote a book, If anybody builds it, everybody dies. I don't think that that's plausible. And I think it undersells humanity. You know, first of all, it underscores the fact that we're geographically distributed. So, you know, you have to think of this as a war, like you're imagining a war between AI and us. You know, reality is like the Times just wrote a piece about robots having a fight with a person, and then you read the fine print seven paragraphs in, the robot was teleoperated. It's not. And by definition, robots are supposed to be autonomous. But the notion is somehow the robots are going to come after us. We're not going to be operating them to kill humanity, presumably, so they'd have to stand on their own two feet. I wrote a joke seven years ago. I said, if the robots come for you, first thing you should do is lock the doors. Because right now, robots don't even know how to unlock doors. You know those locks when it's kind of a latch, you have to kind of pull on, like, you can't do that stuff.

So, and there's just one more thing, like think about what happened on Dubai Air last week. Right. Like, you know, when the chips are down, some humans are brave. And so we would fight them. Nobody would bomb a data center today. But if, you know, they started killing human beings, we'd all be lining up. Give me a pickaxe. Which data center can I kill? And so, like, all of these nightmare extinction scenarios just ignore what humans would actually do. But bad things are starting to happen. So I distinguish between P(doom). That's the kind of San Francisco hip way of saying, what's the probability of all being extinct? People like, go to a party, what's your P(doom)? My P(doom) is near zero. But I also talk about catastrophe and dystopia. So catastrophe is like, could we have an accidental war caused by a hallucinating, unreliable AI? Answer is yes. We almost had that three weeks ago, four weeks ago, whatever it is. When US AI misidentified a Chinese ship as, you know, going in the wrong place and having nuclear weapons. And it didn't actually, that almost led to a war and a huge amount of coverage that it gets. And then two days ago, I think France had this problem with refrigeration, which was, you know, cyber attacks, probably caused by AI. There was a story in the Times yesterday about all the cyber attacks in Japan. Those are, some of those will lead to catastrophe, right? If some large piece of, let's say, electrical infrastructure goes out, that's going to cause a lot of problems. If a large piece of financial infrastructure goes out, that's going to cause a problem. Some people will die because they can't get food and so forth. So the chance of catastrophe, I think, is pretty high. And dystopia, I think we're already basically here. Right? The companies like OpenAI don't really have a business model. They're losing money left and right. And so they're forced to sell all of this super private data that they've gathered to governments and so forth. And so, like, we're already living in 1984. We just don't realize it.