This week, I'm going to tackle a big and important subject: artificial intelligence and the risks it could pose to society. Now, I'm going to be honest here. I've been putting this subject off for months because it's so big and so technical that I felt intimidated by the idea of covering it, but the subject is increasingly dominating the news, and I'm a journalist. So, it's time to face the music and do my job. The good news is that my guest this week, Gary Marcus, is a real expert, someone who's been theorizing and writing about AI, including its dangers, for more than a quarter century. He's a well-known American psychologist, cognitive scientist, and book author, who also runs the popular Substack Marcus on AI.
But before I run that interview, which I recorded last week, let's review some key terms so that the discussion you're about to hear makes the maximum amount of sense. First of all, and I'm grossly simplifying here, AI comes in two big categories. The first, which you'll hear Gary often refer to as symbolic AI, basically operates like a very big and very sophisticated version of a conventional computer program with lots of rules and decision trees and if-then statements, just like you remember from middle school computer class, except on a much, much bigger scale. This kind of AI is great for, say, creating a database or setting out rigid decision trees for an airline pilot to follow, but it's really bad at a lot of basic tasks that even small children can do with their eyes and ears, like, say, distinguish a dog from a cat or a plate from a Frisbee.
The second kind of AI, which you'll hear us refer to as neural networks or machine learning or deep learning, which are overlapping concepts, is completely different. This kind of AI, which became ascendant in the early 2010s, isn't created with explicit human instructions. Rather, the algorithms teach themselves by reference to massive troves of pre-existing data, including through a technical process you'll hear me refer to as back propagation.
By way of example, let's say you want to create a neural network to identify a cat. Instead of writing thousands of lines of code relating to fur density and whisker size and meowing noises, you just show the neural network millions or billions of pictures of cats and millions or billions of pictures of things that aren't cats, and then instruct the computer to teach itself the difference between the two by a computational process of trial and error. The idea here is that the neural network will start with a completely useless algorithm, then compare the results to the data set, and then try again with tweaked parameters, teaching itself through a lengthy process of machine learning to iteratively refine its algorithms so that it better aligns with the data you've already given it. The humans aren't giving the computer rules about what a cat is or isn't. They're just providing the data trove and then hitting the go button, and the computer teaches itself.
Now, these neural network systems are easy to scale up by supplying them with more data and computing power, which is one of their big selling points to investors. But the problem, which my guest identified 25 years ago, is that there's a drawback to self-taught artificial intelligence systems that literally play by their own self-created rules. The iterative, self-constructed nature of their mechanisms means they can sometimes hallucinate artificial realities, a problem that my guest predicted in a 2001 book. Even something as basic as a simple database of public figures, the sort of thing human resources departments at large companies were creating on 1970s-era mainframes, well, that's alien to the neural network information architecture.
As a result, purely neural network-driven AI systems aren't very good at following user-supplied rules, even fairly basic ones like, 'Write me an academic paper with citations, but make sure those citations aren't made up.' Or, 'Find and download some information for me, but please don't hack into any proprietary servers in the process.' Or, 'Make a trillion paper clips for me, but make sure that none of them are made from the bodies of humans.'
I know that sounds weird, but I promise it'll make sense after you listen to the interview. Anyway, all of this has set off a movement known as the AI doomers, who don't just think AI will take our jobs and access our banking information, but that it will cause the extinction of humanity itself by, say, launching nuclear weapons or creating and unleashing some kind of super powerful bioweapon of its own creation. While my guest is definitely not a doomer, he does call himself a skeptic, especially when it comes to what he regards as the allegedly irresponsible and dangerous business practices of certain companies, OpenAI in particular.
Unlike US Senator Bernie Sanders, Gary Marcus doesn't want to ban hyper-advanced AI systems that greatly surpass human capabilities, sometimes called superintelligence. But Gary also doesn't approve of Donald Trump's apparently laissez-faire attitude either. And he'd like to see a more balanced approach from government, an approach that holds AI companies to account if they don't place appropriate safeguards on their products and align those products with human values.
And what would those safeguards look like? Well, this gets us back into the technical sphere. Gary has long advocated something called neurosymbolic AI, which, as the portmanteau nature of the name suggests, combines the extraordinary power of neural networks with some of the safeguarding and fact-checking features that can be implemented through the rules-based, predictably algorithmic features of so-called classic symbolic AI. It's all very complicated, I know, but I'm confident I have the guest who can walk us through it. Please enjoy my interview with Gary Marcus, AI expert extraordinaire.
Gary Marcus, thanks so much for joining the Cool podcast.
Okay, so you self-identify as a generative AI skeptic. Tell me the difference between a skeptic and a doomer, or is doomer just a term of abuse heaped on skeptics by people who aren't skeptics?
Sometimes people call me a doomer, but that usually means they have no idea who I am or what I actually believe. I've been a skeptic about a number of things, including claims about AI imminently taking over the world and stuff like that. Doomers actually love claims like that. You know, the sooner the nerd rapture or our demise comes, the better from the perspective of them getting funding or being vindicated or whatever. Doomers believe in literal extinction of the human species at the hands of AI.
Just to be clear, they're not advocating for extinction.
They're not advocating for it, but their core belief is that AI might lead to the literal extinction of humanity. I think that that's very, very unlikely. We can talk about why, but that really is what defines the doomers. I am concerned about harms that AI can cause. And so I think some people get confused and they think that if you issue some alarms, even though I have denied other alarms, I therefore must be a doomer. But that's silly.
I think in 2001 you wrote a book called The Algebraic Mind: Integrating Connectionism and Cognitive Science, and it discussed two theories about how the mind works or could work. One is as a computer as we conventionally think of it, that manipulates symbols kind of like a giant calculator, I guess, for lay people, and the other is more like an organic brain, which is a large network of connected neurons. And you basically said it can be both, and that a system of connected neurons can actually function effectively as a formal symbol processor. So it's like kind of a hybrid system. Is this hybrid system of a neural network and a formal algorithmically based giant calculator, is that hybrid what modern machine learning duplicates?
Roughly speaking, the thesis of the book was that neural networks, they're not literally models of brains, but wouldn't work on their own the way people were thinking about them unless you brought these classical ideas from symbol manipulation into the mix. And for 20 years, I was variously ignored, ridiculed, etc., by people like Jeff Hinton, who really thought you could use neural networks by themselves without having any of this classic symbolic stuff.
From Google. Actually, I interviewed him here in Toronto. I'm not sure if he's still in Toronto, but he's considered kind of a leading figure in the machine learning community.
He's often called the godfather of deep learning, which is accurate, or less accurately the godfather of AI, which is not true. I mean, AI existed when he was four years old. Media often is sloppy about that. Deep learning is one form of AI. It's not the only form. It's the one that he's most associated with. But yeah, he used to say that we don't need to have any symbolic stuff at all. That it's stupid. He was very militant about this. A lot of the people in the field were. Everybody seems to have forgotten this, but that's what it was for some 20-plus years.
If I'm summarizing his view, he'd just use back propagation and just give it terabytes of data and it can teach itself.
That's right. This whole idea of scaling was that all you needed was more data, back propagation, and the system would learn everything it needed to know. And I kept saying, no, you can't do that. Your system is going to be unreliable, unpredictable. It's going to make lots of mistakes, and eventually you're going to run out of runway to do this. And what happened quietly, the industry really doesn't like talking about this, is they actually moved to exactly what I said you needed to do, which is they started actually adding symbolic systems. The craziest version of that is this morning Jensen Huang announced a new safety system, and maybe we'll get into that. It's completely symbolic AI.
Nvidia announced this earlier today. Maybe we'll get into it. But what basically happened since 2023 is people started putting in things like harnesses and symbolic tools and if-then statements. They just took everything from classical symbolic AI and decided scaling wasn't going to work. And now they are actually building neurosymbolic systems. So it completely vindicates what I wrote in that 2001 book that you couldn't do it only with the neural networks. We're not doing it only with the symbolic stuff either. And that was the point of the book, is we needed to bring them together. The field, for various reasons, does not want to acknowledge this. The reasons mostly have to do with funding. It's easier to sell the scaling story and so forth, but that's not what's happening anymore.
In that 2001 book, you also talked about, which I guess maybe was more theoretical then than now, how neural networks can lead to full-scale hallucination.
Yeah, I warned about it then. The argument that I made was you really need to have database records to keep track of particular individuals. The same way that a classic symbolic AI database would have a record for you, a record for somebody else. Here's your name, here's your address, here's your phone number, etc. Like in your contacts database, for example. And large language models don't actually have that, and nor did their predecessors. And I gave an example of an earlier model in 2001 and showed how it basically blurred together different people's information such that hallucinations, I called them overgeneralizations then, were inevitable in that kind of architecture.
And then when this stuff started getting popular in 2019, I said you're going to have this problem. And when it became really popular in 2022, I said you'll continue to have this problem. I wrote an essay on Christmas Day 2022 saying what to expect when you're expecting GPT-4. Everybody was wild about GPT-4. And I said, 'No, it's still going to hallucinate.' Because that's inherent to how these things work. They don't have database entries. They do this statistical approximation. They're going to hallucinate. And even now today, they do continue to hallucinate. People use harnesses and things like that to reduce that. But your core, very large-scale model continues to hallucinate. That problem has not gone away. You know, any number of times I've been told, 'Oh, it'll go away in a few months or whatever.' Like Reid Hoffman told that to Time magazine in September of 2023, but they're still here.
So, is the rough idea here that the neural network is the base of the model in terms of the power of the model, but the more algorithmic, symbol-based stuff, that's how you construct the guardrails?
That's some of what's going on. Now, the symbolic stuff does a lot more than just the guardrails, though. For example, in all of these math discoveries, the symbolic system is checking, is verifying whether the math is good. So the LLM is actually proposing many candidates, most of which are wrong, and then a symbolic system goes through them one by one and says, well, this one might be legit, this one's not, and so forth. So there's actually many ways that you can put these two systems together, or these two approaches, these two frameworks together.
One is use symbolic stuff as a kind of guardian. That's what we see in the system that Jensen introduced today. Or you can have it do a bunch of checking. And that's what's going on in some of the math cases. If you just want an agent to keep trying something until some condition is met, you use symbolic stuff to guide the agent to do what it wants. There are probably better ways to do this. Those are some of the ways people are doing it.
Now another example is Waymo is actually neurosymbolic. So the overall architecture is symbolic. It's going to take in perception. It's going to define the objects that are out there. It's going to make a decision. It used to be purely symbolic. Now they put in neural networks inside of it to speed up certain processes, to make them work better and so forth. So there's actually, even though nobody wants to talk about it, neurosymbolic AI is kind of everywhere now. It really won. You know, the thing that I argued for 25 years ago won. The industry doesn't say it, but the reality is that it's not even close anymore.
I interviewed Geoffrey Hinton, I guess it was about a decade ago, and this was, I think, maybe still the triumphalist phase of back propagation and neural networks. And he gave me the example of how symbolic-based algorithms, for say, if you show a computer a picture of a cup and say, 'Is this a cup?' you can pull your hair out. Just purely symbolic approach, you're going to get false positives and false negatives. And there was some competition he talked about where different systems never got beyond 70 or 80%. But as soon as you got back propagation, where you just fed it a lot of data about what is a cup and what isn't, putting aside the rules, here's a billion photos of cups and here's a billion photos of not cups, the hit rate went up to like 95%. And the idea, as I understand it, was that that was the future.
Well, so I mean, it's very interesting to reflect on that. It is true that the neural networks are better at that particular kind of computation. You know, what you see in biology is we have different systems for doing different things. The pure neural networks are great at pattern recognition of individual objects. They are much better than their symbolic counterparts. And it's not just visual perception but auditory perception. They're much better, mostly with some asterisk, at speech perception, at recognizing words and things like that. So they're better at that. These two kinds of systems have complementary strengths.
So nobody has actually put together a system that can really understand a complex scene and understand the relations between parts and so forth. But the symbolic stuff is, generally speaking, a little bit better at that, I would argue. But it remains to be seen. In math, what we found is that the neural networks couldn't do basic arithmetic by themselves. That pattern recognition wasn't general enough to deal with complex multiplication problems. And there you needed symbolic systems. Symbolic systems, they're still way better at arithmetic. And it turns out these fancy theorem-proving things that people have been talking about later actually require hybrids. If you do a neural network by itself, it'll come up with hundreds of proofs or thousands of proofs, most of which are wrong. And it's only by filtering them through the symbolic system that they're right.
So what Hinton did was actually to overgeneralize. Right? He found a problem that his technique worked well for and assumed that it would work for all problems. And that's false. And that's what I think the field empirically discovered over the last few years and what I showed theoretically in 2001.
I remember he was working for Google at the time. Maybe he still is. I don't know. But it happened to be a fantastically useful problem to solve. As anybody who's used Google Lens can tell you, image recognition has all kinds of uses.
Yeah, there's plenty of use for both techniques. The problem was, and I think you used the word yourself, was the triumphalism and the imperialism. So, it does work for some things, but it would be like saying that you can use your visual cortex for your entire brain. And this is just not true, right? And so, you know, the kinds of pattern association you do in low-level vision to recognize objects is not even enough to get you to understand complex scenes, and it's certainly not enough, for example, to give you language. A technique can be useful for one thing and not others. This is a point that a lot of people don't appreciate. A lot of people think essentially you just need one mechanism to solve cognition, but it's wrong.
And you should be able to connect that with your experience and realize, for example, that people can be good at the verbal SAT but not the math SAT, or vice versa. You know, cognition is not one thing, but many. There are many things that go into it. And it was naive, I think, to think that the thing that was good at object recognition would be good at everything. If you're walking on the street and you hear a sound and your neural network says that's thunder, and then you look up at the sky and there's not a cloud in the sky, then the symbolic logic takes over and says cloud cover is a necessary but not sufficient condition for precipitation and thunder, and your algorithmic system overrides your neural network.
I think it's a good example. I would call that inference to the best explanation. I once walked in Manhattan when I was an NYU professor and I saw snow on the street in like the middle of the summer. One possibility was it got really cold, but a much better one was, oh, they must be filming a movie here, which was the correct... I would have guessed the Zamboni. I'm a Canadian, so I guessed the Zamboni dumped it out of a rink because you see that all the time.
Yeah. I mean, I would have... Yeah, if I had been in Canada, that would have been a reasonable explanation. And that process of considering alternative hypotheses is something that large language models are not particularly good at. But humans are good at it. And presumably symbolic cognition plays a role. Though we don't fully understand how humans manage to do it. As complicated as both of these systems of information processing are, even more complicated is this sort of meta overlay that you have to put on it saying, 'Well, this is a good problem for a sort of database-based symbolic, and okay, well, this is image recognition, so we're going to farm that out to the neural network.'
You've spoken about something called neurosymbolic AI. Is this a kind of meta way of task allocation?
I wish it was. So neurosymbolic AI is just finding ways to integrate the two. But I would call what you're describing orchestration. You think about an orchestra, you have all of these different potential contributors, right? So you have your string section and you have your percussion and so on, and the conductors and maybe the composer are bringing them in at different times, coordinating them. That's what orchestration is, right? The brain certainly does a lot of orchestration.
If you look at this long literature that started, I guess, in the 1990s, late 80s, doing so-called brain imaging, color pictures, this is your brain doing this, this is your brain doing that. The one thing that really comes through from that literature is that the brain does a lot of orchestration. It brings in different parts at different times. There's that old line, it's not really true, about you only use 10% of your brain. The one way in which it's kind of true is for any given task, you might only use 10 or 20%. It's not a fixed number, but a lot of your brain is idle in any given moment. And the brain is brilliant at that meta-level orchestration that you were just describing, figuring out which things you need at a particular point.
You probably need neurosymbolic AI to do that effectively. But I would say the neurosymbolic AI just gets you into the game on that. And then we haven't really figured out how to do that well yet, how to do that orchestration. So, like, you know, you'll pull in your frontal cortex when you need it and maybe your amygdala when you need it and so forth.
There's a brilliant study by David Hauser and some others at NYU where they compared people watching a Hitchcock movie, where everybody in the audience is coordinated because he's so good at manipulating the audience, versus a control group. And I'll get to the control group in a second. So in the main part of the study, what he found is that everybody in the audience had the same orchestration of their brains. They would go to, you know, they see this scene, they're terrified, they use their amygdala. They see this scene and they're more thoughtful, like how did this work and so forth.
The control group was you just showed people a movie of people randomly walking through Washington Square Park where there's nothing happening. Then people's brains weren't correlated anymore because one person's thinking about laundry and the other's wondering if they're going to have sex tonight and so forth. So they're all, you know, they're uncoordinated. But if we have a stimulus that moves over time in ways that engage us, that orchestration is very similar from one person to the next.
Now it would be great if AI could do that in an effective way, orchestrate with respect to novel problems. We don't have that in a general way yet. There are some special cases I suppose you could consider that way. But I think that's one of the important open problems in artificial intelligence is how to take a complex system with multiple resources and get the ones that you need involved in a particular moment. There are some ways people have worked on that, like there's something called mixture of experts, which is actually very popular and useful now, that does a bit of that, but I think there are important discoveries there that we probably have not yet made.
When people talk about the term alignment, which is this idea of sort of guiding AI toward non-destructive, non-crazy, non-hallucinatory paths?
Yeah. Can I give you a slightly different definition? I would say alignment is just about getting AI to follow instructions. Usually people talk about it with respect to obeying human values.
But is it different from orchestration?
Yeah, it's a different thing. I mean, orchestration is a kind of description of how you get different pieces of a system involved in a task.
It's a technical process. Alignment, at least abstractly, is so far not a technical process. It's a problem. The problem is how do you get AI to follow instructions? You might need some good orchestration as part of it, but I would say it's not enough. The central case of alignment has always been can you make a system that won't, for example, turn us all into paper clips. So there's this famous example where you tell an AI system make a lot of paper clips and it does, and then it runs out of raw materials and it starts melting down people to get more stuff to make its paper clips. And so you would say that system is not aligned with human values because human values would say don't turn people into paper clips. So that's an example. There are less fantastical...
is not right the word. They're more prosaic examples. So it's also alignment to say to an LLM, don't make up any references, and the systems will be told that in their system prompt or in a prompt from the user, whatever, and they do it anyway. That's also a failure of alignment. It's not as dramatic as turning us all into paperclips, but it's actually, in my view, an instantiation of the same problem, which is the kind of technology that everybody is focused on right now is not very good at instruction following. And if you can't follow instructions like don't make up references, you're probably not going to be able to follow instructions like don't cause harm to humans.
There was a famous case — I guess it's well, famous, it's a couple of weeks old — where there was this scholar who wrote a paper where like 50% of the citations were just completely fictional. But the eerie thing from a human perspective is each citation on its own sounded completely credible. They were from journals that don't exist, but the names of the journals were all very credible. This has happened probably millions of times over the last few years. I wrote a whole essay about one of these systems with respect to my friend Harry Shearer, who's an actor. You might know him from Spinal Tap.
And he does voices in the Simpsons. Does Mr. Burns and so forth. So...
I'm friends with him. And I had written — I'll tell you the whole story.
