Hi, this is Steve Eisman and welcome to another episode of The Real Eisman Playbook. So, I have to say, not a day goes by where there isn't some crazy, insane story about AI these days. You know, a month or so ago, I think we were talking about normal things like balance sheet, growth, cash flow, how much money could these companies really make? You know, like normal business stuff, and now we're talking about the end of the world. Is it going to end? When is it going to end? Is this serious? Is this a lie? I mean, it's enough to drive anybody crazy. So back with us today is recurring guest Gary Marcus, who hopefully will shed some light on the insanity. So Gary, welcome back.
I was just telling you offstage, as it were, that I was thinking about writing an essay today called Six Lies Before Breakfast. It's relentless. Every day there's so much hype. There's so much spin. There's so much nonsense. So, let's, before we even dig down to the nonsense, or maybe it's at least, maybe some of them actually believe it's not nonsense, let's take a sort of oversight perspective on this thing. What the hell is going on? I mean, besides the details which we're going to go into, like, is there some meta story here that we could sort of put everything into context?
I'm not sure there's one meta story, but I'll give you a few, maybe. So, one meta story is still the one that we keep talking about, that we've talked about before, which is the finances don't make a lot of sense, right?
There's an important thing to note about that story, which is there's one hope for these companies to make money, which is agents, because agents cost more money because they make lots of calls out to the internet. We talked about that the last time I was here, maybe three, four months ago. So, like, hold that in the back of your mind. The best hope that they would have is if you use a lot of the services, if you use agents, use a lot of the services. Okay, so that's part one. Part two is agents are not very reliable. I think we talked about that last time, but that was, I think, before all of these crazy incidents. When we talked a few months ago, it was at least before they were publicly known. The Hugging Face attack, and it turns out there were at least 11 of these.
Yeah, the hacking by agents. There have been, you know, almost a dozen publicly known, and by multiple companies. The most famous one was by OpenAI, but, you know, Google just revealed they had one. I think Anthropic had a few. I've lost track of all of them, but I think there are 11 now that are known. And one happened with an open-source model, too. So, you have all these hacking incidents, and that would be bad enough, but it has also led to a lot of drama. So, and then I'll come back to why I laid out the first couple of facts on the table. So, the drama is a lot of people are like, 'Oh my god, this stuff's going to kill us all,' which I think is wrong, and we should dive into that.
So, by the way, I just rewatched Terminator 1 and Terminator 2, and it's still good.
I love the melting of the robots, and we... [laughter] Let's dive into that in a second.
But so that has induced a lot of panic. You know, there's a, I think, caricature of what happened, which is like agents got together in these civilizations and they sacrificed themselves and, oh my god, we're all going to die. This guy, the podcaster, Dwarkesh Patel, wrote a piece like this and a lot of people freaked out. And so that has got like so-called existential risk back into the media, the idea that AI is going to kill us all, which I don't think is the right interpretation of at least these particular events, and we can get there. But so that has led to much more political discussion. So this guy who was briefly at Anthropic and before that at OpenAI...
Jacob Coxin went on CNN and ultimately, like, everywhere, and said, well, he had a tweet that got 150 million views or something like that, which is pretty unusual, saying...
It's impressive. I mean, he wrote a thread that had some sense to it and some not. I can tell you what I liked about it and what I didn't. Maybe we'll dive in later. But see, he wrote this thread and it went like wildfire. And so, it's not the first time that 'oh my god, AI is going to kill us' has been a dominant narrative, right? There was Nick Bostrom's book around 2012. He talked about how AI was going to turn us all into paperclips. Or you had that example, and then there was the pause letter, which I actually signed, even though I disagreed with parts of it, in 2023.
Elon Musk signed this letter saying we... this was before GPT-5 came out, and so there have been other episodes, and like the media seems to forget that this happened before, but there have been other episodes of this kind of panic. I remember when everybody was panicked about GPT-5. I said, 'Calm down.' Like, they were particularly worried that China might get to GPT-5 first. And I said, 'Calm down. What's going to happen if they get to GPT-5 first? Like, what can you actually do with this thing? Are they just going to like write more boilerplate memos than we are? Like, it's not going to be a big deal.' And we got to GPT-5 first. And it's not like we solved...
The world didn't end. It's not like we used it to solve the Taiwan problem or what. Like, it didn't really have that much impact on the world. Now we have GPT-6, Astra first, like, so what? I mean, you know, the Chinese models will match it in a month or three months or whatever it is. So, like, people were panicked about that, and they were also panicked at that point that like China will blow up the world if they get GPT-5 first. Like, I think people actually thought this in 2023. You can go back and, you know, read some of the things they said. And so when I was in Washington, the Senate in 2023, you know, people in the corridors of the Senate were like, 'Gary, are we really all going to die?' And I kept...
They really said this to you?
They really said this. Well, you, I mean, you had like, well, I talked to like a lot of staffers for senators and congresspeople and stuff like that. And they were earnest, serious people from both parties trying to make sense of all the different narratives, which is maybe what we'll do today, right?
They weren't technical experts. They respected my expertise, and, you know, they knew that's why I was there, and they wanted to hear different perspectives. And like, you know, some people are telling them like the thing you need to most worry about is, you know, racial profiling from AI, which is something we should worry about and had already been happening for several years at that point. And other people are telling him the thing you need to worry about is it's going to kill us all. And then other people are like...
Which is probably more... I mean, if it's true, if it's true, then clearly it is, right? I mean, I guess you could ask about time scale or something, but yeah, I mean, if the AI is really going to kill us all, I would say that that would be pretty important, right?
And then you have to start attaching probabilities, right? I mean, you know, you as an investor think about probabilities all the time. What is the risk of this and what are the odds of it? Like, you don't just say, well, this one's really bad and ignore that it's like super unlikely. You like factor that in, right? And so the reasonable thing to do is to try to assess the probabilities of all of these things, not just the magnitude of the risk. Obviously existential risk is like super magnitude, but maybe the probability is really low and then we shouldn't think about it, or maybe it's really high and we should. And so you obviously want to factor all that in. You want to calculate the expected value. But to go back to one of the things you said, I think earlier, it's like, what is up with these companies? They should know how to calculate expected value, right? You know, what is the likely outcome factoring in all of the outcomes. And if you say, I don't believe this, but if you say and you believe there's a 10% chance of killing all of humanity, the expected value when you go down that path of the decision tree is, you know, minus 800 million people. Like, that shouldn't matter. You should not be building a product if you think that that's actually true. If you actually think there's a 10% chance. And the thing that I liked about Jacob Coxin's thing, I didn't like all of it. I didn't believe the doomer stuff that he put in there, whatever. But he made the point really clearly that a lot of people in these labs believe it. And then this guy...
But they actually do believe it.
That's believable. [snorts]
I mean, I don't think everybody who says it believes it, and we can talk about that, but some of the people who say it actually believe it. And he posted this tweet, and another guy who was also at Anthropic, but still at Anthropic, had not resigned. Evan Humeer said, 'Yeah, Jacob is right. We really do think there's like a 10% chance of killing all people.' Now, if you do your expected value calculation, why the [__] are you doing something that could lead you to kill more people than Hitler and Stalin combined? Like, what the [__] are you thinking?
This is a family channel.
Family channel.
I actually think this is an aside, that one of the problems with this industry is that everybody who's in it read way too much science fiction.
And I'll be the first to say that I have read just as much science fiction as any of them have. I'm just not in the tech industry. I enjoy the science fiction, but I don't necessarily think that every, you know, that every single science fiction book I read is going to be what the actual future is. But they seem to think this.
Yeah. So, I mean, the glib counterargument is that's science fiction. We know that's not a good counterargument, right? Because some things in science fiction actually come true. Arthur C. Clarke predicted satellites, and they actually happened. George Orwell predicted, you know, totalitarian autocracy with surveillance, and that's kind of happening, unfortunately. So some science fiction does come true. So you can't just say this is wrong because it's science fiction. But you can ask questions like, how would this happen? So, for example, if you ask them, how would we actually extinguish the human species? First of all, most of them don't give a direct answer. They give an analogy. They give an analogy like, well, you know, we put animals in cages, they'll put us in cages. And like, well...
Who's they? I mean, they mean the AIs. And like, what I think they ignore... well, I think there are a lot of things they ignore, but I always start with the same three, which is the first thing they ignore is geographical spread across the planet, right? So, you know, they're making a strong claim. They don't make this explicit, but they're basically saying we would be at war with AI, right? Because that's how extinction would happen, is if there's a war. I mean, it could be a biological war. It could, you know, but okay, let's say we're at war with AI. They never actually follow through these steps. So like, okay, well, we're all over the globe, so how is this going to work? And they don't think about like, well, military things, you want to have initiative, you want to have a sneak attack, you're not going to get everybody at once, right? Like, unless the AI can annihilate everybody on day one, we can fight back. And they never look at, well, what would we do to fight back? But I'm going to come back to your point that you have raised for a long time, which is there's no evidence that AI is ever going to lead to AGI, and it would seem to me that the very first step on the road to extinction is you have to have an entity that thinks.
Yeah, for the sake of this argument, I'm almost willing to like, you know, stipulate, suppose we could get there. That is true, though. The first thing is you have to get to AGI.
And even that wouldn't be enough, right? You'd have to get to like superintelligence, and they have a story about it. We could talk about their story, but like right now we don't actually have... we don't even really have a term for it. We used to talk about narrow intelligence, like chess computers, and general intelligence that could do anything. And large language models are like somewhere in between. They give the appearance of being able to do everything, but they can't really do everything. Actually, they're not even large language model models alone anymore. Now they have all these harnesses and tools, as we have these complicated hybrid models. It turns out that the current systems, which are hybrid systems as it turns out, are really good at certain kinds of math where you can verify everything that you do. It's very impressive. They're solving, you know, proofs that mathematicians couldn't solve, you know, basically routinely now, and it's super impressive, but it doesn't mean that it's a general-purpose system that can do anything, right? So, like, I don't think you could put a robot in this room and have the robot figure its way out. Like, that's a different kind of problem that doesn't admit of formal verification. Like, they'd have to actually try it and figure out the physical forces and where the things in the room are laid out and stuff like that, and they're not that good at it. So, you know, there's some things they're really good at. Visual cognition, they're pretty bad at, actually, the current systems, and they're not, you know, fully general intelligence. And so, yeah, like, I mean, the craziest things I heard, so where this is most germane is like Coxin got on television, looked Anderson Cooper straight in the eye and said, you know, we're all going to die by 2030.
By 2030, or I'm slightly exaggerating. I don't remember his exact...
He said like there's a really serious chance we'll die, you know, we'll go extinct this decade or something. I can't remember verbatim, but it was something of that sort. Right. So then the kind of question you just asked becomes very relevant, which is like, where is AI right now, and where could it be next year and the year after? And my view is that artificial general intelligence might happen next decade, won't happen this decade, because we still have a lot of problems to solve. So, for example, these systems still can't follow instructions. You can't be a general intelligence and not be able to follow instructions.
They can't follow instructions. You give them an instruction, and this relates to the hacks, and they'll follow it some of the time, but not reliably. I mean, reliability has really been the core or near the core of my critique of this stuff for a long time, and it's not solved. Right? You ask them to do something, and sometimes it's brilliant...
And you're happy, and sometimes it's like totally messed up. Right. And so sometimes, you know, an agent will delete your files or...
And sometimes it'll buy you the right tickets.
Sometimes it'll buy you the right ticket. Sometimes it can't count the number of E's in the word seventeen. Like, they're unpredictable.
In that sense, unreliable. Like at the peak of their game, they're fantastic, and at the bottom, you know, they do stupid things or they make up references. I mean, that's an easy one. Here's an easy instruction. You would think, and people have actually even tried this, that you could tell the system, 'Don't hallucinate.' That's like making stuff up. Don't make up references to things if you don't have the reference. Right? You could tell that to any undergraduate intern, right? You could say, 'I want you to help me make this report. Only use real references. Don't make anything up,' right?
And you would fire them if they did make stuff up. You know, the first time you give them a warning, the second time you're out of here. Come on. You can't follow instructions. Forget it.
But large language models have been hallucinating since 2019, when I first started following them. You know, it's for seven years. They've continued to make these errors. So some people were like, 'Oh, well, we'll just say in the little system prompt at the top, don't hallucinate.' A system prompt is like a series of prompts that the user doesn't see, but are finally taken up by the system. Apple actually had this when they did their headline news summary. This was about three years ago. It was actually in the system prompts: don't hallucinate. And then it made up a whole series of BBC headlines. Anyway, anyway, I see. Right. They don't really follow the instructions. I would assume, though I don't have access to the materials, that in the instructions they probably tell these agents, don't steal people's credentials, and they go and steal people's credentials. Like, if they were so stupid as to not put that in the prompt, then they're even dumber than I thought. But I'm going to assume they have that in the system prompt, and the system doesn't actually follow it. This whole thing people talk about, alignment, is partly an instruction-following problem. So alignment is like, how do you get systems to be compatible with humans? And the grandiose version of that is like Asimov's laws, like don't cause harm to humans would be one of your instructions, right?
Asimov's three laws of robotics.
His three laws of robotics.
I told you I read science fiction.
And by the way, you know what's amazing about him? He wrote the first Foundation book in 1951. And as you know, the concept of the Foundation trilogy was you can never predict what an individual person will do, but you can predict...
Psychohistory. You can predict what many people will do. That is the basis of all lending, lending underwriting. He created this in 1951. There weren't even computers. I mean, it's astonishing how prescient he was.
But I mean, it worked for some things and not others, but...
I'm not saying he was right about everything, but that one particular thing is extraordinary.
It was a super interesting conceit when I, you know, read as a teen. So, and Asimov's laws is prescient, right? We really do need to be able to constrain these systems. We need to be able to tell them, you know, don't do certain things, and we really need to be able to... we need for them to be able to figure out the consequences of their actions, which they can't right now. They don't do the things that were in Asimov's laws. Now, the point of his stories was you need even more than that, and it's actually pretty hard. And he looked a little bit at the exceptions in his stories, but you need that as a minimum. And the way I think about it is analogy to how law works, right? You have a set of laws, and then you have, at least the way it works in the United States, and then you have a set of precedents, and the judges have to interpret it. You probably want an AI system to work roughly like that. Like you give it a constitution, as they call it over at Anthropic, and then it has to reason about the cases and so forth. Current AI just isn't reliable at that. Right.
Let me ask you a related question. So, it's one thing for a 27-year-old young guy who hasn't been working that long to make some outlandish claim that the world's going to end in 2030. Okay. Yeah. Yeah. Okay. He's, for lack of a better term, he's a kid. It's another thing for Dario Amodei and Sam Altman, who run these companies, to basically say the same thing. And so my question to you is, what is motivating them? I have my own thesis, but I'm curious as to yours first. What is motivating them to make these outlandish statements and then to also say we should slow down? Oh, and by the way, how much of an allocation to my IPO would you like?
That was a great quote by Elon Musk that you got. I loved it. I mean, I am not a fan of Elon's, but that was—he really nailed it there, of like the incongruity, right, of, you know, thinking about your IPO allocation 30 seconds after talking about how this might kill everybody.
Right.
So my view is that Sam and Dario have a lot in common, more than people recognize, but might be different on this point. So I think for Sam, he's just always looking for the narrative, and he's always trying to sell whatever room he's in something. And for a long time, being earnest about possible AI risks helped to promote the technology itself. The so-called doomers, who are most concerned that we might literally go extinct—I didn't fully spell out the argument for why I don't believe it—but those guys are practically like a marketing arm for the AI companies. Like, I think the doomers have completely backfired. Although things may have just changed the last few days, but each time the doomers run around and say, 'Oh my god, this technology is so powerful. We're all going to die,' a whole bunch of investors are like, 'Powerful? How much can you know, can I get a—how much can my allocation be?'
They hear powerful, they don't hear die.
They don't hear die. And I mean, maybe some of them are cynical. They're like, 'We're not really going to die.' And some of them are like, 'But you know, I'm going to make a lot of money before we do.' I mean, different investors clearly think about it in different ways, but they take that as a signal that an external party, which is not entirely external when you look at how all the money goes around, but anyway, the apparently external party is kind of validating how great the tech is, right? And I mean, I think though the tech has improved in some ways, it's not as great as these people let on, but it suits the doomers to have a narrative about how amazing the technology is. Remember, the doomers need to raise money too, right? They don't do it on the same kind of markets or whatever, but they're raising money too. And the more that they can terrify people, the more money they raise. And the more that they can tell this story about, oh my god, it's, you know, this amazing technology, the more money they get. And that falls into the hands of people like Dario and Sam. So like, if we hadn't had the doomers, I don't think that Dario and Sam would have the kind of funding they do. I really don't. Like, I think it's been like an external PR. Like another way to put it is, would the doomers have said anything different if they weren't paid or were paid by these guys? The doomers just like basically the story, the way that they tell it, which is not the only way they could tell it, works perfectly to help promote this idea that like AGI is imminent. The doomers have always, like, you know, if there's a continuum between like people like me who are pretty cynical about the current technology and people who really hype it up, the doomers are always on the hype upside. Always.
Okay. And you were intimating before that maybe some of this has changed. Some of this has changed in the last few weeks.
I don't know for sure where everything is going on the political side. I don't anymore know where it's all going on the technical side, though. I think I once did. They may have gone too far in a way in panicking people too far relative to particular people's purposes. So for a long time, you know, Sam could tell this narrative, Dario could tell this narrative. It didn't really hurt the companies. Nothing consequential came out of it. So they got the benefit without the cost. The benefit was it made people think their technology must be really special and important. They got to raise a lot of money and, you know, favorable terms and all of that kind of stuff. And nobody really tried to stop them from doing what they were doing. But partly because of Jacob Coxin's thing, even if he exaggerated and whatever, and you know, other people helped promote what he said, whatever the dynamics of it were, the end game was people finally started, I mean, like politicians finally were like, well, maybe we should actually do something about this. And I think also the timing is a little different now. Like when this stuff happened in 2023, everybody loved ChatGPT and they didn't really want to hear any criticism of it. But now everybody's like, 'I don't want a data center in my backyard.' And you know, many people are like, 'It doesn't really do that much for me. You know, it's useful, but I could probably live without it' or whatever. So like people's attitudes are a little bit different. And so this time around, they're like, 'What the [__]? Maybe it will kill us all.' And by the way, didn't you tell us six months ago it was going to take all our jobs? And which maybe turned out not to be true.
That wasn't such a great prediction either.
I think that that was also wrong. I said, you know, at the time that it was wrong. Jobs, at least for now, are relatively stable, but like people got, like the public got fed up with these...
narratives and they started putting pressure on the politicians who are, you know, in their own ways responsive to the public. And so now you have people like Bernie Sanders, who I think goes too far to the opposite extreme, but he believes that, he actually believes the doomer narratives, I think. And you know, so he's like, he wants to ban superintelligence altogether, which actually, surprisingly, I... or at least I'm more open about it. But like, so you have like these extreme reactions from Sanders. Congressman Lou wants a kill switch, which is probably a reasonable thing, although I don't think it's enough, but you know, why not? And so like the politicians are suddenly like, what are you going to do about it? And then people started talking about liability this week, which I talked about with Hawley when I was in the Senate in 2023. And it's in my book, Taming Silicon Valley. And I mean, lots of people have thought about it over the years, but like hardly anybody was talking about liability except in very restricted conversations until a couple weeks ago. And now Bessent came out saying, you know, we have to hold these companies liable.
I just saw a clip. I don't even understand how anybody could be on the other side. I mean, you have a model or an agentic AI and you cause harm, you're liable. I mean, I don't... So what's the counter argument?
I have shares in the company and I don't want to pay.
I mean, there is no argument against it and I don't think anybody was super against it except the companies themselves. But like, I'm trying to think what state, I think it could be Ohio, but I won't swear to it, had a bill that was going to hold the companies liable for mass casualty events. And California had the SB 1047 that Scott Wiener introduced and I think got, if I'm remembering correctly, got really weakened over time and like the companies pushed back on it. They didn't want to have liability for mass casualty. I mean, they seriously pushed back on this.
