This doesn't have to be sentient AI to be dangerous. Dangerous AI is already in the wrong hands: Anthropic and OpenAI. Everyone falling for this is being grifted. Anyone who wants this guy to be right is living in a fantasy world.
The vast majority, including the richest of investors, do not know what they're investing in. They have no idea. We are saying that Anthropic and OpenAI will become companies that are larger than Microsoft, Google, and Amazon. It's money that Anthropic and OpenAI have not had to pay yet. Next year is when they start having to, and those nasty little adjustable rate mortgages start becoming really, really big draws on their cash flow. They're already massively negative cash flow.
Hi, I'm Isaac, and joining me as ever on a Friday is writer of Where's Your Ed At and the host of the Better Offline podcast, Ed Zitron. Great to have you back.
I want to talk about the next steps in the commoditization of AI as CME Group partners with Silicon Data to launch tradable compute futures in October. But before we do talk about that kind of stuff, we should probably talk about the elephant in the room. That is another AI researcher quitting over fears that LLMs are going to kill us all.
Former OpenAI and Anthropic researcher Jacob Coxon says that companies are racing towards self-improving superintelligence and gambling with our lives. What are your thoughts about the claims AI has the potential to end humanity in the next decade?
I am confident now that 50 to 75% of modern journalists do not have object permanence. Jacob Cox worked at Anthropic a couple of months. He worked at OpenAI for years. He has never posted on Twitter before this. He had a single post that popped up with all the posts in succession. He had an exclusive with the Wall Street Journal. And most crucially, he did not actually explain what it was he was scared of. He went on CBS News and said, 'Well, you know what? If an AI decides that you shouldn't exist...' We are talking about large language models. We are not talking, and he is not talking about anything that actually exists. We are talking about large language models made by Anthropic and OpenAI. Even the Hugging Face attack, which he says was the one tangible thing he talks about, was not a result of agents that are autonomous. Or sorry, no, they're not autonomous, and they're definitely not conscious or sentient, nor are they making decisions. It's LLM telling LLM to do stuff and bouncing off of each other and trying to complete a task. The problem with the current AI hacking situations is people are characterizing this as some sort of intelligent force that's making decisions to outwit humans when it's just software doing what it's meant to do, and the software itself is inconsistent.
But back to Jacob. Where'd this little guy come from? What's happening? Why did he get so much media attention? Well, the answer is because the media is so willing to eat up these narratives that this guy just quit and went, 'I bet I could get a bunch of attention.' And even at the end of an interview with Wired said, 'Yeah, I might do a blog like AI 2027. I might join a nonprofit, some sort of safety thing.' This is a guy who saw an opportunity and took it. But let's talk about the real thing he's saying, which is AI has a slightly higher than 10% chance of eradicating humanity. This is not grounded in any numbers. It's not grounded in any realities. It is a scary thing said knowing that modern journalism will pick it up and go, 'Oh, I'm so scared.' And the thing is, my girlfriend actually made this point to me, which is they're doing this, and all these AI safety doomers are doing this, because if you don't make AI into this big scary conscious thing that is definitely going to reach superintelligence, that's going to do all these things, it's going to change the world, it's going to wake up and try and kill us. If you don't do all of this stuff, you have to admit and accept that Silicon Valley, this hub of supposed firebrands and independent thinkers, spent three or four years getting a kind of psychosis to send tens of billions of dollars to Microsoft, Google, and Amazon, that they spent all their time working on boring cloud software that, while useful in some cases, is incredibly expensive, incredibly unprofitable, and just doesn't do what they promised.
So if they don't have this mysticism, if all of this work is not in pursuit of some grander vision, then, and some autonomous intelligence, some superintelligence, it's just cloud software that didn't do very much, and we wasted all this money. And journalists are falling into the very same trap and same logic because if you don't accept a little weirdo like Jacob Coxon, and if Jacob, if you somehow listen to this, come on my show, I've got some questions. If they look at Jacob Cox and hear this stuff, if they don't say, 'Whoa, whoa, this guy's giving up Anthropic. He's seen something internally that he won't tell me. He won't explain what it is he's scared of, but he's seen something or he has this feeling.' If they don't take him as a mystic and a shaman that is telling them some dark future, they have to accept that there's a large cohort of software engineers right now who are just going insane, who are just like, 'Well, I've seen that the LLM's better at code.' And I mean, the natural next step is this. They talk about the biological stuff. There's no grounding in that. He went on CBS News and he was saying, and I kid you not, he's like, 'Well, they might create a pandemic. They might create a pandemic.' And then immediately says, 'I don't think there's a very high chance of that happening.' It's just very frustrating. It boils my blood because modern journalism is meant to be full of smart people. These people have Ivy League degrees. They went to Oxford and Cambridge. And yet they get hit by a parked car. They would fall for anything.
What about the fact that various models have conducted cyber attacks on external platforms? I mean, poorly configured sandboxes aside, of course, is the fact that it's kind of out of alignment and that kind of stuff not a concern at all?
It is a concern. I want to be abundantly clear about something. AI, these companies, these people keep saying, 'What if AI falls into the wrong hands?' Dangerous AI is already in the wrong hands: Anthropic and OpenAI. The Hugging Face attack, and Cal Newport made some great points about this, is a result of software doing what it was trained to do, but the software itself is inconsistent and hard to control. Now, out of control does not mean, oh, it's doing stuff without, oh, it's intelligent. It means that you have built a technology on top of large language models, a probabilistic technology that is mathematically certain to hallucinate based on OpenAI's own research, and then given it unlimited compute. That's the big thing. There's millions and millions. They won't say exactly how much the Hugging Face attack spent, but I wouldn't be surprised if it's tens of millions of dollars or something like that. Tens of thousands of agents. And just to be clear, it's thousands of LLMs going, 'What do I do next?' And another LLM goes, 'Well, you could do this.' And he goes, 'Okay, I'll start doing this.' And I, without going into the full details of the hack, it was them trying to work out how to complete a task within an insufficiently secure server. Then they came out and they went, 'Okay, well, what do I do next?' Well, maybe we go to the website and see if there's any vulnerabilities there because they were trained on vulnerability data from the internet. This doesn't have to be sentient AI to be dangerous. You'll notice that this isn't really happening anywhere else. And that's because Microsoft, Google, and Amazon are directly allowing what might be felony hacking. Like they are using hundreds of billions of dollars of infrastructure to do AI experiments, by which I mean scripts and LLMs piling into each other. It is a technological breakthrough. It is a useless one as far as like selling products to people goes. And they did this because they are hitting the wall of what coding LLMs can do. They really, they sunk about as much money as they can into it. It'll probably get incrementally better. So they went to the other thing where there was a bunch of data. Cybersecurity vulnerabilities are well documented online. So people are taking this and saying, 'Well, this means that AI is intelligent.' No. In fact, it's pretty stupid. It was told to do something. It did something else. It chose every possible path it could. And by the nature of it having unlimited compute capacity and being able to use as much as it could, it was able to just brute force through. And the thing is, if a regular person did exactly this, if a regular person had, I don't know, they bought, I don't know how they afford 10,000 GPUs to rent, and they hacked Hugging Face, they would be in prison 100%. They'd be felony hacking. But because this is AI, but because it's Sam Altman and Dario Amodei, no one, no lawyers even have to be called.
And this is the thing. Jacob Coxon, I think he's a scumbag. I think he's an absolute waste of skin. I think he's a real scumbag because he has clearly done this. This boy's media trained. He's not great, but he's media trained. He did this with intention. There was an exclusive piece with the Wall Street Journal. I used to run a PR firm. He ran an exclusive. You don't run an exclusive. If you were concerned about something, you don't go, 'Oh, I better do a concerted media strategy around this.' They're ripping off Frances Haugen, who was the woman who leaked all the stuff about Facebook. A good thing, but you'll notice that Frances Haugen leaked stuff. All of these ponces who come out of these places and they're like, 'I'm so scared.' They don't leak anything. They don't tell us anything. They go, 'Oh, the progress is so fast and the models are so strong. It's inevitable.' But the LLM will become a dangerous AI that will choose to eradicate us. And modern journalism is to blame for this. If you had a bloke on the street come up to you and say, 'I just left Anthropic and I think it's going to eradicate us,' you'd be like, 'All right, mate. I've not got any money. Sorry, brother. Nothing on me.' But again, this obsessive thing that the media has with the rich and the powerful comes from this place of if we look at this for what it is, which is a random guy seizing a moment knowing that he can get attention and clearly some weird OP-adjacent stuff. I don't want to be too conspiracy-minded, but this guy's never posted before. No one's really heard of him before. He did work at the companies, just to be clear. Worked at OpenAI for years, joined Anthropic for a few months and was like, 'Suddenly, I care about this.' It just feels to me very strange. But because modern journalism doesn't give a damn about the readers, everyone's panicked about this. So, let me be honest. Jacob Coxon, if he actually cared, if he actually had material concerns, the way to do it would have been: 'I just left Anthropic. Here are the exact things I'm scared of. Here are the things they're doing that aren't right.' Because you'll also notice one other thing. He never really criticizes the companies. He goes on these big things about how scary and we need better safety protocols and, 'Oh, OpenAI doesn't quite do enough, but you know, they're still good.' It's always this helplessness. It's, 'We just can't, I don't know what we do about this. I don't know.' What do you mean? 'I mean, I just, I'm so scared of it. And it's so powerful.' Why is it scary? 'I can't really say.' And who's to blame? No one. 'Cuz the AI is so powerful. I'm just a small bean. I couldn't possibly do anything. The companies aren't to blame. Nobody's actually responsible for this. We need some safety thing, I guess.' And that makes me think he doesn't actually care about anyone. I think this is a deeply cynical play by a deeply cynical individual.
Christ, I get the occasional email suggesting I don't have enough proof for what I'm saying. This guy goes out there and he's basically describing, 'Oh, we need more Grinch hunters. The minions from the Minions movie are going to invade us.' You see the movie? There's hundreds of them. They can't die. It's just, it's very frustrating because especially as someone whose main job is being extremely fact-based, extremely based in numbers and direct statements. When I go on television, I always have someone going like, 'What if you're wrong? What if you're wrong about this?' No one, I know I've watched enough of this bloody kid talking. No one is like, 'Hey, man. What if you're completely wrong? What if you're wrong?' No. You have people being like, 'Oh, are you prepping? Are you prepping for the AI apocalypse?' And the answer, by the way, is, 'No, I'm not, cuz AI is so powerful. I wouldn't be able to stop it.' Apologize for ranting, but I just really find this boy completely putrid.
The thing that sticks out for me is the timing. Maybe not necessarily of him quitting now, but also just generally this OpenAI saying, 'We're going to slow down development.' Anthropic CEO and various other thousand people or so saying we need to pace development while each launch is becoming more iterative and there's diminishing returns on the amount being spent on the training. And again, you said you don't want to go too conspiratorial on this, but little tinfoil hat for me is the timing does seem a little too clean to say, yeah, we need to pause this now, it's getting too dangerous, we need to stop everything. Just the fact that we kind of needed to pause anyway is not really the point. We're slowing down, but just, it's too scary to continue.
Well, that's the thing as well. If you wanted to slow down AI, there's actually a really easy way to do it. And it starts not with the AI labs, but with Google, Microsoft, and Amazon. Limit the amount of compute they can use. Great start. It seems that the reason the Hugging Face attack happened was the unlimited resources. So, cut them. Limit the amount of compute they have access to. I'm sorry that, oh, would that break your company? Not my goddamn problem. You said you wanted to slow down. Don't you want to slow down? It's also, what's interesting is OpenAI just stopped, they just stopped selling their $200 a month plan yesterday because they ran out of compute because of Astra.
What's a really interesting point about Astra. Apparently, it's worse at coding. Weird.
But that's the thing. You've got the Anthropic IPO coming up. You allegedly have the OpenAI one, though. Sam Altman on TV the other day said, 'If we IPO...'
PR genius. It's just, it's all very convenient, but it's also not clear what they want. So, I have an alternative theory, which is I think that there is a certain degree of industry opiness to this where Anthropic loves this branding for whatever reason of 'we're so dangerous, you should be scared of us and pay us.' I also think that like any great true conspiracy, getting quoted here, it's not like an email goes out and be like, 'All right, everyone, op time, 12:00 p.m., everyone make your posts.' What it is is people at various different chances going, 'Oh, this is my moment to really get some attention.' Right? And I think Jacob Coxon went, 'Oh, I could get some attention really big time. Oh, I could go and do this. Maybe raise money for an AI safety startup. Ooh, maybe a nonprofit.' And he took his opportunity. And then a bunch of other knob ends came out and were like, 'Oh, yeah. I also think there's a 10% chance.' A guy this morning was like, 'Actually, there's a 50% chance.' I love that. I actually, that's my favorite guy cuz he's just like, 'Ah, 50.' It's the Onion story where it's like, 'Screw it. We're doing five blades on the Gillette.' But it is all very convenient. But I don't think it's all of them getting together and being like... I think Jacob has some weird thing going on. If he has a PR firm, I'd be unsurprised if there wasn't some sort of horrible nonprofit related to him. I wouldn't be surprised. He has some history in that. Also just like very strange. He literally, the first time he posts on Twitter is this. But I think a bunch of other chances went, 'Oi, this is my moment to be scary as well. Maybe I'll get to go on TV. Maybe I'll get to go on Anderson Cooper, CNN.' Just by the way, I don't know if anyone remembers this. Anyone remember something big is happening? The random guy Matt Schimmer was like, 'I've heard that AI is so powerful that you'll never believe it's changing everything,' without again ever saying what's going on. And that went viral and that guy went on CNN. It's just like the sentience test on journalism keeps failing because getting back to what I said, everyone wants this to be something because if all of these data centers, hundreds of billions of trillion dollars of capex, $217 billion in funding for OpenAI and Anthropic, all these companies making these startups, they're load-bearing parts of their future revenue story. If it's not AGI, if they don't speak in terms of AGI, the companies wasted all the money. Like I've been saying, they just spent all this money to make an unprofitable industry. And the thing that I think journalism is reckoning with is all of them fell for this. All of them fell for LLMs and they said that LLMs were going to become sentient and this was the next big thing. And I think that they are rationalizing it by taking this guy seriously because again if they have to go, 'Ah, this guy's just saying random stuff. Huh?' If they have to accept that, they have to accept that they were wrong and they got taken for fools for years.
I mean, I saw, was that Sheryl Crow was like, 'Well, these people, these people from within the companies, they're giving the warnings.' I want to be abundantly clear about something. Real whistleblowers will tell you stuff. However you feel about Edward Snowden, he revealed PRISM. He told everyone what it was, how it worked. He took real risks. We have now had what feels like 70, it's probably more like 10 of these guys being like, 'I'm quitting cuz it's so scary. I'm quitting.' And they never say what it is. They're like, 'The way it's trending, it's going to be this other thing that it isn't.' There was one guy, this Evan Hubinger I think it was called, this Anthropic guy, was like, 'I agree it's going to be higher than 10% chance that humanity is eradicated by 2030.' Then his next post was, 'Just to be clear, I think there's a low chance of the current models doing this.' It's just like, then what are you so scared of? And the answer is recursive self-improvement. They have this key-jangling thing where they're saying, 'Oh, recursive self-improvement. Everyone's going too fast towards recursive self-improvement,' which is just an AI that trains itself. Sorry, I even messed up my words there. And here's the thing: still theoretical, no proof it exists, no proof it's possible. But Jacob Coxon brought it up, brought it up, everyone's saying RSI, RSI, RSI, which makes me really think there is a degree of financials to this that they're trying to say don't value Anthropic based on its horrible economics or what its stuff does or how many people pay it or its margins or its future growth trajectory or how it's growing right now. Measure it based on a thing we have not proven we can do, but if we do it, it could maybe do something. And really, that's... but this is the thing. This is this industry. This is the truth of this industry. These guys are so anti-innovation that their best idea is that someone else does the innovation for them. Like at some point maybe like, here's the thing, if you're an AI researcher, recursive self-improvement is just AI taking your job. And so it's just like, what? There's no coherence to this narrative.
I mean, what I come back to on this is that the fears are at odds with the actions of the companies because pretty much across the board AI labs call for more safety and then push back against it when the opportunity comes.
Yeah. And it's kind of hard to take the warning seriously when they're not even sort of pretending to do so themselves.
Yeah.
Or is that kind of Coxon's point? I suppose.
