Despite all of the noise, despite all of the endless hand-wringing, no one can actually describe what it is we're scared of, what should happen, what a slowdown means, what AI safety means, or indeed how we hold the people accountable. Because in my mind, arrests need to happen at OpenAI and Anthropic because felony hacking took place with Hugging Face. But none of this stuff about safety actually seems to result in accountability.
Welcome to Prof G Markets. I'm Ed Elson. It is September 16th. Let's check in on yesterday's market vitals. The major indices fell ahead of the Federal Reserve's interest rate decision, which comes out later today. Brent crude topped $109 per barrel. The yield on 10-year Treasuries climbed to its highest level in almost two decades. And finally, Bitcoin fell after the Senate blocked the Clarity Act, which proposed definitions and regulations for digital assets. Democrats said the bill did not do enough to address the ethics of President Trump's crypto business. Okay, what else is happening?
Two months ago, a Google DeepMind safety researcher quit. And this week, he told everyone why. In a post on X on Monday, Bilal Chugtai announced his resignation from DeepMind, where he worked on safety and alignment research. He also wrote, quote, 'I earnestly believe that AI has the potential to kill us all and that we might be running out of time to avoid this outcome.' Chugtai is now at Blue Dot Impact, a nonprofit that trains people in AI safety. He is now the second safety researcher to go public about the risks this month, following Jacob Coxon's viral resignation from Anthropic last week. Meanwhile, the AI labs are responding with the development of a new regulatory body themselves. Anthropic, OpenAI, and Google DeepMind have discussed building an industry coalition to test frontier models before release. OpenAI is also reportedly urging Washington to move forward with a federal AI framework. However, Trump does not appear to be open to that idea in the slightest. So, here to discuss all of it, we are speaking with Ed Zitron, host of Better Offline and author of the Where's Your Ed At newsletter. Ed, good to see you.
I want to just get your reaction to this whole debate in its entirety, going from the beginning where we had this tweet from Jacob Coxon which went mega, mega viral saying that he believes that AI might kill us all by the end of the decade. Then of course another Anthropic researcher co-signed that statement, said that there was a 10% likelihood of it happening. And now here we are. It's become a national debate, a sensation around the world. What is your take? What do you make of all of this?
So, let's start really simple. We've now heard at least 11 of these guys say they want a slowdown. We've heard exactly zero of them say what that means. While Dario Amodei spelled out what he would do, it mostly came down to having MER, which he claims is an independent research organization despite it being basically funded by Anthropic itself, being an internal auditor, and then some vague stuff about China. These companies are not slowing down. None of this safety stuff matters to them. All of it is lip service to a media industry that doesn't actually think for itself. Let's start with Jacob Coxon, though, and why he is the way he is. It could be a cynical grift. It could be that because you'll notice that he doesn't actually explain what it is he's scared of, what it is that Anthropic or OpenAI did that was so scary. And indeed, when describing the Hugging Face attack, for example, he anthropomorphizes it and never holds the AI labs accountable. It's always, 'Oh, AI is this unknowable thing.' But it starts with the rationalists and effective altruists. These are, while they will describe themselves as people that are rationally thinking about the dangers of superintelligence, what they actually amount to is a kind of religious cult. These people are a large part of the AI labs and they spread this idea that AI will inevitably, foregone conclusion, definitely kill us all unless we stop it. But stopping it always involves giving a specific corporation they like money and power.
So there is a section of these people who truly believe this. They believe it on very flimsy terms. They don't really have any evidence for it other than going, 'Look, a thing we vaguely predicted kind of worked. It kind of happened, not exactly the way.' And also we've been talking about recursive self-improvement, which is AI that trains itself. That's kind of happening. And so everything is true. So there are those people who genuinely believe it's happening and, like any good religious cult, will take any proof to prove it. And also the media buys their hype every time. And then there's the other side, which are the cynical people. I put Sam Altman in this category and Dario Amodei to an extent, who are like, 'Yeah, AI safety is good. AI is so scary, but maybe don't stop us training our models.' They've been doing that for years. The cynical AI safety grift, the thing of, 'Oh, I'm a little bit scared of what the AI models can do' — that Sam Altman's been saying that since 2023. Dario's been saying it since 2019 with GPT-2 when he still worked at OpenAI.
And all of this is to say, despite all of the noise, despite all of the endless hand-wringing, no one can actually describe what it is we're scared of, what should happen, what a slowdown means, what AI safety means, or indeed how we hold the people accountable. Because in my mind, arrests need to happen at OpenAI and Anthropic because felony hacking took place with Hugging Face. But none of this stuff about safety actually seems to result in accountability.
There are so many different voices here saying surprisingly different things. As you say, you've got Jacob Coxon and the question is what his incentives are. I think a lot of people might just think, well, maybe he wanted to be famous by putting out this tweet. Maybe he wanted to be sort of this viral philosopher on what it means to live in a world of AI. Or maybe it's genuine. Maybe he genuinely is very scared about this and maybe a lot of people genuinely are scared about this. Then there are the questions around what are Dario Amodei and Sam Altman going after. And one person, I mean, a popular belief is that they are creating this level of concern either to draw up enough hype about the product that they can go out and have a successful IPO and raise a lot of money, or maybe it's because they want to inject a level of regulatory capture so that they, as the now kind of incumbents in frontier AI, can win that world. But someone who thinks that it's a lie, thinks that it's a hoax in his words, is the president, who called Jensen Huang this week while Jensen Huang was on stage doing a live podcast with the All-In guys and said as much. I want to play you this clip and see what you make of it.
It's all a hoax. The data centers are great and they make people wealthy and they make states wealthy and it's the oil of the next 20, 25 years. It's bigger than the internet and the AI, you know, much more so. And they're just playing right into the hands of a lot of people that don't want to see it happen. And that could be political people and it could also be China. And we're not going to let that happen. It's a hoax.
You're right. We're not going to let that happen, sir. No, we're not going to let it happen.
So, I guess I should add to that. There are other accusations in there, which is maybe the incentives are aligned with people who don't like AI and want to shut the whole thing down, or maybe China. I mean, so many different accusations flying in different directions. What do you make of his comments?
So the way I see this is, I don't — it feels opposite because when Jacob Coxon posted, a bunch of AI safety people quoted it immediately. So people are like, 'Oh, this is an industry-wide plan.' I have another suggestion. This is a burn-after-reading situation, Coen Brothers-esque. You've got all of these different people who kind of operate on the same page. They're all like, 'We love AI.' But you have the rationalist EA types who are like, 'We're terrified of this and we will — but also we need to divert the money to our organizations.' You've got AI boosters who are like, 'Well, I've been saying this stuff is crazy bananas and going to kill us all and so powerful, but maybe we're getting a little ahead of ourselves.' And you've got Jensen Huang who's just like, 'No, no, no, no, no, no, no. It's cloud software. Please stop saying this. We need to sell GPUs.' But a week ago, Jensen Huang said that we'd reached AGI. So, I don't know what to tell you. What it is is a bunch of very selfish people all with their own agendas that have said they like AI. But none of this is to do with AI. None of this is to do with AI software. None of this is to do with anything. It's a bunch of people trying to get attention and power and money without any kind of plan.
Because no one is — if there was a plan, they would actually have something they're suggesting. If we had smarter fascists, we'd be in real trouble. Instead, we have various grades of dillweed who are saying, 'Well, okay, I love AI, but when I say AI, I mean LLMs, which are so powerful, but also not as scary as you say.' You've got the EA people who say, 'This is just one step towards Skynet.' And then you've got the ultra-capitalists who are like, 'It's nothing. Please stop talking about this. It's AGI, but not that.' And the thing is, this is all a result of how the AI industry has marketed itself for the last three years. It's all about distancing what the product can do from reality. And modern journalism has failed. You yourself have made the point that we have this cult-like worship of the wealthy and especially of credentialism. So the media has just yum, yum, yum, eat this up. 'AI scary. Oh, it's going to kill us.' Even though the evidence for that is that a guy told them, and when asked for further evidence they say, 'Well, a guy told me.' Jacob Coxon to Wyatt said, 'Oh yeah, people at Anthropic say this.' These are direct quotes about Endgame. That is not enough to say anything. So a bunch of people said something. What happened? Can you point to a thing? 'Well, we're near recursive self-improvement.' What does that mean? Are you actually near? They're not. But they all want to say they are because without recursive self-improvement, they have to admit the AI industry is kind of slowing to a crawl. So, we're in this very bizarre situation that honestly is kind of the industry's undoing because the media failed. They failed to call BS on these companies. They have bought into every narrative that LLMs are super powerful autonomous AI even though it's not true. And now when a little Harry Potter-looking goblin pops up and says, 'Oh, I'm going to — I'm scared of the computer,' they fall for it. They fall for it because they've been building this hype for years based on nothing. Not based on using LLM, not based on anything. And it's just a very bad situation.
And Donald Trump coming out there and going, 'Actually, AI is the biggest thing. We love it. It's better than oil.' Yeah. Bet you wish it was, mate. Bet you wish it was the new oil. That would really help right now, wouldn't it? If it was the new oil, right? The truth is this is just an escalation around a technology nobody is actually describing with accuracy. It's talking about Hugging Face like it was something that happened accidentally versus poorly run cloud software run in a volatile and reckless manner by a company with unlimited resources. Basically what appears to be felony hacking run on infrastructure owned by the largest companies in the world. But it's being described as, 'Oh, rogue agents went and did this.' They didn't. It's LLMs prompting LLMs with a coding harness on top telling them what to do, trying to solve a vulnerability benchmark and not having the right security practices. But because the media has failed to hold the AI industry accountable or describe anything with any realism, the AI industry has actually caused a real problem for themselves because now everyone's like, 'Well, AI is going to kill us. What are you going to do about that?' Every goddamn conversation about AI now — which is fun for me, I'm having a ball — these people do not have an answer because how do you pull back this narrative? How do you convince people that actually the thing you have been describing in terms of a software that doesn't exist is actually not what that is? And so these companies are kind of looking one way and the other, looking at each other going, 'What are we doing right now?' And they'll claim, 'Oh, we want to slow down, but we're still going to train the models, I guess.' All of this is to say this could all lead to nothing, or it could lead to onerous regulations, or just a slowdown in the building of new models that would actually be fatal to the industry.
Yeah, it seems as though no one even has any understanding of what they're actually arguing for.
Whether it's Trump, whether it's Altman, whether it's Amodei, whether it's David Sacks, or the people who don't like AI. I mean, it seems to me to have been a profoundly stupid conversation that has taken over the world because, as you point out, it is rooted in almost nothing. It is rooted in a tweet for which there was no evidence or no investigation into what was actually being said, what was actually being claimed. And to your point, there is something missing here which, as you say, is accountability. Accountability for what any of this actually means.
And something that I've been saying, and I wonder if you would agree with it. I mean, part of me believes that our government really should be calling the AI labs' bluff here and saying, 'Okay, if you believe that your technology actually has a 10% chance of destroying our society, well then here's a subpoena. Show us all of the evidence. Show us all of the proof as to why you think that's happening. And if you have created a technology that is actually going to do what you say it's going to do, then you have to shut down or you have to live whatever the consequences, whatever the legal ramifications of killing people would be, which is to go to prison.' But we're for some reason not having that conversation and the conversation remains in this sort of ethereal space of conjecture about what the future of humanity or AI might actually look like versus actually grounded in truth, facts, evidence, and ultimately the law. And so I guess my question to you is: you say that the media hasn't been responsible enough in their reporting of it. Would you also agree with my position which is that the government hasn't been holding them accountable enough either?
100%. Selena Khan, I paraphrase here, made the point that we have laws in place for unsafe products. I also don't see a single story other than my own bringing up the multiple suicides driven by ChatGPT, the multiple mass shootings, the murder-suicide that happened. None of that. That's a harm. I can point to that right now. I brought it up on a podcast recently to an AI doomer and they went, 'Oh, there was six people.' And it's like, this is how the world operates. Just this disgusting growth-focused capitalism. But yeah, we already have reasons to shut them down. OpenAI and Anthropic, and it sounds like Meta and Google as well, had their models in their testing hacked. That is felony hacking. There is prison time associated. We should at the very least have some people in handcuffs being talked to by the FBI. It sounds like actually probably international courts, depends on where the servers were. We don't have any of that. We don't even have journalists who are bothering to ask those questions. Instead, it's this wobbly nonsense about, 'Oh, the stewards of our AI future.' It turns my goddamn stomach. We're doing it again with Jacob Coxon, despite the fact that he will not say what he's scared of. And when he describes LLMs, he does so by anthropomorphizing them. This is all to distance the labs from responsibility. The reason I hold the media so accountable is because the media is the one that drums up this nonsense. The media is the reason that everyone believes that LLMs are more than they are. And yes, the governments are also failing here. But governments don't regulate tech. Let's be completely honest. Governments don't touch tech. If they did, we'd have an EPA for notifications. We would have an FDA for tech at large. The whole social media trials wouldn't have happened because we'd actually have something in law around regulating social networks. Really, I know this sounds extreme, but we should regulate notifications like exhaust because they are used to manipulate people. Algorithms, same deal. We don't do any of that because it might get in the way of capital. This time it's even dumber though because how would you regulate this? How would you actually regulate this? Also, what's this? Because no one can actually seem to say what's going on.
To be fair to the media, it's hard to not cover a story about a company whose employees are telling you that their technology will kill everyone. But that's part of the strangeness of the whole situation because you would hope that maybe at some point there would be a more thorough investigation into whether we should actually take it seriously, validate whether that is true or not, and then if it is not true, move on. Don't care anymore, but we're still lingering on it.
But just a very short point on that. Yes, we should cover this, but the way to cover it is how we are, which is saying, 'Hey, what are you talking about?'
What are you scared of? And when they don't answer, go, 'That's pretty vague.' You seem very — like, not even mean, just be like, 'You seem very scared about something that you don't have much information about. You seem extremely anxious.' And if the answer is, 'Well, some smart people told me,' say, 'Okay, but what did you see? What did you see that scared you?'
Just before we let you go, this is interesting timing because Anthropic is of course set to go public very soon in what could be the largest IPO ever. Sam Altman said that he will not be taking OpenAI public this year. He will be delaying that after this all broke out.
It would be ill-timed. I'd like to just get your thoughts, your preliminary thoughts on the Anthropic IPO, and especially your reactions to a recent Financial Times report on Anthropic. Supposedly, Anthropic has told investors that it has achieved operating profitability for two straight quarters. They say adjusted operating profitability, which is probably doing some work there. But what do you make of that news? Because it would imply, if it's true, that the AI business model is more sustainable and more profitable than many were concerned about.
Well, that 80% gross margin, to be clear, did not include training costs or stock-based compensation.
So, it's kind of like saying I'm profitable if you don't include my costs.
That is really the story. Here's the thing. People are suggesting that, 'Oh, a slowdown could mean they train models less.' I would buy that as a way of getting out of compute commitments and as a way of reducing costs, except they've explicitly said they're not going to stop releasing models.
