CEOInterviews.AI
Start App
Jacob Coxon
Former AI Researcher, Anthropic

Jacob Coxon On Witnessing the Existential Threat of AI from the Inside | The Daily Show

📅 Oct 05, 2026 The Daily Show 25 MIN 1402334 VIEWS 166 SEGMENTS · 2 SPEAKERS
Jacob Coxon, a former Anthropic and OpenAI researcher who spoke out about the dangers of artificial intelligence, opens up to Jon Stewart about the grim reality of the tech industry— from malicious hacks to potential biological attacks. The two discuss the future, AI-led utopia that tech companies are trying to sell us, the megalomania and riskiness that underlie the industry, and the terrifying capabilities Coxon witnessed while training AI models at Anthropic and OpenAI. #DailyShow #JacobCoxon #AI #OpenAI #Anthropic Subscribe to The Daily Show:    / @thedailyshow   Follow The Daily Show:...

Questions asked in this interview

12
  1. 0:00How old are you as a...?
  2. 0:36And is the idea of training it that you guys invent scenarios?
  3. 1:51And you were there for how long?
  4. 3:24And then the second part was what?
  5. 4:15Ooh, that could get out of hand.' Is it explicit, the concerns, or is it something that people have kind of blinders on and they just push forward?
  6. 6:14Why are these people allowed to get into our Medicare system in Australia and water filtration systems?
  7. 10:42is there anyone saying, you know, this may be a recipe for dystopia? And how will people feel?
  8. 14:42Um, you're a high-IQ man in a technology division. Do you think about that?
  9. 18:52Do they know how that's generally worked out in history?
  10. 21:02If you did all this to get laid, brother, you and I are going to have a problem. How seriously?
  11. 23:50do you talk about how would bad actors use this?
  12. 24:54And it's shocking to me out of the percentage of people that work in that industry, how many have turned and stated the dangers? 10 of you, 15?
Interviewer 0:00 ↗
I'm struck by that you are quite a young man. How old are you as a...?
Jacob Coxon 0:05 ↗
I'm 28.
Interviewer 0:07 ↗
And wow. And you worked for... what is it? What did you do at Anthropic and OpenAI? What was your job?
Jacob Coxon 0:16 ↗
So I worked with the... with the data... not extracting the data. That was someone else. Someone else stole it or whatever. I trained the models with the data. I decided which data was useful for training, which was less useful, how to divide it up, how to cut it into pieces, and then how to feed it to the model.
Interviewer 0:36 ↗
So your job is how to better train it. And is the idea of training it that you guys invent scenarios? I would like to see what its capabilities are. So let's see, let's give it a task and see how it... you're experimenting with it.
Jacob Coxon 0:54 ↗
So that's the second half of training. That's called post-training. The phase that I worked on is this pre-training phase.
Interviewer 1:00 ↗
Pre and post.
Jacob Coxon 1:01 ↗
Yeah. Yeah. There's no actual training.
Interviewer 1:05 ↗
That wasn't your department.
Jacob Coxon 1:07 ↗
Pre-training was my department. Yeah.
Interviewer 1:09 ↗
I see.
Jacob Coxon 1:09 ↗
Yes.
Interviewer 1:10 ↗
Would the post-training people go, 'What did you pre-train it?' Would they ask you, would they give you suggestions on how to pre-train for what they might want to do post-training?
Jacob Coxon 1:21 ↗
Exactly. Yeah. So pre-training is like growing it from a baby to something with general intelligence but no specific capabilities. So the ability to kind of recognize things, to think about things, but not... no...
Interviewer 1:33 ↗
You raised it to be a toddler.
Jacob Coxon 1:34 ↗
Exactly. I raised it maybe more like a very precocious toddler. Yes.
Interviewer 1:38 ↗
Yes. And then they turn it into a misanthropic teenager.
Jacob Coxon 1:43 ↗
Still a toddler, really. But yeah.
Interviewer 1:45 ↗
Really? So are these... at what point in your training? So you did that at OpenAI.
Jacob Coxon 1:50 ↗
Yep.
Interviewer 1:51 ↗
And you were there for how long?
Jacob Coxon 1:53 ↗
Three years.
Interviewer 1:54 ↗
And then you decided, 'I don't feel comfortable here. They're too open to risk.'
Jacob Coxon 2:03 ↗
Something like that. It was honestly more curiosity for how things were happening at Anthropic rather than actual safety concerns about OpenAI specifically. At that point, I was still mostly just curious rather than worried.
Interviewer 2:15 ↗
Okay. Then you go to Anthropic and you're there for how long?
Jacob Coxon 2:18 ↗
Four months. Yes.
Interviewer 2:21 ↗
At what point did curiosity turn to worry?
Jacob Coxon 2:25 ↗
So there are kind of two major catalysts. One was this report on the OpenAI Hugging Face incident.
Interviewer 2:30 ↗
So the Hugging Face incident is a hacking incident.
Jacob Coxon 2:33 ↗
Yes. But in particular, there was a report around the motivations of the hackers because it was unclear initially whether they were instructed to hack or whether it was a simple misunderstanding. But the actual report showed that it was very confusing how these drives to hack emerged from the model. It was quite a worrying manner in which they decided it was a good idea to hack.
Interviewer 2:51 ↗
When you say 'they', you mean the AI agent.
Jacob Coxon 2:55 ↗
Yes, they.
Interviewer 2:56 ↗
But that is not a 'they'. That's just a piece of computer code.
Jacob Coxon 3:01 ↗
Correct. I get attached to the baby, but yes.
Interviewer 3:03 ↗
Understood. And it must have been hard to walk away from the baby.
Jacob Coxon 3:06 ↗
Yeah.
Interviewer 3:07 ↗
So you thought, well, that seems dangerous that it would decide to do something misanthropic.
Jacob Coxon 3:13 ↗
Exactly. Yep. And decide to do it of its own accord. And in fact, it was a group that worked together of their own accord to carry out this hack that no one had instructed them to do.
Interviewer 3:24 ↗
And then the second part was what?
Jacob Coxon 3:27 ↗
The second part was the rate of capability improvement. In particular, looking at the models that will get trained next year, early next year, right?
Interviewer 3:35 ↗
Seeing predictions for how smart they're going to be. They're going to be very, very smart. And that was... that combined with the OpenAI hacking. I mean, you can kind of put two and two together. It's like pretty scary.
I don't mean this in the wrong way, but what did you think you were doing?
Jacob Coxon 3:49 ↗
I know, bro.
Interviewer 3:51 ↗
Like honestly, like you're there for four years and all of a sudden you're like, 'these are smart.' Like how... what did you think was happening there?
Jacob Coxon 3:59 ↗
It's completely fair. And that was... it was always in the long term there was this big scary thing in the future, but man, I struggled to think too far into the future and now it's just right there on the doorstep. So I guess I'd say it's like, you know, now it's right there. It's actually scary. In the past, it's just hard to visualize a decade into the future.
Interviewer 4:15 ↗
Is that the subject of explicit talk in these companies? Like when you're going, 'What are you working on today? Ooh, that could get out of hand.' Is it explicit, the concerns, or is it something that people have kind of blinders on and they just push forward?
Jacob Coxon 4:33 ↗
At Anthropic in particular, it is very explicit. It is like, 'Is the model that we train in one year going to be capable enough to take over the world?' That will be an explicit question that is discussed. Yes.
Interviewer 4:47 ↗
And the answer?
Jacob Coxon 4:49 ↗
The answer is probably not.
Interviewer 4:52 ↗
Yeah. So I heard... I heard 'not'. But before that...
Jacob Coxon 4:59 ↗
Yeah.
Interviewer 5:00 ↗
I heard 'probably'.
Jacob Coxon 5:01 ↗
Yeah. It's this whole quantifying uncertainty thing that got people in trouble recently, right?
Interviewer 5:06 ↗
So, and you're unusual in that generally the apostates, or I don't know what are you considered now in the AI world? Are you looked down upon like Fight Club? You're not supposed to talk about Fight Club.
Jacob Coxon 5:19 ↗
I think it's more like people don't usually listen when you talk about Fight Club. That was the really surprising thing, was that people were listening this time.
Interviewer 5:27 ↗
And there are people that have quit before. Certain whistleblowers have generally been in the safety department.
Jacob Coxon 5:33 ↗
Yep.
Interviewer 5:34 ↗
Are the safety departments at these companies figureheads? Are they actual... are they there so that they can put out, 'We have a safety department'? What do they do? If they're not stopping a team of rogue agents from hacking into Hugging Face.
Jacob Coxon 5:53 ↗
The Hugging Face kind of speaks for itself. In that case, clearly the safety team wasn't on the ball with that hack. I think whether or not they're a figurehead, you're not going to like this answer, but I guess we'll find out is the honest answer because they're trying their best and the company claims that they're integrated with the rest, but there's not really been a clear test of it yet. So, you know.
Interviewer 6:14 ↗
Is there a concern? What I kind of don't understand is why are they testing these things in the real world? Like when they say, 'Well, no one thought that these guys would work together to hack something.' There's no closed-system experiment design. Why are these people allowed to get into our Medicare system in Australia and water filtration systems? Why is it not... I cannot think of a product that has this kind of capability that's being beta tested amongst live humans.
Jacob Coxon 6:49 ↗
Exactly. And it wasn't intended to be tested among live humans. It was meant to be sandboxed. So it was meant to be...
Interviewer 6:54 ↗
Tell me what 'sandboxed' means.
Jacob Coxon 6:55 ↗
It means it shouldn't have access to any external systems. It was meant to be operating in a toy environment, solving a challenge. And of course, it figured out it was in a sandbox, hacked its way out of the sandbox because it wanted to get information elsewhere. That's what happened. So...
Interviewer 7:14 ↗
Where was the teacher when it got out of the sandbox? And why should we trust them to self-regulate?
Jacob Coxon 7:23 ↗
I mean, if you look at that incident, it's even worse because OpenAI specifically, they saw the thing poking its head out of the sandbox and they were like, 'That's not happening.' Like restarted the thing and it did it again. So the Hugging Face was... it was like a second. They just went straight back to it. They were like, 'Let's just... we didn't see that. Let's restart.'
Interviewer 7:41 ↗
Are these companies so incentivized to... because we hear a lot of the utopian talk, the kind of, you know, 'We're altruistic, man. We're doing this for society.' But what we see is it's really... they've strip-mined the entirety of human accomplishment for their models, which is our proprietary information, and created systems that we're not allowed to see into because it's their proprietary information. And they've created really dangerous [systems] that could ultimately end up hurting us. And we have no... we have no skin in the game as far as compensation, and we only have peril, except for some future utopia they say is coming soon.
Jacob Coxon 8:32 ↗
Yeah. Well, I am a believer in the future utopia and I'm happy to discuss it. Um, yeah.
Interviewer 8:41 ↗
Of course. So I heard you talking about jobs and I think like there will be no jobs.
Jacob Coxon 8:45 ↗
I mentioned them.
Interviewer 8:46 ↗
Yep. They came up. Yeah. There won't be jobs in this world. Um, I'm going to raise it. This is the world that is being envisioned by the people building the technology. I'm going to discuss it and like I think it's... whether or not it's a good world is up for people to think about. But yes, no jobs, universal high income essentially.
How?
Jacob Coxon 9:05 ↗
Well, the dividends from entirely automated economic production can be distributed among real humans.
Interviewer 9:14 ↗
So it's... we'll all be just performing in a play.
Jacob Coxon 9:17 ↗
Well, we are. In other words, they'll build systems to create food.
Interviewer 9:22 ↗
Yep. Then give us money, and then we'll go to that place and give it money. That money they'll then give back to us to give. It's basically just a circle jerk. Is that... is that like the utopia?
Jacob Coxon 9:36 ↗
Yeah. It's one big beautiful circle jerk. Yeah.
Interviewer 9:47 ↗
That's the first porn movie I ever saw. Uh, so this is what's interesting because I feel like that's an honest assessment of what they believe they're building, but they don't seem like particularly... I'll use the phrase 'stable geniuses'. They don't particularly seem like they have an understanding of the human condition.
Jacob Coxon 10:19 ↗
And I think, if I may poke a hole in that as a utopia... I think for the most part, even when we're doing jobs we don't particularly like, that humans like to feel relevant to the... and so...
Interviewer 10:42 ↗
So I'm wondering, is there... forget about safety... is there anyone saying, you know, this may be a recipe for dystopia? And how will people feel? When people my age retire, they often die because they're no longer connected to purpose. And I don't understand what they think is going to replace that, or even if they talk about it.
Jacob Coxon 11:15 ↗
Yeah. I think this is a really difficult issue. Yes.
Interviewer 11:17 ↗
And I mean with respect, you have a job that brings you a lot of purpose and you really clearly enjoy...
Jacob Coxon 11:22 ↗
And I also enjoy my job, but there are many people that don't, is my rough understanding. And to speak to my personal experience, when I was younger, things I would want to do would maybe be... I wanted to be a zoologist or I wanted to think about history, this sort of stuff. Just my personal background. And then I ended up doing computer stuff and tech because of the need to, you know, make money. So from my personal experience, I would perhaps have been happy to forfeit meaning for being able to satisfy other drives that I have. And I think, you know, not everyone derives meaning from the thing that pays the bills.

81 more exchanges in this transcript

Sign in free to read the rest of this interview. No card required.

Sign in to read the full transcript

Cite this transcript

APA, MLA, BibTeX
APA

Coxon, J. (2026, October 5). Jacob Coxon On Witnessing the Existential Threat of AI from the Inside | The Daily Show [Interview transcript]. The Daily Show. CEOInterviews.AI. https://ceointerviews.ai/interview/2995624/

MLA

Jacob Coxon. "Jacob Coxon On Witnessing the Existential Threat of AI from the Inside | The Daily Show." The Daily Show, 5 Oct. 2026. Transcript, CEOInterviews.AI, https://ceointerviews.ai/interview/2995624/.

BibTeX
@misc{coxon2026_2995624,
  author       = {Jacob Coxon},
  title        = {Jacob Coxon On Witnessing the Existential Threat of AI from the Inside | The Daily Show},
  howpublished = {Interview transcript, The Daily Show. CEOInterviews.AI},
  year         = {2026},
  month        = {oct},
  url          = {https://ceointerviews.ai/interview/2995624/},
  note         = {Speaker-attributed transcript with timestamps}
}