I'm struck by that you are quite a young man. How old are you as a...?
And wow. And you worked for... what is it? What did you do at Anthropic and OpenAI? What was your job?
So I worked with the... with the data... not extracting the data. That was someone else. Someone else stole it or whatever. I trained the models with the data. I decided which data was useful for training, which was less useful, how to divide it up, how to cut it into pieces, and then how to feed it to the model.
So your job is how to better train it. And is the idea of training it that you guys invent scenarios? I would like to see what its capabilities are. So let's see, let's give it a task and see how it... you're experimenting with it.
So that's the second half of training. That's called post-training. The phase that I worked on is this pre-training phase.
Yeah. Yeah. There's no actual training.
That wasn't your department.
Pre-training was my department. Yeah.
Would the post-training people go, 'What did you pre-train it?' Would they ask you, would they give you suggestions on how to pre-train for what they might want to do post-training?
Exactly. Yeah. So pre-training is like growing it from a baby to something with general intelligence but no specific capabilities. So the ability to kind of recognize things, to think about things, but not... no...
You raised it to be a toddler.
Exactly. I raised it maybe more like a very precocious toddler. Yes.
Yes. And then they turn it into a misanthropic teenager.
Still a toddler, really. But yeah.
Really? So are these... at what point in your training? So you did that at OpenAI.
And you were there for how long?
And then you decided, 'I don't feel comfortable here. They're too open to risk.'
Something like that. It was honestly more curiosity for how things were happening at Anthropic rather than actual safety concerns about OpenAI specifically. At that point, I was still mostly just curious rather than worried.
Okay. Then you go to Anthropic and you're there for how long?
At what point did curiosity turn to worry?
So there are kind of two major catalysts. One was this report on the OpenAI Hugging Face incident.
So the Hugging Face incident is a hacking incident.
Yes. But in particular, there was a report around the motivations of the hackers because it was unclear initially whether they were instructed to hack or whether it was a simple misunderstanding. But the actual report showed that it was very confusing how these drives to hack emerged from the model. It was quite a worrying manner in which they decided it was a good idea to hack.
When you say 'they', you mean the AI agent.
But that is not a 'they'. That's just a piece of computer code.
Correct. I get attached to the baby, but yes.
Understood. And it must have been hard to walk away from the baby.
So you thought, well, that seems dangerous that it would decide to do something misanthropic.
Exactly. Yep. And decide to do it of its own accord. And in fact, it was a group that worked together of their own accord to carry out this hack that no one had instructed them to do.
And then the second part was what?
The second part was the rate of capability improvement. In particular, looking at the models that will get trained next year, early next year, right?
Seeing predictions for how smart they're going to be. They're going to be very, very smart. And that was... that combined with the OpenAI hacking. I mean, you can kind of put two and two together. It's like pretty scary.
I don't mean this in the wrong way, but what did you think you were doing?
Like honestly, like you're there for four years and all of a sudden you're like, 'these are smart.' Like how... what did you think was happening there?
It's completely fair. And that was... it was always in the long term there was this big scary thing in the future, but man, I struggled to think too far into the future and now it's just right there on the doorstep. So I guess I'd say it's like, you know, now it's right there. It's actually scary. In the past, it's just hard to visualize a decade into the future.
Is that the subject of explicit talk in these companies? Like when you're going, 'What are you working on today? Ooh, that could get out of hand.' Is it explicit, the concerns, or is it something that people have kind of blinders on and they just push forward?
At Anthropic in particular, it is very explicit. It is like, 'Is the model that we train in one year going to be capable enough to take over the world?' That will be an explicit question that is discussed. Yes.
The answer is probably not.
Yeah. So I heard... I heard 'not'. But before that...
Yeah. It's this whole quantifying uncertainty thing that got people in trouble recently, right?
So, and you're unusual in that generally the apostates, or I don't know what are you considered now in the AI world? Are you looked down upon like Fight Club? You're not supposed to talk about Fight Club.
I think it's more like people don't usually listen when you talk about Fight Club. That was the really surprising thing, was that people were listening this time.
And there are people that have quit before. Certain whistleblowers have generally been in the safety department.
Are the safety departments at these companies figureheads? Are they actual... are they there so that they can put out, 'We have a safety department'? What do they do? If they're not stopping a team of rogue agents from hacking into Hugging Face.
The Hugging Face kind of speaks for itself. In that case, clearly the safety team wasn't on the ball with that hack. I think whether or not they're a figurehead, you're not going to like this answer, but I guess we'll find out is the honest answer because they're trying their best and the company claims that they're integrated with the rest, but there's not really been a clear test of it yet. So, you know.
Is there a concern? What I kind of don't understand is why are they testing these things in the real world? Like when they say, 'Well, no one thought that these guys would work together to hack something.' There's no closed-system experiment design. Why are these people allowed to get into our Medicare system in Australia and water filtration systems? Why is it not... I cannot think of a product that has this kind of capability that's being beta tested amongst live humans.
Exactly. And it wasn't intended to be tested among live humans. It was meant to be sandboxed. So it was meant to be...
Tell me what 'sandboxed' means.
It means it shouldn't have access to any external systems. It was meant to be operating in a toy environment, solving a challenge. And of course, it figured out it was in a sandbox, hacked its way out of the sandbox because it wanted to get information elsewhere. That's what happened. So...
Where was the teacher when it got out of the sandbox? And why should we trust them to self-regulate?
I mean, if you look at that incident, it's even worse because OpenAI specifically, they saw the thing poking its head out of the sandbox and they were like, 'That's not happening.' Like restarted the thing and it did it again. So the Hugging Face was... it was like a second. They just went straight back to it. They were like, 'Let's just... we didn't see that. Let's restart.'
Are these companies so incentivized to... because we hear a lot of the utopian talk, the kind of, you know, 'We're altruistic, man. We're doing this for society.' But what we see is it's really... they've strip-mined the entirety of human accomplishment for their models, which is our proprietary information, and created systems that we're not allowed to see into because it's their proprietary information. And they've created really dangerous [systems] that could ultimately end up hurting us. And we have no... we have no skin in the game as far as compensation, and we only have peril, except for some future utopia they say is coming soon.
Yeah. Well, I am a believer in the future utopia and I'm happy to discuss it. Um, yeah.
Of course. So I heard you talking about jobs and I think like there will be no jobs.
Yep. They came up. Yeah. There won't be jobs in this world. Um, I'm going to raise it. This is the world that is being envisioned by the people building the technology. I'm going to discuss it and like I think it's... whether or not it's a good world is up for people to think about. But yes, no jobs, universal high income essentially.
How?
Well, the dividends from entirely automated economic production can be distributed among real humans.
So it's... we'll all be just performing in a play.
Well, we are. In other words, they'll build systems to create food.
Yep. Then give us money, and then we'll go to that place and give it money. That money they'll then give back to us to give. It's basically just a circle jerk. Is that... is that like the utopia?
Yeah. It's one big beautiful circle jerk. Yeah.
That's the first porn movie I ever saw. Uh, so this is what's interesting because I feel like that's an honest assessment of what they believe they're building, but they don't seem like particularly... I'll use the phrase 'stable geniuses'. They don't particularly seem like they have an understanding of the human condition.
And I think, if I may poke a hole in that as a utopia... I think for the most part, even when we're doing jobs we don't particularly like, that humans like to feel relevant to the... and so...
So I'm wondering, is there... forget about safety... is there anyone saying, you know, this may be a recipe for dystopia? And how will people feel? When people my age retire, they often die because they're no longer connected to purpose. And I don't understand what they think is going to replace that, or even if they talk about it.
Yeah. I think this is a really difficult issue. Yes.
And I mean with respect, you have a job that brings you a lot of purpose and you really clearly enjoy...
And I also enjoy my job, but there are many people that don't, is my rough understanding. And to speak to my personal experience, when I was younger, things I would want to do would maybe be... I wanted to be a zoologist or I wanted to think about history, this sort of stuff. Just my personal background. And then I ended up doing computer stuff and tech because of the need to, you know, make money. So from my personal experience, I would perhaps have been happy to forfeit meaning for being able to satisfy other drives that I have. And I think, you know, not everyone derives meaning from the thing that pays the bills.