a human extinction scenario. Yeah. But in terms of world changing events, I think it is quite likely we could see world changing events next year.
But if something happens, I mean, can we stop these AI systems by just, you know, pulling the plug?
And we'll see just how hard it is to turn something off when it's copying itself around all sorts of data centers.
To me, you know, they were dangerous. Is it solvable? The alignment problem solvable?
I think absolutely. It just will require time and care, and we have to get it right on the first go.
Do we need a kind of, you know, international, you know, regulatory body maybe?
From my perspective, that's the only thing, the only thing we could do.
If, if we have a few years before AI would kill us, how, how do you enjoy your, your, you know, our... Yeah. Our last years as a humanity.
Hello everyone. Welcome to Rakumachi channel. My name is Aminazi. I'm an economist based in Japan. Looking at the stock market, I've been worried about an AI bubble bursting for some time. But probably the most important question is not the AI bubble bursting, but how AI impacts our lives and how AI changes our lives. What is the threat AI is bringing us? And we have today a very famous AI celebrity actually with us, Jacob Coxon. Hi, Jacob. Thanks for joining us.
Hi. Nice to speak to you.
Nice to see you, too. Is this your first interview from Japan?
Yes, it's my first live face-to-face interview with Japan. Yeah.
Have you ever been in Japan?
I was there in 2019 in Kyoto.
Okay, good. What was your impression?
Oh, I loved it. My best friend growing up was Japanese.
So I think like he had like the nice Japanese toilet and the nice like Japanese sort of table tennis table. So I really liked that. And then when I went to Japan, I thought it was amazing. In particular, the people were very polite.
Which I think as a British person I like that.
Okay, good. Is there any particular Japanese food you like?
Food. I mean, I like sushi. I like chirashi. Chirashi donburi.
I liked, I really love. Yeah.
Any, any sort of rice bowl.
Okay. Come to Japan again.
Okay. You became very famous because you tweeted that you left Anthropic and you warned us that AI could destroy humanity by the end of this decade. What was the specific experience, the moment that you realized the threat? Why were you worried so much?
Yeah. So I think highly advanced AI systems are naturally quite scary, and people have been scared of them for a very long time, dating back to all the old science fiction stories about this. I think working in AI, even last year, that stuff still felt very much like science fiction. It felt far away. In particular, things still felt relatively slow. Then this year, looking at the rate of progress in areas like mathematical research and in coding, it looks very likely that next year the models will be able to train their successors. This thing of recursive self-improvement, and that even last year that felt a long way away, but now it seems like we're right on the cusp of it. And at that point things accelerate so fast that I think we enter the science fiction situations very soon, and I think that is just very scary. I always expected that at that point maybe the sort of serious men in suits, you know, kind of some secret service would step in and make it like a government project, but I realized that's just not going to happen. It's really going to be these private companies that take us into this science fiction situation, and that felt kind of unacceptable.
Was there any specific incident? What was the main reason that you're so worried about the AI's progress?
Yeah. I think the Millennium Prize problem solution. So this is quite abstract, but it solved a very difficult math question, and this felt, you know, about as difficult, it's more difficult than my own job. So like seeing that happen, and then seeing also the internal forecasts for the next model that Anthropic will train, the next AI they will train, like the predictions for how good it will be, that was also a specific thing. So there's a very specific model, you know, that will be trained at some point, and then a successor to that, and we can see how good it's going to be, and it's going to be, you know, very good.
Okay. This is actually the first time that I heard you talking about the math, because you know, I watched almost all your interviews. Well, it's been some time. So you also saw the, you know, the AI solved the math problem because you competed in the International Mathematical Olympiad, right? So you won a silver medal. So you know the math. So what was so great about these recent achievements about mathematics?
Yeah. It's again, it's the rate of progress. So like three, four years ago, the height of achievement was solving elementary school math problems, solving like arithmetic problems.
And that felt like as good as we could get. Then a couple of years ago, we started doing the International Math Olympiad. AI systems started being able to get medals at the International Math Olympiad. And at every step of the way, people, mathematicians, all the kind of sensible academics said, 'It's going to stop here. It won't get any better.' And they always came up with reasons that it wouldn't get any better. For when it was arithmetic, they were like, 'Well, language models can't do basic math.' When it was the Math Olympiad, they were like, 'Well, I guess they can do competition math, but they'll never do research math.' And then this year, it not only started doing research-level math, but it actually started doing superhuman research math. So this is solving problems that no human has solved before. And I think if you naively extrapolate, next year it will do strongly superhuman math. So this is solving problems like the Riemann hypothesis or exceptionally challenging state-of-the-art breakthroughs that require superhuman mathematical creativity. Now, even now, professional mathematicians are kind of still coming up with reasons to say it won't do that next year. They're like, we'll stop at this point. And from my perspective, they've been wrong every year. I don't see why they should be wrong this year as well.
I see. So what's driving the progress? Is it the compute power, or the models are, you know, getting better and better? What's behind this drive, this progress? I mean, yes, you were right, people were telling you, you know, the LLMs are stupid and they'll never get the superintelligence or AGI or something like that. But suddenly after your warning, actually thanks to you, people are worried about it.
Yeah, so if it was just computing power, I think I would feel safer because as I've seen some of your videos and you've noticed, like, I don't know, they're spending crazy amounts on compute.
Like it can't be sustained for that much longer, the buildup of compute.
Sadly, it is not just scaling up computing power. It's also the research is making them more efficient.
So they're more efficiently utilizing the computing power.
And every year, researchers, like it was my job to come up with ways to get more intelligence out of the same compute. So that's kind of one thing. The second is, it's not really math-specific. So we do general intelligence improvements, like you'll come up with improvements to the LLM that make it smarter at everything, and then math kind of drops out of that. And math is interestingly one of the first things to fall, but I think everything else will fall like math, which is still a somewhat controversial opinion, but I think time will vindicate this opinion too. So yeah, it's compute and also research.
Okay, so you have worked at both OpenAI and Anthropic, right? So how are these companies different? I mean, how are they taking AI security seriously? What's your opinion about that?
Oh, security, you mean like protecting their intellectual property and protecting the weights?
Well, yeah, that's also one side of security, but...
Yeah, I mean, I can talk about that actually specifically to start with because I think that is a... I haven't spoken about this that much.
But that's also a huge thing. So they don't yet have like nation-state level protection of their intellectual property, which is fine when it's just a startup, but when it's like a military technology, it's pretty ridiculous that the security is not there yet. And they're working. So in Anthropic's S1, they said they delayed training a model for a little bit to prioritize security, but they're really sort of taping over cracks. Like if a really motivated actor wanted to steal the progress, they could.
That's one side. In terms of the problem, is they're basically moving as fast as they can.
They are really going as fast as they can, and part of that is to sustain the revenue to keep up with the crazy compute spending. They really can't afford to not go as fast as they can, which is just incompatible with taking your time to not make silly mistakes. And yeah, silly mistakes in the past were forgivable. They were fine. They may be led right now to a little bit of hacking, but silly mistakes with a military-level technology are pretty unacceptable. And I think neither OpenAI nor Anthropic is like, they're both going very fast in a way that they will make these mistakes in the future.
But is it very dangerous? Like, you know, you see all those frontier models, and then you see the open-source models coming behind, like maybe six months. To me, you know, they were dangerous. If they are dangerous right now, they were probably dangerous six months before. Yeah. And now, you know, any kind of person can get these models for free. It's kind of, you know, scary to me. Don't you agree?
I agree. And one thing I'll say is they're still not that dangerous right now. Like, I'm more worried about the long-term risks of highly advanced superintelligence, and we're not there yet. So I'm personally not that worried about the very immediate dangers. But I will say that I suspect that next year we will see a lot of significant hacking events due to open-source models. I think this is inevitable next year, and I think they will cause damage. And I don't want to say that it's not very scary, but ultimately I think it's less scary than the very long-term stuff.
I see. Actually, it's just the topic I want to ask you about. You know, when people hear that AI could kill humanity, they could destroy us, they imagine like, you know, Skynet from the Terminator. Is that completely wrong, or is there something fundamentally accurate about the fictional scenario? What do you think?
I think it is a useful scenario to have in mind. I think a lot of traditional computer scientists and engineers are quite snobby about not looking at science fiction scenarios, but I actually think it's closer to the truth than a lot of the things that some people are worried about. So I think that's a useful thing to have in mind, and in general, thinking about high-powered robots is not a bad thing to have in mind.
So does AI need to become conscious or develop emotions to become dangerous, or is it just the alignment problem?
I don't think it needs emotions or consciousness to become dangerous. I think from my perspective it's really difficult to tell if an AI system has any of these things. It seems unlikely right now, but it's a difficult philosophical question. I'm more concerned about the very practical question of keeping control over the systems, whether or not they have emotions, any of this stuff. It is a bit sad that we don't have a good philosophy of mind for whether they have consciousness, and I wish we had time to think about that, but from a selfish human perspective, we have way bigger problems to solve before we even have the time to think about that sort of stuff.
So you said you're worried about the future. What kind of time span are you worried about? How many...
In terms of, yeah, in terms of, this is not the full human extinction scenario.
But in terms of world-changing events, I think it is quite likely we could see world-changing events next year.
So to be precise about that, once we start this recursive self-improvement, which we could start next year, I suspect that the US government will get a lot more interested in what's happening. I think the Chinese government will become a lot more interested. I think a lot of the military apparatus will be activated. Examples of things we could see are significant hacking attacks. We could see large-scale espionage theft of recursive self-improvement models, because once you steal that model, you don't necessarily need the AI researchers to make it smarter. So that's something we could see. We could already see potentially some sort of biological threat happen next year. I think maybe that's a little bit aggressive, but it's possible next year. And I think most importantly, we could see a lot of political unrest or just general unrest among people as they start seeing how powerful the systems become. I think in the context of this, I expect there to be a lot more anti-AI sentiment among people next year, in a way that, for example, could cause labs to relocate to more safe locations out of the city. All this sort of stuff could be happening next year. In terms of the timeline for human extinction, I think if we keep going at this rate, it's plausible in the early 2030s, even 2030. That's maybe an aggressive timeline, but if we go fast and do recursive self-improvement, we'll have very powerful systems by the early 2030s.
But if something happens, I mean, can we stop these AI systems by just, you know, pulling the plug?
Yeah, we can right now. We can right now. Like that rogue agent swarm, you know, the OpenAI one, that was contained on OpenAI servers. So they could just shut that down. I think also we'll see next year or the year after, we'll see someone's agent swarm will break loose onto the internet.
And we'll see just how hard it is to turn something off when it's copying itself around all sorts of data centers. It might not be that particularly dangerous. It might be more annoying than dangerous, but at that point, I think people will realize that it's not as straightforward as pulling the plug when you have an intelligent, self-replicating agent.
I see. I see. Actually, you know, the other day I was on TV and I was, you know, talking about maybe the AI will kill the internet before it kills humanity. And so we go back like maybe 30, 40 years past, like the 1980s or something like that. There's no internet, so there's no smartphone. Is there any scenario that you think that we could go back, you know, we lose some technological achievements because, you know, AI is too dangerous and it's banned? What do you think about that scenario?
That's a great point. Which, where did you make that point? Was it on Japanese TV?
Okay, cool. Yeah. No, I think that's a very valid, like that's a very valid scary thing, and I think that could happen as soon as next year. That could happen.
Yeah. Right. You know, it's better than the alternative. I mean, better than, you know, AI will kill us all. We go back to the 80s, you know.
Because, you know, to be clear, I think if that happened, if that happens soon, we could probably rebuild some systems around it. It would just be exceptionally costly and annoying.
But I think something like that is very, very possible.
Is there any good scenario that, you know, AI became superintelligent and, you know, they're better than us, they decided to protect humanity rather than, you know, exterminating us? Is there...
Of course, I think there are... Yeah, there are so many good scenarios. I think like there's any number of good scenarios, including the one you mentioned, also including scenarios where we just cap the level of AI at a reasonable level.
You know, smart enough to make scientific discoveries and help us in healthcare, do robotics, but not insanely intelligent. You know, we cap it at somewhere reasonable.
I think that's a great outcome because I think there's a lot of benefits to the technology. I think that's why I worked on it. I think it's an exciting, powerful technology. So there's a lot of ways it could go well. Unfortunately, like you have to be very careful when you're talking about having a, you know, essentially a successor species and kind of restraining it and harnessing it for your power. You've got to make sure you get it right, I think, is my take.
So you're talking that the AI is already another species, right? So what was the real main reason that you think that, you know, they are entirely different entity, maybe a new species?
Yeah. I think what's weird is because they are still a lot better at some things than others.
It's a very alien thing to have a machine that still makes such basic mistakes in some areas and is, say, superhuman at mathematics.
And yet it can get confused and do weird things in a way that a human would never do. Like you have a very smart human, and sure they're better at some things than others, but they'd never lose track of a name or accidentally delete all your files or something like this. So I think it's an alien mind in the sense that it often makes these very bizarre mistakes. And another thing about it that's kind of alien is you can make these edits. Like with a human, you can't just take a human and tweak them, like turn them evil, but with an AI you can turn it evil in 5 minutes. You know, if you mess with its internals, you can completely change its outlook and behavior. And I think people also don't internalize that. So those are two ways that it's quite alien compared to a human mind.
So do we really know what's happening inside the AI model? Because, you know, I watched your interviews on YouTube and you were talking AI like a baby or toddler or something like that, rather than writing a piece of software. Or it's more like raising a baby inside, you know, the tube in the laboratory or something like that. So do we really know what's happening inside?
Yeah, we don't know. We have some guesses.