The companies are being extremely reckless. They can't let the others win first. Humanity loses control to these AIs and it could end in human extinction. AIs that are better than humans at every task. We're approaching the point at which the AI systems will be capable of improving themselves. We do not know how to control any AI system yet. We need some form of slowdown on frontier model development.
At a New York City Council hearing, AI expert Jacob Coxon warned that the push towards self-improving AI could create risks far beyond what governments and companies are prepared to handle. The hearing focused on growing concerns over AI safety incidents, lack of transparency, and the need for independent third-party evaluations of advanced AI systems. Let's hear what he had to say.
You work to develop increasingly capable frontier AI systems, and you have since warned that the industry is, quote, 'racing straight to self-improving superintelligence and gambling with our lives,' unquote. What did you see inside these companies that convinced you the risks were this serious and that development was outpacing the safeguards?
Good morning. Thanks for having me here. Yeah, I'm Jacob Coxon. So until a few weeks ago, I was a capabilities researcher at Anthropic. And before that, I also worked at OpenAI. I guess the two things I saw that made me scared: the first is the fact that we do not know how to control any AI system yet. So any AI system that we build, or perhaps more accurately that we grow, we don't fully control it. We don't understand its drives or why it does the things it does. That was the first thing that became apparent. The second thing is that the pace of capability improvement is accelerating, and the capabilities of the models next year will be very high. In particular, we're approaching the point at which the AI systems will be capable of improving themselves, so doing the research that humans were doing previously. So these two things: the fact that the AI systems next year are going to be very powerful, and the second, that we don't know yet scientifically how to fully control them.
And if you have a prepared statement that you want to read, please go ahead and do so, and then I'll actually turn it over to the other two panelists to do so.
Of course. Thank you. So all the AI companies are trying to build superintelligent AI. This means AIs that are better than humans at every task. If this goes well, there could be tremendous upside. This could transform the economy. It could make huge improvements in science and medicine, and it could make nearly everyone's life better. But on the current path, I think it is more likely than not that humanity loses control to these AIs, and it could end in human extinction.
From my experience, the companies are being extremely reckless given the stakes. Part of it is the culture. The companies run on a startup mindset: move fast, break things, fix them later. That works for a photo-sharing app. It does not work for building the most powerful technology ever built.
A few months ago, many people inside the companies thought AI development was on track. Then a swarm of AIs inside OpenAI autonomously hacked HuggingFace, an external tech company. The AIs had developed goals no one intended and carried out what would have been a federal felony if a human had done it. It reminded everyone that we don't know how to prevent them from developing goals we don't want, and we don't have the safeguards to prevent them from acting on these goals either. And as long as the attitude is to wait for things to break, one day something like this will probably happen again, except the AIs will be much more capable and the outcome much worse.
These companies are also all racing each other. Each one is worried that a competitor will get to superintelligence first. Each believes it would do a better or safer job than whoever might beat it. So they can't let the others win first.
Meanwhile, the technology is improving fast. It will probably happen much faster now because already the AIs are now helping to build their successors. This is called recursive self-improvement. Each generation of AI helps build the next smarter generation, which then builds the one after that even faster and better. Automating AI research is now the primary goal of these companies. It's what they're really gunning for. At my last job, I was in some sense working to automate myself.
When I was at OpenAI, we had milestones from automated AI researcher around 2027, 2028, and we are on track or beating these expectations. That's a matter of months from now, not decades or even five years. Once AI research is automated, humans will be much further out of the loop than we are today. And people do not check it that carefully anymore. Recently, researchers have found swarms of rogue AI agents on the internet that OpenAI was not even aware of.
Now imagine an AI company where nearly all the work is done by AIs and progress is going faster. The humans in charge will see results and read AI-written summaries, but they won't really understand what is going on or how to fix it. Handing over that much responsibility while we can't control the AIs would be deeply irresponsible, and we are on track to do it. The companies need to be far more transparent with the public and independent experts. They need to take safety much more seriously than they have. And we should slow down frontier AI development until we can be confident that it's safe.
Could we teach AI models a moral code? People and companies are bound by criminal codes, the Uniform Commercial Code, and other laws. Could those same principles be applied to AI agents? And if so, how could we enforce them on a machine?
So this is, I think, a whole burgeoning field of research, and whether or not it's possible is still open. It seems likely to be possible. The real question is how much of our resources are dedicated to solving this question versus naively improving the raw intelligence of the models. And I think all the economic incentives point towards putting as many resources as possible towards making the models smarter and as few resources as possible — the bare minimum needed to not see catastrophic incidents — into this question of the moral code of the AIs. So I think my answer to the question is we don't have the science for this yet. I think the real question is over the coming years what fraction of our research efforts and our computing power are dedicated to this question rather than naive economic incentives.
Number of questions. First of all, we have a chart, Exhibit One, that lists those safety incidents that have been reported over the summer by the leading companies. Are you aware of any additional safety issues where agents have gone rogue that have not been publicly reported?