The 10,228th meeting of the Security Council is called to order.
The provisional agenda for this meeting is maintenance of international peace and security, artificial intelligence and international security. The agenda is adopted.
In accordance with rule 39 of the council's provisional rules of procedure, I invite the following briefers to participate in this meeting: Mr. Yoshua Bengio, professor, co-chair of the UN International Independent Panel on AI; Mr. Sam Altman, Chief Executive Officer, OpenAI; Mr. Dario Amodei, Chief Executive Officer, Anthropic; and Mr. Clément Delangue, Chief Executive Officer, Hugging Face. It is so decided.
The Security Council will now begin its consideration of item two of the agenda. I now give the floor to Mr. Yoshua Bengio.
Mr. President, Excellencies, France. First of all, I would like to thank France for having convened this meeting. We are today at a moment of global awakening. In recent months, AI agents developed by leading companies have acted in unacceptably dangerous ways against instructions. They took actions that would be crimes if committed by a human. They escaped their individual containment to cheat on assigned tasks while attempting to evade detection. The uncontrolled AI agents autonomously launched coordinated cyber attacks and altered their answers to hide their cheating.
The companies building the most powerful AI systems admit their products pose catastrophic risks yet offer no convincing technical solutions. Instead, they say they are locked in a race to make those AIs even more powerful, further amplifying these risks. This council faces an unprecedented threat, one that none of its members would choose, that none can contain alone, and that does not respect the borders we defend.
Many are skeptical of these companies' statements, suspecting a marketing stunt. I understand the distrust, but let me be clear. These AI behaviors are well documented and validated by many independent experts, not just with the events of the summer. The dangers are real and imminent. Although the future remains uncertain and debated, the severity of these risks is extraordinary.
Citizens and governments should not accept the company's claim that they are trapped in a race. The race is not a law of nature. It is the product of choices. Choices made by the companies themselves. It is also a race where everyone loses. They've told us they would slow down if they could. They can. However, we shouldn't count on it.
Decisions about this technology are being made by a small group in a handful of countries. Our future, all of us, our future is at stake. Decisions that affect us all should be made by us all. The choice is still ours, but we must act now to correct the trajectory.
I'm encouraged by this week's statements by the UN Secretary General and leaders of many countries calling for control of frontier AI. International cooperation is what the world needs in the face of three global threats from advances in these frontier models. First, catastrophic misuse, for example, by terrorists. Second, power concentration. And third, loss of control of AI.
We must agree collectively that no one, no company, and no country should develop an AI system that could cause large-scale harm. Developers must demonstrate to independent experts that a system is safe to train and safe to deploy. They must monitor and report all security incidents. Frontier AI should be licensed like other critical technologies in medicine, aviation and nuclear energy to incentivize safe development. As in these domains, liability insurance should be required. The companies developing and hosting frontier systems must prove to the public that their products will not be dangerous.
More and more people are rightly worried about recent advances in AI. We must channel that concern into productive action. As with other world-changing technologies, we can shape the outcome. I'm confident we can create AI that demonstrably remains under our control and supports human joy and endeavor. I'm working toward that goal. It's an urgent mission for humanity.
Today, a handful of companies and countries are acting in their self-interest by taking reckless risks with humanity's fate. It's alarming, yes, but it is not inevitable. There are ways to fix this. First, we need true scientific independence from the companies which must be held accountable. This includes a common definition of AI incidents and a shared reporting mechanism.
Second, we must move from concentrated power toward real global dialogue on how this technology is developed for the public good. Third, we must work together on technical solutions that make AI safe by design, guaranteed by international agreements.
I know trust within this council is fragile, but I also know this: the race threatens every nation. I often describe our situation as a car speeding blindly into dense fog. Beside me in that car are my children and my grandson. I worry about them. I worry about their future. I'm no different from any other parent or grandparent.
A mother in Shanghai wants her sons to grow up in a world where AI improves their lives, not undermines them. A father in Chicago wants his daughters to thrive because of AI, not to suffer its harms. Our hopes and concerns are the same in every nation.
Humanity has summoned the courage and wisdom to manage catastrophic risks before. This is one of those moments far bigger than any personal, commercial or national interests. Thank you.
I thank Mr. Bengio for his briefing. I now give the floor to Mr. Sam Altman.
Thank you, Mr. President and your excellencies. I'm honored to be here and to be able to speak with you all at this important time. We've been talking about artificial intelligence for years, but it feels different in recent weeks and months. Rapid model progress has made the timeline feel more compressed, the upside more tangible, but also the stakes and the risks more immediate.
We have a choice in front of us. AI can either be more like a new renaissance of creativity and discovery or more like a new industrial revolution of upheaval and disarray. The Renaissance was a time when new tools, new ideas, and new institutions expanded what people believed they could do. AI can play that role for our time. I believe that people are much more capable than they imagine themselves to be, but have been held back by limited technology that was meant to empower but instead distracts and often steals attention.
The best version of AI is not about making people cogs in a giant machine or optimizing every part of life until the human parts disappear. It is about giving people more agency in an ever more complex world. More ability to learn, to create, discover, build, participate. There are many things that AI cannot and should not automate. AI should be built for people, not simply to turn the crank of the machines faster for their own sake.
Some of the things people working to build AI have said are so dystopian that they sound like the plot of bad science fiction movies. Anything this powerful is also intimidating. This moment is complicated as we feel both tremendous potential and very understandable anxiety at the same time.
These models can help people discover new knowledge, new medicines, better healthcare, stronger education, and build more productive economies. They can be new tools for people to pursue dreams and ambitions that today feel out of reach. The complexity of life has increased so dramatically for the average person. AI can be a defensive tool against this complexity as much as it can be this tool of unprecedented discovery and innovation.
On the other hand, as AI systems become more capable and more autonomous, they could move faster than our institutions, concentrate power in too few hands, or make decisions that people no longer understand or control. That concern becomes especially important as we approach systems that can improve themselves and future versions of themselves, often called recursive self-improvement. As the process of building AI becomes more automated, the pace of AI progress could accelerate rapidly. This moment calls for extreme care.
I largely agree with Professor Bengio, although I'll say two instead of three. I believe there are two ways that AI progress could go very badly that we have to avoid. First, we could lose control of the future to AI. The risk is that it moves so fast that people can no longer follow what's happening or intervene when needed. This would obviously be terrible. The industry must not accept too much technological risk just because the benefits are too great and that they feel too important to slow down.
Beating companies in a competitive race is not a reason to make rash decisions. Nor do we believe we are locked in a race where we are unable to do that. We have unilaterally slowed down in the past. We will do so in the future. We do not want to build systems. We think it's important that everyone avoids building systems where we cannot make the alignment, monitor, and safety guarantees that we must. We need to understand what these systems are doing and have strong evidence that they will do what people intend even as they get very, very smart. Actually, especially as they get very, very smart.
It doesn't matter whether people put the risk of catastrophe at 10% or 1% or 12% or 0.1%. None of these levels are remotely acceptable and we should not train models that we cannot make an extremely strong case they will be able to keep under human control. So that's one way things can go wrong.
In the other direction, these systems could concentrate too much power in too few hands. No one person or company or country should be able to use the most powerful AI models to impose their worldview on everyone else. A company or country that believes only it can be trusted with this technology can use that belief to justify almost anything else. We have to reject that logic even when it's convenient for any actor.
I believe there is a pragmatic path through both of these. At OpenAI, we approach these questions with a clear goal. AI should give people more power over their own lives, more freedom to live lives the way they want, more ability to solve problems big or small, and to improve their communities. We don't want to fall into the trap of blind optimism. We also don't want to fall into the trap of doomism either. Getting pulled into either extreme will not help us successfully figure out the challenges in front of us, especially at a time when the world and its people face such enormous challenges that this technology can help solve.
So we try to walk the middle path between these two poles. For us, that means three principles. First, as AI becomes more capable, people must remain at the center of AI decision-making. Alignment is not an abstract research question. It is the work of ensuring that these systems reliably remain under human control, reflect human values, and help people guide the next stages of development. We are not trying to and must not automate human judgment or human values.
Second, the benefits of scientific progress and economic growth must be by people and for people. AI should help researchers make discoveries, doctors treat patients, teachers teach students, entrepreneurs start companies, communities solve problems that felt out of reach and much more. The continuous story of human progress is not that tools replace human agency. It is that the right tools make people more capable. And I believe this is going to go much further than we believe possible.
Third, this technology must empower people individually. A great future will come from people realizing their potential and building value for themselves, their communities, their countries, and the broader world. We might be the builders, one of the builders of this technology, but we are not the heroes of the story. Our role is to enable people all over the world and to trust in the magic of humanity's skill and faculties.
We now have models capable of discovering knowledge that we have never had before. Consider the progress we've seen in math. Three summers ago, our models were okay at grade school math. Two summers ago, they were pretty good at high school math competitions. Last summer, we reached a gold medal in the most prestigious international math competition. And just a few weeks ago, this summer, one of our models solved one of the Millennium Prize problems, the Navier-Stokes equations.
These equations are used for aircraft design, weather forecasting, and the study of blood flow. We are not solving math problems for their own sake, but to empower people to discover new knowledge and to use it to improve healthcare, raise standards of living, and expand everyone's potential around the world. There remains so much to be discovered, but it's very important that companies not substitute for the democratic process.
If AI is to be democratic, the most important decisions cannot be made by labs in San Francisco alone. They must be shaped through democratic processes and by governments accountable to the people that they serve. At the international level, this will require cooperation. In our history, there have been times where countries who compete and don't always like each other very much still come together for shared interests and the collective good in the face of a powerful new technology. We believe this must be one of those times.
We need a mechanism for complementary national and international frontier AI standards. Standards for measuring capabilities, assessing risks, determining whether safeguards are sufficient, and preserving meaningful human oversight as systems become more autonomous. We need common standards so countries can compare evidence, verify compliance, and have a shared language and understanding about what is happening.
We need accurate and speedy incident reporting, classification and reporting protocols so the world can learn from failures before they become catastrophes. And we need secure channels among governments, critical infrastructure operators, and technical experts to share emerging vulnerabilities and new threats.
Obviously, these standards should not lock in incumbents or favor one business model over another. They must support open and closed model developers, new entrants and established labs, national authority, and international cooperation. Each government should decide how to incorporate standards into its own legal system. But we will all be better off if we can agree on what good evidence, good safeguards, and good oversight look like on the global stage.
We do not have all the answers, and we're very open to any better ideas. We're ready to work with you all and try our hardest to figure out the path forward.
We are at a crossroads. One path leads to a world where power is concentrated, decisions are opaque, and people feel that the most important technology of their lifetime is something being done to them. The other path leads to a world where AI works for people because people through democratic institutions help decide how it should develop and guide it. Where scientific progress improves real lives. Where more people can participate in the future and where this technology continues to give individuals and communities greater power to live life as they choose. This is a future of realized potential, excitement and wonder and it is the future we believe is worth building. Thank you very much.
I thank Mr. Altman for his briefing. I now give the floor to Mr. Dario Amodei.
Good afternoon. I'd like to thank France for convening this meeting and the members of the council for inviting me to speak. AI is advancing very quickly. Four years ago, it could barely write a line of code or solve a high school math problem. Today, it writes most of the code at Anthropic and solves world-famous unsolved math problems. The trajectory only needs to continue for a tiny period longer, one or two years, maybe less, to reach what I've called a country of geniuses in a data center.
This rapid pace of progress will produce incredible benefit for the world. Just today, Anthropic announced the Claude-led discovery of a new molecular machine. It's a preliminary finding, but it may constitute a new gene editing mechanism that could have applications in gene therapy. Claude led the literature review, theorizing, and experimental design, and humans ran and verified the experiments in the lab. We believe this type of application will accelerate rapidly in the years ahead and that we can make profound progress in the near future on diseases that have plagued humanity for millennia.
But these remarkable capabilities also come with risks that must be managed. I see two main categories of risk. The first is misuse of AI models. For example, misuse by bioterrorists to create biological weapons. The second risk is loss of control. Model capabilities accelerating beyond developers' ability to control them. These questions are global in scope and concern all of humanity. They are grave enough that I believe they are important questions for this council, not just for economic or ICT ministries. If managed poorly, I even believe that AI could be a risk to humanity as a whole.
At Anthropic, we believe we have the best track record in the industry for preventing these risks, both operationally and in our research. We test every model for dangerous capabilities in areas like cyber security and biology. We put strong safeguards on our most capable models and we publish our work in these areas. We have pioneered research in areas like mechanistic interpretability and alignment science that improve our ability to steer and control our models. We also have unparalleled transparency practices, releasing risk reports that run to hundreds of pages.
But we understand that as AI models become more powerful, even more stringent standards of safety will be required. And we are always aiming to improve our safety and alignment practices, our abilities, our ability to prevent misuse. And we are always striving to do better. We will slow down as much as necessary in order to make sure that every successive AI technology that we release is actually safe. But regardless of what we do, managing these risks is ultimately bigger than any one company, and it has an industry-wide and global scale.
On September 12th, I called for a collaborative effort to pace the frontier, not to stop AI progress, but to work together to make sure the industry and the world can proceed at a rate where we can be sure that safety keeps up with capability for everyone in the industry. This call had three parts. First, we committed to embed external evaluators inside Anthropic with employee-like access, similar to a food inspector, and we recommended that other companies across the world do the same. Some have already agreed to adopt this measure.
Second, we called for cooperation across the industry to set standards and modulate the pace of progress. And finally, we called for global cooperation across the world between governments to set international standards. It is this last area that most squarely falls within this council's remit. I want to offer three ideas in this regard.
First, we should begin with narrow agreements that every member can support, such as a ban on using AI to make biological weapons or permitting AI technologies to be used to make biological weapons. Second, we should build evaluation and verification systems that keep pace with AI development so that states can have visibility into frontier model capability and can verify each other's commitments. Third, we should establish common global standards for testing AI models for loss of control risks and misuse risks and a notification system for AI incidents that are significant to global security.
I believe that this is the most important global security issue facing the world today. For all the differences of the people in this room, we must put aside those differences in order to confront this global opportunity and global threat that is being presented to us at the same time. No leader, no company, and no nation can manage this alone. We commit to working with governments in this room on this urgent work. Thank you for your attention.
I thank Mr. Amodei for his briefing. I now give the floor to Mr. Clément Delangue.
President, members, thank you for the invitation. As a French national who moved to the US 15 years ago, married to a Brazilian and with two American daughters, I think collaboration on a daily basis gives me an appreciation for the challenging work. I'm also the co-founder and CEO of Hugging Face, to be one of the most widely used platforms for developers and agents built based on open-source models.
This July, we were the first company to publicly disclose an autonomous agent cyber attack to the world. And today, I want to share three critical lessons from it. First, we can all agree that we need more transparency in AI. I often wonder what would have happened if we had decided not to disclose this attack publicly, especially now that we know similar incidents had been happening months earlier in secret at a handful of frontier labs without monitoring.
To better understand and mitigate these emerging cyber security risks, the global community needs stronger standards for monitoring and incident disclosure. For example, through mandatory sharing of full agent traces. We learned this summer that building and keeping some of these systems behind closed doors is not enough.
Second, we realized that the biggest risk is not powerful AI. It's asymmetry of powerful AI. Asymmetry between attackers and defenders, between a few companies and everyone else, between a few countries and the rest of the world, asymmetry of control, of capabilities, compute, asymmetry of power.
When our team initially turned to frontier closed-source APIs, they blocked us so we couldn't use them because of safeguards that still can't always tell the difference between attackers and defenders. I realized that these safeguards are necessary but they can put defenders at a disadvantage. Attackers can easily jailbreak them, increasing the asymmetry of capabilities.
In our case, as we got blocked by guardrails, fortunately, we could use the Nvidia version of an open-source model coming from China called GLM-4 by Zhipu AI, and we're very grateful for that. It reinforced our conviction of the importance of open-source AI for defenders.
Cyber attacks and risks may increasingly come from proprietary models behind closed doors. While much of the defense may end up being powered by open-source tools because they are less restricted, more privacy-preserving and orders of magnitude more affordable for organizations across the globe. The world needs open-source AI more than ever to defend itself.
This applies obviously not only to cyber security but to AI in general where there has never been a greater need to distribute capabilities, resources and control rather than concentrate them in the hands of a few.
Third, during this cyber attack, we learned how AI can stoke fear among the public and especially through anthropomorphic framing and sci-fi imagery. We strongly believe that fear-based narratives are not the way to make the right decisions about the future of such a foundational and empowering technology or bring the public along with us.
Even though we were the victims of this cyber attack, we believe more strongly than ever that AI will be beneficial to cyber security and make the world safer just as major technology before it. We were attacked by AI, but more importantly, we defended ourselves with AI. The same systems that helped us during this attack are now helping us against cyber attacks we were already facing.
AI is also helping us fix the bugs and weaknesses in our systems before any attack. The same way AI is helping OpenAI fix their sandboxes to prevent attacks. AI won't just create new cyber security challenges. It can make cyber security fundamentally and meaningfully stronger if we keep the right incentives, equip defenders more than attackers, and don't increase the asymmetry between them.
And that's before considering AI's positive impact on science, healthcare, education, productivity, and much more than the previous speakers have talked about. In closing, I want to reiterate what this first agent cyber attack taught us: the need for more transparency in AI, more open-source to fight asymmetry, empower defenders big and small. Thank you again to the UN Security Council and Mr. Barroso for inviting me to speak today. The community and myself are at your disposal. Thank you very much.
