Anne Neuberger - Fireside Chat with Anne Neuberger: AI and National Security [Alignment Workshop]

Fantastic. So good morning, Janet. It's great to be here with you. It's great to be here with everyone here. A couple of thoughts on that. I would say one thing that hasn't changed is exactly what you hit on. Technology moves far more quickly than policy or the law, or as such, we end up with technology adoption or some of the questions that were talked about in the prior panel moving out before law or policy has caught up. And for me, I think about this a lot in the context of I was NSA's first chief risk officer after the Snowden media leaks. And at that point, that issue came up in stark relief. Right? The technology had changed. We were in a digital world where adversaries and terrorists were using the same technologies as everyday Americans and civilians around the world. And what the law and policy should be around surveillance in a democracy hadn't necessarily kept up with what the technology was doing. And when all of that became public, I think many people were surprised by the gap between them. So I think as we look at AI, which is such a transformative technology, thinking about—and that's why I'm so excited to be here and to have seen this set of discussions you had—thinking about how we advance policy and law at least in lockstep as much as possible with technological advancements is really important.

So from a national security perspective, some of the toughest issues we deal with is just the massive increase in data—all different kinds of data, sensors from different kinds of platforms that collect different kinds of intelligence information, from voice to RF to images—and the need to process them quickly and to make sense of them quickly. So there's the opportunity to really produce more, faster and better intelligence via synthesizing of all of that information far more quickly. That's one example of the national security space. I think a second example really is self-healing cyber networks. Many of us are deeply concerned about the advancement in AI-driven cyber capabilities in terms of finding zero-day vulnerabilities. As we all know, finding zero-day vulnerabilities is the first step whether you're an attacker or defender. So quickly hooking that to the second step as a defender, particularly for critical military, nuclear, and other systems so we can self-heal them is key. In a prior role, I was responsible for the security of nuclear command and control. NSA obviously builds the encryption of those systems. They were built in the 60s. You know, quite frankly, as we think about advancements in cyber attacks and capabilities, those are the systems that come to mind first when I think about the need for self-healing systems driven by AI. And to your point regarding the barriers to—as the technology, particularly in AI, we see the capabilities, we see what's possible—we also know that the systems can be fragile. So truly understanding them enough to really utilize them and determining when is it safe enough to deploy in a given space. How do we maintain adequate oversight so if something unexpected occurs it can be addressed, I think is the crux of the core challenge and the barrier, which really the best way to address is use by a set of people who understand the systems, understand the AI, and understand traditional approaches we've used for oversight of rapidly evolving systems.

You know, there's a really interesting report that Scott Sagan at Stanford did after a series of accidents at nuclear power plants. And the reason he did that is because when you think about advanced oversight systems, including multi-step and self-healing systems, nuclear power plants come to mind, right? Because these are systems where the degree of risk tolerance is really low and systems have been modeled and thought through. And he looked at a couple of specific accidents that had occurred in the context of how we think about autonomy and weapon systems. That was why I was reading this study then to better understand as we think about delegating autonomy to weapon systems—whether it's radar control systems, whether it's targeting systems—understanding how much we need to understand them and what degree of oversight and what is the interplay between systems and human oversight. And one of the core takeaways was as systems become very complex and as the natural technical parts of those systems that are meant to address risks become built in and not understood, in some cases the human oversight was disrupting the system-level oversight, and that interaction is in some cases what caused the breakdown. So the takeaway from that, I think, is to both think about and understand—I'll use a particular field because otherwise it's theoretical. So when we think about, for example, what is happening in cyberspace, right—the degree to which AI-driven systems can now identify at scale zero-day exploits and can be potentially exploited at scale—from a defense perspective, we need to be able to at scale also find misconfigurations, find vulnerabilities, and address them. So I think the key piece is to ensure that as things are done, the decisions that are made are documented so somebody can review them. Setting up the left and right boundary space of decisions that we understand so we can see that they stay within certain limits and explore when new ones are needed can also be a way to look at humans being both in the loop but also observing to enable the speed we need.

I think the opportunity for recursive self-improvement is an emerging risk because while it's a powerful approach, it can lead to systems whose capabilities we don't really understand. So that is one of the areas that, as we heard about in the prior talk, the lab simulating and understanding those capabilities, the interplay between the frontier labs and safety areas who can test those in the context of domains we worry about—from nuclear to bio to cyber—is really important.

You know, I think there's been a lot of focus on who's at the cutting edge. In my mind, we need to combine that focus with who's leading in adoption, particularly in areas of the economy that are game-changing, such as drug discovery or areas of national security like advancements in military autonomy that are potentially game-changing in fast-moving conflict. So I think sometimes we focus a bit too much on, you know, who's one step ahead when, to your point, through distillation, it really is one step ahead. One can distill advances in another's capabilities pretty quickly. And I think what China is doing in that way—where you have massive American investment that then China distills quickly—is, while challenging for the American approach, it's a strategy that from their approach makes sense. I think a statistic that we should be watching strongly that I think gets less attention is: 30%, my understanding, of global model use is open source. Of that, 70% are Chinese open source models, which means, let's be honest, these were lessons we learned on all different kinds of technologies in many places: adoption is not necessarily the best mousetrap, but the best mousetrap that's cheapest. And open source models are now getting baked in all around the world, baked in economically and in the US as well. So thinking about what our strategy is for that is critical. I think the second part of that is, you know, I recently was giving a talk with the board of a very global drug company, and one of the things, you know, they said, 'Tell me about some of the risks you think about.' And I said, some of the risks we think about, particularly in a post-COVID world, is where critical supply chains for America have moved out of the country. So today, almost all active pharmaceutical ingredients are made in China or India. That matters both in the context of—it's driving large numbers of drug trials to happen out of the country. 40% of global drug trials are happening in China. So while innovation is still happening in the United States, as we saw in chips, the actual production and manufacturing has moved out of the country. So let's think about the impact of AI on that, right? AI can potentially have an impact. The reason—not to go down into depth on one issue, but I think it's a really important issue—is it talks about national security risks we don't think about in how AI plays out in a given sector and how AI can be helpful in a given sector, because the cost of drug discovery goes down because you can simulate and develop new molecules with AI models far more cheaply and quickly. That can potentially enable us to think about how we bring back key components of those active pharmaceutical ingredients back to the United States in terms of manufacturing them in a way that's more cost-effective. So I think as we think globally from a national security risk perspective, perhaps in more deep, more sophisticated ways rather than saying who's at the edge, is to say: what are the key sectors from a US economy that we need to ensure that supply chains are secure, what has been driving those out of the country, and what role can AI play in ensuring that we maintain both confidence and leadership? And drug discovery, APIs, and other components are just one example of thinking about national security and opportunity a bit in a more targeted way.

And like to your point, it's not without risk, but those are the kind of applications where we think about opportunity and risk in more focused ways, which in some ways it's easier to make progress on thinking through those models and more focused approaches that we can then learn from in broader application.

I think it goes back to the first question you asked me, which was, you know, what's stayed the same and what's changed. And I think what stayed the same is that we have a regulatory system that's built up over time, and there are certain controls that are built up over time to deal with the understanding of technology at that moment. Now, as we see the rapid advancements that AI drives, we need a similarly rapid assessment of where do existing regulations address the mark, where are they a barrier to adoption of really promising ways, both on innovation and frankly on addressing risks, and then how do we adjust quickly. You know, I was having again an interesting conversation recently with—I've gotten people from large national or global health systems reaching out because they want to do AI adoption and they really want to think about enterprise risk in that way, and that's the space we learned so much about in the intelligence community post-Snowden: how to think about enterprise risks. These are the conversations I've been having. So in talking with, you know, the head of a medical system, he said, 'We can never recruit or train enough primary care doctors. So we've been looking at: can we do, you know, prescription renewals in an automated way? What should the protocol be to say, you know, how often does somebody need to be checked up to have they developed a new medical condition so doing automated prescription renewals would be dangerous for that individual?' And I think those kinds of protocols worked across specific sectors, and then the sector comes to the regulatory agencies and says, 'Here are the rules, here's the data and simulations we've done to say we want to leverage AI to make the system better, faster, and safer, here's what's needed.' I think that can bring the laws and regulations of the US into a tighter, more rapid process.

I really appreciate the question, and actually noting your affiliation, you know, I worked with CNAS on CNAS's Insight that I wrote on this particular topic. So, shout out to CNAS, one of the sponsors of this event. You know, it brought back to me strong memories of a very difficult period in the United States, which was the period after the Edward Snowden media leaks. And as somebody was just commenting to me last night at a dinner, that was the sharp end and really was the sharp beginning to significant tensions between Silicon Valley and DC, in that it highlighted the difference of perspectives and frankly the knowledge gap between these two communities at two ends of this vast and amazing country. And I think to me at that time—so at the time that Snowden occurred, a month before, I'd been asked to lead an organization at NSA that maintained hundreds of classified work efforts with companies around the world, largely not large, all related to our intelligence mission, right? There were sensitive efforts with companies related to largely counterterrorism, counterproliferation of weapons, where companies in some cases shared information regarding these kinds of threats as part of their role in trying to keep the nation and frankly the world safe. And when the Snowden media leaks occurred, you know, for many of these companies, they felt that the risk was significant and they broke off these efforts, understandably. A few months later, I was asked to serve as chief risk officer because we saw both the internal divide at the agency, the degree to which it had compromised the mission of the National Security Agency, as well as the divide between the intelligence community and the American public, as well as with Silicon Valley. And it led me to really three takeaways that I'll say in the context of the Anthropic-Pentagon debate. One is: there's one entity we hold accountable for keeping us safe, and that is the national security community—the Pentagon and the intelligence community. You know, after a terrorist attack is missed, after, you know, we had a, as the FBI said this morning, a Hezbollah-aligned terrorist, you know, drove a bomb-laden vehicle into a Jewish synagogue and preschool a couple of weeks ago. Nobody said, 'Why didn't some social media company monitor his account?' They said, 'Where was the FBI?' Right? Could have had 150 kids, God forbid, killed. So that is the community we hold accountable, and remembering that and that they have that role is key. So the idea that a company like Anthropic in the middle of a war would be approving how their models were used—that doesn't work, right? Because they're not accountable. They don't see classified information about the threat. They're not seeing the larger context around what is happening in a given situation, or frankly, for many here who have worked in the national security community, the cultural training, what people go through in terms of learning the laws of war, proportionality, what's acceptable. So the overall decisions regarding use of technology in the national security domain should and can, I feel strongly, only be made by those who have access to the classified information, the training on doctrine and law, and frankly the accountability for keeping people safe. With that said, the technology around AI and its capabilities and its fragility is really uniquely understood by those at the frontier labs. And it's critical that the individuals in the military or intel community making decisions around use understand where it can go wrong, where it can potentially give answers to be more accommodating to those asking, where potentially the systems can break down, where they are fragile. So having a regular, detailed, technical conversation—detailed, technical, repeatable—between the labs and the individuals both using the technology and making the rules around use is critical. And then the final piece I would say is: as democratic countries, yours and mine—I've worked with so many Australians, I deeply respect, wonderful national security community—we are in many cases up against adversaries far less constrained, who don't particularly care about civil liberties and privacy. Whether you look at the Iranian government mowing down its own children protesting in the streets, or whether you look at millions of cameras in China trained to have sophisticated facial recognition models, we should always remember that what we're doing is protecting our way of life. So while competing with those countries and ensuring AI is useful in that competition and fight, we also want to ensure every single American feels confident that that technology is used in a way that reflects the civil liberties and privacy that makes our country so unique. So ensuring that enough is shared in that so the average educated American feels that way is key. And that was my biggest takeaway from the failure pre-Snowden: was that the intel community was a black box. People didn't understand how it operated. The intel community didn't do enough to convey that in ways that protected sources and methods. So those are the three takeaways: roles and responsibilities, a regular way so that there is a deeper technical understanding about what works and what doesn't, and a recognition that we do need to compete with adversaries using these technologies without care, but remembering who we are and ensuring our democratic constituents understand it too.

I think the national—the KC and AICS in the US are important elements because they bring together people who understand concerns around national security areas with the frontier labs. I think, you know, the talk we had earlier, and a shout out to Chris Meserole in the Frontier Model Forum, because that can also do convenings there so that across the labs there's more commonity. I will say just a key point on concerns regarding the Department of Justice and antitrust concerns. After China compromised large numbers of telecom companies, I convened a group in the Situation Room at the White House, and the Department of Justice affirmed that companies working together to ensure the cybersecurity and the trust and safety of their products do not have antitrust concerns because generally there's a larger national interest for them working together. So I would encourage—and frankly we see it in banking, we see it in the healthcare sector—there's regular gatherings of ISACs, Information Sharing and Analysis Centers, that come share information actively, work on trust and safety efforts. So anyone who has concerns, I encourage just asking the Department of Justice or conferring with those sectors that do it successfully each and every day, because from my experience, it is not an issue. Although it appears to be a concerning one, the Department of Justice has said repeatedly it's not one. So I think that removing that barrier that I think some feel—and this was particularly important in sectors that have had high amounts of regulation like telecom. So even in that sector that has had large amounts of telecom regulation because of concerns around companies working together, setting prices, etc., the Department of Justice said, 'No, we need you working together. You have shared vulnerabilities and a need for shared resilience.'