Military AI/ML in China and the United States with Greg Allen

China's commercial AI ecosystem in many ways is a global leader. And China's military has undertaken a really incredible series of reforms, you know, over the past 10-15 years or so, that have made them stronger and better in a lot of different ways. So, if the Department of Defense fails to reform, it is totally possible that we would lose our military technology edge.

Hey, thanks very much for having me on the podcast. I think the best way to understand the Jake is that it is the organization that the Department of Defense set up back in 2018 to accelerate the adoption of artificial intelligence into all aspects of the DoD. And one thing that's a little bit unusual in terms of the organizational construct is that they had folks who were actually building the technologies working alongside folks who were helping steer policy and strategy for the entire department. And that was the role that I had as the director of strategy and policy.

Sure, I think to start here it helps to have a little bit of historical context, which is that as commercial technology has been getting more and more impressive over the past three decades, the defense industrial base, the types of companies who make weapons for the US Department of Defense, have actually been getting more and more isolated from that part of the economy. So, way back in 1991, defense specialists, companies whose primary customer is the DoD, got about 10% market share in what the Defense Department spends its money on, and these days that's closer to 60%. So, these companies are primarily and almost exclusively focused upon the Department of Defense. That is a story where military technology was increasingly divorced in many ways from commercial technology, and commercial technology was seen as sort of secondary relevance to military priorities. The Russian invasion of Ukraine has sort of flipped that on its head because across AI, drones, cyber, and space technology, these are all areas where commercial technology capabilities have been dramatically and immediately relevant for military-type performance. So, a few areas where that's pretty interesting.

The Ukrainian side, which we know more about for obvious reasons, right? Because the Ukrainians are bragging about all of their successes with AIs, and the Russians are not loudly proclaiming their failures on the same thing. So, the first thing to understand is there's a bit of an information asymmetry here, which gives us an imperfect understanding of what's happening in the conflict. But, from what we know, we can see that there's a few instances where AI is already making a pretty big difference. To begin, commercial satellite imagery. There's a lot of space companies up there who are taking a lot of pictures both in the visual light spectrum, but also in other areas like synthetic aperture radar. These are capabilities that used to be reserved to nation-states, and now there are a lot of companies who are performing these types of activities. And part of the reason why these companies are able to pull it off is because they don't have to hire, you know, the armies of imagery analysts that the US government has used to perform this mission over decades. They can rely on artificial intelligence to share a lot of that workload burden. So, the photos that you're seeing, you know, on the front page of the New York Times or other news sources are coming from these commercial satellite companies where AI is a huge part of their business model.

Another area where AI is making a difference is also in the intelligence side of the equation. So, the Russian military has in many cases been communicating over unencrypted channels because they've done a pretty lousy job of setting up their communication networks for this invasion. So, that means there's a lot of Russian radio signals going out with Russian language being spoken on those signals. So, Ukraine does not have, you know, an infinitely large army of folks to capture all of those signals, transcribe all of those signals, and throw them into some kind of database. But, as anybody who's been using Siri or Alexa for a long time knows, AI is pretty good at voice recognition these days. And so, one commercial company, you know, Primer has been working with the Ukrainian government to intercept a ton of this Russian military radio traffic, automatically transcribe it, so convert the audio to text using machine learning, and then make that into a searchable database where you can actually flag things and get the right information to the right folks at the right time. And that kind of capability again is something that, you know, large intelligence agencies have had for a long time, but what commercial AI has done has, you know, lowered the barriers to entry and increased the performance at decreasing cost for these types of capabilities.

It depends on which topic you're going at. You know, when it comes to spy satellites, the United States government has stuff that is considerably more advanced, right, than what is in the commercial market. And sometimes that's just a matter of law. You know, the major companies who do the most advanced satellite imagery are not allowed to sell imagery beyond a certain precision, or at least not allowed to export imagery beyond a certain precision. But in other areas that's just because the DOD has been doing this for a long time. But when it comes to AI, the sort of general purpose technologies that sort of backbone of AI capabilities, there's plenty of areas in which the commercial industry is ahead of the Department of Defense.

Sure. So I think the first aspect of this is the role that this is going to play in the Russian economy. So the first package of sanctions that the United States Department of Treasury announced in response to Russia's invasion, literally like the first thing they mentioned was cutting Russia off from semiconductors technology. And semiconductors as we all know are foundational to a bunch of different parts of the economy, but they're especially important in training and using advanced AI capabilities. You know, if you want to use fancy machine learning in your military systems, you know, chips from the 1970s, 80s, you know, early 2000s are just not going to cut it. It's a field that really does demand bleeding edge performance in a lot of cases. And so the Russian industry, you know, does not have a credible path to reconstituting their own domestic semiconductor manufacturing capability that is at a peer level with that of the United States or the global semiconductor industry. So instead of coming up with an amazing domestic industrial base, the challenge that they are going to attempt to overcome is how to build up an amazing smuggling operation. And China is a critical part of what Russians are going to be attempting in terms of evading sanctions on semiconductors and other AI-enabling technologies. So, the part of the government that's responsible for export controls is a chunk of the Department of Commerce, known as the Bureau of Industry and Security. And in very large part, as a result of Russia's invasion of Ukraine, the budget request for that agency has jumped from $133 million to nearly $200 million. And that is because they have the most ambitious to-do list ahead of them since the end of the Cold War. And so much of what they're responsible for stopping is what Russia's going to try and run through China.

So, I think the really important date is 2030. So, back in 2017, the Chinese government put out what they called their next-generation AI plan. And this is very colloquially referred to as China's national AI strategy. And it set all kinds of goals for various aspects of China's economy, their government, their military in terms of taking advantage of the potential that AI has to offer. And they set a series of, you know, milestones for when they wanted to match world state of the art and when they wanted to actually lead the world state of the art and when they wanted to dominate, you know, the global AI industry. And 2030 is the time frame that they set in place for that final marker in terms of dominating the global AI industry. I would say, you know, it's been 5 years since they put out that strategy. And they have essentially achieved their first 5-year goal, which was reaching and matching the state of the art. There's a bunch of different ways that you can measure success in this area. All the metrics are imperfect in some way or another. But one that I think is useful and illustrative is the share of the most highly cited papers in AI research. So, these are the papers that leading AI scholars are building upon in their own work for the next round of work. The share that China occupies in the global AI research ecosystem has just gone on up and up and up and up over the years. And they are on track, you know, either this year or next year if current trends continue, to actually be the number one publisher of some of these most highly cited AI research papers around the world. You know, I recall going to, you know, Beijing in 2011, back when the sort of cliché was that China cannot innovate, they can only copy. And I remember seeing like the sort of counterfeit electronics supermarkets in China. But, those days are very much in the rearview mirror in terms of the overall story of what's happening in China's technology ecosystem. There are a lot of companies who are generating a lot of revenue and are building really impressive stuff. And there's a lot of universities who are putting out a lot of impressive research in AI.

So, I don't know that it's right to say there is a really solid measure of who's winning and who's losing, but I think it is clearly the case, right, that China is making significant progress. And not all of that progress is zero-sum. So, there's a paper for instance from Tsinghua University that found that of the papers that are most highly cited that include Chinese researchers, something like half of those include co-authors from outside of China. So, research is an area in which, especially in the AI sector, there's a great deal of international collaboration. And the United States benefits significantly from research collaboration with China. The thing that becomes frustrating from a national security perspective and a foreign policy perspective is that some of the organizations who do genuinely state-of-the-art AI research in China are also deeply involved in some things that the US foreign policy strongly opposes. This includes the domestic security and repression system being set up in Xinjiang where AI plays a significant part in the surveillance state over there. And also in actual genuine military capabilities. There are Chinese, you know, tech companies who the United States will experience as sort of purely innocuous commercial business stuff, but actually have reasonably deep ties to the People's Liberation Army and are suppliers to there. And so the fact that AI is, you know, not just a dual-use technology, but a general-purpose technology that is weaving itself into all parts of the economy, society, and military makes it a real challenge when thinking about what is the right balance of, you know, cooperation versus prioritizing security.

Absolutely. So you raise the point, right, about what is the connection between research and actual ability to commercialize or the ability to militarize on the technology. I'll point out first that when it comes to commercializing, China is quite successful here, too, right? There's a number of Chinese startups or large Chinese tech conglomerates that are actually building very successful products both domestically and abroad. TikTok, for example, right? A lot of the secret sauce of TikTok is in the recommendation algorithm that determines what to show folks next so that they, you know, continue staying on the platform and engaging with the platform. And that's an area, right, where China has been tremendously successful. Your point, though, is absolutely right. In general, there are certain types of AI technologies like the neural networks algorithm that are highly generalizable and are openly available to anyone on the internet, right? And you can take advantage of that sort of general purpose types of technologies. Similarly, like Nvidia GPUs, which many companies and research organizations use to train their neural networks from a processing power standpoint, those are very widely available. Although, as I mentioned, right, we try and cut them off. We try and cut off access of those to Russia as a whole and we try to cut off access of that to military end users in China. But there are certain aspects of the artificial intelligence sort of technology stack that are not perfectly generalizable. You know, folks have pointed out that data is the new oil and that China seems to have most of the largest data sets. There's parts of that story that are true, but, you know, oil you can turn into gasoline, you can turn into a lot of other things. If you have a massive amount of Chinese facial recognition data, well, that doesn't necessarily help you build a missile guidance system, right? Because you need missile guidance data in order to train a machine learning algorithm for that kind of things. So, there's plenty of aspects of the advantage in data that are application specific. What I do want to point out here though is that the overall strength of the Chinese AI ecosystem does have a strong impact in the Chinese military's ability to harness that ecosystem, right? The success of Chinese facial recognition AI or social media AI or financial data AI, that kind of determines how many universities are going to be pumping out how many graduates with these types of skill sets. That determines the size of the workforce, the familiarity with these types of underlying concepts. And so while the data sets might not be fungible, the size of those data sets also sort of translates into the size of an overall ecosystem that the Chinese government has worked very hard to take advantage of. The Chinese policy of military civil fusion is one that very deliberately seeks to take advantage of China's success in the commercial technology sector and find ways for the People's Liberation Army to make use of that.

Yeah, so I think, you know, this is a podcast where I can trust the audience understands that China is not a monolith. And the fact that China is not a monolith and China's government is not a monolith sort of plays out in unique ways in terms of what they're up to when it comes to military harnessing of artificial intelligence. So starting with the diplomatic part of the story and the Chinese Ministry of Foreign Affairs, there have been negotiations at the United Nations in Geneva on regulating in some way the use of autonomy in weapon systems. So lethal autonomous weapons is the term of art that is most commonly used in international discourse. And back in April 2018, I believe it was, you know, the Chinese Ministry of Foreign Affairs put out a paper that said that the Chinese government would support a ban on the use of AI-enabled autonomous weapons, but would not support a ban on the development of AI-enabled lethal autonomous weapons. So the challenge in all of this, right, is that you have folks who are keeping track of these dialogues who are not AI experts or not national security experts or perhaps even not an expert on both. So the headlines that went out, you know, around the world were China supports ban on lethal autonomous weapons. And there's a bunch of very important asterisks there that relate to what we were talking about in terms of the complicated, you know, Chinese foreign policy org chart. So the first is that, you know, the Ministry of Foreign Affairs is part of the Chinese government whereas the, you know, the People's Liberation Army is the military of the Communist Party. Now obviously both of them report up through, you know, Xi Jinping, but the story there is a little bit more complicated. You know, in the United States when the Secretary of State says something on a defense issue, around the world people basically understand, right, that the Secretary of State is authorized to speak on behalf of the United States as a whole. The same is true of the Russian Foreign Minister. But in the case of China, you know, the Ministry of Foreign Affairs does not always speak for the People's Liberation Army. And so actually the People's Liberation Army has said relatively little in terms of what it would be interested in or what it would put up with in terms of diplomacy related artificial intelligence, whether that's coming up with some kind of international norms around what types of technologies are allowed to be used and what types of contexts. In general, you know, it's been radio silent for the PLA. And the weapons manufacturers, some of which are state-owned enterprises, some of the largest ones, have been incredibly bullish on the use of AI in weapon systems and the use of autonomy in weapon systems. So, there's no formal international definition for what an autonomous weapon is, but there is an official DOD definition of this, which is the ability to select and engage targets without further human intervention. So, once the human says, "Hey, I'm putting you in autonomous mode," then the thing can go, you know, find its own targets and engage and perhaps even kill them, right, without further human intervention. That's what we in the DOD define an autonomous weapon as. Well, there's plenty of Chinese weapons manufacturers who are, you know, in their marketing documents describing capabilities that are consistent with that definition. Some of which are even being, you know, we have records of them being exported internationally. So, for example, the Ziyan Blowfish A2 is one weapon system that falls into this category. So, this makes it, you know, really hard for a diplomat or somebody, you know, such as myself in the DOD to really understand in which cases is, you know, one part of the Chinese state not talking to the other. And it also, you know, makes it a little bit confusing for folks in the United States, whether that's in think tanks or academia or the technology industry, to kind of understand what's going on with China. Because when you see a headline that says, you know, China supports a ban on AI-enabled autonomous weapons, you think, "Oh, that's perhaps a favorable development for world peace." But, the unfortunate reality is that a lot of the actions are not backing up, you know, that type of posture.

Yeah, so, you know, these conversations are not meant to be, you know, made public. They're meant to be confidential, but one aspect of the track two dialogues, which for those who are not familiar, a track two dialogue, a track one dialogue is when our government talks to their government. The track two dialogue is when our academics or perhaps retired government officials talk to their academics or their retired government officials. And this is a sort of series of formal and informal dialogues that were very helpful during various phases of the Cold War and that, you know, tradition has sort of been continued and kept up in the case of China. One thing that makes it, you know, a little complicated in this case with China is that a lot of their universities who are authorized to speak publicly on security issues have active military affiliations. So, you might get, you know, somebody who is presenting themself as participating in this track two dialogue, you know, who is a general in the People's Liberation Army, which makes it seem a lot more like a, you know, a sort of a track 1.5 dialogue. But, those types of institutions and those types of dialogues, I'm still in favor of them continuing. It's a tool, but it's important, you know, not to assume that just because somebody in a track two dialogue said something that we can infer that the, you know, the Chinese government or the Chinese military actually agrees with that sentiment.

Oh, I would say pretty clearly it's more like the Ministry of Foreign Affairs dialogue. But, you know, when I was in the Department of Defense, you know, we raised the issue of having a dialogue with China on risk reduction in the case of military AI, and the Chinese always refused that overture. So there has not been that type of diplomatic dialogue outside of the process that takes place, you know, in the United Nations, where again it's the Chinese Ministry of Foreign Affairs.

So, I think there's an element of it that is. So the reason why everyone is so interested in artificial intelligence and machine learning technology fundamentally comes down to the fact that it delivers improved performance at reduced cost. You can do a lot of stuff with machine learning that would not be possible with traditional software. Just for example, you know, a lot of folks had spent hundreds of millions of dollars trying to crack facial recognition using sort of traditional computing and software techniques. But once we figured out how to apply machine learning to the problem and especially the subfield of neural networks, the performance of facial recognition systems really went through the roof. And so it is that improved performance that has everybody so excited and chasing the opportunities of AI. But alongside that improved performance, comes new failure modes. Machine learning software breaks and sometimes it breaks in really strange ways that organizations don't have a great deal of muscle memory preventing, because it's relatively new and the number of folks who have this expertise is obviously limited. So, in general, you know, the Department of Defense, the United States Department of Defense, is in favor of our stuff always working and in favor of, you know, other militaries who are not our allies, their stuff not working. That's how we, you know, tend to think about this. There's a very important exception, though, right? And that is when certain types of technical failures might lead to unintentional escalation or unintentional hostile engagements. There's some pretty, you know, scary stories from the Cold War about certain types of technical failures, you know, leading to folks thinking that they were under attack when they were not under attack. And obviously in a nuclear scenario, that can be incredibly scary. So, the Department of Defense, you know, has spent a lot of time thinking about how to reduce these types of accidents, including in machine learning, you know, and we've gotten really good at it. And one of the things that we want to have, we have wanted to have a conversation on is, is there some kind of, you know, norms we should be considering or other types of just conversation around how do we mutually take advantage of our shared incentive to reduce the risks of these types of things. That's what I think about in terms of risks that are worth paying attention to.