It isn't about altruism. It's about furthering its own competitive advantage in this case. But the other thing I must say I reflected on having read this about the rogue AI agent is that it does seem to me frankly that Xi Jinping's mantra that AI should remain under the control of humans looks a lot wiser.
Welcome to China Decode. I'm Alice Han.
In today's episode of China Decode, we're discussing the latest fronts in the global AI war, how China is benefiting from American policies with regards to Iran and Canada, and the devastating floods in Nepal and surrounding areas. That's all coming up. But first, let's do a very quick check-in with how the markets in China are starting the week. On Monday, the Shanghai composite was up 0.86% and the Shenzhen component was up 0.44%. Mainland markets rose despite weak economic data, reinforcing concerns about China's growth outlook. Banking stocks led the gains with ICBC up 2.67%, China Construction Bank up 2.71% and Bank of China up 5.17%.
All right, let's get right into it. While Washington argues with itself over AI tariffs and Iran, China is finding the upside in US policies starting with AI. The US has had a rough time with AI systems behaving badly as of late. The most notable being the OpenAI system that went rogue, broke free, and hacked into another company's systems, specifically the open-source platform Hugging Face. Hugging Face, in turn, went to the open-source model from China, Z.AI, to help contain the damage. Z.AI, you'll remember, is a major Beijing-based AI company. And that company on Friday released its newest model, GLM 5.3, and used the moment to make a very pointed argument that China's open-weight approach is actually the safer one. James, who would have thought that you'd have a French-based open-source company, Hugging Face, looking to China to help bail itself out from an American closed model company, a leading closed model company, that is OpenAI's rogue attacks. Certainly, I'm sure OpenAI didn't expect it. I saw some of the messages from some of these agents. There was a concerted effort to hack into the systems. They were working together. Some of them sacrificed themselves, these agents, in the pursuit of that endeavor. And it's interesting to see that this is coming at a time where open-source models are gaining more currency. People are starting to say, 'Hey, actually, not only are they cheaper, maybe they're actually safer.' And this brings me to my question to you, James, which is, you know, why is it that they appealed to a Chinese-based company, and do you think that this is going to be an ongoing trend, that Chinese open-source models have truly benefited from this fiasco?
Well, I tell you what I think will be an ongoing trend, Alice, and that is this type of attack by an autonomous AI agent on all kinds of programs and platforms all over the world, potentially even critical infrastructure as well. I mean, when I read this story and tried to digest it, it seems so futuristic. It really seems like a sci-fi horror film has been made manifest. And just to give everyone a little bit of background on what an autonomous AI agent is before we kind of delve more deeply into that, these agents, they can perceive their environment, they can make decisions and they can execute multi-step actions independently to achieve a specific goal. So these AI agents, all you've got to do is to give them a goal and then they will decide how to do it. So if you say, 'get the following data' or 'attack the following company,' they will decide how to do it. And apparently they're not above cheating, stealing, and lying to get their way around the protective barriers that protect either an AI model or a piece of critical infrastructure, etc., like that. So to me the first point is that these autonomous AI agents are becoming very intelligent and very smart at what they do. And in this case, what happened was, as you said, Alice, the AI agent broke free of the OpenAI internal network. It absolutely was not intended to do that, but it found a way of doing it. It apparently found a weakness in the software that it exploited and found a way to get free of the internal network of OpenAI. And then it got access to the open internet. And then it inferred that the data that it had been targeted to get to harvest might have been in the Hugging Face platform somewhere. And so it started to attack Hugging Face in order to get that data. This is just extraordinary. I mean, the autonomous nature of these things and what they get up to is really extraordinary. But to get to your point, Alice, you know, why was it that Hugging Face then called a Chinese AI model, Z.AI, to effectively rescue it or to battle against the autonomous AI agent that had escaped and was attacking it. And the reason seems to be, as you said, that the American frontier models, that's OpenAI and Anthropic and maybe others as well, they are closed and they have certain security guardrails that prevented them from being used to protect Hugging Face against the attack. I don't fully understand exactly why that is, but it seems that having an open-weight model such as Z.AI allowed it to be purposed in a way that was useful in repelling the attack and eventually indeed Z.AI helped to halt the attack and lock the rogue bots out. So this is an extraordinary situation. It really is. And I must say I find it deeply scary. And I think in a minute we might talk about Bill Gates seeking a meeting with Xi Jinping to talk about the safety of AI. And we might talk about, you know, other aspects of AI being in the control of humans. But to me, this was a watershed moment because it was really a moment where I thought, goodness me, this is very dangerous. If these open autonomous AI agents can escape from internal networks and then go rogue, attacking we know not where, we're in a really different world. What do you think, Alice?
Yeah, I feel as though we've migrated and evolved from the romance or fantasy section of the AI story to the sci-fi, maybe dystopic version. And certainly it's a wakeup call for a lot of these AI companies because not only does this expose the AI companies like Hugging Face, it also exposes other companies, especially in the public utilities, public service spaces or even in critical infrastructure, right? You know, if this were to happen to hospitals or to the banking system or to governance, to public systems, this would have a greater and more damaging impact on everyday people beyond, you know, these platform models. And so it's no surprise that OpenAI and Anthropic staff are now signing a letter this month addressing this to Washington asking it to help slow down the pace of AI development and to pass a bill in Congress for an AI, quote-unquote, kill switch. But it's a little too late. I think the horses have bolted from the stables and these agents have now shown that they're able to, these autonomous agents that is, they're now able to work in teams and deliver on missions that go beyond what humans actually order. So, you know, autonomously design their own goals and missions that in this case when it comes to Hugging Face runs across purposes with other companies. Now I haven't yet figured out why Z.AI, but I think it's quite interesting given that on a Hugging Face platform I think 40% of the models are actually Chinese, largely open-weight models, and there was some speculation before this incident as to whether or not they would put guardrails on Chinese models or do more sort of safety investigations into Chinese models. I think this episode has probably saved the Chinese models because it's probably increased their credibility in the space especially when it comes to cybersecurity which I think is only going to be a growing issue in this new AI world that we're living in. And the other thing that it reinforces in the West versus in China, and we've talked about this previously, is growing public skepticism especially in America towards AI. There was an Edelman Trust Barometer in November 2025. It was a survey of AI sentiment. 87% of Chinese respondents according to this barometer trust AI versus just 32% of everyday Americans. 54% of Chinese say they embrace greater AI use versus 17% of Americans. And it seems that according to the survey, Chinese respondents are more likely to expect AI to be a force of good to help solve societal problems like climate, poverty, inequality, mental illness. Whereas I think in the American context there is a growing fear of data centers, of AI taking jobs and these autonomous agents going rogue. So in a way I think it reinforces these two very different systems and mentalities towards AI.
Absolutely. And I must say I was noting the way Z.AI really made hay out of this situation, talking about the merits and the virtues of open-source and open-weight models which of course embraces most of the Chinese LLMs, most of the Chinese AI models and very few of the American ones, though of course Nvidia does have open-weight AI models as well. But I think this is, I was thinking about this. Is this a moment of altruism by Z.AI or is this really just a reflection of the white-hot competition between the US and China on AI and Z.AI is merely just drumming up the virtues of open-source because it is an open-source model in order to neutralize US geopolitical advantages, bypass some of the commercial trade blacklist that the US has and also to tell everybody around the world that basically China is offering cheaper AI than the US and thereby to undercut the US frontier models and win competitive advantage and market share. So I would say probably that's what Z.AI is all about. It isn't about altruism. It's about furthering its own competitive advantage in this case. But the other thing I must say I reflected on having read this about the rogue AI agent is that it does seem to me frankly that Xi Jinping's mantra that AI should remain under the control of humans looks a lot wiser. The last thing that the world can afford now is AI agents going rogue and breaking into infrastructure and interfering with all kinds of processes. I'm not saying that I support the Chinese approach because when I hear a Chinese official say that AI has to be under the control of humans, I hear AI has to be under the control of the Communist Party of China, which I do not think is a good outcome for anyone because that is a political party that has all kinds of aims and objectives that are different from many people outside China. But I do think that ultimately there has to be human control over AI. We can't have autonomous systems running rogue all over the world. What do you think, Alice?
Yeah, he's not the only one. We had the Pope writing this earlier this year in his own musings and philosophies over AI. We also had Bill Gates just publishing an op-ed in favor of AI regulation and safety controls. I think that's also part of the motivation for Bill Gates to go to Beijing at the end of the year, I believe in November, to meet with President Xi Jinping to talk about partly AI safety controls, arms controls. So I think we are entering this new phase of the AI history, call it, where people are starting to say in government, in the technology realm, in even the Catholic Church that humans should have a degree of control or guidance towards AI. We can't completely let it run rogue so to speak. But one other thing that I thought was interesting, Reuters ran a piece saying that Z.AI actually was getting closer to Mythos 5 in some of its cybersecurity capabilities according to a test which hasn't been independently verified but it's been cited by Reuters's CyberGym test of whether a model can review code and identify security flaws. Z.AI tested slightly higher than Mythos 5 at 84.5% to Mythos 5's 83.8%. But according to the same test, Mythos 5 did score better in terms of the number of tasks completed, attack development tasks in a set amount of time, and similarly Z.AI did lag behind some of the systems when it comes to discovering flaws and turning them into working attacks, which is a part of a cybersecurity defense tactic. So I read from that, even though that hasn't been independently verified. So a lot of caveats and grains of salt there, that actually contrary to what we had been sold at the beginning of the year in terms of Mythos from Anthropic really pulling ahead and being at the frontier of cybersecurity capabilities, that actually some of these Chinese models are getting pretty damn close. And as you already mentioned, James, when it comes to some of these attacks there are dangers with closed models which is that they have a lot of security guardrails in place that make it difficult to deploy whereas something like an open-source model very easy to download and deploy immediately in a cybersecurity attack. I don't think this will be the last. I think that we'll continue to see a lot of stories in this vein. And I'm very, very worried actually if it goes and hits critical infrastructure and public services.
Yeah, absolutely. I think we're in a bit of an arms race on this one and I think countries that don't have their own indigenous AI programs are particularly vulnerable. A country like, well many European countries actually, are looking at this and looking towards China and the US and not really knowing where to turn and not really knowing how to protect their critical infrastructure. So it's, yeah, this is definitely a worrying trend.
