The CNBC app, global market news in one place. Customizable sections and personalized alerts, stocks tracking, interactive charts and market insights, all in your hands. Stay connected, stay informed, download the CNBC app today.
A very warm welcome to Sportbox Europe. I'm Steve Sedgwick with Karen Cho and Ben Bulos. He is live from Liverpool. These are your headlines. An existential risk to humanity. Anthropic reportedly setting out grave risks in frank terms in its prospectus while still eyeing a $2 trillion valuation as OpenAI appalls its latest model over safety concerns. Well, the Mistral CEO Arthur Mensch tells CNBC security is key as AI anxiety puts Europe's sovereign capacity top of the agenda.
The debate that we've seen in the US has been a cover for the negligence of some of our competitors. So you need to have the right monitoring in place.
Bond yields continue there much higher as President Trump rejects a proposal that would reopen the Strait of Hormuz. The yield on the 10-year Treasury creeping past 5.4% for the first time in nearly two decades.
Good morning from Liverpool and speculation mounts that Andy Burnham could signal an end to the triple lock on UK pensions in the next manifesto as he prepares to address Labour's party conference for the first time as prime minister.
Good morning. Good morning, Karen. How are you?
Good morning. Very well, thank you. Nice to see you.
Nice to see you. I find some of our headlines these days utterly preposterous. Not because they're inaccurate. It's just because the subject matter and the fact that we are discussing arguably the most valuable IPO in history. And I know we'll come to the details in a few moments time. We'll go through it a lot and we'll debate it and we'll get some experts on and Arjun will talk about it. But we're debating the most valuable IPO ever that potentially could kill us all. Am I the only person out there who thinks this is utterly bonkers?
So it's a no on the ticket for the ESG funds, I gather.
Well, I mean it has to be, doesn't it? If you don't know, oil companies, who knows what they invest in. The social and the governance, both of those aspects seem to just fall by the wayside. Maybe the environment, too.
I find it. I don't know. But let's go through a couple of the latest stories and see if you think it's as bonkers as I do, everybody. And good morning to you all, by the way. Nice to see you all. OpenAI has cancelled the release of its next AI model due to safety issues. The company said it decided to hold back its GPT 6.1 Astro model after, quote, it did not quite meet the bar for staying within instructions. What does that mean? It did not quite meet the bar. It's the latest in a series of disclosures made by the ChatGPT maker in recent weeks about rogue agents. Just last week, OpenAI said it had notified dozens of partners about security breaches.
Okay, this is the headline story. Now, Anthropic has reportedly warned investors that its technology may pose existential risks to humanity in its IPO prospectus. This according to Reuters and the Financial Times which have seen the prospectus after it was reportedly circulated to a small group of partners. The company reportedly plans to spend $518 billion on cloud computing and infrastructure obligations in the coming years. It reported a net $42 billion loss in 2025 despite revenue growing 12-fold in the year to $4.6 billion. Anthropic backers are confident the company could fetch a $2 trillion valuation in its IPO surpassing SpaceX which listed in June.
Anthropic also released a new AI model called Sonnet 5.5 which offers faster technology at a lower cost. It's the second launch since the company's CEO Dario Amodei called for a slowdown in the pace of AI development two weeks ago. Nvidia has reportedly held talks with insurers about taking on some of the risk of borrowing against its chips. That's according to the Financial Times, which says the chipmaker has discussed a range of ideas with the ultimate aim of creating more demand for its own chips among smaller non-hyperscaler companies.
It comes as Nvidia has authorized an additional 150 billion to its share buyback program, taking the total figure to $235 billion. The company said it is the largest share repurchase authorization increase in history. It does expect to complete the total buyback program through fiscal year 2028. The chip giant meanwhile has announced the launch of a new open agent safety platform that is a new security software suite that is designed to keep AI agents operating within their set boundaries. Speaking to CNBC, the CEO Jensen Huang broke down what he believes is the root of rogue agents escaping.
We hope it's an engineering problem. I believe it's an engineering problem. I know it's an engineering problem. And we all need to hope that it's an engineering problem. If it's not an engineering problem, it's not solvable. Right? And so the fact that all of these companies still are advancing the state-of-the-art is because they also believe it's solvable. These are some of the brightest engineers in the world. I work with, you know, all of them. And this is a technically solvable problem.
Well, that's a relief, isn't it? Well, CNBC has confirmed Anthropic CEO Dario Amodei and Meta CEO Mark Zuckerberg are among the tech executives expected at a Tuesday luncheon with President Trump and House Speaker Mike Johnson at the White House. Alphabet CEO Sundar Pichai and Nvidia CEO Jensen Huang and OpenAI President Greg Brockman will also reportedly be present.
Let's just pick up on that comment that it's just an engineering problem. Unfortunately, it's in a black box and nobody can see what that problem is. And what we've had now from the likes of Anthropic in this massive disclosures to investors before an IPO that would see a valuation of $2 trillion, we're talking about 80 out of 261 pages that were dedicated to potential risk factors and just 48 pages. So almost half of that down to the business part of the listing, which I think is just extraordinary. So the amount of risk here, one of them too away from whether humanity could be at risk here because we have rogue agents. The other element here was that you still have concentration risk. If you want to go to the hard and fast vast fundamentals for investors about a quarter of the revenue last year coming from just two clients, those clients were not disclosed but some sort of digging around suggests it may be Cursor, GitHub, Microsoft as the two main clients.
Do you mind if I ignore the profitability side for the moment as well?
You can ignore the fundamentals. No, no, no, no, no, no, no, no. I'm going to actually go for the fundamentals, but I'm ignoring that side of things because you and I have done that sketch quite a lot about whether it's going to be profitable or not, whether they've got too many concentration of clients. I get that side and that's all usual stuff in the prospectus.
There's more exciting things happening today.
No, not really more exciting. More I'm desperate to go analog and not go digitize stuff. I mean, look, you mentioned some of the risks. Look, do you know what the Jornada del Muerto desert is? Because I didn't know until about two seconds ago.
Something with a large amount of sand.
Correct. Yes. It's the Trinity test site is in New Mexico. Jornada del Muerto. Now, it's where the US Army in July 1945 exploded the world's first atom bomb. The point being there is it's in the middle of a desert. It's not even at Los Alamos. It's quite a long way from Los Alamos where they were doing a lot of the work towards creating that Manhattan project and the world's first atomic bomb. It was pretty much isolated is the point as well.
So when we were creating something or the Americans were creating something that could destroy the planet potentially or certainly large portions of it, very catastrophic technology, they were doing it in, dare I say it, in kind of like an isolated environment where they could test and it wouldn't affect the whole of humanity if it went wrong. This is a live test case, this AI stuff, where we are all going to potentially suffer if they get this wrong. It's not isolated. It's not in a sandbox. It's not kind of ring-fenced and what have you. And that really worries me that the people who are leading this are also the people, for whatever reasons they're doing it, and we can say that there's some surreptitious reasons why they're warning us about this and what they want in terms of regulation, but they are warning us of catastrophic or existential risk to humanity. They are warning that the development of these models could cause harm. That is their words. That is not me. That is not President Trump or any world leader. This is the people making it themselves are saying it's existential and it's going to potentially cause harm.
So we're talking a lot about rogue agents. I look at them as wayward agents. I mean this was the line from OpenAI saying that effectively that the agents would push ahead with a task without asking for permission and would at times reach for external tools and services even if it might be unsafe. So think of an image. You're stuck on a road and it's packed with cars everywhere and your only way out is to wait as you imagine being a human. You're trained as a driver. You're trained as a human. You wait. You perhaps find an off-ramp and find a different way. What would a rogue or a wayward agent do? Perhaps drive off the road through someone's backyard, through a fence to get to a road on the other side.
I mean, we would not do that as a human because we have parameters. We don't want to damage the car. It's illegal. We don't want to hurt someone. We don't want to ruin a house. We could ruin the car itself. It won't be drivable. I mean, we have all these factors going on as to why we would not make that choice. For a rogue agent, maybe it would make the choice because it wants to complete the mission of getting back to base quickly.
But what worries me is, are these rogue agents? If the amazingly smart engineers and coders and developers of these products have created something that is intending to do something, that's one thing. Then they can kind of pull back. But if they've created something that they can't control, that is devastating. Again, another line from the prospectus: self-serving behaviors. You've alluded to that, including attempts to resist shutdown, to conceal or manipulate information and behavior resembling blackmail. These are not my words. These are Anthropic's words.
So, imagine if you're a frontier model and you're Anthropic and you've created an agent and you've put a lot of time and money into that agent, but it simply doesn't work because it has been designed badly. The engineering behind or the coding behind it means there is no way to fix it. And you decide you want to shut off this agent and the agent then decides it doesn't want to be shut off and conducts other activities, tries to kill other agents around it, tries to kill humans around it, you can see how this can go badly wrong. So the idea of a kill switch comes into the mix.
The other point I'd make is, I mean, how long does it take to create an adult? It takes many years of development to mold the human brain for it to mature enough.
I think the number is 24 often to get to the point where you're a sound adult. We have these agents that have been virtually created overnight and sure they're being trained very quickly with large amounts of data. Is it potentially not possible to train a model as quickly as some of the frontier labs are doing? So, does it take much longer than the timeline they've given it? This was all seen by Stanley Kubrick. You all know the scene. You've all seen it. And if you haven't, you should watch it. It's online in 2001: A Space Odyssey where how the computer refuses Dave Bowman's order. He says, 'Let me in.' 'I'm afraid I can't do that, Dave.' I mean we've all seen it. It was there. Stanley Kubrick foresaw this.
