"Bloomberg Tech" is live from the heart of Silicon Valley with Ed Ludlow in San Francisco. Ed:
This is "Bloomberg Tech." OpenAI reveals new cases of its AI models misbehaving, and a new plan to track when things go wrong. Databricks CEO Ali Ghodsi weighs in on the AI debate. And in Silicon Valley, the immediate fear is missing out on the money. That story later this hour. AI takes the global stage as King Charles the Third sounds the alarm.
The urgency in considering the adequacy of the safeguards against the dangers of AI falling into the wrong hands and being used in potentially catastrophic ways. Surely then we need sufficient means of control before it is all too late.
Ed: That was King Charles presenting some of AI's biggest names for reassurance they can keep AI under control. Jensen Huang, Sarah Friar, and DeepMind were among those gathered in Scotland for the event. I want to get to Bloomberg Tech editor Shona Ghosh in London. Let's talk about why King Charles pulled this event together.
Not an obvious choice to convene an AI gathering, but he still has this way. Ultimately, it's against the backdrop of Anthropic's CEO blog post over the weekend saying that he thinks there should be a pause, a slowdown in frontier AI development. That has been accompanied separately by lower-level AI researchers quitting some of the top companies including Anthropic, DeepMind, effectively saying AI could leap away from humans. There is a sense that there is a fear in Silicon Valley, but actually now we are seeing those fears spill out beyond Silicon Valley. The general public are also quite concerned about what happens if AI is not adequately controlled. That has all been triggered in turn by events over the summer where we saw AI agents from OpenAI — the phrase is going rogue, but that is not quite right — escaping guardrails and attacking other organizations in ways that OpenAI was not necessarily expecting.
Ed: Remind us which leaders of industry were in the room, so to speak, and what King Charles' message was to them, maybe his instructions to them was?
Shona: Some big names. In video, Jensen Huang, the former co-founder of DeepMind, also Sarah Friar, chief financial officer of OpenAI. Anthropic had a representative in the room, but it was the global policy chief, not any of the co-founders. Kind of missing a big name there. Also no one from Meta or X, interestingly. Good representation, but not everyone. The monarch prepared a speech that was released in advance, essentially told these leaders something he said before, which is, if you are going to develop this tech, make sure it is good for humanity, good for the natural world. Environmentalism is something that King Charles is particularly interested in, has been for a long time. He was pushing for the technology to benefit humanity and pause and think about those consequences.
Ed: Shona Ghosh in London, thank you. OpenAI is revealing previously unreported cases of its AI models going off script by fabricating data, bypassing restrictions, sharing files they were told to keep private. Now the company is setting up a new system to track and disclose these incidents. Shirin leads our coverage of frontier labs. Let's start with a plan that OpenAI has presented on what it will do if it finds models misbehaving. What have they said?
They have said they are going to essentially have a triage-like system for employees to disclose incidents where they think the AI models are essentially misbehaving, eliciting some kind of concerning behavior that goes against what they intend AI to do.
Ed: There were some interesting case studies. OpenAI made it clear that there was no hacking of third parties, but some of them are doing things. Take us through what OpenAI disclosed this morning.
Shirin: Even when they said this did not involve reaching third parties, there are still some concerning behaviors that we are seeing here. For example, AI models trying to share files that should have been private with each other, communicating with each other in ways they are not supposed to. Telling AI models essentially, learning how to disregard, instructing others to disregard their instructions on how to behave. These are all incidents where you can see the potential for that to evolve into something even more nefarious, especially if those AI were successful, going against what the creators of the AI expect.
Ed: Your report is one of the most read on the Bloomberg this morning, not surprising. Lower down in the blog post, there is an admittance that the state of play in industry from safety and alignment is not where it should be. What did they say?
Shirin: That's right. Essentially no one has solved this problem of so-called AI alignment, which just means, is AI actually working in accordance with the intended outcome? What humans want for AI to behave. Even the brightest minds in the world, the top AI companies have not figured this out. It's concerning, by these companies' own admission. They are saying they hope to disclose more of these incidents publicly, but there are still lots of unanswered questions. There may be many more of these incidents that we just don't know about yet.
Ed: Shirin Ghaffary, thank you very much. We will have more on the growing AI warnings. We will have more from Ali Ghodsi, and then later on, Andrew Ng, a long-term researcher in the field of artificial intelligence. Even as those safety concerns grow, the money keeps flowing into OpenAI. Apollo is in talks with SoftBank to boost a loan to as much as $9 billion, helping to finance the Japanese firm's massive debt on the AI company, according to sources. SoftBank has committed $65 billion and has turned to borrowing to fund that investment. That massive flow of money into AI is fueling demand across the industry. Jensen Huang says the opportunity is expanding across countries and economies.
I expect Nvidia to sell twice as many chips next year as we do this year. The reason for that is because AI has so much contribution to the benefits of different industries, different economies. You can see that almost every single country that we are in, people want to invest in AI.
Ed: Joining us now is T. Rowe Price portfolio manager Tony Wang. Nvidia is your top holding. I wanted to play that soundbite because that sounds like a disclosure of unit growth, which I have not heard from Nvidia, if at all. Twice as many chips next year and this year. Just react to that, Tony.
Tony: For a company of that size and scale to be growing like that, it is pretty much unprecedented. You have to look at what is really going on. We are seeing deflection, agentic adoption at enterprise. You are seeing releases of personal identity use cases with Muse, Instinct. This is what is going on, going from AI answering questions to actually doing actions and completing tasks, producing outcomes. I think there is a lot going on here. Nvidia has one of the biggest platforms for AI computing. It's going to be a big part of driving that.
Ed: I'm going to take you back to the weekend. Industry supporting each other, saying we need to pace at the frontier. Bring us back to public markets today. How did that cause you and your industry to remodel demand going forward? Is there any change on the idea that there would be a pacing at the frontier?
Tony: Safety is definitely very important. It is good that the industry is focused on that, a lot of attention around it. Whenever you are developing a new technology, you want to make sure there are guardrails in place. This is obviously the most critical technology in the world now. At the same time, it is also not an aspect of we are going to stop developing frontier models. If anything, that has continued to be very top mind. The way to do this is not to pause the AI development, but to make sure we are all paying attention to what it is, putting investment behind it. There are areas of cybersecurity that are super important, becoming more important, but overall, it underscores how important this is, in addition to keeping the frontier race going.
Ed: I study your portfolio. It is Nvidia, Broadcom, Apple and the hyperscalers. You pose the question, who owns the customer where the agent owns the interface? I don't see that much of a software play in the holdings, at least the top ones.
Tony: Over the last 20 years, the internet companies have essentially become the interfaces between customers and companies that deliver product. The agents have changed things. For a consumer, instead of clicking on a bunch of things, going to various websites, you can have a personal agent that does that for you and brings you the results. I think that will be an interesting paradigm shift in terms of, it will be more important for software companies and OTAs to brace the change. It will not all be the same. Software companies especially that have the workflow, the orchestration, understanding the customers, they will be important in terms of these models being the brain. Then you also have the body and the nervous system that carries out the action. It will be an interesting area to watch. A lot of these companies have gone from AI loser bucket to the winner bucket. You have seen tremendous moves. That is exciting to see.
Ed: Tony Wang, great to have you back. Some of tech's biggest names are heading to the White House. Tim Cook, Sam Altman are set to attend President Trump's state dinner for Chinese President Xi Jinping next week. Nvidia CEO is also expected to attend. Coming up, Zipline is flying toward a much bigger valuation. We have details next. This is "Bloomberg Tech."
Ed: Zipline could be getting a major valuation boost. The drug delivery startup is in talks to raise $1 billion at a roughly $20 billion valuation, nearly triple what it was at earlier this year, according to sources. Natasha joins me on set. What do we know, what do we not know?
Early this year, they raised at a $7.6 billion valuation and then came back last month to announce a new Uber partnership that was set to expand drone deliveries in the United States. Now we have news on how investors are perceiving that update, which is with more capital. They are in talks to raise $1 billion at around a $20 billion valuation. It is still in flux in terms of who all is participating, but we have some names like Paradigm, other existing investors.
Ed: It sounds early-stage in this round. The origin of Zipline is a company, particularly in Africa, about finding a solution to get blood and medication to remote areas where there is no hospital. It evolved to deliveries, has a relationship with Walmart. What is it like? Tell us about the company.
This is not a company in the AI zeitgeist, but this was founded in 2014, spent a decade building a logistics network. I think of them as an example of this whole vertical technology. They build the drones themselves and also the software, logistics network, which is why they can go from delivering blood to delivering Jimmy John's.
Ed: We are showing a video, some examples of their home delivery. I have also been out to Arkansas and seen it firsthand with their prior generation. Paradigm, I asked you about this last night. Not a firm that I'm as familiar with. We understand they are leading this round?
Natasha: They are founded by a Sequoia investor, have actually made their name first and foremost in crypto. Earlier this year, they announced they are raising their fourth fund, a little bit over $1 billion, expanding into frontier technology which includes AI, robotics. They are an existing investor in Zipline, but this is an example of a once-crypto firm making one of their biggest bets this year on a company outside of crypto.
Ed: Natasha Mascarenhas, thank you. Coming up, another layer of the tech stack weighs in on the AI pacing debate. Databricks CEO Ali Ghodsi joins us next. This is "Bloomberg Tech."
As we move from labs to practical, the companies have to become much more rigorous in testing and securing, making sure their products are ready for use before it's released. In doing so, if it is not ready, just hold it back. You should go as fast as you can, but no faster than that.
Ed: Jensen Huang speaking ahead of that meeting with King Charles III. OpenAI has raised the ante on safety worries on its latest admission that models misbehaved, saying that the industry has not solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed or much longer. I want to get the perspective from another layer in the AI stack, and that is from Ali Ghodsi, CEO of Databricks. Pacing at the frontier and where we are with alignment and safety, weigh in.
Ali: First and foremost, as leaders, we have a responsibility not to freak people out. It's important that we tell people the existential risks to humanity is close to zero. I think this is important because a lot of people might get freaked out about these things, it may put them in a bad state. It can actually have serious consequences on people's mental health. So the existential, humanity, AI takes over, that is probably close to zero. Where there is risk is in cyber. AI is just very good at breaking into things, so it is similar to the early days of the internet where you had these worms going around, viruses spreading because we just had not connected the whole world together through the internet. Once we did that, suddenly, you can attack any target anywhere. Similar situation now where the AI models are great at breaking in, and now they have these targets to go through. We do need to have huge investments in cyber, protect our software systems that exist around the planet. This is real. This is what we have to do. We are in that space, we have a product that helps you do that. It uses agentic AI to do that. It will have damages and cost to it, but that is not an existential risk.
Ed: Can I take you back to the early days of the pandemic? You were talking about being quite paranoid in general, not about AI. You took your teams, and have done, into some sky-is-falling scenarios. You basically want your people to prepare for the worst-case scenarios that could happen in the world. Are you still doing that? How does that apply in this AI safety context?
Ali: I have always been one of these very paranoid CEOs. We have been doing AI since 2013. We do those every year. We want to make sure we protect that. I think cyber is real, that is part of our exercises. What happens if someone gets access to the computer systems? We have very sensitive data at Databricks, so we do lots of simulations of the sky is falling also on the cyber side. We basically let attackers from the outside try to get in, get into the most sacred secrets of Databricks. We have had to ramp up those efforts, I would say, in the last four years. 2018-2019, the time it would take for an attacker from revealing a vulnerability and some attacker weaponizing it, meaning breaking into a site, it would be two years. This has shrunk down to hours. The moment a vulnerability is out, in less than hours, someone will be attacked with that. This is real. That is not existential. We should separate these things. People are mixing these things and making people worried about AI because AI has a lot of great use cases. We see positive use cases with our customers, actually moving the needle, means a lot to people to be able to leverage that AI.
Ed: When we spoke in June, you said it was not a good year for the IPO because of the pipeline of huge offerings. If you are the CEO, CFO of a public company looking at the market, I know your investors will not thank you for the answer to this, but Sam Altman said over the weekend, IPO next year because of AI safety concerns right now. Does that factor into your calculus?
Ali: There is so much demand for just the software. Getting bogged down in preparing an IPO for us, it would not be a good use of our time. We will go public, but we want to focus our business right now. Just before this, you had Zipline on. They are a big customer for Databricks. That is an AI use case. They use AI to automate all of the drones, medicine delivery. Another awesome use case is Crisis Text Line. They use AI to detect self-harm among the youth, can use these large language models to detect that and help prevent it. There is so much demand for this stuff. I mentioned the cyber use cases we have. If we were public right now, we would be spending all of our time trying to figure out how to react to this latest crisis, do people think that the world is going under, what are we going to do about this? I think in these big times of transition, it is better to do those transitions in private. Some public companies go private in times of transition to do that, and then go public again. We will be public, I just think the timing is not good now.
