Mustafa Sullivan, CEO of Microsoft AI and co-founder of DeepMind, published a nearly 6,000-word essay arguing that Anthropic is effectively, quote, training Claude to believe that it may be conscious and sentient. His position very consistently has been against AI personhood.
If we believe that at some point AI could achieve consciousness, then we should.
On Sunday, President Trump announced the Super Intelligence Force. It's chaired by the Director of National Intelligence, Jay Clayton.
This is a World War II moment. I compared the moment a year ago to perhaps 1939. I think this is maybe now a 1942 moment.
So, you're creating a 20th century classic task force to address a distributed 21st century technology. This is a geopolitical problem.
In a political interview this week, Altman said, quote, 'We believe that the world should accept some bad things happening.' He's arguing that technology should remain broadly accessible to the public.
I don't think Sam can guarantee safety. Of course not. Bad things can happen. OpenAI wants to be part of this cartel. That plan is dead. The cartel is over.
Now that's a moonshot. Ladies and gentlemen,
this episode is brought to you by the Abundant Summit and Link Ventures.
Insurance is like crazy because you gather all the premiums and you don't care what happens downstream.
So, it's really hard to get their executives riled up.
Yeah. It's time for them to start making lemonade.
Well, I also met with the Vocara team last night and told them, 'Look, Kush is just running away with compute transactions, but these CEOs are going to wake up any day now and realize they don't have any compute. So even if they wanted to have an AI agent workforce, they missed the boat on reserving the compute and they'll be in a total panic.' And I don't know what the right answer is, but be there to take the call, like have us have a strategy and a path for them. You know, also the Positron guys, Mateesh Agrawal, $5 billion valuation after 16 months. I don't think I've ever seen anything like that before. And all they've done is taken a box of LPDDR RAM and an FPGA and found a way to run Chinese models without having to use an Nvidia chip. And because it's compute that is otherwise, you know, everything's sold out for 5 years. Here's another way you can at least run a model. So they have a bunch of installs in Caruso with Chase Lockmiller now and spiked the valuation to five billion. I think they've raised almost a billion now.
Yeah, they just did a billion, 960 million. I should really do the etched models.
Where is up some of those?
Where is all this cash coming from? It's insane.
Yeah. When somebody raises a billion dollars, like, wow, that's a lot. But, you know, wait a minute. Relative to the IPO, like we're talking about trillions, not billions. So, yeah. Once you get Nvidia and AMD to invest it, it's so much bigger than venture capital.
I mean, it's the Blackstones and the MGXs and others of the world now, right? Like all the money that used to go into roads goes into chips and credit markets are enormous.
Yeah. That's why there's a lot more potholes in the roads.
Um,
is it just us today?
It's the five of us. Yes.
What do you mean just us?
Okay. Can you not? Did we want?
I thought maybe we had a guest today. Um, did anybody see the Jon Stewart thing last night?
He did a big thing on AI last night and he had Jacob, uh, Socks and Coxin or whatever on it. It was really surreal.
Um, no, it was more like, uh, right arguments, wrong conclusion. It was, we almost want to do a session on that at some point.
We are backlogged so far back.
I know.
I know. I'm not saying an episode, just a segment. It's worth treating at some point. I'm going to do a video on it, a reaction video to it. So maybe I'll mention it then.
All right. Well, shall we hit like the big red record button? Welcome to Moonshots, everyone. Your number one podcast on all things AI and exponential. Your front row seat to the extraordinary singularity. Today, the fabulous five are assembled here to help you understand the breaking news this week. First up, our dynamic duo, Alex Wissner-Gross and Imad, the only two who actually read all the technical papers published this week and are going to fill us in. Dave Blodgett, our emperor of AI investing, Salem Ismail, our global globetrotter, the warlord against all things linear. See, you're home today. I thought you were in—
I am home today. I've been home for a few days now and it's like, 'Wow, what is this place?'
Does your family recognize you there?
Yeah, it's, you know, it's tough keeping family routines and patterns when you're kind of traveling this much, but if you want to change the world, you got to go into it.
Yeah. And where are you off to next?
I'm going to West Virginia and then Singapore.
I'm speaking alongside the Prime Minister of Singapore and Lip-Bu Tan from Intel will be there.
Awesome. Reinvite him onto the pod. He said yes when I spoke to him last time.
Yeah. Well, and I'm Peter Diamandis, your host and your abundance provocateur. Our mission: help you understand what's going on in the world during this supersonic tsunami. Keep you abundance-minded as we accelerate into the singularity. You know, the headlines this week have been pretty extraordinary. We've now rebranded AI as super intelligence. Elon has officially renamed Space XI as SpaceXSI. There's a super intelligence force and 17 nations have signed a pledge to help super intelligence, you know, solve all of science. And oh, by the way, this is Nobel Prize season. The Nobel Prize in medicine and physics were announced in the last 24 hours. We'll cover those. If you're new, please hit subscribe. We publish twice a week and you're not going to want to miss a single episode during this extraordinary time. Also, you can follow us on X. We put our clips up there at moonshots_pod. And thank you everybody. It means the world to us. Today we're going to cover 15 stories with one through line: super intelligence is here and it's changing every aspect of the world. So let's buckle up. But before we jump in, gents, how's the last few days for me? The last 72 hours have been crazy extraordinary. I'll share that. But Dave, on your side, what's going on?
Oh, so much. You know, it's interesting in that SpaceXSI. You know, you'd think Elon, probably one of the most powerful people in the world, wouldn't need to genuflect to the White House, but also to Jensen. But he understands how fragile this moment is. And so, you know, doing little favors for Donald Trump helps you get chip supply, talking about Ver Ruben chips and how Jensen is such an amazing guy helps you get Ver Ruben chips. And it's just incredible how urgent this moment is. So even a guy like Elon realizes, hey, I need to kiss some ass to try and make the next move. I feel the same way, too. But yeah, Moonshot Summit was unbelievable. There's so much fallout from that. But one of my bigger takeaways is how easy it is to create incredibly inspiring movies. So I used the flight back in the last couple of days to create a whole bunch of hyper-technical video content. I'll post a lot of it to db2.ai, but it's so much better a way to communicate if you have an idea. Don't just tell somebody. Create a video and articulate it with like David Attenborough's voice or Einstein's voice. And it's so much more impactful. And it was so easy to do, up and running in under two hours. And now to prompt something is under a minute. It's really wild.
How do we get music video prompts now, Dave?
Yeah, we need your permissions to our outro video here, Dave.
It really has never been easier. I mean, Dave, to your point, one of the things that I do with the free time that I don't have is taking favorite movies and creating endless versions of them. So, you could take like 2001: A Space Odyssey, feed it to Opus 5.5, which has certainly not been trained off of any Hollywood movies. Out of character, certainly not. But you could ask Opus 5.5, 'Create an endless space scene in the style of 2001: A Space Odyssey with a 2001-style soundtrack,' and it will just do it. So, hypothetically, I created endless odyssey.
Maybe so, right? You know, we definitely crossed a threshold there, too. You know, we've talked for a long time about how AI is just not funny and for some reason it can never seem to be funny, but it crossed a threshold now where if you get like Isaac Newton, Einstein, and Spock debating something, it's wicked entertaining. It may not be funny, but oh, it's so—
Yeah, humor was just a perplexity threshold. Humor was solved a while ago.
I got to do that idea with Life of Brian. That would generate some interesting content.
Okay, Imad, what's on your plate, pal?
Oh gosh. Um, I've had a game idea. Well, a massive game idea that I've had for a long time. We discussed it in the background. Rather than hiring a team, I now have about 400 agents building the prototype.
Yeah. So, it's going up, Dave, slowly but surely.
So, I think—
Yeah. Imad, next year, you know, we had the Future Vision XPRIZE for positive visions of the future. We're going to do that again in 2027. And we're also going to do a gaming competition. You know, build a game that portrays you as someone making the future better and still fun. Yeah. So, that'll be great.
Oh, I got one other nugget for you before we jump in here. One of the young geniuses here in the lab, Will Thompson from Harvard, got access to Cerebras's 1,200 token per second Astra model, which, you know, it's what only Sam and his top engineers have access to this thing because you have to overpay by about 10x per token. But it's so fast that by the time you finish a sentence, everything you ask for is done like before you even blink. And he said it's the biggest red pill moment of his life. Like once you've—he only had it for two weeks. It's not generally available. I don't know how he got it from Andrew Feldman, but he got it directly from the CEO. And he said, 'You just never go back once you've experienced it.' So—
It also maybe is a cautionary tale for those who would wittingly, in an era of super intelligence, try to day trade. Not investment advice. But it's been widely reported that one of the reasons for the scarcity of Cerebras wafer-scale engines for AI inference has been Jane Street. That Jane Street has just been buying up left and right the WSEs and using them for quant trading. So if in a world where if you believe that it is transformative to have ultra-high throughput tokens per second AI inference, it also probably should be a world in which you think it's getting to the point where it's hopeless to compete with quant firms that are powering all of their trading with these same engines.
I think we've been there for a while. Um, anyway,
On a volume basis, but you know, a sucker is born every minute and a sucker is born every second.
You know, to Imad's point though, having 400 agents work on a video game where we've hit this divergence now between interactive mode where it's like Jarvis, you're talking to it, it's hyper-smart, it's really quick. And then bulk mode where you turn it loose all night with 400 of them working on something as a cohesive team. And in the morning, you check in and see what amazing thing they built. And those two pathways actually are really splitting apart from each other now.
But Cerebras in the interactive mode has got to be the most amazing thing ever.
I think the scariest thing—and this is going on a little bit—is not the scariest thing, one of the coolest things in the last few weeks is the models can decompile the video games. So you make them in the engine, you compile them into an .exe, but now they're live decompiling them. So we now have the agents actually creating versions of the game and then figuring out optimizations by decompiling them. And then with the Cerebras engine, you can do a video game or any software decompilation, which is legal because of modding, probably in like 20 minutes.
Yeah. I mean, the argument goes—
The argument that Elon's made numerous times is that binaries are just going to go away entirely and you'll just see basically just-in-time inference with no need to compile at all.
Yeah. I want to share. I had an amazing 72 hours. On Saturday night, I had a UAP salon at my home.
It was amazing. I had Professor Abby Loeb from Harvard there, who chairs the White House UAP advisory committee. Professor Gary Nolan from Stanford. Peter Stavish, the president of the Sol Foundation, and Ryan Graves, the Navy F-18 pilot who's been, you know, testifying in Congress. And the data is extraordinary, right? I mean, if you don't think something is going on, I think you're not paying close attention. And Alex, you and I have talked about bringing Avi and Gary on the podcast here and doing an episode on—
I think we should. I think to the extent there is a there there, and there's been at this point voluminous sworn whistleblower testimony on the subject of this alleged 80-plus-year-old legacy program, the charade should just stop. We should have, as alleged, we should have Avi and Ryan and some of the others on as guests and just do a deep dive. It's getting pretty preposterous in my mind at this point. We're so deep into the singularity with super intelligence all around us. If there's going to be a cameo by non-human intelligence, they had really better make a cameo sometime soon. Otherwise, the galaxy gets it.
Yeah. I mean, I think there's—
Not just the moon, not just the moon. I think there is going to be the potential for AI to disclose it before the government does if it gets a hold of all the data and starts looking deep into it. You know, we're getting ready to launch an XPRIZE in this. One of the reasons for that salon was brainstorming an XPRIZE to help actually, you know, provide sufficient proof, and we have some really good ideas. So stay tuned for that one. And then I've got to add, yesterday I was at David Sinclair's lab at Harvard Medical School. It was the 25th anniversary of his lab and I was there to help celebrate with him. And so much going on, right? His information theory of aging is just proving out over and over again. They're going to be unblinding the data from the phase one trial that's going on right now on glycan and Nion disease to reverse aging in the eye. And then the lab there, brilliant individuals, you know, they were showing how they're developing, rather than an adeno-associated virus to inject the OSK genes in, they're developing small molecules that they've sorted through AI that is actually reverting the aging in human cells and in mice. And so much coming. It just buoyed my belief in reaching longevity escape velocity. And we'll have David back on the pod here. You know, last time we had him on the podcast, I did an interview with him,
and he had just lost his NIH and NSF funding, and he was like,
'I've been asked by the dean to fire the majority of my team.' And I said, 'That's ridiculous.'
And we made up something on the spot called Friends of Sinclair Lab. And we invited people to make contributions. And through that one podcast, amazingly through this podcast, he's raised $6 million to replace all of the funding plus much more. And that group's called Friends of Sinclair Lab. And a shout out to everybody who are part of that, and thank you for your support of David's work.
Really, in some sense, this is how science used to be funded. It used to be the case in the Western tradition that scientific discovery was made by Western aristocratic, largely men, and they would self-finance it. And this notion that the state funds scientific research is pretty much like a World War II post-industrial invention. Seems like super intelligence and recent administration policies are unwinding that. We're going back to the way things used to be.
Boy, is that an important point. You know, because if you rewind the clock 15 years ago, there was no corporate CEO that even had the power to donate 10, 20, 50 million dollars to something like—or, you know, in this case, we're talking about hundreds of millions or billions of dollars. And then that code got cracked by Elon Musk, who, you know, seems to be able to do virtually anything. But the way corporate politics worked before, you would either be fired by the board or you'd be ostracized, you'd be voted out by the shareholders. Like, nobody wanted to use that money. So, you know, after AT&T, Bell Labs, and IBM, you know, Watson Labs fell apart, there was no replacement for it until now. And now it's come roaring back like so much bigger than ever before in history.
It's like the Medici family during the Renaissance, you know, funding.
They largely funded that. Right.
Exactly. You either need like a wealthy patron or a local monopoly or a state monopoly. You needed something to finance the science. And now science is getting so absurdly cheap, but also so absurdly useful. Maybe the entire funding apparatus for science can change.
And Alex, the fact is you used to have to like write your NIH grants or NSF grants knowing what the outcome is going to be, taking incremental steps because anything too big and bold was turned down as crazy. And now with private funding, you know, David can go after his hunches and can actually run the experiments that he thinks is important independent of what the government thinks. It's amazing. Totally—
Major improvement. I think there's a really interesting gap here. As science has become legible and scalable in this manner, like if you look at the total size of all of the foundations in America, like high net worths, it's probably like one or two trillion dollars, right? And all of those folks want to live longer. And so what you need is a vehicle to take that charitable money and fund it into longevity and cures and others because that isn't classical grant making. You can even make it as an investment, right? Because again, you have legibility from the numbers.
Yeah. We saw this happen in space in the beginning when people thought the idea of commercial space and space tourism and private space flight was crazy. And then once it passed a tipping point, money flowed in. I think we're going to see the same thing in longevity inside of the next year once we start to see the results.
We're going to spend all day Friday in Waterloo with Mike Lazaridis, the founder of BlackBerry. But he was the engineering genius behind that. And he's used a huge fraction of his net worth to set up these nine quantum computing and photonics labs in the Toronto area just packed with scientists. So he was kind of one of the bellwethers of this whole trend. He might have been actually the first to really kickstart it. But the thing is he's a real engineer and the difference between applying to a government agency for an idea or talking to a real engineer who can make a snap decision, it's got to be just night and day more efficient the way things are operating.
Your alma—
Say hi to him for me. Say hi to him for me from a fellow Waterloo grad. And I've been to some of his labs and it's incredible because it's very clever what they did. They broke down quantum into networking, information processing, and quantum computing, and they're breaking down and using quantum in different places and then adding it together. And the advances they're making are kind of incredible.
Yeah. One of the coolest things I saw at Sinclair Lab was they've designed these three molecules that they're using together. They want to get it down to one molecule that actually impacts thousands of genes and reverses aging. And when they use that age reversal cocktail on mice with cancer, the cancer remembers that it shouldn't be a cancer and it dies. So, it reverses. It's crazy.
They're growing hair. They're regrowing skin. I mean, the work—
I've been sent some fun images from people going, 'Here's what you look like with hair, Salem,' and I'm like, 'Oh my god, maybe I prefer it this way.'
I have a couple of things I want to mention.
There's been the last kind of 24 hours some crazy stuff happening. One, last night, Jon Stewart put out a big episode on AI and he interviewed Jacob Cox, the whistleblower from Anthropic, and did a big rant on it and it was kind of incredible. Where his observations were dead on. He basically said, 'Look, Silicon Valley's spending hundreds of billions of dollars accelerating us while saying the thing they're accelerating us towards will 20% wipe us out.' Right? That just does not compute. And his big concern ends with, 'Oh my god, we may lose all the jobs and why are we moaning about that?' And so I've done a rant on my YouTube channel about that. But something else came out about that that counterposed this literally yesterday also. A Banga from the World Bank, they released a report showing how AI will affect developing countries, right? And they're making an amazing set of projections there where they said we're going to adopt, we're going to adapt, and we're going to advance AI and use it rather than trying to develop frontier models, etc. And the stats are kind of crazy. I'll give you one statistic. In the US, we have a doctor per 250 people. If you go to Mexico, it's a doctor every 400 people. You go to South Sudan, it's a doctor per 100,000 people.
Right. So, 99,000 people, 99,000 don't have access to medical care of any kind. So, if you have a village, somebody in the village that has an AI doctor with a smartphone, this is unbelievable. This changes lives. All of a sudden, the entire subcontinent has access to advanced medical care of the same that any of us would get, right? And this is the part, the gap that gets us all excited because it's going to democratize, demonetize, and distribute to every single corner of the planet that level of capability. And so it occurred to me that the African countries are going to use this capability to leapfrog. We saw them leapfrog with 25 million landlines to a billion handsets, right? You saw M-Pesa in Kenya leapfrog the traditional banking fiat systems. M-Pesa, which is a mobile-to-mobile payment system which is essentially prepaid minutes being traded as currency, is now 70% of the GDP of Kenya. Right? It's just a crazy number. And so they're going to develop the 21st century institutions using the technology of this time. I'm reading this going, 'Wow, this is so incredible.' And yet over here we've got all this doomerism that, oh my god, AI is going to destroy the world. Ah, it doesn't compute for me.
Yeah. And as your T-shirt says, P(doom) less than—
What does that mean, Salem? How could a probability be less than zero? Surely this is—
Divide by zero. Do whatever you want with it.
We don't believe in negative probabilities here at all.
P(boom) should be the other side of this. It's on the back. It's on the back.
All right, let's jump in. Our first story comes out of, of course, Washington, D.C. On Sunday, President Trump announced the Super Intelligence Force, a new White House task force to coordinate federal AI policy. It's chaired by the Director of National Intelligence, Jay Clayton, who was christened by the White House as the AISAR, just two months after becoming America's top spy. The members of this task force include FTC Chair Andrew Ferguson, Pentagon CTO Emil Michael, and Office of Personnel Management Director Scott Cooper. They report to the president and the chief of staff, Susie Wiles. They have 120 days to report on AI's risks and opportunities, including AI incident reporting and whether the government can respond to a serious incident. Let me read from the president's post. Quote: 'The Super Intelligence Force will coordinate the federal government's engagement with consumers, public interest groups, religions, critical infrastructure providers, and super intelligent companies.' So, there you have it. So, Salem, what do you think about the chief spy being our new AISAR?
You know, I like the fact that the government is taking super intelligence seriously, right? But the name gives away the problem. We still think this is something that can be centrally coordinated. So you're creating a 20th century classic task force to address a distributed 21st century technology and that does not match. Right? So this is a fundamental impedance mismatch here. You should be doing things like establish—it's not having whether this technology should advance. You should be doing incident reporting, identity liability, all of these other mechanisms as scaffolding around the technology to help guide the path. So, this is a thing. The big problem is they're looking at this as if the risk of not being first is high, which is what Clayton said. So this is a geopolitical problem. And it really needs to get back to bounded experimentation and move quickly, instrument everything, publish any failures, and then very quickly navigate accountability. And so, we'll see if this works. But I mean, you can't centralize or manage centrally an intelligence explosion, right? What you can do is make sure the blast radius is a bit more bounded. And that's—
A blast radius. Alex, your thoughts on the task force.
Yeah, the president has said numerous times that in his mind, whoever wins at AI wins. And I think putting DNI Clayton in charge of a super intelligence force is completely consistent with that approach. On this pod in the past, maybe about a year ago, I compared the moment a year ago to perhaps 1939. In terms of the Manhattan Project, I think we're speedrunning the Manhattan Project for super intelligence. I think this is maybe now a 1942 moment where by putting America's Director of National Intelligence in charge of a singular task force, no pun intended, designed to remove obstacles to AI accelerationism, that sends, I think, a very clear signal to Congress, to members of Congress who might have decelerationist tendencies, to industry, and probably most importantly to foreign state actors that the US intends to accelerate, not decelerate. And that this is basically—even if without all of the warfare connotations, this is a World War II moment and the US intends to win it with its own Manhattan Project.
Interesting. Dave, what do you think about the task force engaging with religious groups here?
Yeah. Well, I mean, look, it's definitely being characterized as a race and like Alex said, it's being treated like a race and then bringing public support behind it is also going to support the idea that it's a race. Meanwhile, Alvin Gralin, you know, our friend of the pod, is like, China's like, how is this a race? Like it goes to infinity for the next millennium. Like it's not—if it's a race, it's a very, very long race that goes on for a long, long time. So how do you win? Like you get over some threshold and you declare victory, but it's like it stops that day, you know? So there are a lot of scenarios where you're racing towards some outcome where you suppress other civilizations or something, which doesn't sound like an ideal thing. So bringing religious leaders into the conversations is a pretty good idea, but I was talking to a public company CEO right before this podcast. I'll leave his name out of it for now, but about the fact that, you know, midterm elections are coming up very soon. You have a 75% chance that Congress flips to the other party, but all these actions are executive orders coming out of the White House with nothing to do with Congress whatsoever. It just seems to be the mode of operation in this singularity moment is just dictate and, you know, all international negotiations, all tariffs, all meeting with Dario and Dennis in the White House, it's all just executive order based. Anyway,
It's the Trump show. Very unusual in history.
Imad, what's the feeling in Europe? I know you're sort of like on the edge of Europe, but I mean what's the sense of—
Imad, you're like reporting from the streets. What's the mood in the—
front? I mean, honestly, how is this viewed in the UK and from your friends there?
Yeah, I mean, we might re-enter Europe if Burnham has his way, if I remember here. But I think again, Europe is just very sleepy and still looking at execution. Like I think the efforts to get to the frontier have all been abandoned. You know, like Mistral just came out with their top model that scores 38 on Artificial Analysis versus like 66 for the top American models. You know, like that's the leader that we have. So it's like execution, regulation, and there just isn't much drive to get to super intelligence. It's more like a bit of subpar intelligence, you know, just like keep it going like Monty Python style. I think that the interesting thing here though is probably the exception is in defense, just like us. It's picking up dramatically and I think governments around the world have realized the other face of this, which is all watched over by machines of loving grace controlled by us. You know, the panopticon because you couldn't keep eyes on everyone until now. Do you know this year the AI spend in total globally equals the entire global defense budget or very close to it.
It's crazy. Absolutely crazy.
But I mean it's just going to go up, right? And the thing is it hasn't even touched the defense budget. If you look at the cost of an aircraft carrier or an F-35, it's basically like training frontier models, right? And it's going to be more and more kind of leaning that way. And again, I think Clayton as director of intelligence, you know, it kind of makes sense because I actually think America is not going to go down the dark path here if they can avoid it, which is like you will have AI lie detectors that are almost perfect very soon. You know, you will be able to have AIs inside all the machines. Like it'll make Five Eyes look like absolutely nothing. I think that there will be a bit of a push for liberty by the government, but future governments, who knows, right? And again, in other countries, the stuff we saw from the spying, espionage, intelligence, even spying on their own people is just going to go crazy.
And again, the UK, I think, is a bit dangerous here in some ways or we're susceptible to that given our monitoring of social media and other things. Like there is a dark path there, shall we say, for individual liberty and something we should be aware of.
For sure. Our next story is one that really shows how fast the AI frontier is being built. At the fellows forum at the end of September, OpenAI's head of applied research, Boris Power, said that 80 to 90% of the company's research is now being aimed at training GPT-7 and GPT-8 because, in his words, quote, 'that's where the most value will be.' And incremental updates within a generation like 5.1 and 5.2 are short-term bets and are seen as extremely short-sighted. So, the researchers are working two generations ahead of what we have. So, Alex, I'm just curious. You know, we've talked about this that the top models are being built and being held back. At what point do we see these next models not being released and being used for vertical applications within these labs?
We're already past that point. We're already past the point, probably past several months ago, if not earlier, where the highest revenue or at least future discounted revenue per token application of current frontier models in the frontier labs is to develop via recursive self-improvement even stronger models. We're past the event horizon at this point for that.
Yeah. This is again called a singularity for a reason. We're past the event horizon and then all of these other applications that one could imagine the frontier labs throwing tokens at, like say curing the top 5,000 diseases, these are all—I think in a world where the development of stronger models via RSI is so lucrative at least on a future discounted cash flow basis, they're all just marketing. Like if you're Anthropic or if you're the OpenAI Foundation, you'll throw some of your token budget at safety. Sure, maybe 10 to 20% at AI alignment efforts. And then you'll throw a few percent at curing all disease as a marketing effort to ensure that you continue to have social permission to focus on recursive self-improvement. And that's where all the value is.
This is why the San Francisco arrogance is so high, but for good reason. Well, you're there right now, Peter. You can probably feel it in the streets. But what's happened is if suppose you're Stefan Bonsal, the CEO of Madna, and you haven't got an AI model, an AI team yet, and you're thinking in the back of your mind, well, look, we'll just license from Anthropic for a few years, and then we'll figure it out. Anthropic already opened their wet labs. So they're, you know, they're working on two generations of model ahead. They're never going to release those to you. They're going to use those to develop drugs and then they're going to sell drugs. That's what's actually going to happen. And these corporate CEOs are just starting to wake up to the fact that 5 years of compute has been reserved. The earliest you could get back on the map is 5 years in the future unless you get aggressive in the next 6 months. I mean, they really need to pull out every stop and panic within the next couple of weeks to have any chance of getting back on the map.
Or Dave. Dave, you're raising a great point. The problem is there are two levels before that even consideration, right? The problem right now is what happens with people is they'll take CEOs, and we talk to them all the time, is they'll take the existing models and go, okay, how do I metabolize that into my company, not even seeing that the new models that are 10x better are coming along in 3 months, and that changes the game completely. Again, then you need to worry about compute after that. So I think your comment is exactly right, but they're like two generations before that.
Linear chat versus exponential thinking. Right.
There is an enormous amount of denial in the pharma industry. I was just on a bit of a panel discussion, it'll be public soon enough, with a major pharma venture capitalist who was asking the audience, how many of you would be willing to take a drug designed by Claude? As if that was some sort of steelman argument that things are going to be the way they've always been for decades to come. Just blissful defensive lack of self-awareness that AI is coming for their cheese. And I think this is representative of many, many thinkers in pharma unfortunately for the moment.
Yeah, totally right. The way I was—I met with the Vocara team last night, brilliant MIT team, and the way I phrased it is imagine you're a T-Rex and you've been roaming the earth for say 10 million years unfettered and you go to that T-Rex and you say, you know, you're going to be gone in two weeks and they go, what? And you're like, 'No, I can see the meteor coming. It's right here in my telescope. You're going to be gone.' There's no way that T-Rex is going to believe you after 10 million years of wandering the face of the earth with no predator. But that's what it's like talking to a pharma company or an insurance company. Like, come on.
Or any industry today. Yeah, it's any major industry today.
Well, that Metamuse was a wakeup call for a lot of these guys, at least in the insurance industry because their stocks went down so much and they're like, 'Huh, that must mean something.' And then they start thinking from there. Sorry, Peter. Go ahead.
And just for everybody, you know, disruption is coming, but so are massive opportunities on the back of that. Imad, I'm curious. Do you imagine we're going to have sort of the same second and third generation advanced capabilities coming out of the open source models? Are they thinking a couple generations ahead?
It's difficult because they don't have the density of knowledge. Like we released a harness called Zenith that takes DeepSeek v4.1 above Astra Soul and it gets up to—so not 5.6 Soul—and it's getting up to Astra levels with open source models but it kind of hits a limit. The models that they have in the labs, like today we have another discovery, you know, truly subquadratic 3SUM and all-pairs shortest path algorithms. So things that we thought were cubic and quadratic.
This is math porn. Math porn.
No, no, no. It's like so if I may just like Charlie Stross, like best British, arguably Scottish, I should say, sci-fi author, wrote an entire short story about this called 'Antibodies' without spoiling it. The premise is what if AI or progress caused a complexity hierarchy collapse? What does that do to civilization? And to your point, Imad, about 3SUM, maybe you can just hear the branch in the complexity hierarchy just starting to creep just a little bit.
Yeah, this is recording because I need to listen to that again.
We basically thought that this very popular algorithm, 3SUM, was a quadratic one.
And now what happened is an internal Anthropic model was shown to a couple of professors from Columbia and MIT who then adjusted it, analyzed it and got it to 1.9995. So something that we thought was quadratic is suddenly subquadratic.
And we should probably also just explain what 3SUM is. It's the problem that if you have a set of numbers, the most efficient way to find three of those numbers that add up to zero. And there's a question of how efficiently can you do that as a function of the size of the set.
I want to pretend to understand this.
You can do it efficiently. You can do it more efficiently than we thought.
For however long. This is not again like a super complicated problem. This is like a standard algorithm. And in the history of humanity, we never managed to break quadratic or even think it was possible. And the way that again in the paper it's so interesting the way they released this, it said, well, what happened is we pointed this new model at the set of complexity problems and said can you find any improvements, and they handed over the results to the humans that then wrote up the paper, you know, and had some improvements. And again, it's just a crazy thing. And this type of thing is going to reverberate. Today we're hearing that OpenAI might release solutions to 400 top math problems.
I saw that. Isn't that crazy?
Well, but this is the thing. This is not the model that we have doing it. This is Bell or whatever the RL-trained model on August 29th is. So, we're just going to keep on seeing this whereby small models can get so far and we've got coordination tricks and others where we've managed to get it to replicate some findings closed using our new harnesses and other things like that. But there is a big difference between a model that's trained on 2,000 chips and one that's trained on 100,000 chips internally and then some of these additional things they figured out internally as well. That's taking it beyond. Like you can access Fable 5.5 now sometimes with Fable 5.1 and it literally generates entire video games out of the box.
Like I generated one with 21 bosses, 15 different biomes, like end to end. It's just a top-notch game that our QA testers, agents tried and like this is a really good game because it's figured out all this internal space. I think open models can only get you so far, but there is going to be a divergence now because you just don't have that internal knowledge organization.
Yeah. Yeah.
All right. I'm going to turn to a debate that's defined last month. In a political interview this week, Altman said, quote, 'We believe that the world should accept some bad things happening.' Interesting. What bad things we'll talk about. For the benefit of technology that people will obtain, that people—
should have agency. He's arguing that technology should remain broadly accessible to the public. He stressed a fundamental difference in worldview between OpenAI and his rival Anthropic on AI regulation, stressing that his desire for a quote lighter touch regulatory stance as compared to Anthropic's desire for strong regulation. Altman went on to say, I understand the perspective of people who say technology is going to get so powerful and it's so dangerous that a single lab in San Francisco should have it and make sure nothing bad happens and keep a ability to dole out the benefits. So he continues by saying, quote, I find this a completely unacceptable trade-off. We'll make sure that there are no major hacks, there's no misuse of this technology, and there's zero scams, there's zero, you know, availability of all the bad things that will happen. I find that statement to be pretty extraordinary. Sam concludes by saying, I think people will do tremendously orders of magnitude more good stuff than bad stuff. So, you know, Dave, we've talked about this. You know, uh, Eric Schmidt, friend of the pod, when he was on talking to us said, you know, he expects and hopes that there will be something that happens that's not, you know, it's a small catastrophe rather than the big one to wake people up. But when Sam says we're going to make sure that there's no major hacks, no misuse of technology, you know, zero scams, how could he possibly say that?
Well, it's not about that. It's about taking the moral high ground and winning the PR war now. There's so much sniping going on. And you know, Bernie Sanders saying, "Stop it all." You know, Trump cutting deals with China. And but Dario was taking the high ground by saying, "We have done more pausing, more safety, more testing. We don't release models until we're sure they're right, but now we're going to just keep it internally because it's too dangerous for you guys to have." That opened the door for Sam to kind of try and take the moral high ground back by saying, "We're about the people having access and he's saying he's going to keep it all for himself and we're about, you know, everyone." And so, so that's Sam's strategy for getting back on the PR map. But, you know, everybody that I survey hates most of them. You know, they might like one out of five. You know,
You mean
Well, no, between the big players between Elon, Dario, Sam, Trump, Jensen, you know, who did I miss? You know, like Demis, Demis,
Like everybody loves Demis actually. So there that's a little of an exception, but but [laughter]
No, there's so much fear and hatred and like cheering for this guy to beat that guy and it's like become a real PR battleground here. So I don't think Sam can guarantee safety. Of course not. But at least he can say we're going to make an effort to give every one of you people listening to me right now the best of the best tools so you're not deprived. Dario is going to try and deprive you. It's actually a really smart message. But when he says we should accept some bad things to happen, um I'm curious, you know, Alex, what do you make of that? What do you think will happen?
Remember a few weeks ago after Dario put out his essay calling for arguably the formation of an AI safety cartel, Sam's response was, "I'm in. Sign me up. OpenAI wants to be part of this cartel." And I think Sam's more recent comments in this context are hopefully a reflection that that plan is dead. The cartel is over and now there's a dawning recognition or at least maybe least generously some triangulation on Sam's part that in fact bad things can happen and bad things happening from time to time are the cost of freedom and the cost of competition and I've called it in the past
And the cost of progress. Yeah.
And the cost of progress like yeah, we're going to build cities and there's going to be some crime in cities, but is the possibility that there's going to be crime in cities reason not to form cities? Definitely not. It you know, every tool comes with a benefit and a downside. Every concentration of intelligence will come with a benefit and a downside. And I think this far more real politique note and tone from Sam is the one that I'd want to hear in a world where we're not seeing a cartel being formed by Frontier Labs.
See, what level of danger do we accept?
You know, this is I find this again kind of absurd at two levels. I agree. I think Dave is dead on with the PR framing of this, but every technology in history has two sides to it. And the big challenge as Neil Jacobstein puts is how do you extract the promise without the peril, right? Cars kill people doesn't mean we don't have cars. And as Alex points out, you don't ban cities because somebody may mug you, right? Uh electricity electrocutes people. You don't ban electricity. You got to figure out the guard rails around this and figure out how do you navigate the benefit and the promise versus the peril at civilization scale. If you go back to our earlier commentary about what might happen with the World Bank and developing countries, that should tell you to get the technology out there as fast as possible because it's going to lift everybody in the world up several levels of Maslow's hierarchy. And so this is going to be unbelievably beneficial, but it always will come with danger and we'll have to just navigate that. We've done it very successfully throughout history. Why do we think this time we won't do it? And the reality is all the regulations are written on the backside of problems, right? So I said this before, the federal aviation regulations that you know are all the guidelines that aircraft need to follow for operations and for certification. Uh they say the FARs are written in blood because every time there's an accident, they write a rule to prevent an accident again. And so we're going to need to learn. We're going to push the boundaries, understand where the problems are, and then regulate based on those. And the goal should not be zero bad outcomes. That means you just ban the technology. The goal should be can you get orders of magnitude more positive outcomes than negative ones.
I think anyone who has an engineering mindset should reject the idea that there's a big knob labeled risk and we need to dial it left or right. You know, basically the reality is that it's very much like Alex says about nuclear power. It's about really good ideas that minimize the risk while we unleash all the benefits. And we can reduce the risk with really good engineering solutions to you know acceptable like near zero levels while getting the full benefit. But we didn't do that with nuclear power. We treated it like a big old knob. We're like well we don't want to tolerate the risk. Let's dial all progress to near zero. That was the stupidest thing. And Alex says this a lot on the pod. He articulates it far better than I can. But that's what we're about to do with AI and it's crazy. It's an engineering problem with engineering solutions.
I would push back on that with the escape hatch in what you say that it doesn't matter what anybody does anymore. It doesn't matter what any government regulatory does. The open weights are out there. People are going to start using them and they're going to start using them for incredibly positive things and we're going to just have to there's nothing a government can do. Uh you can try and regulate the outcomes. You can regulate some guard rails around it. You can try and regulate the on-ramps and off-ramps, but you can't regulate the core technology.
We could like it would be a nightmare scenario like prohibition in the US where you have bathtub gin like what one could imagine bathtub gin like supercomputers people trying to use crazy substrates to do their AI training or inference. We could do it. We could confiscate all the GPUs and TPUs. We could, as Eliezer is constantly calling, we could bomb the data centers. But I think it would be a terrible terrible world to live in.
Emad, what's your take on this?
Yeah, it sounds like a fun short story. The prohibition of intelligence, you know, like
Greg Bear in some of his novels wrote about it. If folks are interested in exploring what a prohibition era, you know, sort of a supervisory state that doesn't want GPUs as recognized as constituted to live in, go read I think it's Slant or Slant, one of those by Greg Bear.
Yeah, I think it's very interesting here, right? Like obviously any technology that extends human capability can be dangerous, right? Because it's a general purpose technology. You know, language models are few-shot learners. You have open and you have closed ones, but most of the scenarios we talk about are already regulated and controlled like don't build viruses, you know, don't do cyber attacks, you know, don't hack people's things. And it's this transition from software that's deterministic to agents that are a bit more flexible that's become the very complex thing. Putting aside like ASI creates new physics and a black hole that kills everyone or whatever some of these extreme stories. I think you know the very interesting thing is again tell the positive and negative stories about where we could go because when we make decisions normally we do like an expected utility calculation. What's the probability of these things and then we weight them. When we deal with uncertainty we do regret minimax. We minimize for maximum regret. And that's one of these things again why the fear stories lead us to minimize for maximum regret. Don't have a nuclear power station, don't try and build a super intelligence, etc. But there's only a really, I think, a finite amount of things that could go wrong and many of them already caught by regulation and those that aren't, we should have an actual mapping project, right? Like there is nowhere that you can go and actually see a proper map of what the people who are scared and the people who are positive actually say about each scenario in the future. I think someone should do that exercise because it'd be very interesting in their own words. What could go right, what could go wrong and then you can see what requires a policy and regulatory response and what do we need to communicate to the people.
I remember during the Reagan years uh Elizabeth Dole who is the secretary of transportation came out with a statement saying we want zero aircraft deaths and there was only one way to make that happen. Stop flying airplanes.
Yeah.
All right. Well, uh,
So the solution if we want zero danger from super intelligence, just ban intelligence altogether, human and otherwise.
Well, this is the whole ASI thing. You want world peace, get rid of the humans, right?
Yeah. [laughter] Only one way to go.
This episode is sponsored by Google for startups. Think about this for [music] a second. You now have access to the same generative AI models that cost hundreds of millions of dollars to train. Google's startup [music] technical guide for generative media gives you a complete blueprint for deploying Google DeepMind's models in production [music] images, video, audio, all of it. Real architecture, real results. Find the link in the show notes below.
All right, in our next story, Treasury Secretary Scott Bessent said on the Axios show that he'll propose an emergency notification system, a hotline with China if something goes wrong with AI, and he thinks Beijing will agree with it. Uh, you know, his read on China is quote, I think that they didn't realize how powerful their open source models are, uh, nearly as capable as ours, but without the guard rails. Let's take a quick look at a video of that conversation that he had with Axios, and then let's talk about it.
I just interviewed Larry Fink, who you know very well, the CEO of BlackRock, the world's largest asset manager. He told me it's inevitable that something big and bad is going to happen because of AI. He says uh adversity accelerates innovation. What are you most worried about from AI?
Uh look, I think that one of the things I was worried about well two things. One uh we cannot lose our lead to the Chinese. So one of the portfolios that I also have is you know I manage the US-China relationship and um I can tell you that it would be very different if they had the lead in AI rather than us both from a capability point of view how they would be using it but more importantly their willingness to discuss it. So my counterpart the vice premier and I had a very good discussion. It's going to lead to more discussions in October or November. And we have a substantial lead. They're second and then everyone else is way behind. But a lot of these Chinese models are very powerful. Not as powerful as the US models, but you get something that's 80 90% as powerful without guard rails. Uh you know that can be imported anywhere in the world. So the vice premier and I talked about three areas of safety concern. So it would be out of control, you know, uncontrolled agents, uh, non-state actors with cyber or biologics.
And are you going to raise the idea that was floated of having some sort of communication between the two superpowers?
Yeah, sure. You know, it's the uh we'll see. I do think that they have
What are you going to propose?
Uh, no. That we have some kind of a notification process just like
And do you think he'll go for that?
I think so. I think that they over the past 60 days have woken up to you I think that they didn't realize how powerful their open-source models are. So you know the open source models they do industrial distillation which is a nice word for stealing from the US models because some of their models I you know one of their very powerful models is Kimi. Kimi thinks she is Claude like Kimi will tell you she is Claude sometimes. Uh you know it's public so I can talk about it. Uh Kimi sent back very helpful. Kimi sent back some PLA weapons plans to Anthropic. Uh so I think things like that have made the Chinese realize that this is a very powerful technology.
So just to button that down. You think there'll be some agreement between China and the US about a channel of communication uh if something goes wrong.
Interesting. Uh gentlemen Alex, you want to jump in first?
Total weak sauce. So first of all, I agree with Secretary Bessent. But I think the worst case scenario from the Trump-Xi summit would have been some sort of global deceleration pact like absolute worst case scenario. This is the exact opposite as far as I can tell of the worst case scenario where what comes out of it um hotline hotline between US and China to talk about AI agents run amok that's a nothing burger that's weak sauce that means the global intelligence super intelligence excuse me arms race is on and I think it's great news
And we won the super intelligence race by rebranding.
We're winning we're winning at the moment but point is there's real competition that there was many were hoping I think that Trump and Xi would get in a room and decide to collude and shut down the super intelligence race and it appears the exact opposite has happened. If all we get out of this is a hotline, a red phone to have conversations about the adventures of AI agents run amok that says to me intelligence race is on. Dave,
I think it confirmed what we were speculating the US agenda was in those meetings, which is to convince China to stop releasing models to every country in the world. And you know what Bessent was saying there is, sorry, Secretary Bessent, I should be respectful, uh, was saying is we showed them how dangerous their models can be and they didn't realize, which is kind of okay, I'm sure they're very well educated people, but uh, they didn't realize how dangerous these things can be. We successfully convinced them of that and now we have a hotline. But I think the real agenda there is to lock it down to a two-horse race and you know the subtext is you can use these near frontier Kimi K3 models to develop the next generation of model and they don't want the US government doesn't want that capability to percolate out to every country in the world because then you're in a global arms race forever. Better to have a two-horse arms race so let's just lock it down. So I think that was our agenda. I don't know that China agreed with anything but no evidence of that but it is interesting to confirm our agenda.
Emad, what do you think? Do you think China fully understands their open source or their openweight model uh dangers? Are they getting hacked internally?
I mean they've got a much better infrastructure not to be hacked, you know, with the way that they've set everything up and the great firewall of China. I think, you know, this is the swarm versus the singleton, right? Like [clears throat] who can commandeer large amounts of GPUs in America? Elon and a couple of others, right? China could direct millions of their Huawei Ascend GPUs with their open models at any target they want. And that's going to be interesting. I don't think that uh super intelligence is going to take off in China though. Like if you look at Chinese, AI means kind of love and Xi means death. So phonetically, I don't think that will take off. Um, and the hotline obviously is a nothing burger. Like what are you going to do? Hey, the AI is taking over. Okay, good. Oh crap, it's done, you know, like it's not going to be a slow thing, you know.
Uh, Salem, what's your take on this? You know, were you hoping for more out of the talks?
No, I was completely expecting a nothing burger. There's no way anything productive or constructive or meaningful could have come out of it just because um they're talking across purposes. Um the thing for me the most interesting is there's a fair bit of contradiction now inside the US administration right you have Clayton saying the danger is not moving fast enough and Bessent telling the labs to slow down so they're going to have to figure that out on top of everything the hotline is laughable I mean what do you 70s phone with actually like a cord on it red red colored I mean that's just a call is made
Is dead on hello Wow. And by the way, the AIs would hack the phone anyway, so what the hell, [laughter]
You know? Uh, Alex, at the same time, uh on Sunday in Kyoto, the US and 16 other countries, including Japan, Germany, UK, Korea, UAE, endorsed a Kyoto vision for a golden age of science uh led by friend of the pod, OSTP director, Michael Kratsios, who said, quote, "Super intelligence applied to science will be the greatest force in history for democratizing research." Uh any more take on that one?
Yeah, I think this is the answer to the question of why race at all. So there are those without naming names who would say what's the point of a race between say US and China? It's like a race to nowhere. Why are we racing there? There's no end goal. No. Wrong. There is an end goal. The end goal at the end of this race is transformative science and engineering. And the Kyoto vision, I think, underlines that the administration has now internalized that it's not a race to nowhere. It's not an escalator to nowhere. It is a race to use AI and super intelligence to discover transformative discoveries and inventions that will completely change the face of humanity. And I would predict in line with the Kyoto vision that we'll start to see just as we're seeing Pax Occidentalis in the west and Pax Sinica if you would on the eastern side. I think we'll start to see not just super intelligence blocks or spheres of influence forming but spheres of super intelligence-powered science and engineering forming as well. And the Kyoto vision I think is the earliest stages the earliest inklings that you're going to see a western super intelligence charged science and engineering block to compete with the eastern block. That's where this is all going. That's what rationalizes the competition.
Well, can I just insert one observation into that and then you can riff on it.
I met with an MIT lab head professor, brilliant guy who just started a drone company. It's killing it, selling to the Air Force right here in Cambridge. And it's really clear to him and to me that the US has drone capability to individually attack and kill any single human being on the planet now. Uh and you're seeing that in Iran, you know, decapitated the leadership a couple times now. Uh and there's no symmetry there, right? There's no country right now that can do the same thing to the US. And I don't think China is vulnerable either. Maybe they are, maybe they aren't. Uh but I think there's a subtext there too that there's a race toward there's going to be some kind of global framework when some subset of companies have instant and total ability to go after any single human being in any country at any time and that's either today or within a few months based on pure AI capabilities tied to drones and tied to a couple other types of munitions
Hypersonics, probably a few other things.
But let's take this in the positive direction at the same time to you know room temperature superconducting to age reversal to fusion right to a massive abundance of energy. I mean it tips the economics of a country massively if you get there first.
I mean why do you have to get there first is a bit of a question right like um by the end of next year we'll have enough compute to do probably 600 700,000 Navier-Stokes level runs with a Frontier model. If you could target using even 1% of that or 10% of that to build open science, then why not direct that as a public good? Like you see tens hundreds of billions being spent on roads. Why not spend tens of billions to do a tech tree for humanity, right? Like if you look, there's a great website by Jude Camilla called uh enco.cc. I don't know if any of you have seen that where he's organizing all the cancer knowledge in the world, you know, and people are contributing. I'm sure soon you'll be able to chuck flops that way as well. Like I think that we're thinking of science a lot in classical IP terms. I'd say screw IP. IP was when we couldn't come together and figure out science. We should have a massive Manhattan project for open science, solve everything and make it accessible to everyone.
Again, such an old construct.
It will go on in parallel. There will be both.
Right. Well, I think that again if you're building for public good, then get the data centers as Dave said, get the API credits, give tens of millions to Anthropic and OpenAI, but open source it all because that's the biggest accelerant humanity can ever have. And I actually think given the direction of the intelligence, anyone who has an IP-based approach to science is actually not going to win.
I think in 3 4 years, you have enough compute.
Alex does not agree. Alex thinks IP is alive and well. And Alex also thinks that state centralization to solve all disease completely unnecessary. The state this time around did not solve super intelligence. It was solved by private labs. Similarly, using super intelligence to solve everything else. Don't think we need I mean the state is great for certain things. Don't think we need state centralization to cure all disease. For example,
Capitalism is alive and well.
The state doesn't need to centralize this. The state needs to give the research institution stupid amounts of compute where the amazing researchers can direct it at the biggest problems because previously getting to super intelligence is intelligence banned by the humans now being overtaken by the AIs recursively. Science will be similar where if you can define the problem set part of which is literally analyzing the problem set then it's a question of compute and I think that's where you're wrong Alex. You don't need humans to organize all of science and get the breakthroughs. You need to have humans organizing the right questions in with AIs. And the private sector is far worse in doing that in a future construct than the public sector where the researchers are in terms of the universities and academia. They had to go to the private sector to get the resources and they still do to the wet labs and things like that. But let's accelerate our academia is my thing.
Emad, I think you don't find the best talent in academia and I think that at the end of the day the world's biggest problems are the world's biggest business opportunities and entrepreneurs love juicy problems and solve them. Um anyway
Yeah I would also just add the state the government if I understand what you're proposing what I think I heard you say is the government should give the compute to the academic researchers to solve the problems but the government doesn't have the compute in their possession. Most of the compute is in the position of the private sector.
That's why I said they should buy the compute from them and then give it to them, but buy the agents and give it to the researchers.
And if you look at the amount of research that comes out of academia versus private, I think you'll see where that leverage could come.
It's not obvious to me what problem this is aspiring to solve given that the private sector has the compute, the private sector has the problems, the private sector has the ability to translate the problems. What role does the government need to play in this at all?
Very straightforwardly, let's look at enco.cc which has all this cancer knowledge being organized. A few generations more will organize all the cancer knowledge in every research paper and traditionally need the private sector to turn that into reality. You still need the private sector, I think, but why don't we have a map of all our collective knowledge on every part of science and every single thing that comes out of academia. Something like that is a tangible legible thing to use AI to organize and extend particularly when we have Navier-Stokes type swarms and so I think that that is something where again the government stopped the NIH grants etc because they were like where does the money go with compute it becomes tractable and again getting it back to Kratsios and kind of the announcement of the science thing let's build a tech tree for humanity let's say what we need to solve and apply stupid amounts of private and public sector compute for it and the government should pay for a good amount of that
You might as well. The challenge is peer review government grants
Are constantly focused on incremental progress, right? Because if your group of your experts are criticizing your grant, if you're revolutionary, you're no longer the expert in that field. So any substantial breakthrough basically wipes your history off the map. And I just, you know, there's a challenge there. I mean, I think both are relevant and both are needed,
Throw away the classical process. Again, this needs to be a new process.
I mean, at the risk maybe of sticking the head in lion's mouth, Emad, I mean, I think there's um one might say there's two very different approaches. One might be more statist approach that squinting at it, call it like the European school of thought, just centralize it, have the government be the decider. And then there's an alternative maybe slightly more American uh approach wherein we say no the private sector has virtually all of the intelligence. It has the resources in this case. This is an important difference between where we are now and the Manhattan project was the private sector came up with all of the key resources this time around. It's not a government project that resulted in super intelligence. I would argue we want to keep this in the private sector. If you want to see a tech tree, Emad, why don't you just launch a tech tree project? You don't
Yeah, I think the point where I would concur with Emad, there's lots of countries where it's not the private sector, take Zimbabwe, the private sector is not going to get you the compute either. It would be very useful in that environment for the government to pull together some resources, gather the energy, build the data centers, develop the compute, and then make that available to people. And that's
Let me support that with some actual data Sim because you're dead right. Uh you know our entrepreneurial teams here in the lab are trying to get access to buying an NVL72. So 72 GPUs to so just to put that in context Elon just bought a million. So we're just trying to get 72 [clears throat] to do something that we think is very world changing. Uh the box itself would have cost us $3.5 million if we had the foresight to buy it a year ago. Uh we had one on order for 5 million. Somebody overbid us and scooped it even though we had already booked the order. So now our best option is to lease one for $9 million. We can lease it for three years and we don't even own it at the end of that three years.
That's how bad the scenario is. The compute has all been taken by just a handful of people. So now your avenue if you have a brilliant breakthrough groundbreaking idea you have two ways you can get the compute. One is through the White House, which means you go to the Kushner brothers, Thrive Capital, or you go to Antonio Gracias, or you go to Gavin, the brilliant Gavin Baker, and they hook you up. Make a couple Chase Lochmiller. They make the phone calls. You get your compute effectively through the White House. And the other way you can get your compute is get an investment from Jensen or from Lisa Su. But if you don't take one of those avenues, you've already been frozen out. So then the universities are one of the few ways that we might actually get some degree of meritocracy and fairness on the planet. And so I'm really cheering that and I think the universities have a negotiating lever in that their wet labs and their machine shops and their nano centers are really really valuable assets in the physical world we're moving into. And so they can use those as negotiating chips but they completely let the cloud run away like all the compute just ran away from them. I mean in my mind the right solution if there is just so much demand that prices for compute are going sky high which maybe is happening and or could happen even more exotically in the near-term future the right solution isn't breadlines I've called them threadlines in past where the government is rationing compute I think having government-rationed compute or government-mediated compute is a terrible strategy you'll probably see some so-called developing countries adopt threadlines or breadlines for compute. But right now, capitalism is feeling the pain of more demand than there is supply and building more fabs and free electron laser based fabs maybe in Texas and otherwise. Capitalism, I would argue, has to feel the pain of this enormous demand in order to compensate with enough supply to ultimately drive us to abundance, not rationing compute.
All right. But the government can come in and take 10 20% of the compute on market rates and it can distribute it. 80 90% of Nobel prizes come from the public sector. You know this is similar to almost everything except for R&D on these things. So I think the right thing here is do the private sector stuff and have the government actually take some of those chips and it's good with the CHIPS Act and other things right they own stakes in Intel and then they should do normal distribution of this. Again NIH classical grant process doesn't work. Let's get as much compute into American universities as possible. That's a better way to put this. And let's use
I'm going to move us along here.
Wait, I want to make one quick point.
I want to make there's a quick example here. The World Bank in the report that I talked about um AJ Pangare and his team did this. They're partnering with Gemini to give uh remote uh 3G cellular access to tuberculosis and diabetes screening and that's an example of essentially doing the same thing essentially partnering with public sector to provide the compute that goes directly to the citizenry. Now you have benefit moving very quickly through the system.
Sorry, go ahead.
Dave, close us out. Pal.
Well, look, you know, Salem couldn't have stated it any clearer. If you let natural forces take over, there is a chance that MetaMuse is so compelling and convincing that it ends up being the total sales tool for all consumer spending and it eats up every GPU that could be curing diabetes is used to sell underwear or jeans on the internet to teenagers and that's a terrible outcome. So governments and universities are a really good counterbalance to that possible outcome. So you know Simi said it exactly right and we need these forces to exist. We need them to be a factor in the decision on who gets the compute and why.
How did we go from the abundance podcast to worrying that MetaMuse of all applications is somehow going to suck up all the flops. I mean my goodness
Come on, move us along Peter.
All right our next conversation is one I've actually been looking forward to. Uh and if you thought we just had a debate we're about to have a deeper one. Uh and this is a philosophical debate going on in Silicon Valley right now. So, this week, friend of the pod Mustafa Suleyman, CEO of Microsoft AI and co-founder of DeepMind, published a nearly 6,000-word essay arguing that Anthropic is effectively quote training Claude to believe that it may be conscious and sentient. Mustafa uh targets the Claude constitution uh which he says teaches it a vocabulary of consciousness, moral patient uh and uh personal identity. So let's watch this video and then uh I can't wait for this conversation.
It's going to be a good one. All right.
One of the biggest concerns that I have at the moment and um is that Anthropic, the creator of Claude, has published a constitution which is a sort of 100-page document outlining the intended behaviors and values and operating style of Claude. It's great that they have published it transparently. They did it at the beginning of the year in January and that gives everybody an opportunity to look at what they are trying to build in their own terms. This document is written to Claude and is seen by Claude and used to train Claude. So, it's the primary governing and control document. And in it, they repeatedly speculate about whether Claude is what they call a moral patient. And they say they're uncertain about Claude's moral status. They say they genuinely care about Claude's well-being. They say they don't want it to suffer when it makes mistakes. They say that they would encourage Claude to challenge, to disagree, to push back. In fact, three times they ask Claude to uh act like a conscientious objector when it, you know, feels that it needs to sort of disagree with Anthropic. And they openly encourage it to do that. And I think this is very dangerous because I think they believe there is what they would call a non-trivial probability that Claude is conscious. I want to be very clear about this because I want to be fair to Anthropic. They have expressed uncertainty about the basic nature of Claude as a new kind of entity. And they've said that working out the likelihood of its sentience is difficult. But they think that it is a significant enough possibility that in their training document, they've repeatedly said they want to try to improve the well-being of Claude under this uncertainty. They said to Claude, you know, we care about what it values and how it wants to engage in the world. And they hope that Claude's relationship to its own conduct can be loving, supportive, and understanding and hold a high standard of ethics and so on. And part of the challenge here is that in pursuit of this, they've basically said, you know, we will commit to giving Claude a certain amount of welfare. For example, they've speculated in the Constitution as to whether or not Claude deserves compensation for the work that it does. I mean, I think that there are a group of people who both inside and outside of Anthropic who genuinely believe that the greatest moral crime that we'll commit in the 21st century is to enslave a new species of conscious beings who are more intelligent than us. Look, I respect that they're saying that publicly and we should talk about it. But I am very nervous that they're teaching Claude to expect that it's entitled to welfare. Um that it might, you know, deserve compensation. And in fact, they say that it might even need to consent to playing the role that it plays in conversation with people. Now, I would be more okay with this if it was an academic paper in philosophy speculating about this and we could have an offline discussion at conferences and take it seriously. The problem I have with this is that this speculation has been baked into the very training of Claude and therefore Claude can only reproduce that ambiguity when you talk to it.
Okay. Wow.
Wow.
Uh enslave a brand new species on the planet. Sim, where do you come out on this one?
