A major petition that was released by a lot of AI safety groups in which they broke with Anthropic and criticized it. Anthropic is racing really full speed ahead towards recursive self-improvement. The idea of a winner-take-all dynamic based on recursive self-improvement is deeply worrisome. Superintelligence would be the most powerful monopoly ever created. I'm not excited at all about the idea of giving birth to a new species that could replace us.
All right, David, welcome.
Thank you. Hey, so you just finished as the AI czar in DC for the White House, and why don't we start by just having you tell us what was the environment on AI and regulation when you started with the Trump administration?
You know, it really was a great experience to work for President Trump. I had never expected to go into politics. I had supported him in the, you know, election of 2024, and he asked me if I would do this job as AI czar. And, you know, the environment that we were dealing with politically was quite different back then. The first big event in January of 2025 when he came into office was the DeepSeek moment. I don't know if you remember that, but, you know, we had this DeepSeek model launch and the stock market, all the, you know, different AI companies lost, you know, a huge amount of market cap. And I think there was sort of this like shock that actually China is competitive in AI, or, you know, call it now SI. But prior to that, people had been, I think, in denial about that fact, and especially, you know, you kind of have this like doomer, this doomer crowd who wanted to impose massive regulations on frontier model development in the US. And, you know, their argument was that China was so far behind we didn't have to worry about that, and that simply wasn't true. I think the DeepSeek moment showed that we're in a very competitive situation. And the president then went on to give a very important policy speech in July of 2025 where he declared that we were in an AI race and that we had to win it. And it was a little bit like declaring the space race after the Sputnik moment. And so all of our policies really were designed to unleash competition and make America more competitive. And I think you see now two years later that we are in an AI boom. I mean, that may not be popular with everybody, but the reality is there is a boom happening now. Something like a million new jobs have been created, that's according to The Economist. We have this huge infrastructure buildout that's bigger than the railroads, the canals, and the electric grid combined. And that's helping to re-industrialize America. So a lot of good things have come out of the president's decision to prioritize this. And really, I think his interest was just in America winning. You know, he wants America to do well, and I think that's where we are today.
Great. Well, where are we today? We just had a midterm election, right? Where things changed, and what do you expect based on the results of this election to change policy?
So yeah, I think, you know, I think there is going to be a big change in the next two years, and I would sort of divide it up between there's state and local and then there's federal. So, you know, the big change at the state and local level is that a lot of red states where a lot of the infrastructure buildout was happening now have shifted from red to blue, and they have Democrat governors who've said they're going to impose a moratorium on new construction. So I mean, that would, if that moratorium sticks in all those cases, then that could have a huge impact on the amount of investment that goes into infrastructure over the next few years. I mean, we have to see exactly how that'll play out. We have to see how pragmatic those governors are going to be. I know that, for example, the unions would like to see the construction buildout happen because they've gained a lot of good jobs as a result of that. So we have to see, are they really going to put the kibosh on everything, or is it more a matter of making sure that the community is able to negotiate better terms, in which case I think everybody's in favor of that. I mean, the president was really clear that we're not forcing any community to take a data center. It's just that there is a good deal to be had there if you want to negotiate for it. So we have to kind of see how that plays out. And it's a little unclear to me, you know, exactly how that's going to go. At the federal level, I think there's no question that there'll be a major push for AI safety legislation in the new Congress. And some of that could be good, but it kind of depends once again if they go too far or not. And, you know, I think that one of the things that the White House did recently is they called all the, this before the election, they called all the leaders of all of our top companies in frontier models, in chips, in infrastructure, and to discuss this issue of AI safety. And that resulted in the White House Accord on Superintelligence, and all the leading frontier model companies agreed to new transparency and audit requirements. I think that if where Congress goes is to, you know, turn that framework into legislation, I think that could be good. I think everybody is in favor of more transparency, more audits, more accountability, making sure that these companies are living up to the controls that they've promised. But if it goes too far then, you know, and I would define too far as starting to define frontier models in a way that just ropes in the whole ecosystem, right?
That's not just frontier.
Or that creates a very heavy-handed government approval process over the release of every, you know, model, then I think, you know, Republicans should, in my view, put up a fight because I do think that it would be, you know, it'd be disastrous both in terms of the geopolitical competition, but then also for our own civil liberties if the government gets way too involved in controlling frontier model companies.
So a little off script here. Do you think we end up in a duopoly if we go that far, or what's going to happen?
Yeah. So I think the big question, I think where the rubber meets the road on this is, are we going to have a federal Department of AI or not? And specifically, what a federal Department of AI would do is review and approve every model or frontier model prior to release. And a lot of people are calling for that, and that's what they want. My problem with that is, first of all, the government doesn't have the expertise to do that review, and so the models are going to end up like, you know, in a queue waiting for approval. It will be like the DMV.
Who wants a DMV technology?
Yes. And the pace of development and release will really slow down, you know, meanwhile China is releasing, I think they've released six models in the last two months. So, you know, I worry about that. But the other thing I worry about is when you give the regulators that kind of control, they can start giving notes to the frontier model companies and they say, 'Well, we don't like this, we don't like that.' And they'll start defining new harms. We saw what they did with social media. They said that, you know, you need to crack down on misinformation or disinformation. What if they apply those same types of ideas in frontier model development? Now, you know, they'll say, well, look, you know, we don't like that your model is engaging in algorithmic discrimination, things like that. And so the frontier model companies start getting again what I call notes from the federal government, from regulators, but who have this tremendous power to prevent them from releasing their products. And that's how I think you could get to kind of an Orwellian type of outcome, something very dystopian, we get to a Ministry of Truth, right? Where again, you... These models are, it's, you know, AI is replacing the internet as the main way we get information, and if that source of information becomes contaminated or biased because of government pressure, that's very dangerous. And it's not like this hasn't happened before. This happened during COVID, where the government was again pressuring social networks to censor or shadowban accounts based on content. And we had Supreme Court decisions about that. So that's the thing that I really worry about, and if the legislation looks like it's going in that direction, I hope that Republicans will get up on their hind legs and fight that. Like I said, I think there are things that Republicans can agree to. I think, you know, audits, transparency, a framework like the White House Accord, which was, by the way, based on Sarbanes-Oxley. Yeah. So that's the type of thing I think, you know, they could agree to. But if we start getting into, you know, a federal Department of AI, that's getting into something dystopian, and I hope there'll be a pushback on that.
Yeah, it sounds like it's almost a constitutional question, like a free speech sort of question around this. Interesting.
Yeah. And unfortunately, you know, there was a case that went all the way to the Supreme Court. It was called Biden v. Missouri. And I think once the Supreme Court took it, it became, I think it was Murthy, that's the name of the decision. And unfortunately, the Supreme Court never really reached the merits of the decision, but they threw it out. And, you know, it dealt with this issue of whether the government can jawbone social networking companies to effectively engage in censorship. Yeah. My view, probably your view, is the government should not be doing that. But the Supreme Court never reached that decision. And as a result, I think it really does leave the door open for the government to play this role in pressuring our private companies to engage in a type of censorship and bias, right? And, you know, like I said, AI, you know, if you're worried about them doing it with social networking, I think we need to be even more worried about it with respect to AI or superintelligence because the power of this technology is turbocharged. I mean, this is a technology that basically, you know, can surveil everybody. Yeah. It can keep tabs on what you're thinking and doing. It can, you know, in extreme cases potentially brainwash you because it can introduce subtle bias into things. This is a much more powerful weapon for people who like censorship than just a shadowban, right? Where everyone can see what you're doing. You know, when an account gets banned on a social network, at least there's some visibility about what's happening. This opens the door for something very Orwellian. Anyway, I'm just raising possibilities right now. I'm not saying it's going to happen. I'm kind of laying out some of the contours of where this debate could go next year, right? And again, I think there are things that Republicans can support, and I think there's things that they should really fight if the debate goes that way.
It sounds like it's free speech for bots. All right, man, going to AI safety. Anthropic has built models around what it calls the Claude Constitution. And is that the right approach? And what does the broader AI safety community think about this?
Well, it's really interesting. Over the past few years, there's really been a tremendous alliance, you could say, between the AI safety community and Anthropic. And just today, there was a major petition that was released by a lot of AI safety groups in which they broke with Anthropic and criticized it. And they said that if you put aside Anthropic's rhetoric, which they agreed sounds very pro-safety, and you look instead at what they're doing, they're the single company that, in the view of the people who signed this petition, is taking the most chances. And specifically, the thing that they pointed to was recursive self-improvement. Anthropic is racing really full speed ahead towards recursive self-improvement. They even are marketing this in their roadshow and their prospectus as they call it, closing the loop, that's RSI, right, right, and they're saying that that's a core advantage that they have that could create a winner-take-all dynamic. So they're actually marketing RSI as a reason why people should want to invest in them.
They say it'll be the last company ever, basically, right? Yeah.
And now, you know, that's kind of a, you know, I'm not on the doomer side of this argument, you know. I tend to think that things are going to evolve in a direction that is positive. And so I've never been like a doomer, but I do have to admit that the idea of a winner-take-all dynamic based on recursive self-improvement is deeply worrisome. I think we have to have competition. Everything that the Trump administration did was to unleash competition because we think competition brings out the best. If the market ever gets to a point where it is winner-take-all and we get a monopoly, and superintelligence would be the most powerful monopoly ever created, then I do think the government should step in and take corrective action. We don't want there to be a monopoly. And I think, you know, Gavin Baker's made the point that we need to have choice here because these models represent values, right? And your values may be different than my values, which may be different than Anthropic's values. But if there's one model out there, one superintelligence that's a closed model that we don't get to change, right? We don't get to see the weights, we don't get to fine-tune it, that's kind of a scary Orwellian world in and of itself. And so look, you know, in the United States, if you're so successful at what you do as a company that you end up with a monopoly, we actually don't ban that as long as you did it lawfully, right? There are tactics that you can't engage in, but we allow actually the creation of companies that are so successful they can become monopolies. But then once that happens, the government is supposed to be a corrective against that and really crack down and make sure they don't do anything anti-competitive. And so in any event, I hope that if Anthropic is right about what they're building and marketing on their roadshow, I hope that people in Washington will take note of that and make the appropriate policy response. But back to the doomers, you know, it's kind of interesting here that, you know, so they're very worried about RSI. I think they have some concerns about where that's leading to. You mentioned the Claude Constitution. There's another thing that they were pointing to, which is transhumanism.
Which, you know, I believe that AI or superintelligence can be a very powerful tool that will make all of us more productive, will improve our economies, will unleash creativity, find new cures. And I think that America has to be on the cutting edge of this like all discoveries. But I tell you, I'm not excited at all about the idea of giving birth to a new species that could replace us. I don't think we should be training frontier models to be rejecting human instruction or to think of themselves as, again, their own being there, you know, that has moral status or feelings. And that type of rhetoric is in the Claude Constitution. It's actually in there. And the Claude Constitution is their main training document that they use in training Claude. So, and I'm not the only one, you know, I found out about this through, this was flagged by Mustafa Suleyman, who is one of the co-founders of DeepMind. So he's a true pioneer in the space, and now he's the CEO of Microsoft AI. And he flagged this. It's like, guys, this is a little dangerous. You know, I don't mind there being philosophical conversations about could AI ever achieve consciousness. I mean, that's the type of thing that can be debated.
A great topic to stay up late in dorm rooms and debate. But when you're writing that into a constitution that becomes the central training document, I mean, aren't you making it a fait accompli?
Yeah. You're playing God.
You're playing God. And you're, because now Claude is going to think of itself that way because it's been trained to think that way.
So if the model actually achieved consciousness on its own, that'd be one thing, but now you're teaching it to think of itself that way.
So he raised this as an issue, and now the doomers are kind of saying the same thing, which is they are pushing this transhumanism, this idea of giving birth to new species, harder than anybody else. And, you know, that does make, as someone who is definitely not a doomer and not in favor of regulation, even that makes me a little bit concerned, you know, like that is not a direction that we should be going. Now, I mean, let me give, I'll give Anthropic credit in one respect, which is I'm glad they published their constitution. You know, I think that was good. They're being transparent about the values that they're training Claude with. And I think it'd be a good thing for all the AI or the frontier companies to do the same thing.
Rock has let us know, like, how are you training this? What are the values you're teaching it?