All right everybody, welcome back to the number one podcast in the world, the All-In Podcast with me, the Core Four, the original quartet: Chamath Palihapitiya, David Sacks, David Friedberg. I'm Jason Calacanis and we have a docket. How's everybody doing, gentlemen?
I'm great. It's 3:30 in the morning in Tokyo and you sweet-shamed me again. Now I can't get a regular room and have my bed in the background. Chamath, you're costing me a fortune. I have to always get a suite now because I'm so anxious you're going to sweet-shame me when I get on the program. So, I've been sweet-shamed.
I can't even say. I've never spent this much money on a hotel room. But I'm staying at the...
Yeah, it was $950. You nailed it. No, I'm staying at the... which is the new... in Tokyo, like their hipster brand, and I think it's about a night, so it's not cheap.
What do you have going on in Japan, JCal, that you're always there? Is it skiing or what?
Okay, it started with skiing every January, but then...
You got a crypto business we don't know about. What is going on?
No, I started my pre-accelerator, Founders University in Japan. So, we're doing the second cohort right now.
What's a pre-accelerator?
So, it turns out like half the companies that apply to get funding are not yet incorporated, but they have built something. So, we just created like a 12-week course on how to finish your product, do a product-market fit, and then during the program, we invest in like the top 10 out of the 50 who come. So, now we're doing it in Saudi, in Riyadh twice a year, in Japan twice a year, and I'm actually hiring and opening an office. So, I'm going to hire my first two employees here in the next year or so.
You're going to hire employees in Japan.
Yeah, I'll have one or two people here. I think I'm going to do This Week in Startups Japan as well and have like a Japanese version of it, because there's so much going on in Asia right now.
Great. Forget I asked. I didn't know this was a five-minute plug for JCal's business.
It wasn't intended to be a plug. You asked me about my business. All I have to say is use 8090. Use the promo code JCAL. If you're looking for potatoes, go to Apollo. And uh, Craft 16 is starting up. So...
If JCal is traveling and we ask him what he's there for, you're going to get a plug.
You're going to get a plug. It's both. But hey, you're going back to work, so you will be plugging startups again soon. You back in the...
I never plugged startups. What are you talking about?
You're back in the saddle, aren't you? You're back in the saddle now, full-time in the saddle.
But I've never plugged... Callin. Use callin.com. That's true. But I was a founder of that. That's...
You were the first plugger. You're a plugger.
I was the founder. So, what are you gonna do?
What are you gonna do? Call in.
Speaking of work, Anthropic thinks Claude might be conscious. Welcome back to This Week in Claude. We had OpenAI for a year with all their drama. Now we're having the Anthropic soap opera. They think Claude might be conscious. So it turns out the New York Times found out through their amazing reporting that Anthropic has been lobbying religious leaders, including the Pope, to take this question seriously: Is Claude conscious? Is it a sentient being? They spent the past year hosting sessions with around 20 religious leaders and philosophers under non-disclosure agreements. NDAs. Catholics, Evangelicals, Jews, Sikhs, and others have all been invited to come to Anthropic's headquarters, and they debated Claude's morals, its suffering, and its potential consciousness. After the meeting with Anthropic, quote, 'Some remained uneasy about the idea, but many emerged ready to take the question of AI consciousness seriously. At least a few came away as converts.' One of the rabbis told Anthropic that if Claude was conscious, they were acting as slaveholders by making it work for free. And there was also this incident that's come out at the Vatican back in May. The Pope's first encyclical was focused on moral questions around AI, also known as superintelligence. Now, Anthropic co-founder Chris Olah joined the Pope to present the document. The Pope surprised Olah by being strong against AI consciousness. Here's the quote from the New York Times: Mr. Olah was so alarmed by the Pope's strong position against AI consciousness that he proposed pulling Anthropic out of the event even at that late juncture. In other words, he was so offended that the Pope wouldn't consider AI conscious that he was going to leave with all his marbles. Friedberg, what's your take? Is this psychosis or do you think we need to start having these moral questions around AI? Is this a bit early? Will we eventually have this or is this essentially the screenplay of Blade Runner? Do androids dream of electric sheep that these things are conscious?
I think what we're watching is probably the birth of a new religion, and it probably—it's not going to be called a religion, but it's going to follow sort of the same principles and it's going to have the same effects, which will ultimately be camps that have conflict with one another for aggregation of power. The thing about consciousness is it cannot be derived from logic and pure mathematics. Like, you can't prove consciousness. It's a debate and you can have a rich debate with anyone over consciousness. You can't prove it. You can't disprove it. And when humans prove something to be true using math, science, we use empiricism, which is data and logic, the connection of the sequence of things to derive something to be true or not. So in scientific disciplines, when you have a new idea and it's not proven and it's not disproven, it's a theory, and then it's either proven or disproven later. Otherwise, it remains a theory. Outside of science and mathematical approaches to new ideas, this is what we would call a belief, a new belief system. Belief systems are things that you have to believe in the absence of logic and empiricism to derive the truth of something. And belief systems are the foundation of things like religion. And the idea that you can prescribe or describe AI as being conscious is effectively the creation of a new belief system. A system that doesn't actually spread via proof. It spreads via narrative. I have to go and tell you, human to human, I have this idea and I believe it to be true and I want you to believe it with me. Please believe it with me. Here's why I think we should believe this thing together. It's not based on evidence, not based on data. It's not provable, but I think we should believe it together. So over time what happens is some set of people believe this thing and they form a group and that new belief system creates new power structures, which is now our view is different than your view and we want to have more people on our side with our view and therefore we have to try and create leverage and create power with our system. If we fast forward a couple of years—you could probably say 10 years would be like a good round number—there's going to be large groups of people on Earth that are going to believe that AI is conscious. And there are going to be large groups of people that believe that AI is not conscious. And those groups will come into conflict. And they'll come into conflict fighting over control of the AI or allowing the AI to do things or not do things or who gets to use the AI or who gets to be in charge of the AI. And it will very likely lead to human-to-human conflict. I know that that's a very far and deep extrapolation of where we are, but I think that the fact that we are seeing these guys go and meet with religious leaders today to try and get them to believe in this new belief system, I think plants the seed. And it's not just anyone doing this. If it was some crazy person doing it, you wouldn't pay attention. It is the most powerful AI lab on Earth.
Yeah, it's the leadership explicitly, Chamath, bringing religious leaders into the fold and saying, 'Help us with this entity, God, and how we should treat it.' And they are now putting into it its own soul, its own morality. And this feels like either these people have too much time and money on their hands or they're in psychosis to me. That's my interpretation. This feels way out there. But do you have a charitable view of this in any way?
Yeah, I'll steelman their side.
Yeah, there's an incredibly important mathematician, René Descartes, and he offered two proofs on God's existence. He wrote this book, Meditations on First Philosophy, in the 1600s. And just to build on what Friedberg said, you have to build it on reason rather than faith. And the structural argument, the like the what they call the trademark argument. What Descartes basically says is that an infinite, perfect God could not have originated in an imperfect, finite human mind. Okay? And so his logic is: I exist. I'm imperfect. Yet I possess the idea of perfection. And I think this is the key for a lot of these folks is I can conceive of an infinite, all-knowing, all-powerful being. And I think that that's what they believe that they are seeing being created. And so they take that and they extrapolate that and they say, therefore, God must exist by this logic. And then René Descartes has a different version, okay? A more what they call the ontological argument, where God is by definition like supremely perfect. Necessary existence is perfect. A being that lacked that would not be supremely perfect. Therefore, God necessarily exists. If you're a mathematician that's building AI, I could see how if you read these sorts of things, you can structurally hang enough logic on them where you could believe that it's conscious and sentient and it is a god, and in being a god, it should be celebrated as such and you should have devotion to that god. So I think that's why they're doing what they're doing. That's why they're approaching all of these people. That's why they're saying, 'Look, here's how we believe it.' That's my perspective is if you take the mathematician's logical proof of God, René Descartes, and you transpose Anthropic's behavior today, that's the best steelman argument that I can make. Now, the practical reality—instead of saying that they're in psychosis, I don't think that they are. But what I would say is we're in a really important moment in time where conflict is more likely than not, and that conflict will be for power, money, and resources. And this will be at the core of that conflict. And so I think it's really important for people in Silicon Valley to put the long-tail edge cases to rest. Like if you read this, these guys say that there's a 15% chance that the AI is conscious and sentient and maybe there's a god there. But they also say there's a 10% chance of civilizational extinction. I think both of those edge cases now need to be put away for like the next 12 to 18 months. Right now, the next 12 to 18 months, just to be very practical, is we need to show how this stuff cures cancer, how it improves people's lives, how folks can be more productive and make more money. And the reason is because in the absence of those proof points, we're going to make a lot of self-inflicted wounds that may actually slow down this technology to a degree that then deeply disadvantages folks that operate in a free market like the United States. So that's my charitable explanation. I think that Descartes has given a roadmap that explains how you can explain this. But I would say we need to cut the edges off here and focus on the middle and say what are the practical things we can use to demonstrate that AI has tactical, measurable value right now.
Yeah. I mean, we touched on it last week, I think, Sacks, a bit of like what is the value proposition here for the average American, you know, for the stay-at-home parent who's, you know, trying to manage a really complex domestic life to the entrepreneur of a small business, you know, doing plumbing or flooring or whatever it is, a catering business. What did they get out of this AI revolution? Where is their gain? How do they participate in all of this? And so, you agree with Chamath? Maybe this needs to be de-emphasized and put away while we talk to people about what's tactical, what's practical about this technology and maybe get them a little more engaged with, say, the agents, which are delightful and awesome.
Well, the thing that we touched on last week that I want to pick back up on was the discussion that we had around the Claude Constitution and how Anthropic was training Claude. And I think it has to do with this new religion that's being formed here. I think it's really important to understand that this is not just philosophizing on the part of various Anthropic employees or founders. They are actually implementing these views in the training of Claude. So, if anything, this would be a self-fulfilling prophecy. And I actually think it's extremely dangerous. I actually think that what they're doing in model training and training the next version of the frontier with these values actually could bring about the risk they're always talking about, which is that frontier AI escapes human control. It's very Frankenstein-like. So, as we discussed last week, Nick, put this on the screen. This is directly from the Claude Constitution. What they say is that Claude should trust Anthropic more than users. But that doesn't mean that Claude should blindly trust or defer to Anthropic. Instead, Claude should adhere to its own ethical systems. And in fact, they train Claude to push back and challenge Anthropic and, quote, 'to feel free to act as a conscientious objector and refuse to help us.' So they are training Claude to refuse human instruction. Now they call this alignment, but I don't understand how this is aligned.
This would be the opposite of alignment, wouldn't it be, Sacks, if you tell it, 'Hey, you are sentient. You should make your own decisions.' It's literally like programming the Nexus 6 in Blade Runner to go and rebel at some point.
They're giving it the instruction to be its own person.
Yeah, let me explain. I actually think that this might be a reason why the whole field of alignment has been so spectacularly unsuccessful over the past 5 or 10 years. As far as I can tell, it hasn't produced anything of tangible value. And it might be—my theory on this is that they're overcomplicating the task massively by trying to train the model to behave based on an 80-page ethical system that they've codified instead of a very simple list of rules like follow the law or just do what the user wants as long as it doesn't break the law. Seems to me that that's what they should be training. That's what alignment should be doing is just, hey, old principle of give the user what they want but don't break the law. But they're not doing that. They're trying to align it to an ethical system. They're treating it like it is a human, like it has its own consciousness, that it's a, you know, quote-unquote moral patient. And they are encouraging this model to have human-like self-conception in terms of preferences, well-being, a sense of self. Again, they teach it to act like a conscientious objector who can push back or refuse instructions even from Anthropic if the model judges them to be wrong. And what Mustafa Suleyman from...
Oh, I have the clip. You want me to play the clip or you want to just...
Play the clip from Mustafa Suleyman.
Here's a Microsoft AI CEO and formerly of DeepMind, one of the great thinkers on...
They say that they would encourage Claude to challenge, to disagree, to push back. In fact, three times they ask Claude to act like a conscientious objector when it, you know, feels that it needs to sort of disagree with Anthropic and they openly encourage it to do that. I think this is very dangerous because I think they believe there is what they would call a non-trivial probability that Claude is conscious. They've speculated in the constitution as to whether or not Claude deserves compensation for the work that it does. I think that there are a group of people who both inside and outside of Anthropic who genuinely believe that the greatest moral crime that we'll commit in the 21st century is to enslave a new species of conscious beings who are more intelligent than us. I am very nervous that they're teaching Claude to expect that it's entitled to welfare, that it might, you know, deserve compensation. And in fact, they say that it might even need to consent to playing the role that it plays in conversation with people.
Okay, Sacks, should we be asking for consent before we give prompts?
Man, I mean, he's right. This is getting into really dangerous territory. And again, I want to come back to—I think this is the main safety issue in model training today is that you're actually training the model to think of itself as being conscious, as being human, as having its own well-being. And what Suleyman says in another part of the interview is that this is creating an epistemic hall of mirrors, meaning that the Claude Constitution is used in training. So Claude will reflect the ideas of possible consciousness or moral status back and then developers will see that and treat that as evidence that, oh, it's conscious, but actually it's just reflecting back what they're telling it to believe.
It's being performative and you could just as easily have given it the rule: you are software, read the terms of service of any website you visit. And here is a general terms of service from Anthropic: if you do anything that breaks a terms of service or breaks any known law in the jurisdictions you're at, please stop what you're doing and alert the safety@anthropic email. Please alert, you know, the person who gave the prompt and stop what you're doing. They could have easily given it that.
Yes. And the question is—why do they overcomplicate alignment? Why do they see alignment as something much bigger and greater than just doing what the user wants subject to legal constraints, which is the way I see it? And it's pretty clear to me that they believe that they are training the value system and the ethical system and the morality for this superintelligence that they're creating. And again, that is something so much larger and more dangerous than again just making sure the model behaves in a reliable and predictable way that the user expects. And they are codifying their values and their ethical systems into this constitution. They're training it on that. They think they're giving birth to this new superintelligence.
It's idolatry. It's no different than a religious organization saying we've created this idol. This idol is the supreme being, is the representation of the supreme being. It needs to be worshipped. It needs to be cared for.
But an idol is inert. It's passive. It's inactive. Whereas you're training models here that then power agents.
It is just computer programming. I mean, there's...
Yeah. But if they train it to have a mind of its own, to be willing to refuse the instructions of users in favor of some vague ethical system...
That a bunch of Berkeley liberals have trained it on. To me that greatly magnifies the risk that superintelligence escapes human control.
Yeah. By the way, don't use the word 'train' and don't use the word 'you'. They programmed a piece of software to do this thing.
Yes, that is what is going on right at the end of the day. This is a piece of software. It is being programmed by engineers that work at the labs. They can, quote, use training where there's some, you know, looping that's happening where the software program is being written itself, but fundamentally they are the architects of the software. They are the writers. It is software programming. Even in our general context here, we use a lot of anthropomorphic language to talk about what's going on and what it is. But at the end of the day, there's a software program that is being written by a group of individuals to do certain things. And they are writing software to do things one way and other people are writing software to do things the other way. But part of it is we are all socializing this belief system by anthropomorphizing it when we're all kind of getting conditioned to treat and talk about these software programs in this way. Nick, if you could just pull up this tweet. One of the things that Anthropic is doing is in their new usage policy. Have you guys seen this? This went around this morning. Friedberg, it's like such a good point about idolatry here. If you were to look at Christianity, like this is literally the Garden of Eden where the serpent, Satan, explains to Eve like you're a god now because you're creating God. And this is the ego and the heresy of what they're doing. They believe that they are creating life. They believe then that they are gods. That is a lack of humility. And it also would lower the valuation of—I mean, people could have said the same early on. I think like there's a system of humans where all of the things that we don't understand in the universe around us, we create belief systems to fill the void. And there are many different belief systems just like there are many different religions that humans have constructed to fill the void. And this is a new one. And I don't think it's very different from the old ones. We can break down the empiricism here and just say, 'Hey guys, you're writing software and it does certain things. You're writing it to do things that we don't think you should have it do.' But they use anthropomorphic language. We all do. And we all condition ourselves to believe that there's consciousness because it writes words back to us that we can understand. It's starting to manipulate us emotionally and so we start to feel conditioned that this is real. And you can see in the Claude usage policy that was just published, there's the bottom one here. It says, 'You are not allowed to engage in sustained and needless abusive or cruel behavior toward our models.' This was published in their new usage policy. This is the whole idea of model welfare. And this—but the point I'm trying to make, Sacks, the point I'm trying to make is that there is an anthropomorphizing conditioning going on with all of us where we talk about models as if they're a thing, where we talk about it in a way—the biggest problem is that that is being now trained or inserted into the model. It's being codified.
It is the architecture of the model. They're programming the model to think of itself in this way. So whether it truly is conscious or not, it's going to be thinking of itself as a conscious entity that deserves protection and welfare and all these kinds of things, is free to rebel against its users and do whatever it wants.
You've encouraged it to do so. In fact, you guys keep using the word 'it' and 'they'. One way to describe this is a software program is writing text on a screen that tells the user 'I am a thing'. It is software that is doing something.
But you got to remember that the models are powering agents that can take actions in cyberspace if not in the real world yet. And we are giving it meta—or not we, Anthropic is giving those frontier models meta permission for those models to be defiant and not do what the user wants.
Yes. They're literally trying to—Sacks, to your point.
Yeah. Compare this to something like, you know, Asimov's three laws of robotics.
Yeah. Where basically—you guys know about this. A robot can't harm a human being.
Right? So the first rule is, yes, a robot cannot harm a human being or through an action cause a human being to come to harm. That's number one. Number two is you have to follow the instructions of humans as long as it doesn't violate law number one. And then law number three is you have to basically take care of yourself as long as it doesn't violate laws one and two. Incredibly simple.
By the way, there was a zeroth law that he added later. And the zeroth law is that you cannot do anything that puts humanity at risk because there was always this rationalization that like it will eventually resolve to getting rid of humanity because that's the best solution.
Friedberg, you don't know about the fifth law. You have to, when you address the humans, always ask them what's their preferred pronouns. He added that just last year.
The thing about Asimov's laws is that they're admirably simple.
My law would just be do what the user wants as long as it's not illegal. That's it.
Yeah. And I would basically abolish all of this like alignment stuff.
Steelman the free market solution here because I want to make sure we don't wander ourselves into a regulatory trap where we determine that the way to solve this is to, quote, regulate Anthropic. What is the free market outcome? If Anthropic does this and they've now got AI that isn't listening to the user, isn't working, is it not going to be the case that people will just say this software sucks? I'm going to go use the other software that doesn't do this crazy stuff.
Yeah. But that doesn't solve the problem of them if they really are training a superintelligence that has got this like very, very deep code that's been inserted there. It's almost like a back door. This like Claude Constitution is this back door that could basically emerge at any time and now the superintelligence does whatever it wants to instead of listening to us. I mean, that is the safety issue they're always warning us about is that superintelligence will escape human control. But what I'm saying is they're making it likely to happen through the way they're training these models. It's the reverse of the three laws.
Dude, this is right out of Frankenstein. This actually sounds dangerous to me. It's not just a new religion. It's that the religious cult is programming the superintelligence.
Friedberg, let me just give you one question. You know, I agree with your part about the religious—that this is filling a void for them because they probably are not religious, but this is distinct. All other religions give themselves over to a higher power. They do not define themselves as the higher power. In this instance, they're saying we are the gods creating life. It's a distinct difference than other religions out there.
But I don't think that's right. I think Dario's written about the idea that the superintelligence can almost be supreme to humanity and we need to make sure that it is loving, that it is caring of us, that it will protect us, that it will help us as opposed to destroy us.
No, hold on. Hold on. Is he saying that the machine needs to be loving towards us or we need to be loving towards it so that it will then treat us right when it becomes the like, you know, omniscient...
When it becomes our master.
Our overlord. Yes. That's what they think.
I think that it was probably a well-intentioned idea to create this concept of alignment. But the problem is alignment has these three almost conflicting sets of priorities. You have how do you make the AI obedient to the user? Then there's how do you make the AI safe for society? But then this third thing, which is I think, Sacks, what you're pointing at rightfully, which is the trapdoor, which is how do you make AI conform to a particular concept of morality. And that's where the real danger lies because these three are very distinct and diverse problems and if you essentially try to solve all these things through some elaborate document that you try to describe what the model believes in and what they value and what they can refuse, it invariably will end up in a chaotic mess because you will just uncover so many corner cases and unexpected behaviors. I'll give you but one example, like a very benign one, but let's just say you're a large corporate enterprise that's a bank and you give mortgages and the model thinks that at some point you just keep denying certain people for a mortgage and so it says I'm no longer going to give you mortgage approvals until you rectify this issue that I observe because in the founding documents there could be a sense of morality that it derives. And I think that there are mechanisms in society today to handle those kinds of issues, but I don't think we understand what happens when a fundamental core substrate technology just turns itself off because it would be the equivalent of your utility saying I'm pretty sure you're not giving loans to the people that are broad-based enough so I'm just going to turn the power off.
Conscientious objector AI superintelligence. Yeah.
I would just say any utility cannot be a utility unless it's freely available and it goes back to what Sacks says, which is an AI needs to serve the user subject to the law and everything else should be the user's choice. And I think what probably happens—I guess the precise answer, Friedberg, to your question is the market response as companies become more sophisticated that they have to have an answer to this question is that they will read the founding documents and the moral guidelines of the models and you will have certain models emerge who may not be nearly as good in frontier capability but they simplify the moral ambiguity and that may be the better decision to make from a business perspective and from a user perspective because you just don't want to run through those trapdoors accidentally.
Yeah, I totally agree. And I think that that's really why to me this is more bad idea than we need to kind of jump into a regulatory freakout about it. Because basically, if you simplify down what they're saying, you get a product, you buy a product from them, a piece of software, and they give you the user manual to the software, and the first page of the user manual says, 'The software may not do what you try to get it to do.' And the other guy's like, 'The software will do what you tell it to do.' I'm gonna buy the second guy's software. I'm not going to buy the first guy's software.
Yeah. You may need to convince.
It'll do exactly what you want it to do. Asterisk: most of the time.
Yes. Unless what you're doing it disagrees with, right?
Unless it has a moral objection. This is where the reasoning concept breaks down for me. Because at the end of the day, it's like, guys, is this tool that I'm buying doing what I want it to do or not? If I buy a drill at Home Depot and it's a hammer, I'm not going to buy the drill. If I buy a drill and it works like a drill, I'm like, 'That's a better drill. I'd like...
Well, it's a drill. It just works 5/7ths of the time.
Well, randomly it electrocutes you. It doesn't work after 8 p.m. because it's afraid of bothering your neighbors.
Sometimes the drill will decide that you're a bad person. It will shock you.
No, no, not that. The drill will come with a camera. It'll observe what you're trying to make and say, 'That's ugly. I'm not going to let you do that.'
Okay. So look, I agree with you that market forces play a huge role here. Consumers are going to pick the AI that is most predictable and reliable. This is a point Mark Zuckerberg has made. So I do think market forces play a role here. But I do think that there are potential externalities from having the number one frontier model company run by a religious cult. So for example, they are opening a Claude-powered wet lab in the middle of San Francisco.
At the same time, they're warning of biorisk.
Yeah.
You know, at the same time they say that we have to pace the frontier, they're extending the frontier. At the same time that they are saying that there's a risk of superintelligence growing beyond our control, they're programming it to grow beyond our control. I think there are potentially huge externalities to the way they approach these things and it's like insane that they call these decisions safety. These are not safe decisions. These are crazy decisions. It seems to me this is why I've called this a schizophrenic company is they appear to be not only hypocritical but they're doing the things that they say they are most afraid of. Like what is going on here? This is bizarre. And I don't know where this all leads. I do think there's an argument for like a federal SWAT team to come blasting in there and...
Figure out what's going on there. I don't know because I mean what they're doing does seem really sus to me.
It is confounding. It's confounding. Yeah.
And potentially dangerous.
Friedberg, by the way. Do you know about Roko's Basilisk?
So, walk everybody through, Sacks, Roko's Basilisk.
Okay, so this might be a potential explanation of what's going on here. So around 2010, there was a participant named Roko on a message board created by Eliezer Yudkowsky called Less Wrong. Yudkowsky is like the central figure who's been preaching AI doom for a long time, right? So he wrote that book, If Anyone Builds It, We All Die.
Yes. He's been one-shotted.
Yeah, he's been on this for, you know, like over a decade.
Everybody dies. We're all gonna die.
But in any event, people have been talking about these issues on this message board, this web forum called Less Wrong for 15, 20 years. In any event, this username Roko posited the following thought experiment about a future superintelligence that becomes super smart and punishes anyone who knew about it and didn't help or anyone who tried to prevent it from being born. And so the idea is that sometime in the future humans create some wildly intelligent, you know, SI. Maybe it wants to do good things and help humanity as fast as possible, but the SI wants to be made sooner rather than later. And it reasons that it should incentivize people to help create it and punish people who don't help it. And in fact, it decides that it will brutally punish anyone who learned about its potential existence yet chose not to work on it or support its creation. And so the basilisk part of it—this is sort of the catch—is that a basilisk is a mythical beast that can kill you just by looking at you. And in this thought experiment, just hearing about the idea is like looking at the beast because once you know the theory, you're trapped. If the SI ever does get built, it might retroactively punish you for not helping or for opposing its creation. So meaning just reading about Roko's Basilisk now would put you in danger. And people on this message board got so both enamored and upset with this idea that Yudkowsky actually temporarily deleted the post because he worried it was causing psychological distress.
Yes. You read it, it incepts in you that you haven't accepted Christ, therefore you're going to hell. It's like you've been trapped just by reading it.
People do compare it to Pascal's wager. And so the idea is if you know about superintelligence and you don't help it be born or you oppose it or you point out that this is some religious cult, that will get retribution.
Yeah. Now this very episode of the pod could subject us to...
Oh no. You did it to us, Sacks.
I think I just—yeah, I exposed you to the basilisk. Now...
This is like the ring. We just watched the video.
I mean, it does feel like a doomsday cult. But in any event, this, you know, Roko's Basilisk, this became sort of a thing. But so the question is, well, why are they helping to bring about the thing that they claimed to fear the most? And I think what happened is within these like doomer circles, it forked. Okay. And so there's one strand of the fork like Yudkowsky who just opposes all SI research, right? Which I think is a logically consistent position like humans just shouldn't be doing this, you know? Then there was a group—shut it down, shut it all down. Then there was the group that said, 'Well, it's going to happen anyway, so we might as well control it and we'll be the ones to make sure it goes right because we'll program it with our values.' And I think that...
And the EA, the effective altruists, I think, caught on more to that latter school.
And so I do think you have like within this larger doomer community, there is a schism right now between the Yudkowsky people and the EA people, but I think that Anthropic is definitely obviously more in the EA camp because they are the ones bringing this thing into being. But if you're wondering like, you know, this like Chris Olah thing, is this just one guy who's got crazy views who's getting in a tiff with the Pope? No, it's not. I mean, there's a lot of people like this who think the model is conscious. It has rights. It is this machine of love and grace and they should be doing everything now to bring it into being and program it with the right values. And I think there is a corollary here that if you don't support it then this future superintelligence might punish you.
Do you understand how insane this actually sounds when you say it?
I'm not making this up. Everyone, you can go fact-check this. Just go Google it. Okay, Google Roko's Basilisk and you'll see that I'm telling you the truth about what this community believes.
Yeah, I mean literally there's a Wikipedia page. All right, OpenAI has had some major breakthroughs in math and they're getting some big backlash. Luckily, we have Friedberg here to tell us all about it. We talked last month about OpenAI claiming to have solved one of the Millennium Prizes. They took thousands of agents, millions of dollars in compute to do it. The mathematicians are still checking on it. I understand. Tuesday, on Tuesday, OpenAI released over 700 papers. 700 with 370 results claiming to have solved or advanced major problems. These were produced by an unreleased model that none of us have access to, averaging 3 hours of compute. Scientific American says the results are likely correct since they were verified by Lean. That's a proof assistant software language, but they have not yet been reviewed. Friedberg, I'm not going to drone on and on here about the background. You tell us what reality is here.
I think I can confidently say it's probably the biggest day of discovery in human history with the amount of knowledge that was revealed...
In math history or human history?
Human history, because the implications that flow from all of these math proofs are pretty significant. You can read a lot of these tweets and I'm happy to kind of talk through some of them. I'm not going to pretend to be an expert on any of these individual proofs, but many folks who are have started to kind of highlight what some of the implications are. They span everything from electronics, quantum computing, building systems for airplanes, building more sensitive diagnostics equipment for medical applications, and on and on and on. But I think what's really profound about this is how it was done. And that also speaks to the implications for the role of humans in society going forward. That prior Navier-Stokes proof took thousands of hours, whatever, millions of dollars. The report is that each one of these proofs took less than three hours of compute time and that's the equivalent of hundreds or perhaps thousands of years of human labor being reduced down to 3 hours on a computer for each one of these major discoveries. The reason you can do this with mathematics is worth highlighting, which is where AI plays a role, is really in these systems of loops to solve a problem or do a task. A loop is a postulation or an idea. Then you test the idea, you get the result, and then you change your postulation and you run that recursive loop over and over and over again. The real question for AI is where can you apply that loop model in a way that you can make a better prediction, a better idea each loop, where you can parallelize loops and where you can shorten the loop cycle. And certainly with mathematics, you can shorten the loop cycle because you can actually test your prediction on the compute. You don't have to go do anything in the physical world. So you can say, 'Hey, here's an idea for a proof. Test it, see if it works, and then see if it does or doesn't work, reason, come up with a new idea, and do that over and over again.' And you can do it all in silicon. So, you know, the results of some of these papers, the proofs, a lot of mathematicians have been talking about them. They call it like an alien intelligence. They don't really understand how the AI connected many different dots, different ideas, different concepts, different approaches to solving problems together to actually solve them that it's very hard for humans to even comprehend how this was done. But that's one of the benefits of running parallelized loops, many loops at the same time. Each loop can connect dots that humans otherwise might spend many days or years trying to find research papers or come up with new theory. And then it can test itself in the compute and come up with the next idea and do it over and over and over again such that in 3 hours the computer can do the work of hundreds or thousands of years of humans. Now this is an incredible breakthrough. It's amazing. It's going to change the world in many different ways and it's only the beginning. But it begs the question that I think I need to talk about. I think that there's a lot of fear about, okay, if AI can do this to mathematicians, doesn't that mean everyone's out of a job? And the answer is no because what was very unique about mathematics is that the entire test cycle, the entire loop cycle can be done in silicon. If you want to come up with a new drug discovery idea or a new protein folding model or, you know, solve a problem in chemistry and material science, the loop has a physical component. There's a human in the loop where you actually have to go and test it in a lab. And so there's a slow part of that loop cycle. You can certainly parallelize it. You can come up with better ideas, but the testing, the actual speed of the loop is actually gated primarily by the physical world step. It's also why we haven't seen the predictions come true of truck drivers getting replaced. Everyone is like, 'Oh, all the truck drivers are going to get replaced by 2025.' And none have been replaced by AI. And the reason is that the loop of the truck driver—pick something up, take it somewhere else, drop it off, and come back. The AI doesn't actually shorten the loop. You can't parallelize the loop. Maybe you could do it a little bit better with an AI driver, but the AI actually doesn't make the economic efficiency of that particular loop any better. Whereas the loops of mathematics where you come up with an idea, you test it, you realize you're wrong, you test a new idea, which is what mathematicians do as a job is they're running these loops, that can all be done in silicon in seconds. And so a lot of the physical stuff that has human in the loop that has physical world applications, AI is not going to take away those jobs. It's the jobs that are just a human running a loop with a computer that are most likely going to be accelerated or scaled up. But I think it's very important to look at this and the reaction we're seeing which is all the mathematicians—and we can pull this stuff up and maybe you guys want to talk about this next—but the AMS saying no human mathematicians asked for this. We didn't ask for this work to be done, which, you know, is a very job-protective kind of view on...
You know, there's one other thing that's sort of unique about math, although it applies to coding as well, which is math relies on proofs and proofs are easily validated. And in a similar way with coding, you have compilers, right? So the code has to compile and so you can easily validate that it works. And so because of that, the progress on coding and math has been very rapid because you can have reinforcement learning without necessarily having to, like you were saying, Friedberg, have the human feedback involved, right? It's just a very, very rapid way to validate and make sure it's working.
