I got an email at 10:59 yesterday saying, 'Hey, do you want to do something impromptu in Rap Park at 11:00?' And Scott, who's finished his talk, got a question at 11:04 today about whether he'd like to do something impromptu in Rap Park at 11:00, and we both said yes. So, here we are. Here's a second mic.
Thank you. The low battery one. Great.
Oh, great. Do you want one that maybe needs to be plugged in but isn't? Random roll. Hello. It might need to be turned on.
All right. So people in the front can hear Scott's answers and people in the back.
There's one here. You may just need another batteries for it.
We can also just trade mics.
So I've had 12 hours and one minute to prepare. Scott has had negative 12 hours and 1 minute to prepare. Scott has had negative 4. So we're hoping that you guys will have questions which would make it easy on us.
So tell me about all this AI stuff.
If we build it, will everything go okay?
Yeah, I think it will probably not go okay if someone builds super intelligent AI. I like the current AIs. This is my way of getting hated by almost everybody on the planet. Yeah, it looks to me like they will not wind up with good values.
Sorry. Raise your hand if you do not know who at least one person on this stage is. Okay.
All right. Let's introduce ourselves. I am Scott Alexander. I write the blog Astral Codex 10. I'm interested in forecasting in AI just like everyone else. I'm also a psychiatrist and an amateur miracle investigator apparently.
Yeah, I'm Nate Soares. I'm the president of the Machine Intelligence Research Institute. I'm less funny than Scott. I recently wrote the book If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, in keeping with this not being very funny thing.
Yes. Look, and seeding the fact that like super intelligent AI will likely be a bass bank but will likely be miss. I think one of the things I struggle with was is like how do you get from here to there? Like most systems to me seem all induction based without any like ability to actually prop solve conjecture reputation, David Deutsch, all of that. And I'm just curious if you have any thoughts on like what will need to happen and in what timeline would it need to happen for those characteristics to emerge? Does it come purely through scaling? How do new algorithms need to be deployed? Any thoughts on that?
You want to go first and I'll go second.
Sure. I would challenge that. You're probably aware that...
We're asked to repeat the question. And the question, which I will now butcher, is what's needed to get from here to super intelligence. In particular, aren't today's AI systems somehow limited by, you know, being like coming from human training and aren't they unable to sort of, you know, go off on their own and solve hard things that aren't heavily shouted by training. Is that roughly?
Yeah. So I'll challenge that. As you probably know, AI recently solved some very exciting mathematical problems. I'm sure that some people will still say that that's still like not creative reasoning, just like massive at-scale application of uncreative reasoning. I think these are a gradient. I think that creative reasoning is applying reasoning, applying boring reasoning steps extremely well in the right order. An analogy I would make is that human brains are basically monkey brains that are scaled up a bit. Monkeys are not doing that many interesting things. But if you do those same boring things with more fidelity at higher scale, then we start being pretty creative and interesting. Likewise, GPT-2 was not doing that much that was interesting. I think if you had said GPT-2 scaled up would like start solving Erdős problems, people would be quite surprised. One thing that I've learned that kind of shaped how I think about this is that a reason scaling works is that AI is really just performing like one million different tricks. Like to some degree there may not be such a thing as general intelligence. There's just like being so big that you can learn a million different tricks and know exactly the right time to do each of them. So I don't think there is a difference between, I think the difference between kind of these leaps of creative reasoning versus doing boring reasoning very well is just a matter of degree and that as we scale AI up it will increasingly look like the output of its boring inductive reasoning is creative and exciting.
Yeah, I agree with most of that. I would add a couple things that I think a lot of people miss about how AI works. One is training an AI to predict human text does not limit it to human abilities. Often predicting text generated by humans is much harder than generating that text. If a nurse sees the doctor administer epinephrine and then sees the patient's eyes open and writes down the doctor administered epinephrine, the patient's eyes opened, and the AI is predicting, is filling in the blank on the doctor administered epinephrine, the patient's eyes blank. It is harder to fill in that blank if you can't see the patient. There's actually a lot of prediction problems about human text that are harder than generating that text. Anytime humans wrote down observations about the world, predicting that text really well involves learning things about the world, which can include things humans don't know about the world. So whether current AIs can be trained on pure prediction and wind up more powerful than humans, that's a question of the architecture, that's a question of the training methods. But whether they can't because it's prediction. The prediction is not limiting here. You know, saying, 'Oh, AIs can't become smarter than humans because they are trained on prediction text' is like saying, 'Oh, a house cat can't kill you because it's made of biology.' It is made of biology. It can't kill you. And tigers are very real. You know, is this a tiger cub or is it a house cat? Is this as big as it gets? That's sort of a different question. Separately, AIs today are not trained on pure prediction anymore. You know, AIs today are trained on like you give them a hard problem. You give them a thousand tries, you have some human or some AI come in and tell you which try got closest to solving the problem. And then you have some automated process tune trillion numbers inside the thing to make it, you know, more like whatever was getting the better answers to the problem. You repeat this process a bunch. Next thing you know, it is solving these big open problems. And you know, this process can sort of push the thing to develop abilities that go well beyond humans. And then sort of tying back to Scott's point a bit, you know, one of the jokes here is like, oh, it's just learning how to understand the world, comprehend the world, and develop new insights like it saw in the training data. It's just repeating those past training data. Like, yeah, okay. Doesn't really matter if it's picking up these sufficiently deep ability to generalize. You know, those deep abilities generalize as Scott says they may come from, you know, having a million tricks and knowing properly when to use them, right? It's not like there is a binary switch between them. And you know, like when the AI is learning how to solve various math problems, it's both memorizing answers to a bunch of math problems and, you know, learning some of the deeper skills about addition. You know, you can have it add numbers that have never been added in the training set and it has learned some deeper thing about addition. And then, you know, probably it's also a little bit learning deeper things about even more general skills. It's not learning those deep skills as efficiently as humans, but it seems to be learning them at least a bit and, you know, we can probably expect that to continue.
If someone wants to donate money to AI governance, where do you recommend they go?
I'll give a very vague answer first. So, oh the question is if you want to donate money to help with AI governance, where should you donate to? So, some considerations. Dustin Moskovitz has at least 20 billion dollars. He is also hoping to donate money to AI governance. So the important question is not what is the best AI governance charity. It's what is your story for why Dustin Moskovitz hasn't already filled this opportunity. I think there are many great stories. One is you somewhat disagree with Dustin Moskovitz. I think many people here do. Dustin is like a very legitimate work within the system type of guy. If you want to fund radical firebrands, that's a good story for why Dustin hasn't gotten there first. Another good story is in US campaign finance. Watch out. You can, billionaires may donate an unlimited amount to super PACs which are allowed to do some things, but they can only donate a small amount to candidates who are allowed to do their own campaigns. So, if you can donate, if you like, for example, Alex Bores or someone in that category, you can donate directly to his campaign, you can donate, I think, a max of $7,000. Dustin can donate a max of $7,000. And you're exactly equal there. So, that's another great opportunity. There are a couple, most billionaire foundations are kind of scared to donate to, I think I'm getting this right, C4s, which are the nonprofits that are allowed to be quite politicized. So again, if you want to have some value above and beyond the billionaire foundations, you may find it useful to donate to these more politicized groups. You can also just look at the specific things the billionaires are donating to and donate to something else. For a while, Open Phil wasn't donating to digital minds research. I think that's now no longer true. But that would be a good place to donate because of that. I know Nate has many more thoughts on this. I was giving him some time to collect his thoughts. What are your thoughts?
That covered a lot of the basics. One reason a lot of people are shy about C4s is that they are not tax exempt. So if you donate to a C4, you still have to pay tax, you donate post-tax money. If you can, and you can't donate that from a DAF. So if your funds are already in a DAF, you're sort of stuck there. But if you can donate to C4s, yeah, I think direct donations to campaigns are good. Some C4s that, for example, coordinate direct donations to campaigns I think are often good. There's an AI policy network which is basically a loose network of individuals who donate to individual campaigns and often wind up having, you know, dinners or going to socials with a lot of these politicians which I think can also be pretty effective. It's one way for them to sort of like hear that real people with actual money and like a large number of small donor individuals with actual money care about these issues. I think that can move the needle. I actually, this is where I donate most of my own money these days. The other is in human enhancement tech. But, you know, if we know each other socially or if we don't and you can somehow find my email or phone number, you are welcome to just be like, 'Hey man, can you like let me know when you donate to a political campaign so that I can follow it.' I'm happy to like make a group chat where I'm like, 'Hey, I happen to be donating to these guys. You're welcome to follow. I think they're good for these reasons.' And I think, you know, 10 people doing that can actually make a pretty sizable difference in politics. You would be surprised especially because of these money limitations on, you know, politicians can tell when you're getting a ton of money not from PACs that sort of has to be because it's from a lot of individuals.
All right, two more things I thought of while Nate was speaking. Number one, many of the billionaires associated with us are Democrats and refuse to donate to Republicans. If you know of good Republicans who you like and are willing to support, both in terms of think tanks and in terms of particular candidates, that could be high leverage, especially considering that Republicans control all branches of government right now. I believe Foundation for American Innovation is good, although I'm not sure I'm getting the name right. You should double check. What was the other thing I was going to say? Oh, yeah. And if you like, like many people here have different opinions on institutional EA and their networks for doing things, but if you like them, they have some networks set up like this. I would talk to Eric Nyman or Sydney von Arcs are good input people who can connect you to those.
Yeah, I endorse all that. And, you know, I have a list of Republicans who I think are best-in-class on this issue if anyone wants it. Yeah. So the question is will the AIs be able to debunk the miracles? I'm going to start off with yes and then pass to you.
So I have asked AI its opinion on this particular miracle many times. I think it did as good a job as the average person here would do after reading let's say two or three books on the topic. I didn't feel like it was superhuman at it in particular. The one thing I'm really convinced of in my explanation is the ultraviolet chromatopsia and the AI didn't pick up on that. So, I don't think it's there yet. There's this question of given that AI knows far more than humans, why haven't they connected more dots? And I think the answer is they're not working on that until you prompt them to. And you got to do good prompting. So, this is all prefable. Who knows how it would have done, but I wasn't that impressed with Claude's ability to solve these things on its own.
Yeah. But of course, this is, you know, as dumb as they'll ever be.
I mean, when I talk of prompting better, I mean like directing it to the right area already. Like I think if I had, I can't even remember if I got the ultraviolet chromatopsia stuff from an AI eventually, but it was after asking it in like 20 different ways. Okay, I'm pretty sure this is some kind of eye damage caused by ultraviolet light that causes you to see colors. What could that be? Whereas that's like very different from being like sun miracle of Fatima. What's happening?
Are there any immediate takes on the export ban on Yeos and Fable? And what if you truly believe that we need to slow everything down?
I mean, I think the jury is definitely still out. Oh, yeah. Sorry, repeating the question. Immediate takes on the export control, Fable, Mythos, effective ban. My take is, you know, we'll see how it plays out. It's, I think it's definitely evidence that people should not write off government intervention on AI. Which, you know, I've seen a lot of people say, 'Oh, it's inevitable. No one will be able to stop this AI stuff. The governments are never going to get in the way. You know, there's too much money. Like the industry is going to be allowed to go as fast as they want and there's no way to stop that.' I'm like, well, I've never thought that was true and it sure isn't looking any more true this morning than it was yesterday morning.
Yeah. I think my thought is if you are not a pure libertarian minimalist who believes in no role of government then you have to believe that it's possible for government regulation to do good things. However, in fact even though the government could do good things with these powers, it rarely does. So like I don't know, as somebody who believes in AI safety I have to believe in some sense that it's good for the government to have the power to do something about AI but that coexists with the fact that our existing government is just kind of led by personal spite and random whims and so probably like it looks like it's shaping out that they mostly did this through some combination of they had no idea before this that AIs were generally jailbreakable and they're learning this for the first time and doing crazy things and also like various people have personal grudges against Anthropic for standing against them. So like we are stuck in this situation where if we give government the power to regulate AI they will do it in ways that are extremely stupid and counterproductive and petty and if we don't maybe we'll all die. I have no great solution to this except that like first of all, we all need to be like working hard to improve government in any way we can. And second, I do still trust that there are many functional parts of government like the courts that will probably sand off the worst offenses of this administration. And I have not currently been battered all the way back to government should not be allowed to regulate anything. But man, it sure makes it hard to be a person who believes in government in any way.
Yeah, I agree with a lot of that, but I also expect the situation to develop a lot. I sort of... Yeah, we'll see where things go. And yeah, I echo the point that the hard part here is making the governments respond well and respond promptly. And, you know, if you're really interested in that, see previous comments on how to support AI safety policy work.
Since you wrote your book, do you have any like big updates that you changed your mind on or do something else or same with the AI 2027? Like you're right, what would you change about?
You know, I think one of my biggest updates is in fact on it being, it looks a lot more possible to me right now to get a global coordinated response on AI than it looked to me when the book was coming out in September. And some of that has been things like, you know, Bernie Sanders sort of noticing the issue and getting real concerned about it and sort of trying to do everything he can on this issue, which is more of a response from any US senator than I expected. Part of it is things like watching the Trump administration downstream events like Claude Mythos sort of starting to, you know, break their taboo on AI regulation in ways that, you know, maybe we're like, yeah, that looks like a pretty bad way to implement that policy, but we're sort of like exiting the regime of like nobody can talk about this. We're exiting the regime of like it's not in the Overton window. And we're doing that maybe a little faster than I expected, which I found quite heartening.
I'll answer your question, but first I want to give a disclaimer. I just did the writing, like literally the literary writing for AI 2027. I was not one of the forecasters. So I'm taking this from what the forecasters have said because I have read most of their public statements about it about six months after it was published. They lengthened their timelines from early 2028 to like mid 2031. Then like a little while ago they took that back a little and now I think it's somewhere around 2029 to 2030. Other than that, there is a section about how China really wakes up and realizes it needs to deal with this more and centralizes all their compute. I think that was supposed to happen around now. There's no sign that that's happening. So maybe China is going to be less relevant than we thought. In which case we will get what appears on your shirt. Sure, sure, check. Okay. And then I think we had the part where the US government like restricts foreign nationals from working in US AI companies in like 2027 or 2028 or something, whereas that already seems to be kind of starting although hopefully it will be a false alarm.
So do you have also like the general population and their role here like buriat likes as homing more people for likely concerned about all the data centers. I just wonder like, you know, project like playing it just...
Yeah, it's hard to say. One somewhat interesting phenomenon is when I talk to sort of, you know, random Uber drivers or people on the street, and mention AI, a lot of them seem worried about, you know, the environmental impacts. A lot of them also seem to me like they're worried about AI killing everybody. And like often I'm the first one to bring it up in conversation. And so, who knows, maybe there's a biased sample, but they're like, 'Oh, what do you do for a living?' And I'm like, 'Well, you know, I try to make AI not kill us all.' And they're like, 'Oh, I'm worried about that. You know, I think the AI is going to come for my jobs and then kill me.' And I'm like, 'Yeah, you know, that seems, you seem well calibrated.' And one reason I find this kind of interesting is that, you know, a lot of it feels to me like there's this weird thing where journalism doesn't really track the maybe this will actually kill everyone stuff nearly as much as the common folk. Like from my perspective, you have all these people being like, 'Hey, this AI is going to like suck up all the water, take my jobs and then kill me. And I don't like it.' And then the journalists report this as like people are concerned that AI will suck up the water and take their jobs. It's like, well, you know, sure. And again, maybe I'm in a bubble here. But it looks to me like people are worried and it looks to me like people are worried for generally okay reasons. It's like I think the average person isn't like, 'Here is my ranking of the EA list of cause areas. And here's how much I agree with Yowski versus Cristiano.' But I think the average person is like, 'Look man, this AI stuff smells fishy. Like I don't know whether it's going to like make me dependent on corporate overlord handouts to survive because they own the entire economy now or whether it's going to kill us all. But it's headed somewhere that doesn't have room for me and I don't like it.' And I think that that's a decent take. How will it affect things? Very hard to say. It could be that the common person sort of like continues to be malcontent and then recursive self-improvement kicks off in 6 months and everybody dies. It could be that there's some watershed moment, there's some, you know, emperor has no clothes moment where suddenly the fact that the populace is like pretty anti-AI sort of like breaks out and that leads to a bunch of civic unrest which leads to an international treaty. I have a lot of trouble predicting this sort of thing.
Yeah. So Nate's getting at this weird contradiction which is that many people if you ask them directly will say yes I do worry about this but also like the world is not acting like it's worried about this. I read a poll the other day which was amazing. It said that I think like 25% of people said they thought they would be killed by AI. But if you ask these people, what issues are you most concerned about? It would be like the economy, the environment, the transes, the gays, and then like 500 pieces down, I might be killed by AI. So I worry that we are having trouble coming up with coherent narrative because even the individuals are not coherent. They're not thinking about this in a coherent way. They're not thinking like AI might kill me and this is a real thing, therefore I should be concerned about it. They're sort of slotting it in to like is this something I should talk about? Not really. What's the right answer to this question? Okay. What's the acceptable answer to that question? Okay. And that maybe at some point all of these different neurons containing these different things will connect and they'll be like, 'Wait a second. I think AI is going to kill me. Maybe I should be concerned about that.' But right now I think this is like deep in the weeds of psychological mystery. How people can hold all of these different beliefs and not have them connect or do different things than they already are. And I don't really think we can do much to like solve this mystery in the sense of here's what people believe because they believe a thing which kind of seems incoherent.
I agree with that. It seems very weird to me. And one quick addendum is one of my guesses a lot of this comes from people thinking there's nothing they can do about AI, that it all feels inevitable. And one of my guesses is that if we get to a point where it feels like there is something that can be done maybe you'll suddenly see a lot of those neurons connect. You know, if it looks like an international treaty comes within striking distance maybe you'll suddenly see a lot of support for it. I'm not sure.
I'm still a little bit stuck noticing this miracle of Fatima background behind you. And so clearly people are very open to, you know, mass perceptions of things that seem completely miraculous and impossible, like people are open to totally impossible things randomly arising and that can sway public opinion rapidly. Any thoughts about staging some kind of miracle that would tip the public opinion towards AI governance?
It's called Claude Mythos.
