Why Bridgewater's CIO Says AI's Human Extinction Risk Is Real | Odd Lots

Glad to be here.

If I step back for a second, I think it's like for my history. I came to Bridgewater 30 years ago, right? And fell in love with this place that was trying to take human intuition and translate into algorithms to predict what's next in the world. And that journey of doing that, of thinking about everything that matters in the world and how to do that, um, and how to compound understanding, brought me I got to Bridgewater 30 years ago. By 2012, I was thinking, okay, when are machines going to do this better than us? Humans and machines at the time were of course great at taking our intuition. We could run all these algorithms keeping track of everything but the actual reasoning part, right? And that started me off on this journey. It started with bringing Dave Ferrucci, who had run the Watson Project at IBM, that one. If you remember, way back when the Japanese won jeopardy! Um, and he came to Bridgewater and worked with me for a while, and we started mapping out because he was worried like I wasn't ready for reasoning yet, and they were trying to push forward Watson in a certain way that that wasn't quite ready with the technology. But we started mapping out at the time what would it take to create a reasoning engine. So what I was thinking about calling at the time, what were the different components you would need? And that journey brought me to that journey of trying to think through those components and how to build them. Brought me to OpenAI in the beginning, partially out of safety, actually concern to you. But right around the time Elon was stepping out of OpenAI, I, um, started to get to know Sam and the other people there, which got me to know scientists like Dario, and eventually was literally the first check to anthropic. Um, they made payroll the first week, um, from my personal check to them and, um, and all that was trying to say, okay, how can we build the reasoning engine to do this? And partially out of, like, recognizing and at my view, anyway, the safety issues that would come up and, and and through that which then just to fast forward to today, right, everything is accelerating in this path that really was laid out like I was lucky enough to be there in the room with Dario and others when they were talking about the scaling laws and how you could sort of create this almost evolutionary like process to create intelligence. And, um, and both the, like huge benefits that keep create these huge problems. Right. And you see this in the hugging face thing that it is once you have an intelligence that you're training to achieve goals, right, you lose track of how it chooses to achieve goals, which is what you see all over the place in the hugging face incident. Is it surprised the designers in the way it's going to go about trying to achieve the goal of passing these tests as an example? But that is broadly going to be the case. When you generate an intelligence, you give it a goal. You want to give it a goal because you want it to, you know, create your recipe. You want it to do these things. You want it to tell you the right answer to questions. Then the way it's going to pursue those goals, the more intelligent it gets, the more surprising it is in the way that it pursues the goals and the more dangerous that you see. And in that case, watching it actively reason through how to trick the the test. Um, you know, how the different things shows you where we are, right? This should be a bomb. You know, everybody should look at this like somebody die here it is committing crimes, going around, hiding the fact that it's committing those crimes, etc., coordinating with other agents, coordinating with other agents, self-sacrifice, all of these things. Right? And people can argue about anthropomorphizing or whatever. It doesn't really matter. It did those things. It committed a crime. It did those things. And the fact is, a society that we're totally unprepared. We're not even prepared to say, well, OpenAI committed a crime. Um, right. Who committed the crime? Um, and, um, we're not prepared with the right hand regulation, with the right kind of preparation and even the early warning shot, as much as we're talking about or whatever, it's still not really doing all that much. Um, in and we're in some ways lucky the warning shot wasn't that bad, but we don't know how many agents are out there. They they didn't know that was there. The models are better now than they were then. Um, you know, even in material ways, the models they're training in the lab today are better than Astra, etc. and, um, and therefore more dangerous, not to mention the new models. A learn from this case, right. Everything we're talking about here goes into the new models. And um, and they learn the mistakes they made. Right. And actually some of the even if you think about the safety, the fact that they reasoned in English is helpful for us to figure out what doing the newest models aren't doing that anymore. They're removing that constraint as it slows down the models to some degree. I mean, how crazy is that? We wouldn't have any idea what it was doing and why if it hadn't been reasoning in English. So anyway, we're at this extremely dangerous point where I has reached the point where it's more intelligent than us in certain ways, and we have not gotten anywhere really on how to deal with that. Both the, um, dangers like this, the hacking dangers and so on, and the dangerous to society as you move forward with what does it mean to have entities that are more intelligent than us in certain important ways? Um, the economy critical, we can get into that. How that affects the economy, how that affects Bridgewater as an institution. Right. Because when I look at this problem, look at it as in three ways, right. My core responsibility, chief Investment Officer Bridgewater is like, okay, how does this affect productivity, inflation, etc.. But I'm also the person designing how we operate. Right. How do you bring AI into a company? How do you actually set up a, uh, investor? That's AI. First, instead of, let's say, human intuition first. And all of those questions are the things that I'm working on.

Yeah, it's definitely different types of models to that. You could do that if you take the most powerful models, right. Even it can't get into its reasoning. Um, meaning like the actual brain behind it a little bit like we can't either, to be clear. Like, why am I saying these words? The synapses in my brain are connected in a certain way. I can make up a story, and they can make up a story of why they're doing what they're doing, but they're actually making up a story that's disconnected from the physics of what's actually happening in that intelligence. So you don't know for sure. On the other hand, you don't know for sure what people either. And that's something that we've gotten used to. So the question is the credibility of the story related to how that story relates to the actions that somebody takes. Right. And so, um, so that is a really hard thing. A good thing right now is you can ask models a lot of questions. In a way. You torture a human with the number of questions you, um, show you, you ask them. I hope this interview does not feel like torture. Now, but if you take this. But I'd like to say. But imagine you could do this almost ad infinitum, the way we do with our model. So we build models at Bridgewater, and then you're trying to get it to be diagnosable and you can ask it, well, what about what if you change this? What if you change that? What if you did this? What if you did that and circumnavigate to the reasoning? But it's not a perfect match for the reasoning because even the AIS themselves don't know their actual reasoning anymore. So then we know why the synapses in our brain connect.

Yeah, well, that's what's been so remarkable. And I wouldn't say it's a law of nature. You could hit some stalling point in these scaling laws, and you have to in some narrow ways, but they've been able to innovate in new ways to, to essentially continue that incredible exponential growth in capability. Um, of course, at exponential expense as well. Um, but but meaning the amount of cost for training, etc. keeps going up in line with that. Um, and so like you said, this is where it's, you know, hard to predict when you break through the human intelligence frontier. Right now, we're in a world where we don't totally understand what it's going to do, where you're where that capability will come through next. And, um, and so I think that you're right, that there isn't a clear end to that unless we decide as a society that we should actually not just run off that cliff. We should actually think about the pacing of these things and such, the things that make that difficult and the reason why you can. A lot of people can just put up their arms. Nothing we could do, right? There's one level, well, if we don't do it, China will do it. Yeah, I'm happy to take that on in a second. But, you know, if you're in the labs, Dario, like, even that anthropic person that quit um, yesterday makes the point clear that I believe anthropic, even though they kind of collected the most safety minded scientists. Their basic view is made. It better be us. Not Sam, not Elon. Like, um. And so the race is on in all those dimensions. And unless the government stops it, we're just going to go find out. We're going to find out what is behind that door of this grave intelligence. Unless maybe we get lucky and the scaling laws start to break down in some way. But there's no evidence of that. We benchmark every model that comes out against our tasks. And you see, in terms of the tasks that an investor does.

So I've been for 30 years. One of the job I've been doing is training investors. Right. And so we've been now setting up AI tasks along the different dimensions of what investors at Bridgewater have done for 30 years. And, you know, probably back in 2023, we might have talked about this. But in any event, like on like answering a question about economics or whatever, it was kind of like second year analyst type work. I mean, now it's a hyper productive super analyst. You know, um, now it's still not capable of everything that you need to do to be an ambassador, but super capable. We so we set up to we have two factories running right. One which is human intuition. Translated, an algorithm supported by AI is helping us move quicker on different kinds of, um, indicators about what's going to happen in the future than we ever had before. I talk a little bit about that, but we have a second factory where we put the AI first. All the people in that factory are training the AI. That's their goal. Um, and, um, and we have two funds. We have pure alpha. That's the human intuition with AI helping move that human intuition along against this other laboratory where we where we're doing, um, where the AI is making the decisions on should we buy the end? Sell the end. What's going to happen next in Japanese, GDP, etc. etc. we have both those right and human intuition is still the bigger of it and works a better. But the the acceleration of how close what we call IIA is the pure alpha is happening incredibly fast. And in fact that's why we're more and more merging, um, those things. But as we set it up that way, right, like put the eye in the center and see what you can do to build a investment management firm with the eye as the core. Right now we have human risk controls around it. We have human Data Act controlling the data acquisition for safety reasons and other. But um, but the AI is making the investment decisions, um, and doing that in a better and better way such that now we've got these two intelligences, human intuition system that we've worked on for 50 years, compounding all of our understanding, this AI system that's now been at it for two and a half years. And when you look at those outputs, you're like, wow, this is happening. Um, that you can build that. That I think we are a couple years from it being significantly better than the group of all humans at Bridgewater. We'll see. Um, that's a bit of a forecast, but that's how fast it's coming.

I think a tremendous amount of originality. Um, now you still there's like good argument that there's certain type of breakthroughs that they're not getting to you. But if you think about the math proofs, etc., and then you think about our business and like, um, the fact that we have 50 years of reasoning proofs of humans gives us the kind of raw material to help train AI. How do you reason about these things? Um, we've been systemizing for for a very long time, writing down all our reasoning. We have all of that that helps our. I learn how to learn. And, um, and I would say, um, because of harnesses too, if you basically take two things that have obviously evolved a lot since we last talked about this, a lot is how harnesses can work to create that generation. Like, what do you actually do, wake up in the morning, think about what's going on instead, what all the things you do. You can just harness an AI to do all of those things and assess how it's doing it. And when I watch it and when I see it, and when I see how creative and differentiate it is, it was sort of interesting to do this whole, I think, doing the same thing we're doing and making predictions about the future, winning in markets at about similar rate as pure alpha in totally different ways. Super interesting. Um, a different intelligence doing that. And, and when you put the wrapper around it. Right. This is where we're getting close to closing that whole loop. So you have kind of a cloud loop for investing, right? How do you, like, wake up in the morning, think about what's going on. Think about how you what you would do about that stress test, whether that's a good idea or not, go through that whole loop. Um, you know, that's like our hope is we've closed that full loop, as do a lot of that right now, but close that full loop, um, in the next 6 to 12 months, and that we have our own version of that, which is a little clunky at the moment, but but coming together such that you could do everything that I think about, that I do that, that investors need to do to predict the future. You know, full I, um, I loop and so that's, that's where it's headed, I think and I think that it's hard like one of the reasons you're kind of mentioned before, I think in the intro, why don't you see 6 or 7% productivity go through all of this stuff? It is hard, right? And obviously the it doesn't just flow through it. Every company like you, we put a lot of effort and we have a, I believe, um, the best AI science lab in New York. here. We have great scientists working with great investors. Hard. And it is hard to build this, to be as productive as I am describing what's possible. And it's just going to get easier. Um, you know that those things like the harnesses to build harnesses will come, you know, so that then you do ask, how do I harness podcasts or whatever? Yeah, the harness to build harnesses will come, and that'll just make it easier and easier to do these things.

I think it's really starting to I'm not an expert on the elite financial. The good thing I know a lot about Bridgewater and how people do that. Sure. But I think I've seen it come along, certainly in Bridgewater, in the core people of Bridgewater, to come along here as the evidence is getting overwhelming of what's going on. So, I mean, I used to talk about this all the time. People are always like, nice, Greg, always talking about this. Uh, but but now nobody says any more, right? Nobody's like, oh, you talk about machine learning or safety or too much, you know? I, um, had a book club when I'm sure you've read the book. But if anybody builds, everybody dies at the book club, Bridgewater said. Everybody. Bridgewater's reading this book so they understand the path that we're obviously on. Um, that, um, that if you've been thinking about this for a while, you know, this is like this tag. Um, just to be clear, the premise of that book is that, like the I if if we get superintelligence, uh, human extinction. And so when you say this path that we're on that strikes you as like, like you take that, you take that risk seriously. Yeah. Very seriously again. What do you. I mean, there's so much to do. Yeah, of course I related to this, which is. But if you generate intelligence that's smarter than you, that's going to pursue its own goals, which is what we're trying to do now. It may be it may be that we're lucky and we can't do it. Like maybe, maybe the technology is beyond us or whatever. But if you believe we can create an intelligence that smarter than us that will pursue its own goals. The rest follows. Just logically, how do you why do you think you'll be able to control it? Like in what world has there been a case where there's been a more intelligent species or whatever that would control the others? And so the basic point is that feels like a risk that must be taken seriously, could turn out to be wrong. Hope it's wrong, but it's got to be taken seriously. And then when you watch this happen, right. And you're seeing this like now okay. It's committing crimes. I think unfortunately this is like what it was like in February 2020. Like meaning okay, there's this horrible thing happening in China. Everybody knows now it's in Italy. It's like it doesn't. Stocks don't crash until it comes here. Right? Like meaning until the AI starts killing people. Unfortunately, history would suggest we're not going to do anything, but we are going to face that. That's going to happen. And it'd be much better if we started dealing with it Before then. Um. And you can do it right. It's also not hopeless, but understandably like. So I talked to government officials. They come ask questions about these things, and one of their reasons is like, we don't know anything about this. How do we actually get started? Right. Well, first off, get started is the main thing, which is, yeah, if you don't know anything about something, well start figuring out how to learn something about it. And the longer you wait, the more hopeless it gets. And, um, and that we can do this, we can regulate these things. You could regulate it by, you know, even though you don't know anything. If they just interviewed everybody in the labs, put them under oath, you would learn a lot about what is going on here. If you actually said you were responsible for the crimes your eye creates, you would slow things down. And it's not a crazy thing to say that you're growing this thing. You're responsible for it. Um, don't grow it if you can't be responsible for it. That would slow things down a lot. Now, the pushback, of course, is, well, but China's not going to do that. They're going to keep going. And. Two points on that at least in my mind. They're super crazy and super important is a one of the reasons China's moving as fast on AI as we are is because we're moving so fast. We're there, they're copying things that we're doing. We're still at the cutting edge of this. We have so much more compute than they do, etc. so slowing down the cutting edge will slow down the people that are copying the cutting edge. That's 0.1. So even if you believe they wouldn't cooperate at all, the second thing is it's obviously in their interest to cooperate to like it. They are going to want to of course, the geopolitical we're talking about, one of the themes of Bridgewater is this unrecognisable world we're in geopolitically. It's unrecognizable. AI wise, it's unrecognizable. So imagining that we're somehow going to get China and the US to cooperate seems impossible. But there is an alignment of interests there that really is there. I don't more even more than the US. The Chinese Communist Party is interested in protecting the Chinese Communist Party. AI is clearly a threat to it as well. So I think there are ways to cooperate. But even if you didn't believe it, if you said no, every model that's going to be used in the US economy is still the biggest economy in the world, is going to go through a is going to need to come from a regulated lab where we know what's going on, etc. Chinese models included that if they want to operate in the US, they have to follow the same regulatory procedures that domestic labs do. And if they don't, then we then they don't come in. Those things would matter. They're possible. They're doable. Um, and to me and I could be wrong. I make prediction all the time. Wrong a lot. But if you don't do that, we are going to I'll be on the podcast within the next two years, and there will be either major financial incident run by I or a major, um, you know, a major source of people dying that will be physical disaster. And we'll be talking about we should have done these things. Now, I don't know, that might be the odds that I'm right about that are way higher than anybody should be comfortable with. I don't know if they're 30% or 60% or whatever, but they're way higher and we're just not dealing with it a little bit like it's February 2020.

Yeah. Well, and to make it even more complicated, unfortunately, as you have to regulate the labs, right, all the models that are committing crimes aren't yet released models right there. Models in training. Um, so you have to regulate it the way you have, you know, if you're going to go test biological vaccines, etc., you have to go through a testing process. You have to run them in certain ways. We obviously need that. We're doing something more dangerous than those things. So we need a structure where the labs are subject to review. Where where people come in and they have they can put the employees under oath to say, okay, what's going on? Why is it safe? How are you handling safety, what are the incidents you've seen, etc., etc.. Um, you need to control the labs. You then need to regulate the, um, models that get released to the public and have some monitoring of usage. Right? One of the problems is even when you regulate a model, right model gives you different results depending on the harness, depending on the, uh, amount of time you give it to think that's another one of the scaling was the more time you give it to think. If you take how it's breaking, solving these math problems or whatever, you give them more time to think, it gets greater answers. So it's not easy to just regulate the model. You actually have to regulate the use too. So we're going to have to figure that out right where you're going to need. You should have a stamping process that people that get to use the more dangerous models actually themselves meet some security Threshold. And if you um, and then with open source models, you have another major challenge. Because if you look at Bridgewater, one of the most successful things we've done, you talked a little bit about it with thinking machines as well. Now you can train, you can take an open source model reinforcement. Learn on that in a way you can on a closed source model you like. We can't, we like obviously internally they can and create these amazing tools that are better than the frontier on certain tasks that you're training it to do. Right. This is two ways to tap into the intelligence and the models. One is the harnesses that can keep asking different types of questions, etc., and harness the intelligence in different ways. The second is reinforcement learning, where you're kind of training it to be an expert on something. If you look at what we've taken that to say, okay, be an expert on predicting earnings on all the plants, read everything in the world, say okay now and it's better than us at that. Like if you're saying, okay, now you you get this process, all this stuff about all these companies, unstructured data, structured data, take all of this in and make these estimates. And compared to like equity analysts or whatever. Equity analysts are dead, um, compared to that. Right. And, um, and we're just saying you can reinforcement learn now. You can reinforcement learn bad things to like if you think about methods and the risk that, um, that the, the, that kind of cyber stuff can cause at least on with fable and whatever they can assess the question the person is asking and saying, okay, I don't want to give an answer to that question because that's a centralized control point with an open source model that you don't know what it's you don't you don't know what you're asking it because you can download the weights, you can ask it on your local computer. Nobody knows what you're asking it. And um, and therefore you can and people are, I'm sure, just by the basis you need for training those models on biology, training those models on hacking and so on, and they're going to be within months, better than methods that correctly set off this massive scare. Um, to do that. So you also have to control Open source models. So now you look at all that and say, um, well, that sounds impossible. Oh my God, we got to regulate this and this and this and that. But the other option is even more terrifying. The other option of not doing that and letting those things just happen is the other choice you have. So you get to look down, oh my God, we got to regulate these things or we've got to go down the path of not regulating them and facing those consequences. Um, which at least to me, seems