Building OpenAI’s Stargate Data Center w Crusoe's Chase Lochmiller | Inevitable Podcast

Thank you.

That's right. You know, data centers were scarcely a thing, honestly, when I was growing up. But, you know, I grew up in Denver, Colorado. Coley, my co-founder, and I actually went to high school together in Denver, a school called Kent Denver. And, you know, I think growing up, I was always very interested in, you know, math and science. I was very drawn to math and math competitions. I was a very competitive kid. I played a lot of sports. Loved soccer. You know, we got the World Cup coming here next year. I have very fond memories of the '94 World Cup and just following that very closely and that really helped inspire me to just fall in love with the game of soccer. You know, I played so I ended up playing soccer through college, you know, where I played at MIT. And actually there's a number of, you know, there's people from like every chapter of my life that have been part of Crusoe and including, you know, there's a number of kids that were on the MIT soccer team that actually work at Crusoe, including our CTO, Nathan.

Yeah, Nathan was our sweeper. You know, he's a great...

I was a center midfielder.

Kind of the CEO of the soccer team. You know, but I think, you know, I originally went to MIT, I had this, you know, I was convinced I was going to become a theoretical physicist and, you know, help unlock the mysteries of the universe and sort of expand the human knowledge set. And, you know, I spent a lot of time doing various research things and did some research at MIT, did some research at Los Alamos National Lab. And I think one of my realizations during that era was that basic science research just moves pretty slowly. And I just had a ton of energy. You know, maybe today that's changing with AI, but certainly at that point I just felt like a very slow drawn out path and I felt like I wanted to do something faster paced.

I think that was the path. Yeah. And, you know, academia or like, you know, lab or something. Exactly. I mean, you know, ITER had just been announced, so, you know, this big international collaboration to build this giant tokamak to, you know, hopefully have a big breakthrough in fusion. You know, I remember kind of my first day working at Los Alamos National Lab and my principal investigator I was working under, you know, he told me, you know, Chase, this is really, it's a really exciting time for fusion and plasma physics and, you know, hopefully in 30 years we're actually going to have fusion. And I was like, wow, that's pretty cool. That's cool. And he's like, yeah, you know, they told me that when I started 30 years ago, but, you know, here we are and, you know, we're still 30 years away. And so anyway, that was always the joke in fusion was that it was 30 years away.

Exactly. But yeah, so, you know, when I realized I wanted to do something faster paced, I felt like in certain ways I had lost a purpose. Like I was like, oh man, what am I going to do? Like I had my whole life set on being a theoretical physicist.

I mean my dad was kind of an entrepreneur in certain ways.

Yeah.

He's a total character. He's a one of a kind. You know, he's certainly had a big influence in my life, but, you know, entrepreneur in a very different sense. And, you know, what's funny actually, I think there's this aspect where sometimes if you have a parent that has a lot of success in one vertical, you tend to be like, look, I want to make my own path, my own sense of success and something completely different. And, you know, I felt like I was doing that for many years and, you know, building things in technology and, you know, I worked in quantitative finance for a long time. And just doing things that my dad never would have done. Like it was just never his career path. He's been building in real estate. He's a multifamily and commercial real estate landlord, owner, operator. But the funny part is like this whole big boom in AI and all these big AI data centers that are being built. You know, it feels like I'm coming full circle where it's like, okay, all of these things that I've learned in technology that have inspired how we're building out all of this AI infrastructure to, you know, power intelligence in the future economy and yet I own a building and I'm a landlord.

Just like my dad.

Sure. Yeah, you know, in middle school, I had this teacher. And her husband was this blind guy named Erik Weihenmayer. And he was attempting to climb the seven summits, so the highest peak on each continent, and he was blind.

And at the time, you know, I was in seventh grade, and he was training to climb Mount Everest. And, you know, the David Breashears' Everest documentary had come out, Into Thin Air. The Jon Krakauer book had come out just about the account of the disaster that had happened. And, you know, something like really, I was like, man, that's so cool. It's the first time I'd heard of this concept of the seven summits. And we have a lot of 14,000-foot peaks in Colorado. It's like a very Colorado thing to talk about climbing 14ers. But, you know, I think I was just kind of inspired to climb big mountains and spend a lot of time outside and, you know, I grew up going skiing a lot and spending a lot of time outdoors. And so after I graduated college and I started, you know, sort of stopped playing sports very competitively, I was like, all right, what else can I do? And, you know, I became sort of an alpine style, alpine mountaineering kind of hobbyist for lack of a better word. And so I did a bunch of different expeditions, been sort of all over the world. You know, at this point, you know, I've climbed five of the seven summits, including Mount Everest. I had a couple different expeditions to Mount Everest. You know, once in 2014, there was a very large avalanche that happened that year in the Khumbu Icefall. And tragically 16 Sherpas were killed in that accident. But for, you know, some reason I felt compelled to go back and ended up summiting in 2018. So, had a very successful summit that I learned a lot from sort of the first time. And, you know, I think maybe eventually I'll get to the final two summits, but it's just kind of a...

Been deprioritized.

You know, Everest was like this challenge because it was this crazy expedition and there was so much preparation that went into it and there's so much emotion as a result of just like shifts and things. And, you know, it's just kind of like the world is so focused on what's happening on Everest. There's, you know, mass media coverage for like every little thing that's going on in Mount Everest every climbing season. So, you know, I think kind of coming out of that, we actually, you know, when we started Crusoe, you know, one of our core company values that we created is to think like a mountaineer, right? So it's like, you know, Cale spent a lot of time mountaineering as well. We were climbing a mountain when we sort of were ideating on the origins of Crusoe. And, you know, the sense there is really about trying to have a lot of the same principles that you have in mountaineering applied to your life and your work at Crusoe. And it's not like everybody has to go climb mountains and, you know, go want to be a big alpine enthusiast. It's more about just the sense of, you know, an incredible amount of preparation, like thinking through, okay, here's my plan in terms of how I'm going to get to the summit and then I'm going to get down safely.

And here are the different things that could go wrong.

And if those things go wrong, here's my plan for them and here's my plan B. Here's my plan C. Here's what I'm going to do if the weather changes. Here's what I'm going to do if I have this piece of gear fail. Here's what I'm going to do if my partner gets sick. Here's what I'm going to do if, you know, like...

Exactly. Exactly. And, you know, there's this notion in mountaineering that, like, you know, getting up is optional. Getting down is mandatory. And, you know, we actually try to instill a lot of the core safety practices in mountaineering. You know, just really thinking through safety as a critical aspect. And, you know, in our business where we're dealing with both hardware and software, you know, there are a lot of, you know, there's millions of man-hours that have been worked in Abilene, you know, we have to be thoughtful about sort of safety culture so that we can build these things at scale, at speed, but also...

Well, that's, I mean, there's an element of, mountaineering is actually an interesting thing to study because you have kind of like these legacy styles of mountaineering that are like these big expeditions where you bring like everything, you almost like build a whole city and it's like, you know, you sort of are very intensely prepared. And over time what people found to be like the most successful ways of climbing big peaks and sort of breaking records is sort of this alpine style of light and fast, right? Where you're basically moving quickly, you have everything you need and nothing more. And, you know, you have this robustness in your planning and, you know, you're thinking about things safely but oftentimes speed is actually a mechanism to be safe in mountains especially when you're sort of...

So maybe let's fast forward to today and where Crusoe is. Crusoe's had, you know, from where I sit, one of the most unbelievable sort of evolution stories. And for folks who really want to dive into that, we recorded an episode with Chase's co-founder Cale a couple months ago in Austin and we really dive into the path that the company has taken to get to where you are today. But with you, I really want to focus on where are you today and where are we going.

Yep.

Sure. So Crusoe is a vertically integrated AI infrastructure business. Our goal is to help make energy and intelligence more abundant, right? Just really accelerate this abundance of both energy and intelligence which we think are going to be critical to uplifting humanity in the future and really driving progress for everybody around the world. Now how we got here is like pretty interesting. You know, I think we just have a culture of curiosity, right? It's like, you know, we just sort of are constantly asking questions about how we would do X.

We actually just celebrated surpassing a thousand.

In Abilene, we have 7,000 contractors that are working there every day.

But, you know, a thousand full-time employees across, you know, a couple of offices.

Yeah. So, look, I think what's interesting, you know, we've taken this very vertically integrated approach to AI infrastructure. And so what that means is we're doing the hardware aspects of building the infrastructure needed to power AI, needed to run these large scale intelligent workloads. So it's everything from land and power development, data center design, construction, cooling, mechanical, electrical, like the whole stack of like the physical infrastructure to stand up AI workloads. And then we're also in the business of deploying these large clusters of GPUs and, you know, this is our Crusoe Cloud business where we build, operate, and manage these large scale clusters of GPUs and integrate them with critical services like storage and high performance networking. And then we also have these managed services to extract intelligence from that infrastructure. So things like a managed inference product, a managed auto-clusters product, things that basically help people create intelligent results from these massive investments that they're making in infrastructure. And, you know, let me just, we cover a lot of stuff, right, from like hardware to software. I like to talk about our business as creating these AI factories, these factories that produce intelligence and doing the hardware of building the AI factories and then actually the software of actually how you utilize the AI factory to produce intelligence. And the reason we believe that it's the right moment to build a vertically integrated business is what we're seeing unfold is actually this massive new category of infrastructure. And whenever you have this paradigm shift, being able to move quickly, being vertically integrated enables you to move quickly into a new vertical. And, you know, this infrastructure for AI, you know, what I call the infrastructure of intelligence, it's dramatically different than anything that we've ever seen before, right? It's different than traditional data center infrastructure that serves the internet and sort of traditional cloud computing. It's different than traditional power systems that have served us scaling to date. And a lot of that stems from the actual workload, the design of the systems ranging from the way the clusters are built out, the way they're utilized, and the way users interface with the services. It just merits a completely different design and it's at a massive scale so it sort of warrants this investment in vertical integration.

Well, I will say that we are seeing a convergence and I'll get to that in a second. But, so training is basically where you're setting up the model that you're going to be utilizing. It's basically like you're making the investment and you're amassing all of this data, all this training data, this historical data, into your data center and then you're sort of setting up this large scale model, you know, it's going to be like a tens of billions, hundreds of billions, trillion parameter large language model or other foundational model depending on your application. And then the training process is basically trying to fit a large scale nonlinear statistical model to that data so that it would be able to with new inputs be able to produce results that are intelligent so to speak and sort of learn from that data.

Kind of, it's, this is where I was saying it's changing, it's converging. So, there's this notion of pre-training which is basically trying to create structure from all of this unstructured data and that's like what's historically been producing these original GPT models, so GPT-1, GPT-2, GPT-3. Where we're seeing massive compute scaling is actually in the post-training and sort of test time compute scaling where you're actually having, you know, you have this foundational model that has been pre-trained and then when you're serving a workload, you know, OpenAI sort of released this o1 model about a year ago and, you know, there's since been a lot of innovation on these different chain of thought reasoning models that actually produce results and then take those results and feed them back into the model and what it ends up doing is it's a bit like training on new data.