Chase Lochmiller, Crusoe, Co-Founder & CEO on AI Infrastructure

Hey Grace, happy to be here.

So early lessons that I've learned, I think, you know, it is interesting that there are a handful of folks that work at Crusoe, you know, that, you know, either I played sports with in college or you know were athletes in college. Well, you know, I think there's actually an interesting aspect of, I think there's an interesting aspect from being an athlete that, you know, comes into, you know, building a business or pursuing an opportunity. And you know, this I'm not saying this is like the only thing that's required, but you know, there's an interesting thing with team sports where when you're working as part of a team and you have this confidence and expectation of winning.

And yet oftentimes you lose, right? And so when you're playing sports like you know you lose a good amount, right? You know if you're playing in a competitive league like you're often times not on the winning side. And you know, being able to pick yourself up from those losses and then learning about how to, you know, how do I get better? How do I not feel this, you know, this feeling of loss next time around? I think it's like an important, I don't know, it's been important to me over the course of time in terms of, you know, building Crusoe and building this business is that, you know, I haven't gotten everything right the first time around. And you know, I've made tons of mistakes along the way. You know, we've had our fair share of failures throughout the entire building of the business. And you know, each step of the way every time we've failed, I've always felt like I've learned something very important that has helped me not fail the next time around. And you know, to me, you know, the lesson, I think a lot of the lessons learned have come from, you know, my experience in team sports in terms of, you know, working with a team effectively towards a shared collective goal and learning from your losses so that, you know, you can win the next time around.

Yeah, I guess you know I've always been attracted to, you know, things where I can solve interesting problems. So I think that's like my key motivator is like, you know, is there an interesting problem to solve that can, you know, create value and, you know, create benefit for the world. You know, I think in the case of quant trading, you know, I think my original like how I landed there was sort of it was a little bit random honestly. I initially thought I was going to become like a theoretical physicist or, you know, working as like a more of an academic path. And I was doing, you know, I was doing research both at MIT in physics and then in Los Alamos National Lab, you know, doing some work in plasma physics. And you know, what I came to realize was that like I had a ton of energy. I wanted to move very quickly on things and like the basic science research space just moved incredibly slowly and I just didn't feel like I was going to, you know, be a good fit long term for doing that. And so, you know, I sort of felt like I had lost my purpose, right? I felt like my purpose was to discover the secrets of the universe as a physicist. And once I sort of lost that sense of purpose, I was like, man, what should I do with my life? Well, I guess like absent not having a purpose, like making money like feels like a good, you know, filling the void of purpose. And so you know there were a bunch of head hunters that sort of, you know, came to campus and, you know, the quant trading space was something that was a very lucrative financial opportunity to like go solve hard math problems and that was like kind of what attracted me to it is like hey I can get paid a bunch of money to, you know, solve interesting, you know, math and technical problems, you know, that sounds great. You know, the meritocracy aspects of it were very compelling to me. And you know, there were a lot of things that I learned from the trading space. Again, you know, it comes back to this like winning and losing thing. You know, in the quant space like, you know, particularly in market making, your win ratio is very, very, very low, right? It's like it's just above 50%. And that's enough to make, you know, millions, tens of millions, hundreds of millions, billions of dollars, by just having like a slightly better than 50%, you know, odds of winning something. And, you know, again, I think one of the things that I also learned from, you know, being a quant trader was viewing the world very probabilistically. I think that's something that I've taken with me throughout like various points of my career. You know, just the future isn't static. And, you know, it's not like a foregone conclusion like what the future is going to be. And so you sort of have to be prepared for multiple different, you know, futures to exist. And, you know, we try to channel that with a core value here at Crusoe called thinking like a mountaineer. And, you know, mountaineering is kind of one of my hobbies and it was a big passion for me for a big chapter of my, you know, 20s and 30s. And, you know, one of the things about, one of the aspects of mountaineering is that, you know, you can have a plan in place to be successful. And then your plan can be completely changed by virtue of, you know, the weather changing or a piece of gear failing or just, you know, conditions changing materially and you have to be prepared for those conditions to change. And so, you know, I think, you know, thinking probabilistically is not too different from this in the sense that, you know, you want to be ready for multiple different, you know, futures. And, you know, there's a saying in mountaineering that, you know, getting up is optional, getting down is mandatory. So you definitely want to like, no matter what the conditions are, you want to, you know, be in a position where you're going to be able to, you know, walk away from the mountain safely. I think, you know, that's a very similar philosophy in quant trading in terms of like, you know, you don't want to, you know, if you get completely wiped out like there's no coming back from that. Like if you have a, you know, drawdown, you know, you live to fight another day and, you know, maybe your model works better under some, you know, different market conditions. You know, but, you know, getting wiped out is like not really an option. So, just I think building resilience and robustness in, you know, having a very probabilistic view and Bayesian view of kind of what the future holds is like a big takeaway. And then another thing that, you know, I learned a lot from being a quant trader was, you know, the importance of technology in, you know, driving competitive advantages. You know, there was this whole piece of the networking stack that was, you know, built out by the quant space and, you know, I think, and just the importance of having data and sort of the speed at which you had that data became a critical aspect to, you know, being able to make money in the quant space. And so, you know, there was a period where, you know, we were trying to find the fastest fiber path between Chicago where the futures exchange was and New York where the stock exchange was and there's, you know, obviously this tight correlation between, you know, the way the futures trade versus where the underlying stocks trade and, you know, if you had that information shared between those two matching engines faster, you know, you actually have a very significant competitive advantage compared to the market. And so, you know, we initially worked out this, you know, very fast fiber path. And then people tried to figure out ways to actually move faster than fiber. And they figured out, you know, data transferred through fiber actually goes slower than, you know, microwaves going through the air. And so people figured out ways to actually move data between Chicago and New York via these large microwave networks. And that created a huge competitive advantage. And, you know, I think the reason I'm sharing this story is not to say that, you know, microwaves are sort of the key solution here but it's more that, you know, being at the cutting edge of technology and technological innovation is critical to, I think, moving the world forward and creating business success. And I think you're seeing a very similar thing unfold in AI today in terms of, you know, the next generation of data centers looks very different than historical legacy generations of data centers. The next generation of chips is unlocking incredibly new capabilities, new transformative technologies that are really advancing businesses at an unprecedented rate. And I think just, you know, the importance of staying at that cutting edge is definitely a key takeaway that I had from, you know, my experience as a quant.

Abilene. Yeah. Yeah.

No no no I think you're on the right track there. The way we originally started the business was and the reason we named it Crusoe actually was we named it Crusoe after Robinson Crusoe who was stranded on a desert island and had to be innovative with his resources to survive. We wanted to channel that same sense of innovation towards unlocking value in stranded energy with computing. And that led us to markets, you know, by taking this very energy first approach to developing computing infrastructure it led us to markets that historically hadn't been markets for compute but we felt at scale energy becomes the bottleneck. So one of the first sources of energy that we were utilizing was natural gas that was being burned off from oil production and it was basically being produced as a byproduct of oil production. And so it was the byproduct not the product for certain types of wells and absent having a pipeline the most economic thing to do with the gas was to light it on fire. It was a complete waste and an incredibly horrible, horrific, you know, environmental cost of producing oil. And so we said, you know what? If we what if instead of them burning that gas and just lighting it on fire and emitting it into the atmosphere, what if instead of doing that, we just brought demand for that gas directly to the site and instead of, you know, venting it or burning it and flaring it off. What if we actually used it to generate power to power a mobile and modular data center on site? And so, you know, that was like one of our first products was, you know, what we called digital flare mitigation. And the primary use case that, you know, we started with was actually Bitcoin mining where you had this very effective way of turning low-cost energy into money via computing infrastructure. But, you know, our goal all along in starting that business was really, you know, how do we actually, how do we, you know, AI was sort of something that we had been focused on from, you know, the very early days of the business. Because of my background as a quant and, you know, having experience building out some of the early deep learning models, I really felt like AI was going to be a transformative technology for the world and, you know, it was a critical thesis for us from early days of building the business. You know, this energy first approach led us to other unique energy resources. So, you know, places like Abilene, Texas where there's abundant wind energy and before the AI boom, you know, Abilene, Texas was not a data center market. But it was a market where people had built a tremendous amount of renewable energy because of the government incentives via the production tax credits and it was actually a market where you had too much energy where people had actually built out all of this energy production capacity and there was no demand for it. And we said, you know what, what if we again, you know, channeled this, you know, sense of energy first approach to computing and actually brought demand for, you know, that energy to this stranded location and, you know, that's really what catalyzed us to build this gigawatt scale AI factory there that we're doing to support OpenAI and Oracle and, you know, was the fact that we could access low-cost, clean, and abundant energy resources.

No, here's what happened. So, there's something called production tax credits. And the way it works is renewable energy developers get incentivized by these production tax credits. And, you know, the incentive was basically something like $30 a megawatt hour, like 3 cents a kilowatt hour, to basically produce clean energy and sell it. So those are the critical dynamics here like you're producing the energy and then actually selling it to someone. It doesn't say anything about like what price you have to sell it at or anything like that. And so what resulted is with intermittent renewable energy resources like wind and solar, you're making these large fixed capital investments in the wind turbines or the solar panels. And if you're getting paid this production tax credit that's based on how many kilowatt hours you sell, you want to basically build your renewable energy plant in the place where you're getting the highest capacity factor, meaning you're producing the most number of kilowatt hours per, you know, unit of capex that you're deploying, independent of the demand, right? Because you essentially have this incentive. And so what that resulted in was people like scouring the country saying like where is the most consistently windy area of the country and there's this whole corridor that flows through, you know, sort of West Texas that's incredibly consistently windy but there's not a lot of industrial activity there. There's not a lot of, you know, there's not a lot of demand for power there. But, you know, people had built tons and tons of wind farms in this corridor throughout West Texas and Oklahoma, where, you know, you can produce energy from wind with a very, very high capacity factor and collect those production tax credits. What ended up resulting though before AI was that there was all this, you know, renewable energy production there and no demand for power, which meant that people would be producing energy and then selling it into the market and willing to sell it at actually a negative price when there's no marginal buyer because they're getting this kickback from this production tax credit. So, they're willing to sell it, you know, up to like negative $29 a megawatt hour. You know, they're willing to sell power because, you know, still on a marginal basis, you know, absent having a storage solution, you know, it's still marginally beneficial to them to basically, you know, sell the power at a negative price. That's sort of the genesis of, you know, what caused people to build all this stuff out and created a real opportunity for us to bring demand to these markets of oversupplied renewable energy resources.

Northern Virginia. Yeah. Yeah.

Yeah, I mean, I think an interesting trend to follow is how like crypto in a lot of ways kind of and Bitcoin mining in particular paved the way for a lot of, you know, AI infrastructure. And, you know, sort of the evolution that unfolded was that you had Bitcoin miners relentlessly seeking out low-cost energy resources to power computing infrastructure. And, you know, if you look back at the history of Bitcoin mining, it's sort of like, you know, when the Bitcoin network first launched, you know, you sort of had hobbyists that were mining Bitcoin on their laptops. And then over time like folks professionalized a little bit and said hey this is actually quite a bit more cost-effective if I actually do it with something that, you know, has a higher flop per watt ratio. I can do this with a GPU, you know, far more efficiently, you know, running these hashing functions. So people started it with GPUs and then people said, you know what, we could actually just build, you know, specialized chips to do this, these ASICs, application specific integrated circuits that were just designed to do this one thing super well, this SHA-256 hashing function. And what you saw was this evolution where, you know, as costs were eliminated from the equation, the cost of securing the network and providing hash rate to the network converged on the cost of energy. And people also rethought the entire infrastructure stack because, you know, originally like people were, you know, mining Bitcoin in like tier 4 data centers that had five nines reliability and then they said do we actually need five nines reliability? What if we only had four nines? What about three nines? What if we only had one nine of reliability? Like is that still okay? Like if I was able to, you know, eliminate, you know, a significant portion of the cost and, you know, the answer really became yes for that very specific compute application like the interruptibility tolerance was quite high for the Bitcoin network. And so, you know, what you saw was infrastructure that might have cost $10 million a megawatt to support, you know, hardware, you know, you were able to eliminate call it 98% of the cost and, you know, people were able to produce capacity for $200,000 a megawatt or, you know, $100,000 a megawatt, like, you know, something very, very, very cost effective but, you know, not nearly as reliable as what you would get for a traditional web application data center. And so what people were doing with this infrastructure was like seeking out the lowest cost energy resources in the world with this very cheap infrastructure. Why do I bring this up? Well, I bring this up because I think in certain ways we're seeing similar trends unfold in AI where one, energy is becoming the bottleneck. You know, the pathway to scaling AI, the pathway to scaling super intelligence is fundamentally being governed by our ability to access energy. So, we need to build a lot more energy to support this. And we also need to seek out existing energy resources that we can make more efficient utilization of for what we have in today's ecosystem. And that's pushed a lot of data center and compute infrastructure out of traditional markets like Northern Virginia and into markets where you can access this, you know, abundant energy resources. So that's point number one. Point number two is that I think people are starting to really rethink the data center infrastructure itself that's required to support different AI use cases. And, you know, this is definitely the case of like, you know, training infrastructure but people are starting to question it for inference infrastructure as well which, you know, fundamentally has a very different footprint than a lot of the training infrastructure. You don't need, you know, giant superclusters you can do a lot with much smaller clusters for inference. So, I think, I think studying the evolution of Bitcoin mining is an interesting parable to what we see unfolding today for AI and what we see may unfold in the future. And, you know, it also is an interesting historical case to study because there's been a huge trend of people that were mining Bitcoin that are now focused on delivering AI infrastructure because of this energy bottleneck that's unfolded and, you know, you see, you know, not just folks like us that, you know, had sort of AI as a thesis from early days but, you know, a lot of other folks that were pure play Bitcoin miners that are making a huge migration into being AI infrastructure businesses. And, you know, there's a whole bunch of public companies folks like, you know, Core Scientific and Hut 8 and Iris and, you know, increasing and folks like Terawulf, you know, there's a whole list of companies that basically had a big Bitcoin mining footprint. Applied Digital is another one that are now shifting into AI and so I don't think that, I think that trend is worth, I think that trend is worth noticing and appreciating because of, you know, the similar energy first characteristics that made Bitcoin miners successful.

Sorry, the question is how do I...

Um, I want to start with how did you do it back then? Because I think right now if you're convincing any AI startup to use anything it's like super easy because everyone's an AI startup, but back then in 2018 I feel like it's a different climate and then when people are still focusing on crypto even when it was like 2021, 22 people are still doing crypto stuff. So how do you kind of like find the first batch of customers back then?

So the early days of Crusoe, when you know we were kind of trying to build out the early versions of the Crusoe cloud platform, we didn't know what the product should look like. You know, we didn't know the key things that we needed to build. And so we went through this whole product discovery process where we were evaluating what to build, what would be valuable, you know, why people would use the platform compared to other offerings that are out there. And really what we learned from that process was, and sort of some of the feedback we got early on was, you know, just initially standing up some compute clusters, trying to figure out ways that we could deliver that as a product to customers and the different things that would actually make it valuable to them. And so the way we did it initially is we actually gave compute away for free to a lot of research labs. And this wasn't the big AI labs that you hear about today. You know, this was research labs at universities. So, you know, I had a number of different close connections at MIT that were doing various AI workloads and we worked very closely with a bunch of different folks in CSAIL, the Computer Science and AI Lab, as well as the Department of Physics that were using AI for a lot of things like simulations of the early universe, you know, the picoseconds following the Big Bang for instance, or using AI in computing some complex partial differential equations and fluid dynamic calculations. So there were anyway, there was a lot of early learnings that we gathered from a lot of those different use cases. And then, you know, I think there were, pre the launch of ChatGPT, there were a lot of people that were talking about using AI but weren't using it in the sense that we think about it today with really large deep learning models. But we had to basically find customers that could actually benefit from massively parallelized computing infrastructure that would accelerate their workloads. And we had kind of a set of early customers that really helped us understand the features that we would need to provide. And I think to give you an example, we were initially very compute focused, right? So we were very focused on the GPUs and delivering GPUs in all of their glory and we actually didn't have a storage product. You know, the only storage we offered was the NVMe storage, roughly one terabyte, that was available on the actual compute nodes. And there were early customers that were like, this is great but we actually need a lot of, we have a huge data set and we need to be able to store it in some other persistent storage solution. So that led us to, we said, oh yeah that makes a ton of sense, of course you have data that you need to run your compute workload on and you need some other persistent storage solution. So that led us to launching our first block storage solution. Anyway, just it was interesting learnings like that along the way.

When ChatGPT launched November 30th, 2022, it was kind of this moment that changed everything, right? Where it was this seminal moment for the world where people realized that you could do these incredible things with these deep learning models and it caused a flood of new startups to get going, a lot of people building on top of the OpenAI platform, just all these things that have sort of happened over the past three and a half years that have gotten us to where we're at today. But we happened to be in a good position at that time because we'd been investing in this platform for a long period of time leading up to that.

So at that point, you know, the Bitcoin mining business was the lion's share of our revenue. And that was a nine-figure business for us. At that point, we had a number of great customers early on. You know, one I want to highlight, there's a guy named Matt Mirus that is the founder of this company called Oasis. They were one of our early customers that gave us a lot of just great feedback. And they were trying to do some really cool stuff with computer vision and using AI and deep learning models to realize that product. And I think that really helped prepare us for this moment. And we had invested heavily in, by virtue of having the success that we had and my ambitions to build something important in AI, we'd invested quite a bit in terms of having a software engineering team that was building this AI cloud platform itself. So we were really well positioned to capture the moment. You know, in retrospect it's the kind of thing where we should have bet bigger. You know, I think for everybody it's like we tried to go all in but we should have maybe gone more all-in or something. You know, we should have tried to get more resources to make a bigger bet on the compute infrastructure required for this. And I think that's been a consistent trend is people actually under-appreciating, under-valuing how much compute was going to be needed for this at scale. And anyway, what happened was, as that development unfolded, we got a number of wins that ultimately led to compounding success, right? So as we would sign interesting customer contracts, it would give us more confidence to invest more into this opportunity and grow the business more, be more aggressive. And a lot of that success just ultimately compounded into building a fairly sizable platform that we have today.

What is something that I would say has changed the company trajectory?

I think the big bets that we made that really changed the trajectory of the company is like one, having the foresight to say that taking an energy-first approach to building the infrastructure has been instrumental throughout the history of the company. And two, actually having the foresight that the data center piece of this was going to become a critical piece. I think with the growth in the data center industry, there was a lot of stagnation of rinse and repeating and kind of growing a lot of the data center infrastructure that supported the web application buildout. I think our experience kind of having witnessed everything that happened in Bitcoin mining that I described earlier, really led us to have this high conviction that the infrastructure to support the evolution of AI chips was going to look different than what the evolution has been to support the growth in CPUs and web application cloud infrastructure. And so getting really smart and building the best data center development team in the world would give us a key strategic advantage in terms of if AI really took off and we were able to really benefit from that. And having that very first-principled thinking in terms of how the data centers get built out led us to rethinking the data center infrastructure from the ground up and be able to deliver some of these ultra-large campuses that ultimately would come into demand like what we're doing in Abilene, Texas. And that wasn't obvious a few years ago. I think today people realize and appreciate how important the data center is to scaling the infrastructure that supports this boom in artificial intelligence.

I think that there's still, I think we're still in the very early days of AI application scaling. So, the whole process of serving tokens and all of the optimizations that can be done, we're still in the very early innings of that. So I think particularly around managing memory across different layers of memory is going to be incredibly valuable. That's why we've invested in building what we call Crusoe Memory Alloy that stitches together a bunch of different layers of memory and serving and managing the KV cache which is super important to delivering high performance managed inference. So I think memory optimization and scaling of inference is a major theme to pay attention to. And then the other theme that I would highlight is just data centers as we know them today. They're going to look a lot different to support the future of AI. I just think the sheer volume and quantum of data center requirements is going to drive incremental specialization of the data center infrastructure and a massive cost reduction of data center infrastructure we're going to see over the course of time through innovations.

You take out everything that you don't need.

I think when you think about the end use case, I think this is particularly true for inference where serving tokens is a very different application than like running a massive training cluster. And as such the data center infrastructure to support those two things should look very different. And if you're adding in all these incremental components that add stability, that add reliability, that add backup generation and different nines of reliability, you may or may not need those things in other cases. I think inference and application scaling and the usage of tokens can fundamentally be a manufactured process, right? I think we're sort of in this previous domain of just like data centers have to be these big construction projects that require tremendous amounts of labor. To me, what we're doing is we're scaling up something that the infrastructure looks identical or it can look nearly identical. And so what you can do is you can actually scale this through ultra-high volume manufacturing processes and you can eke out a tremendous amount of margin and value throughout the entire value chain.

So think of it like this. Think about like a, you know, there's markets like furniture, right? If I go buy a handcrafted piece of artisanal furniture like a sofa, right? Maybe that sofa is probably going to cost me like tens of thousands of dollars, right? It's going to cost me, I don't know, $30,000 for like a really nice artisanal sofa. That's like handmade by someone that is like a well-respected artisan. But if I'm like, look, I need like 10,000 sofas and they're all going to look identical and they're not going to be special in any way, shape, or form, but they're going to be functional for people to sit on and whatnot, you know, I'm going to go more the IKEA approach, right, which is like high-volume manufacturing of critical components that can be assembled on site and I'm going to try to value engineer every single aspect of the supply chain. And as a result, I'm going to take that couch that was $30,000 and I'm going to produce a couch that functionally will support the exact same thing and it's going to be $200 or whatever the cost is, right? Something very significantly cheaper. I think you're going to see a very similar trend play out in data centers and AI infrastructure where what is needed is a pretty well-defined thing. And it can be dramatically cost-cut through scaled manufacturing approaches. And I think this parable of building the IKEA data center is a good one to sort of hold in your head in terms of what AI data centers look like.