How to Build Your Own Data Center & Why Every Startup Should Do It

Yeah, 100% it's on me. It's the biggest strategic mistake I made in the history of Speechify.

The best way to lose is not to be in the race. Be in the race. You don't want to be a fat manager who is like a general sitting in the back saying, 'Take that hill.' You want to be the warrior who runs up with their sword and engages the enemy first.

My pleasure. Glad to be here as always.

So, in 2022, we bought a huge rack of GPUs from Nvidia. And the reason we bought them is for training, right? We have a bunch of models. The newest Speechify Simba 3.2 model is ranked number one in the world for quality, above all the frontier labs, 10x more affordable than something like ElevenLabs. And we used to rent GPUs, and we found that engineers at Speechify would be parsimonious with how they used the GPUs because they were like, 'Oh my god, I'm costing the company tens of thousands of dollars. I don't want to do that.' And the analogy my brother and I came up with is: imagine you're Michael Jordan and you want to be in the NBA. It's the only thing you care about, and you need to pay $20 an hour just to train in a basketball center. Well, that sucks. You want one that you can go to whenever you want. In fact, you want a hoop in your house. And so our initial idea was: we want a hoop in our house. So we bought a bunch of our own GPUs. And that deal ended up being really good for us. And we ended up training really good models. So with time we invested more and more and more. So that's the first part.

The second part is actually how the economics work out. If you look at it, the transformer was invented inside of Google in 2017. Nvidia came out with A100 GPUs in 2019. Shortly after, they came out with H100 GPUs, right? The original ChatGPT was trained on A100s. And then they came out with Blackwell — B200s, B300s — and now they came out with Rubins, which are the GPUs that Elon is sending to space, and they're liquid cooled. They're very, very cool. And we're like, okay, huh. One, every class of GPU is more affordable per one trillion flops, right? So a flop is addition, subtraction, multiplication, any mathematical operation, and you measure them in how many trillion operations happen per second in a GPU. And so they're more affordable as it relates to this. If I was to buy an H100 for, let's say, $30,000 — that's how much a single card would cost — if I wanted to rent an H100 for one hour spot instance from GCP, it could cost me $5. If I rented it from, you know, Azure or AWS, maybe it'll cost me $3.50 per hour. So if I multiply that times 24 hours and then times 365 days in a year, I'm actually going to end up paying $35,000 to $50,000 to rent that GPU for one year, but I could buy it for $30,000. So it's 1.5x the cost of owning the hardware to rent the hardware for a year. Now, the hardware is typically warrantied for three years to work properly, but it'll keep working up to the warranty for, I imagine, I don't know, 10 years. So the math just makes sense where it makes way more sense to buy them.

The other big part is if you want to do large-scale training like we do, you need the memory to be co-located with a large cluster of GPUs. I can't just rent from Google or Microsoft or even B10 and run the size of training that I want because I need a gigantic memory card next to it with all of my data that all the GPUs are accessing. So that's why we first started buying them. The next thing that we found is actually if you run open-source models for coding, you could pay Anthropic, and then, you know, you're paying for all the tokens and the fact that you're paying the branded rate per token. Or you can run an open-source model, and instead of running it on a spot instance from Azure or anyone else, you run it on your own hardware, and then you're paying a fraction of a fraction of a cent per token. And so for all those reasons it made a ton of sense, but we can go into all the depth that you want.

At Speechify, we still use K80s for a lot of specific operations for inference, and we use older models of GPUs constantly. And there's essentially a difference between when you do inference and when you do training. For training, I'm like, okay, I have this hypothesis. I want to know the answer to this hypothesis as soon as possible. Like every minute that it doesn't come out, I'm in competition with everybody else. And so having a GPU architecture that is much faster by orders of magnitude is a huge advantage. But if you go speech-to-text or text-to-speech with Speechify, I can afford to give you a lower-quality GPU and it'll give you what you need still in, you know, 100 milliseconds. So it's totally good. And so I can always use these older GPU models for inference. That's number one.

Number two, we have so many experiments that we're running at every single point in time. Not all of them need to run on, like, the newest hardware. So the analogy I always give: let's say you bought an iPhone back in 2011 and it's an iPhone 3G, and then you bought another iPhone and another iPhone and another iPhone. You could have a drawer in your house with, like, five iPhones that are collecting dust because you can only use one iPhone at a time. But if I own 100,000 GPUs, I'm still going to use all of them at the same time. And so I'm not losing anything by having more GPUs because not only do I own a bunch, I still rent from the hyperscalers all the time. And I rent both dedicated instances that I prepaid for, and I rent spot instances. For example, more people use Speechify in September because everybody goes back to school. So I need to, like, level out the load. And so the parts of that load that I know for sure I'm always going to use, whether it be training or it be inference, I might as well just own it. And then on top of that, it's also the case that I have so many other friends who are running training and running inference. I can always rent it out to other people if I have excess capacity, which I don't expect to have. But, like, every once in a while you have an interesting situation. So for all those reasons, it just makes mathematical, financial sense.

Lastly, if you have excess capital, really, you either stick it in the bank or you buy a bond, right? Like the best long-tail bond you can buy — I don't know — will yield you, like, 5%. Or you can buy a GPU, and because renting it would cost me 1.5x buying it for the year, the return is, like, way higher.

So let's talk about Rubins, for example. Rubins come in the form of 72 cards in one rack. So we'll buy multiple racks of Rubins, and then on top of that we'll buy B300s, which are, like, the newest form of Blackwells, because we can get them earlier. And then the same thing — like, you know, when we bought our first instances of DGX H100 GPUs, we bought just, like, a bunch of racks of those, and then those get delivered in a truck to the data center. We rent the data center space, and so the data center provides the networking capability. It provides the energy, which is actually the largest constraint now, and it provides, like, physical engineers that take it off the truck, they install it, if it has an issue, they fix it. And then it just runs.

Yeah, ElevenLabs is amazing at this. I think Piotr at ElevenLabs literally bought a bunch of GPUs early on and set them up in his house, and then they just kept building bigger and bigger and bigger clusters. They do the same thing that we do.

Yeah. So number one, I want to explain again, it's very different than buying an iPhone or buying a MacBook. I can only use one MacBook at a time, one iPhone at a time, but I can use all the chips I have at any given point in time, and I still will have more demand, especially when I have multiple teammates and 60 million users who are using inference on my Speechify software that's providing text-to-speech and helping them, you know, read their work and dictate their work and, you know, use Speechify Work, which is our newest product. That's an agent, kind of like Jarvis from Iron Man. And so I go, okay, let's imagine I have 100% capacity. That is the average usage per month that I need for GPUs for training of my AI models and for inference on my AI models. Inference is when you actually make a call to Speechify and you give me text and I give you back audio. Like there's math that happens in the background. That's inference. Training is: I take a gigantic amount of data, I take all the architecture and software engineering that we're doing, and I go, I think that this will give me a better model. I'm kind of baking that model in the oven and I'm going to come out with a new black box. And then when I give you text, that black box is what calculates it and gives you back the audio. So those are the two usages.

Let's say I have 100%, which is what I would have in, let's say, a month like November. In October I'll have 140% because it's, like, a big month for us. In December, you know, everyone's at home, you know, they're not necessarily studying or working. So I might have 80% utilization. Okay. So I go, cool. Well, I can take 20% of the usage that is normal and let me buy it because it's the best deal. I'll take another 25% of the usage and I do long-term contracts with hyperscalers. The rest I'll rent — what's called spot instances — from the hyperscalers. And then I'm still not even close to overcommitting myself. And so that's kind of how we think about the math. And then we go, okay, well, also we have 45 engineers, but we want the team to be 150 engineers. And even inside of my 45-engineering-person team, like, there's a couple of people who are rock stars. They have dedicated, like, DGX racks just for that one person. And 25% of my team are almost, like, waiting, and I want to double the size of the team. They just need — like it's like you have a football team and you just need another field because they don't have enough field to practice on. And so that's how I think about how to allocate.

And then in terms of depreciation of the asset over time, I go, okay, well, these are still amazing GPUs. Like even A100s you can run amazing experiments on. So it's completely valid to use that as long as it's hooked up and as long as it's not stopping to work. And so think about the mileage of a car, right? If a car gets to, like, 250,000 miles, you know, it's kind of going to break at this point. That's not necessarily true for a GPU because it doesn't have as much wear and tear. Yes, it's moving and yes, all these things, but, like, it's in a very clean environment. It's very much cooled. It has constant maintenance because it's not moving around. It's very expensive. And Nvidia just does a really good job. And so that asset is going to stay for a very long time. And let's say it got so not good, so outdated that I no longer can run training on it. Cool. Now I'll use it for inference.

There's one more thing that's very interesting that just happened. So, I believe earlier this month, Nvidia did a huge deal with Blackstone, BlackRock, Apollo, and Goldman Sachs, and they said, 'Listen, we want more people to buy more GPUs. We're going to underwrite for you up to 25% the value of a GPU.' If you lend money to someone who buys a GPU — let's say Google or a startup like CoreWeave — and that startup goes out of business and you have that GPU as collateral against that investment, we'll buy back the GPU for up to 25% of the value of the GPU. And so they're succeeding in creating a liquid secondary market for GPUs that they're underwriting. So now the large banks have an incentive to loan money at much better interest rates. This is actually exactly what Elon did in the beginning of SolarCity. He went to Morgan Stanley and Merrill Lynch and got them to advertise the price of a solar panel over 30 years. So the whole invention behind SolarCity was the fact that you could take a loan against the collateral of your solar panel. So Nvidia has done an amazing job now in creating a clear floor for the value of the GPU over time.

I think that a lot of the things that, about a year ago, like, what was going on between Oracle and OpenAI, like, that was way too much. Like, that was ridiculous. I think the Nvidia stuff is not, because you're talking about a real asset. So if you think, for example, about the logic behind the value of Bitcoin, right? What is the intrinsic value of Bitcoin? I can't really tell you, right? What's the intrinsic value of gold? Well, gold, you can use it for some medical stuff because it's a really amazing metal and, you know, it's jewelry, whatever. But a GPU, it has intrinsic value. Like, you can actually use that asset for something that's really, really valuable and it doesn't matter where that GPU is. It could be in Iceland, it's still useful to anybody all over the world as long as it's, you know, and so actually it has a pretty good store of value. Even if new GPUs come online, really my one question — and this is the math for everybody to come back to — is how many teraflops per second can this device do? And that is, like, essentially a token. That's the value. And so, like, there is intrinsic value. So yes, you can have all these, like, circular things, but at the end of the day, Nvidia is making a product that's real. It's not complete tulip mania. Like, there's real, real intrinsic value here.

I mean, it's not that people are naive, it's just they haven't been in the space. So I'll give you an example. Imagine you're buying a GPU. Well, you're going to buy it. You know, it's an Nvidia-produced product, but Nvidia is not going to waste the time talking to Cliff Whitesman. So who do I buy it for? Well, one of the best-rated vendors is Dell. So everybody thinks Dell is a personal computer company. No, Dell is a GPU rack supplier at this point. And then, okay, I want to buy it from Dell. Well, Dell has a constraint because there's not a lot of, like, you know, Blackwells out there. Well, it happens to be that they have someone in France. All right. Well, I'm going to order mine from France. Okay, shoot. It was supposed to come a month ago and it's still not here, right? Because of whatever, like, you know, there's demand. So then you need to negotiate to make sure that you get it, which is why we're very willing to pay 100k per month extra to get them earlier. So you'll call up Pierre in France and say, you know, hey, we'll give you an extra 100k kicker if you get them here in a month — even more than that. So in that France situation, which is something that happened to me, I was like, Pierre, what the heck? We have a contract. You're not delivering on time. And so it is the case that we had a contract with another company beforehand and they were, I don't know, a few weeks late. And I called them and I was like, listen, I've got a better deal. I'm canceling our contract because you're out. Like, you didn't deliver. So I'm going to go with this other contract, but if you have a better price, like, we'll go with you, but I just need the GPU now. And remember, I'm paying for the renting space of my data center. So the most expensive part of a delivery of a GPU is if it's late, I'm still paying rent for that data center space. Now, that GPU — and so, you know, you put pressure on Pierre to send you the thing when he said he was going to send it to you. And then you go to Nvidia or Dell or whatever, and you're like, look, it's a market. Hey, can I pay more to get it earlier? Skip the queue? Yeah, you can. Cool.

Now there's a truck somewhere in the United States with a GPU whose value is the value of a house that's coming to my data center, right? Well, I should have insurance for that, right? Because if that truck gets hit or there's too much humidity or the GPU gets flipped, like, I lost multiple houses worth of GPUs. So, okay, the insurance is really, really important. And then also the value of the amortization is really, really important. And it's like there's all these nuances of how to do the math through. Then there's the cooling, right? So, like, you're not only paying for the physical space and the networking and the energy — and the energy is the biggest constraint, we'll talk about it in a second — well, how do you cool that thing? Because you have a thing that's just, like, moving and moving and moving. Well, the thing that's, like, most new now is liquid cooling because air is just not enough. And the thermal load of water is much better, and there's other liquids that are even better than water. And so Rubins are liquid cooled, but most of these data centers don't have liquid cooling installations already approved. So we had to do a bunch of research and we find, okay, we could buy what's called a sidecar of liquid cooling that you enter into the data center. Then you pay someone at the data center to install it for you. Cool. Now you can have, like, the rack that you want. And so there's a big difference between running a purely software company and running a company that includes hardware.

Exactly.

ElevenLabs worries about the same thing because for them to train excellent models, they need to have co-located GPUs with a lot of memory available.

You can. It just becomes, one, ridiculously expensive; two, you need to commit for many, many years ahead of time because you need to build a co-located cluster; and then you don't have as much control because you don't own it. So it's like difficult to — like, you suddenly need an InfiniBand cable which allows for the memory to flow from one DGX to the other one. And the answer then is, well, now my ability to train is so much bigger. I can have a larger AI team. Every person in the AI team is leveraged and I could just, I could shoot ahead of everybody so much faster. And let me just make one thing clear. If I want a Rubin, which is, like, these much faster GPUs, I'll get it faster if I buy it than if I wait for Google to buy it and then there's other people in front of me in line. So I'm going to skip the queue by, like, a lot, and then I'm going to have, like, a year of access to Rubin before everybody else does.

The way I think about it is the following. How do you build an amazing company in a world where there's so much competition today? The team is the most important part, but the team is the most important part because the team gets you the other resources. And so what are the missing pieces? The missing pieces are data, compute, and architecture. In a world where intelligence is commodified and no one needs to handwrite code at all anymore, right? Our engineers — really what I'm looking for is 10 really good decisions per day, which is very tiring. Not, like, optimizing random parts of the code. And each one has, like, you know, five to 18 agents running at any point in time, doing long-horizon tasks on these GPUs, coming up with theses, testing them, going back and forth, back and forth, back and forth. If they don't have the capacity to train, the team is limited. If they don't have the data to train, the team is limited. And by the way, a lot of data is cleaning the data, right? You get this raw data in the beginning. Well, you need to organize it into datasets. And so you need the GPUs to also organize the datasets, too. Like, I have one of my best engineers right now who is not even writing models. He's making synthetic datasets to train models, and so, like, it really becomes an indispensable asset.

We have, but, like, small datasets. So I suggest you use Fireworks — but, I mean, Fireworks is amazing. Lin, the founder — she's one of the co-founders of PyTorch. But I had the very obvious realization that you'd have every company having their own specialized models of a certain size trained on their own data. But you would need supplemental data, sort of like this synthetic data or real-world data that you just don't have, and that you would buy that from data providers like Mercor, which is why I was...

100%. Because if you're a Mercor — shoutout Brendan Foody — ElevenLabs or OpenAI or Anthropic is going to make money for the next decade or two on the data that they bought from you, and so they're willing to pay a fraction of that 10 years of revenue to you today to supply them the data. And again, it's all a speed thing. Yes, ElevenLabs or, you know, OpenAI can go and make a team that will get the data, but they don't want to manage it. And so the data is key. All this — like, you need all three things: you need compute, you need data, and you need a team that writes great products. And ideally you need users that use you a lot and a lot of them to have a feedback loop of whether the stuff is good or not, so benchmarking.

So the first problem to understand about the data marketplace is it's not ARR, right? It's not annual recurring revenue. It's one-time deals every single time. So the buyer of the data is not required to buy it from you again. So it's a very risky business. And if you look at early days of companies like Mercor, they didn't raise significant funding off the bat because investors were very skittish about that fact. Let's put that aside. Well, it's very important that the T's are crossed and I's are dotted about how you got that data, right? And so, like, you know, like, you need to indemnify the companies who are using you, and that's part of why they buy it from you as opposed to sourcing it themselves, or you've seen the lawsuits. But it's a great business if you could do it well, but, like, you need to be an ops monster — like, you need to be really, really good at operations. You need to be very fast, and really the key is the company training on your data needs to actually see improvements in their model at the end of the day.

The thing that has always been challenging for Speechify compared to other companies is B2C customers pay a lot less than B2B customers. So ElevenLabs — huge credit to them — leapfrogged us because they sell to B2B. Well, historically we've only sold to B2C. And so our big constraint is we needed to do this on a cost basis of — it needed to cost us, you know, less than $10 per million characters. ElevenLabs charges $100 per million characters. The OpenAI model on the benchmarks costs $196 per million characters. So ours, when we sell it to other B2B companies now — we just launched our API, Simba 3.2 — it costs $10 per million characters.

100% was on me. 100% was on me. It's the biggest strategic mistake I made in the history of Speechify.

So I met Patrick and Mati. I was living in London at the time, in my house in London. I think it was 2022. And we were very impressed by them. And we wanted to use the model, by the way. It was just too expensive for us to use. And I looked at it and my thought to myself was, they're very smart, they're going to do well, but I don't like their strategy because I think that an API for text-to-speech will become commoditized with time, right? You're going to get to the point that you can run that API on your computer and then on your phone and then, like, what are they selling anymore? So I don't want to go into that business. And I made a critical error. What I didn't understand is that the point of an AI lab like Speechify or like ElevenLabs is to continuously innovate, and the first product that you release is your wedge that gets other people to then later use your other technology. So, for example, if you're in text-to-speech, you build the best text-to-speech model in the world for one specific voice. Cool. Well, now you can do other voices. Now you can add emotional prosody. Now you can add voice cloning. Now you can add speech-to-text. Now you build duplex models that handle ums and ahs, laughter, interruption handling, turn-taking. You add a harness for voice conversations. Then you optimize it for sales, and you optimize it for customer support, and you optimize it for all these things.

And so what they did is they first built an amazing API. They were great at launches. They built a really great product for creators. Then they built their best product ever, which was agents. Agents is amazing because the buyer is no longer a software engineer. The buyer is a CTO, CIO, CEO, executive in the company. Sierra has this concept called outcome-based pricing. Brett Taylor is amazing. And so you can start finding the outcome. And having an AI agent is like having an AI co-worker. But it was my mistake to think that an API product was a bad strategy because I thought it was something that would become commoditizable. And I forgot the central thesis about Silicon Valley, which is constantly innovate. Get the user to start using your product. I don't care if it's free, then you sell them other things. And so that was my big, big, big mistake.

Super hard.

You are now competing against ElevenLabs and Sierra. Really? And those two are competing whether they like to admit it or not. They absolutely are competing, and they will, I'm sure, if you ask them off camera. That's Brett Taylor.

I don't want to compete against Brett Taylor. And that is the tidal wave of ElevenLabs. Now ElevenLabs is an unstoppable machine at this point, to the point where it has government buying across all of the large major Western democracies. Actually, it's insane the government buying they have, and they started three months ago.

Yeah. And so, you know, the graph of...

So here's the two things to consider. The first one is: if you go to the App Store and you search text-to-speech, Speechify has 98% of the installs in text-to-speech for B2C. Speechify has served more than 770 billion words to users over the last few years, which in terms of times of listening is, like, 6,000 years of listening, right? If you go from today to zero BC and back, you still have, like, thousands of years left. So we've, like, completely dominated that market, and it's still a business that's growing really, really fast. And we're constantly adding more features into that product. The thing is, we have a pretty big engineering team, and now everybody is capable of doing 10x what they did before. So I have extra staff. I have a huge AI engineering team with the ability to make amazing models. So, like, where is the highest ROI for that to go? Well, it needs to go both to B2C, but it should also go B2B. And one thing that I will never be is a person who doesn't learn. So I might as well just freaking learn B2B.

Now, to your point about competing against giants like Sierra or ElevenLabs: hey, Anthropic came into the market as a second to OpenAI, and they were second for a very long time, and now they're not second. Facebook came as a second to Friendster, and now in my space, and now they're not second. And so the nice part is this space is not a monopolistic space. It's an oligopolistic space. And if you look at what happened with ElevenLabs — I'm going to exclude Sierra because the Brett Taylor effect is huge. It's just amazing to see how good of a business that is. And so it might very well be that for the core offering that they're currently winning on, I will not win. But what did I learn last time? It's fine if I offer my product essentially for free because I'm an AI research lab, and as long as people start to use me, with time I'll be embedded in the system, and I'll keep coming out with more and more and more innovations that are useful to them. And so there's unbelievable demand from all these companies and governments and everybody else for great tools, whether they be AI agents or APIs or products. I just want to be on your phone if you're a user, or in your stack if you're a company, and supply you with the best front-deploy engineer experience and AI orchestration experience and API experience to give you an amazing experience. And there's room for everybody.

But you would then think that OpenAI would be the place where value accrues for voice AI, right? That's what you would have thought three years ago, and that's not what ended up happening. So you can't not go into the race because there's a big incumbent.

I agree. And, like, that was theirs to take and they fumbled the bag across every...

All respect to Patrick. I think they're absolutely amazing, and I love working adjacently to them.

But they have so much in their net right now.

And so much is getting added to the net constantly.

And so you just, you have to, like, go where the football is going.

And so I think that it's too expensive for Speechify not to be playing in B2B as well as playing in B2C. The best way to lose is not to be in the race. Be in the race.

It's not that every startup has to be a compound startup. It's that at a certain point you can't afford not to be that.

Yeah, absolutely. Those are the winners, right? ElevenLabs is an example. Anthropic is an example. Ramp is an example. Speechify is an example. All the companies that have absolutely maniacal leadership teams and engineering teams — like, that's why people care about team more than almost anything else, because the right team will iterate fast, get there, and then figure it out. And now when everything can be turned into a reinforcement learning problem where you can have long-horizon agents and orchestrating agents thinking about the problem for, like, two weeks at a time, if you set that up, of course you're going to win.

Yeah, you lost the glasses.

My favorite type of person to hire is a CTO of another company. We have — when we were 21 people at Speechify, 18 of the folks at the company were previously either CEO, CTO or VP of engineering at their last company. Anthropic — I have never seen a company like this hire so many CTOs of publicly traded companies and other successful startups.

The reason is they built the best, most beloved product for engineers in the history of the world. So it's easy to hire CTOs. By the way, they hire much more CTOs than CEOs because CTOs are the ones who get the most excited about this product. And, like, you're right, they're the fastest growing company ever, especially at the...

Scale that they are. So they're going to keep growing. OpenAI is going to keep growing. You had this very condensed period, like fireworks of growth in both of those companies. Yeah, it's very hard to hire. But remember, they're hiring people that their annual compensation needs to be $15 million a year minimum. What startup is hiring someone and paying them $15 million a year? You're not like, dear seed founder, that was not something one that you were going to hire. And so I will push back against it. The competition for growth stage companies hiring exceptional leadership talent is more difficult. For seed companies, I would say it's the easiest time ever because the impact of even just the founder on their own is bigger because they can orchestrate agents. But the same thing for hiring. So one thing that we have changed about our hiring in the last even 6 months is we really cared that you read a ton of textbooks about software engineering and that your handcrafted code was amazing. I still care that you read a lot of textbooks about software engineering and you understand it. But the thing I care about the most today is technical aptitude and just like raw technical intelligence because I know that we could teach you everything else and in 6 months you could be a machine. And so we hire a lot of math olympiads and LeetCode coders and like Kaggle award winners and people who like studied physics and math like they might have even not coded before because I just need the hunger and the work ethic and the intelligence and anyone can become so good so fast now. And so the pool for hiring exceptional talent is bigger than ever before. And Duolingo did this really well. They love hiring college grads and then coaching them. And so I wouldn't say that it's harder to hire than ever before for seed companies. Seed companies now almost anyone can be someone that you hire if they're smart and hardworking because you could teach them very fast. What is more challenging to hire is for growth companies because you're fighting with just absolute juggernauts.

I totally get that and I completely agree. I will see CRO packages in the 50 million plus range by the way.

15 is like kids' play. By the way, with the greatest of respects, I will even see $15 million on the table for comp packages for seed companies today. That is the dislocation that I think, with the greatest of respects.

Well, I mean you've got to understand a seed round today will be 150, 200 million.

And there are several of, I mean there's 30, 40 companies that at seed have raised 100 to 300 million.