Can you tell us about the dot-com boom? We did much more technically interesting work in the bust than we did in the boom. There's a degree to which innovation requires some level of desperation that good economic times are kind of hard to summon that desperation.
How have AI tools changed how you're working at Oxide?
Certainly we're using Claude Code a bunch and people are doing that, but for a lot of the work that we're doing it is helpful as maybe a polishing tool but less as at the epicenter of its creation.
Can you tell me what it actually means to design or build a computer?
Oh, it's very involved. Yeah, it's very involved.
So, first of all, how have servers and cloud infrastructure evolved since the late 1990s and what is next? Bryan Cantrill was a distinguished engineer at Sun Microsystems during the dot-com boom and bust, built a small competitor to AWS called Joyent, and is now the co-founder at Oxide. Today, we go into the history of servers in the cloud from the late 1990s to today, the challenges of building hardware like the Oxide computer from scratch, how the Oxide team uses AI and why they find it practically useless for hardware engineering challenges, why Oxide builds everything as open source, and how they manage to work remotely as a hardware startup, and many more. If you'd like to understand more about how the cloud works, and learn how a nimble hardware plus software startup operates, this episode is for you. This podcast episode is presented by Statsig, the unified platform for flags, analytics, experiments, and more. Check out the show notes below to learn more about them and our other season sponsor. So Bryan, welcome to the podcast.
Oh, it's great to be with you. Thanks for having me.
I'd love to jump back in time a lot, back in the 1990s, because you're someone who's been around the block, and back then you worked at some interesting companies including at Sun. And if you could give us listeners and viewers a sense of what was it like in the '90s in terms of software servers, what was the vibe like?
Yeah, it was an interesting inflection point because I was interviewing in 1995. I started in 1996. So I would say that the internet, and I mean HTTP had been developed in like '93, '94. We had kind of the first web browsers, but it was still very, very, very new and the internet was just kind of primed for takeoff. Java had come out in maybe 1995. Java had kind of taken off immediately. So there was a lot of really exciting energy, but it was nowhere near what it would become a couple years, even a couple years later. It became very frothy, of course, and it was exciting. It was very clear to me, I went to school actually on the East Coast, but just coming out here to Silicon Valley, the energy was extraordinary and really knew that I wanted to come out here for my career. So at Sun those next couple of years, I mean, I got very lucky really because Sun was in the right place at the right time with the right technology, which you know sometimes you only appreciate in hindsight because it was so explosive. And if you wanted to build a website as part of that dot-com boom, you were buying Sun servers, you were buying Cisco switches.
Now why was this the case? Because again, just taking myself back, just being a bit naive, I would assume that, hey, I'm in 1995, I want to build a website. Could I have not just used a PC and spun up a server? Did it not work like that or how did it work?
I mean, a PC maybe, but you didn't really have an operating system, right? Because Linux is very, very new. Linux is not...
Oh yeah, definitely. Linux would be like Haiku today, which is an operating system you haven't heard of for a reason. It's kind of like a hobbyist operating system. You know what I mean? You'd be like, what? No, you wouldn't. And then you kind of had the BSDs were out, FreeBSD was certainly out there. Also still very much under the shadow though of this lawsuit from AT&T. So the Unices are kind of... there's not really open-source operating system options. There was the... actually this is kind of funny because... so where was the GNU option? It was going to be the Hurd operating system. So Hurd was kind of like the Duke Nukem Forever of its time. It was the operating system that was constantly coming kind of next year and next year and next year, and it was going to be microkernel-based. And so you know, that's kind of amazing, but you really couldn't do it on PCs because of the lack of system software. And actually part of my attraction to Sun was I had used Solaris on SPARC, but I never... I knew Solaris existed on x86 but I never used it. So I was excited to use Solaris on x86.
And so what did Sun build? You mentioned Solaris. That was the operating system.
Solaris is the operating system. We built servers. So we built SPARC-based servers. We built desktop machines. So Sun was a computer company. It was a systems company. So we built desktop machines, built some ill-advised laptops. So basically desktop machines, workstations. But then at that time in the '90s, what was really exploding were everything from those kind of workgroup-style servers up to really getting bigger and bigger servers up to the very large machines, machines that are as physically the same size as what Oxide makes today. And I remember vividly in what would have been like '97, '98 maybe, Greg Papadopoulos, then the CTO of Sun, giving it to the entire company saying, here are the top three applications for Sun Microsystems: databases, databases, and databases. So that gives you an idea of kind of how it was being used. And this is again as that kind of... in that knee of that dot-com buildout where if you again, if you wanted to really build a web presence, you were going to use Java, you were going to do it on Solaris, you were going to do it on Sun servers. And it was kind of... it was a wild time for sure.
And can you tell us about the dot-com boom? Because you know right now I know AI is pretty exciting and it feels like we're in a special time, but what was it like, especially working on... it sounds like it was at the epicenter of it.
And you know what was funny is I did... it was frenetic in a way that was not always positive. So, one of the things that is just a point of fact, and one can take from what one will, we did much more technically interesting work in the bust than we did in the boom because I think that when you're in boom times, you know, everyone kind of like secretly believes that this is because of me, like that it is because of the thing that I am working on. If I... you know, I once had, you know, one of the early technologies behind Java once told me with a straight face, every server that Sun sells, they sell because of Java. And I'm like, you know what? You know what's most amazing? I believe that is actually the more interesting fact. That I mean, it is like obviously false, especially with, you know, databases, databases, databases being the top three applications. But that kind of reflects the zeitgeist of the time that everyone believes that this is, you know, if I work on the microprocessor, it's because the microprocessor is perfect. If I work on the operating system, it's because, oh, this is the operating system that people are buying the machine for. And that doesn't really lend itself to real innovation. I think there's a degree to which innovation requires some level of desperation that good economic times are... it's kind of hard to summon that desperation sometimes. So, I think that during the boom it was... and it was just... it was frothy and it felt like there was a period of time where I'm like, this obviously can't go on forever. And you know, The Economist is having these very gloomy covers about how this is all going to end and it's going to be an apocalypse, which I believed, and then I just stopped believing it. And I'm like, well, maybe The Economist is right. It just went on longer. And you know, one of my early life lessons from the boom and bust is these things go on longer than you think possible.
In terms of the boom, when you're in frothy times, that boom will go on longer than you think possible.
And when it switches, it will collapse faster than you can fathom.
In the boom, do I understand correctly that customers were just like wanting to buy your servers? They were flying off the shelves. All these companies. And on a day-to-day work, what did it mean for you?
So I'll tell you, like in day-to-day, it meant first of all, it meant that traffic was terrible. That the, you know, there is... you couldn't get housing, you couldn't get, you know, everything was in short supply. You couldn't... customers are, you know, they are buying. We had a customer that, you know, was going to buy 19,000 servers, which is obviously a big, very big number.
And these were these massive big servers, right?
Yeah, well in that case those were actually one U servers to build out a broadband initiative that actually was a company called Enron. You know, I remember vividly we were at a dinner here in the city at a restaurant called Aqua, which is a very kind of fancy restaurant, long since out of business, and I don't think Aqua survived the bust. And we were at Aqua with a bank who was a customer of Sun's and they were spending a galactic amount of money every year with Sun. And we were at a dinner and I just remember, I mean it was the kind of like 19th century Gilded Age kind of dinner. People are ordering, you know, nine courses. What I remember is at the end of that having Château d'Yquem, which is a Sauternes. So I don't know very much, I know very little about wine. I know nothing about Sauternes. What I did know is there was someone who knew wine and it's like, we are going to all drink the 1952 Château d'Yquem, which is... and I remember being like, I'm not much of a drinker, but I was like too drunk at that point to really appreciate it. So I have had this Sauternes that, you know, that oenophiles kind of live their life to drink, and I'm sad to inform you that there's one less bottle of this precious vintage because it was poured down the gullet of a twenty-something dot-comer who really had... And I just remember being back in my apartment being literally drunk on Château d'Yquem, thinking to myself, this can't last, this is not sustainable. And I swear the dot-com boom turned to a bust like that night. That is September of 2000. So Pets.com had kind of busted out and the NASDAQ had busted out early in 2000. The traffic got lighter early in 2000. Anyone who was here would be like, the absolute spookiest thing is it went from like gridlock to like COVID-like traffic in the span of like a month.
Without COVID happening, with only the NASDAQ collapsing. And you're like, okay, that's very odd. And then 2000 kind of muddled along, and then that dinner was in September of 2000, and what really stopped was the telco buildout. So there was a lot of telco buildout because people are like, the internet is the future.
And telco buildout meaning the towers, the servers?
The servers, the infrastructure for it, and then all of the concomitant... the fiber, like JDS Uniphase was a huge company. You had these companies that were, you know, Global Crossing and MCI WorldCom and all these companies were explosive, and everyone believed that the internet is the future and this is like an important thing. And they were right. They were right.
Bryan just said how an important lesson from the dot-com boom was that people who believed the internet will be the future, they were right. Today we're in a similar stage with AI. It's pretty likely that AI will be part of the software stack in the future, even if timing is harder to predict. The latest shift is how AI agents are becoming a lot more commonly used for development. And this is a great time to talk about our season sponsor, Linear, and how they think about collaborating with agents. Linear has taken an interesting approach here. Instead of building one proprietary AI assistant and locking you into it, they built an open API and SDK that lets any agent plug into your issue tracker. That means you don't need to wait for Linear to build the features that you need. You can connect the best coding agents on the market like Cursor, GitHub Copilot, OpenAI Codex, and Devin, or you can build your own agent for your team-specific workflow. It's a fundamentally different approach from most issue tracking and project management tools on the market. You get optionality and the experience is surprisingly natural. You assign an issue to an agent the same way you'd assign it to a teammate, or you can simply mention the agent in an issue thread. Cursor then can pick up a bug, understand the context from the issue, open a PR, Codex can explore a fix while you're focused on something else, and root cause analysis when something breaks. It's pretty powerful what you can get these agents to do. And here's what I like: you, the human, stay the accountable owner. The agent works for you, not instead of you. You review the work. You decide when it's good and when it ships. If agents are going to be a part of the toolset of building software, and it feels to me they increasingly are, you'll want a system that's actually designed for them. Linear is a system like this. To learn more, head to linear.app/agents. And with this, let's get back to the point where Bryan was saying how those believing the internet will be the future back in 2001 were right.
This is the other thing. It's like, they're right. And so like a very famous e-commerce company from the dot-com boom is Webvan, right? Webvan was delivering groceries, which many people today are going to get their groceries delivered, right? Instacart. It's like they weren't wrong, but their timing was off and they lost track of the underlying economics completely. And so when it busted out, so in the fall of 2000, in November of 2000 in particular, there were zero orders from telecoms at Sun. Like it went to zero. And you know, you're kind of used to kind of ups and downs, but that's like just off a cliff. And from that point, going into 2000 and then 2001, and it was then very, very grim. I would say that the thing that happened through the bust, and layoff after layoff after layoff, because companies had kind of built themselves and geared themselves around these fat times lasting forever, and now they were gone. And expectations, as frothy as expectations were during the boom, they were that much negative in the bust. People were like, everything is... it's the end of days.
And were you a software engineer back then?
Yeah, software engineer. Yep.
And then so as a software engineer, like both you and also thinking about your colleagues back at the time or friends, how did it impact you? Were you kind of just chugging along or...
So I would say that like lots of people left, and you had like the statistic of, you know, the U-Hauls were ten to one out of the Bay Area. So people moved away. And the thing that I noticed is that the people that had moved out to Silicon Valley because they really had an interest in the technology all stayed and were not adversely affected, honestly. I mean, yes, every one of us, if you had equity in your company, which of course you all did, you try not to overthink it, right? You just try to remind yourself, like, I never had it to begin with, so it's hard to... but it's definitely gone. Sun lost 98% of its value, so it's like definitely gone. And you know, there was something... and I think it also, like, a boom can get you to care about things that you actually don't care about. And a boom can get you to... because in a boom everyone is so financially driven that it's hard not to become financially driven. But it's like, that's actually not why I got into this. And so during the bust, I'm, you know, definitely able to put a meal on my table and a roof over my head. But it was really a reminder about what's important. And again, because we did do better technical work in the bust than we did in the boom. And I think it's because in the bust it's like, okay, now we really have to focus. We have fewer resources. The fewer resources actually force more creativity. So, you know, all of the things that we did, certainly speaking at Sun in system software, so ZFS and DTrace and the Service Management Facility, all these things that were really revolutionary for the operating system, all happened in the same kind of post-bust period of time. So all of those things happened from 2001 to say 2005.
And so what were these specific innovations?
So I'd gone to work at Sun to work with Jeff Bonwick. And as long as I had known Jeff from the mid-'90s, Jeff had wanted to rethink file systems. And now finally in the early 2000s, he and Matt Ahrens were able to really go take a clean sheet of paper for the file system, and that's ZFS. I had a chip on my shoulder about the way we understand and debug systems, by the way we observe systems. So I, along with two other colleagues, did DTrace, which allowed us to dynamically instrument running systems. And you can kind of go down the line, and there were a bunch of things like this where... and I don't know that all of this is related to the bust. It's just that the timing lined up such that it was all happening during the bust, and what we ended up with was a whole bunch of interesting technology coming together actually in a single version of the operating system. And then very... I mean, fortunate for us, and I do think this is a bit of a consequence of the bust, because Sun was definitely open to new approaches, we open-sourced all the operating system. So that happened in 2005, and that was very important to give these kind of technologies eternal life. But I think, you know, we can never predict the future, but to me it is pretty positive in this sense that even in the bust, hearing the stories that innovation did not stop.
Sure. You know, sounds like it was probably harder to get jobs and there might have been fewer of them, but you know, industry kept innovating. And what you said that I didn't expect to hear, that it was a bit easier to innovate.
It's just less manic. We were able to focus more. And so not that... I mean, not that one should necessarily pine for a bust, because busts are brutal, but there is a clarity that you get too. So I mean, ideally you would like to have just like, can we just be like normal economically? But like, nope. Apparently in high tech we've got to be like on or off.
So bust aside, in the early 2000s leading up to this internet boom, the way to, you know, most companies went about buying Sun servers with Solaris installed and everything was hardware and software came together. It was beautiful. It worked well. Again, I heard from folks who did it. What happened then? Because when I got into tech in 2000, I did not hear about Solaris and that was not how it...
No. Right. That's what was the shift. So the shift was first of all open source, right? So then, you know, we said in the mid-'90s Linux was kind of still very much a hobby project. Not so by the 2000s, right? So it grew up. It grew up absolutely. And it grew up because you had a bunch of companies that really backed up the truck. And, you know, the things that... at first IBM and SGI, Data General, some other companies, those companies were very important because they decided to contribute their technologies, like XFS, right? XFS, many people still use today on Linux. That's from SGI. XFS was SGI on IRIX. That was happening in kind of those late '90s. And then in the 2000s, I mean, Google was always built on Linux, right? And so you had kind of the companies that became that next boom were all built on open source and indeed needed to be built on open source. So they economically relied on open source to be able to build what they built. So then it became much more practical to certainly run Linux, and I think the other BSDs, or we open-sourced Solaris. So there were a lot of options that were now available. So that shifted. I think the other thing that shifted is that, I mean, SPARC bluntly lost to x86. And SPARC is a Harvard architecture.
SPARC is a microprocessor.
Yeah. And there was a time in the '90s when if you wanted the fastest microprocessor, it was a RISC microprocessor. It was a SPARC microprocessor, or it was MIPS, or it was Alpha. And x86 was a commodity and obviously available with a personal computer, but was not faster than those RISC microprocessors. That shifted in the late '90s. And we, you know, because we ran the operating system that was in Solaris on both SPARC and x86, we could see how fast these x86 machines were and could see, frankly, how... like, you know, you talk to the microelectronics folks, they really did not... they kind of dismissed x86 and dismissed Intel. And you shouldn't do that. And in particular, Intel was very focused and architected their way around what was called the memory wall. And they were able, in part because they use speculative execution, they were able to actually make these microprocessors that became much faster than the RISC microprocessor. So by the time, say, you are in 2004, 2005, if you want a leading-edge microprocessor, it's x86. So that was a big and important shift. So by the time you're coming up, it's like, okay, yeah, if I want this, I'll just like, I don't know, get a Dell box or Supermicro box and then I'll put Linux on it or maybe FreeBSD and away I go. Then the next kind of big and important shift that happened started in 2006. You could argue with S3, but then especially in those next kind of seven, eight, nine with the introduction of EC2, and now you have like the cloud that starts to come into play. And now like people were like, why would I even like screw around with a server at all? I mean, it was so great to be able to just spin up infrastructure.
Yeah. I remember one of my early companies, mid-2000s, we had a server room. We had server administrators. The server room was always hot. And this was a small company, mind you. This was not a big one. Every company needed to do that. It's kind of amazing to think it's like that every single company, no matter if you were a website, you had your own server room.
And if you were a dev, you wanted to be friends with the server admin because when you wanted to deploy your stuff, they could do stuff for you.
They could do stuff for you. Totally.
And so I think that cloud computing was really important. This is not a deep thought that elastic infrastructure was really important, but the ability to have API-driven infrastructure. And so for me personally, so I was at Sun, and then in 2006 I started a storage group inside of Sun, which was great. Really successful group, but so successful that it actually attracted Oracle as a customer for the first time in a long time. I kind of... this is like a little bit of residual shame that I have, that like, did I attract the marine apex predator that ate the company?
Because Oracle later bought Sun, right?
And then they bought Sun, and that closed in early 2010. I left shortly thereafter because I could see what Oracle was.
Well, I never heard a story of your potential role here.
That's right. So yeah, and Oracle... and I gave some, maybe a year later, I gave a talk in 2011 with some rather unvarnished opinions about Oracle and Larry Ellison in particular. I cautioned people about anthropomorphizing Larry Ellison. You have to treat Larry Ellison as a machine, like a lawn mower. You stick your hand in the lawn mower, it'll chop it off. Well, this is so... all right, so I'm giving this talk in 2011. Again, this is after I've left what was then Oracle. And, you know, I was just saying things that I felt were obvious, but people, you know, the audience is kind of gasping. And, you know, it's like, and people are coming up to you after the talk like, do you think there's going to be like, it's going to be retribution from Oracle? No, you're misunderstanding. Like, there's no... the lawn mower is not angry at you. It's a machine. It doesn't have the mirror neurons to be... I would almost like, it would almost show me that I'm wrong for Oracle to resent what I'm saying about them. Anyway, so but all the videos for that conference go up, and my video does not go up.
And so my colleagues were like, this is an Oracle conspiracy. I'm like, this is not an Oracle conspiracy, which it wasn't. It wasn't orchestrated by Oracle. But what I did, what I underestimated was the fear of the conference organizers. So they themselves were terrified of offending Oracle.
Yes. Even though it probably would have been fine.
No. So the talk did finally go up. Before the talk starts, there is a disclaimer: the views in this talk do not represent the views of the USENIX Association. And you're like, all right, I get it. Like, never seen this disclaimer before, but fine. Then during the talk, you know, the format of the talk is you got a slide and then you've got like a little blank script and then you got this talking head in the lower right corner. There's like kind of this dead space above the speaker. They took this disclaimer and they re-justified it and they put it above my head the entire time I'm speaking. So if you... and I mean, and maybe in this regard they were prescient because to this day if Ellison is mentioned on Hacker News or Oracle is mentioned on Hacker News, someone will immediately cite minute 33 of this talk, which is when I go on this kind of Oracle... again, I don't view it as a rant. I view it as just like me describing what is obviously true that we all know. But anyway, so I had left... I left Oracle after they bought Sun.
So we're now around like 2010. So cloud has taken off. x86 architecture is everywhere. Linux is now winning both for small-time servers but also on the cloud. And then what happens? This was an interesting time when Google started to figure out that, hey, they could do something interesting on their cloud, right?
Yeah, that's right. So this is still a little bit before that. So this is in kind of from I would say from 2010 to about 2014 is when is a period of relentless execution from AWS. AWS is executing so extremely well. There are not really other public cloud options. There's like kind of Azure kind of drifting out there.
I think people forget that, you know, like GCP on paper has been around from 2009, but up to like 2014 it was like, it was almost like a joke.
It was a joke. I would say before it was... it existed but it was a joke. And in particular at every single re:Invent, Amazon would announce a new price cut. And if you were a competitor to AWS, you were like dreading re:Invent because here comes another price cut. If you are a partner of AWS, you're dreading re:Invent because here comes the announcement of a new service that competes with what you're making.
I think people who have not been around have forgotten, but it really has happened, because it's not been the norm the last like, let's say, five, ten years or so.
Well, and in particular, they did a couple things that are just like, man, you got to tip your hat to just... I mean, Jeff Bezos is the apex predator of capitalism. Like, Larry Ellison may be the lawn mower, but Bezos is ultimately the apex predator because the thing that was so impressive is they were able to give people the idea that this was a terrible business. So, in particular, they did not break out their financials. So, everyone's like, oh my god, what an awful business. Like, they're cutting the price every year. Like, you do not want to... like, this is a, you know, a classic red ocean. It's bloody. You don't want to compete. And so, we were at Joyent. We were actually competing head-to-head with AWS.
So you were offering a public cloud.
So we ran a public cloud, and then unlike AWS, taking the software that we had used to run the public cloud and making it available for people that wanted to run a cloud on-prem on their own hardware. So people that would buy Dell or HP or Supermicro, they would buy our software and they would run it on there and get a cloud. So we ran a public cloud and we knew what the economics of a public cloud were. Namely, pretty good. Margins were good. And so what we knew that Amazon... that Amazon wasn't volunteering, but what we knew is that AWS S3 was underwriting a war on big box retail. S3 was paying for your Prime shipping. It was a genius move. And so...
Also some insider information that you had because you did your own thing.
Well, we know that the margins are very good. And then of course, I mean, we did... you will be unsurprised to learn that several of Joyent's most prominent customers were retailers. Retailers, this was not lost. Retailers are like, gee, I wonder what's happening. Retailers are like, if you think I'm going to take my dollars and spend them on AWS so AWS can... so Amazon can go to war with me, like, no thank you. There was a period of time when I felt like in order to be in the cloud, you have to implement every AWS API. So there's this idea that you had to be API-compatible with EC2. There's a company called Eucalyptus that tried to do this. It was just a disaster. And part of the reason it was thought that GCP and Azure could never compete with AWS because they could never be API-compatible. And so I am convinced that the... because what changes in like 2015? What starts in 2015? Kubernetes. And I think that part of that initial attraction to Kubernetes is that people wanted to get some optionality around their cloud and they felt locked into AWS. They're like, I'm not using all this stuff. I'm not using Elastic Beanstalk. I'm not using Greengrass. I'm not using kind of these more... I'm not using Redshift. What I actually want is this kind of basic infrastructure, and Kubernetes now gives me this layer upon which I can deploy and get some sort of true cloud neutrality. So multi-cloud didn't really exist, I would say, before Kubernetes. And I think a lot of that, especially early momentum behind Kubernetes, is around this idea of like, I need to have some optionality in here. I want to actually be able to go to GCP. So I think, you know, and I don't... I think it's giving Google slightly too much credit, but only slightly too much credit to say it is a master stroke.
On the podcast I had Kat Cosgrove, who's a release project manager on Kubernetes, and you know, she's been in the project for a long time. And I asked her... she was never a Google employee, but I asked her, why do you think Google open-sourced Kubernetes? Which, you know, they have Borg, which is amazing, and they kind of built, honestly, a better version for external, and they just released it just like that. They put a lot of work in it, and to me it didn't really compute, like, why would Google... like, what is the business reason? And she told me that she thought, again, speculation from the outside, she thought that they probably thought that it would help Google Cloud...
...to have a container which is now portable, and now you can give the promise that if you run this on Azure, especially AWS, you could come over. So it kind of makes sense. Is this your thinking?
Yeah, absolutely. But I think that is definitely the argument that Kubernetes proponents would make inside of Google in terms of like why they did it. Nobody prevented it. You know what I mean? It's like they kind of open-sourced it.
Google was a pretty cool place in the sense that it was very bottoms-up, as I understand back then still.
Yeah. And then I think part of their... you know, it was Craig McLuckie who really pushed for the CNCF, the formation of the CNCF around Kubernetes to give it kind of a foundation home. I do remember one conversation with Craig was that he and I were talking early as he's contemplating the CNCF, and he's like, well, I think this is going to allow Kubernetes to get the marketing dollars that it needs. I'm like, don't you work for the most profitable company on earth? Like, do you really... isn't it just like gushing cash over there and you can't get like, you know, a couple million bucks for marketing for this thing? But no, apparently you can't. So, but I think that the argument that people were making internally was about we should be encouraging cloud neutrality because we are the ones that have something to win, and they're right. And they did, and GCP is now not an afterthought. GCP is very important. It's a very big business, and I think that they've got... is Kubernetes to thank solely? No, but I think it's played an important role for sure.
And where are we today in terms of the hardware and the software stack running, specifically thinking of these big clouds? What's happening inside the likes of Meta, these giants? As I understand, you know, they're no longer just like, you know, ordering servers from Dell or wherever.
Never were. Never. What they do... so it's kind of funny because for all of these folks, they took a somewhat similar path. They never were, because in Google's earliest days they were assembling machines from Fry's, you know, RIP Fry's, Fry's being an electronics shop that has long since disappeared. But they were kind of famously velcroing machines together and finding...
So they bought like the processor, the different networking switch, whatever.
And they had this idea that like, it doesn't matter what junk we run on because, you know, our software is going to run as a distributed system. It actually doesn't matter. We don't need ECC-protected memory because it doesn't matter if your DIMMs fail. And so it's... I think they learned, well, it does matter a little bit if your DIMMs have rampant data corruption. Like, DIMMs failing, that's actually not a problem. DIMMs, your memory returning the wrong thing, like, that is a problem. You can actually like, you turn that... like, next thing you know, like, your software inserts that into a row into a database and like, yeah, now you got...
Yeah, that is... correctness is a problem.
Yeah. Yeah. Correctness is a problem. It's like, okay, overshot the mark. So by the time they're like, okay, we're not going to velcro machines together, we're not going... but by that point in time, you know, the business was established enough that they actually did... they built the machines that were fit for scale. So they have a great book that was written in the kind of the mid-2000s, The Datacenter as a Computer, where they talk about all the things they did, DC bus bar, really thinking about power across the entire DC. So they kind of... they went from being kind of too cheap for kind of Dell or even Supermicro to then being much better engineered than those systems ever were. So they were never really meaningful customers. And ditto for Facebook, Meta. They were never really meaningful. I mean, they kicked them out very early and did their own stuff.
Bryan just talked about how Facebook built their own servers because off-the-shelf solutions didn't work at their scale. And what's interesting is that companies like Meta and Google didn't just build better hardware. They also built incredible internal tools. Tools for safe deployments, feature flagging, experimentation, debugging, analytics, the whole stack that lets teams ship fast and with confidence. Most companies never get access to this level of infrastructure. You either build it yourself, which takes years and large engineering teams, or you make do with scattered tools that don't talk to each other. That's exactly where Statsig comes in. Statsig is our presenting partner for the season, and they give every engineering team access to the kind of tooling that only the biggest tech companies used to have internally. At its core, Statsig is a toolkit for safer deployments and experimentation. You ship a new feature to 10% of users behind a feature gate. You validate that it behaves correctly, watch the metrics, and expand to the remaining 90% only when you're confident. And if something goes wrong, you can turn it off instantly, long before it affects everyone. And safe deployments require visibility.
Static includes analytics, both product analytics and infrastructure analytics. So you can actually see what your code is doing in production: errors, performance changes, funnels, user behavior, because you cannot ship safely if you can't see what's happening. Companies like Microsoft and Notion run hundreds of experiments per quarter with Statig velocity that used to require entire platform teams to build and maintain. This used to be infrastructure available to maybe 10 or 15 tech giants. Now startups and mid-size teams use Statig to ship quickly without breaking things. If you want to give your engineering team world-class tooling from day one, go to statig.com/pragmatic. There's a generous free tier, a $50,000 starter program, and affordable enterprise plans. And now let's get back to the conversation about the history of computing and what might be coming next.
And this was independent. So like both Google and Meta both came to the conclusion of like we should just build our own stuff, and Microsoft and Amazon all came to the independent conclusion because the scale at which they needed to run was not at all the scale at which Super Micro and Dell and HP were geared. What they were geared to do was to run the servers in your server room where you needed to know the devs, right? Where it's like I'm going to have a little rack. It's going to have six servers. Then maybe it's got 12 servers. Okay, maybe we grow to 24 servers. That's what they were designed to do. If you're like, "No, I want to buy servers by the thousands because I've got a public cloud business." Like, if you want to buy servers by the thousands, there is no product from those companies for you. And in very, very basic ways, well, like the DC bus bar at every juncture, they've been designed to be a personal computer that you happen to be slapping many personal computers together, but they're not designed to actually run infrastructure at scale. So, and that was happening inside effectively all the hyperscalers. And Joyent, meanwhile, was bought by Samsung in 2016. Joyent was bought by Samsung because their cloud bill was off the charts and...
They bought, they bought Joyent to bring it in house.
Yeah. And there was not a product they could go buy. So they went to go buy a company.
So you're like, "Wow." And it's like, "Wow, that's a big AWS bill."
Yes, very big AWS bill. But then that was not a product or company that was available for, you know, the next S. What does the next Samsung do? Like, well, that's one less company available to buy. Um, so when we were contemplating the next thing in 2019, one of the things that we had seen is that, and we felt, we earnestly believe that one, cloud computing is the future of all computing, not a deep thought, that elastic infrastructure, API-driven infrastructure, that is modernity. Two, you shouldn't be able to only rent that, you should be able to buy that, own it, run it in your own data center. Why would you want to do that? Well, you might want to do that for risk management, for security, or for economics, because it, you know, if you're at a certain scale, you'd rather own it than rent it.
And I think, you know, before Oxide or like in 2019 or even in like, you know, 2020, 2021, if you were like a midsize company, you know, like not big enough to build out your own custom cloud and build everything that the hyperscalers did, you could like buy some off-the-shelf like HP or Dell, like a bunch of them. I think that's what Basecamp did. I think they posted that they bought a bunch of these things. They rented a space in one of these shared or I think two different locations. They put in their boxes with all the memory and then, you know, they kind of set it up and put it together. So I guess those were the two options, right?
Yeah, those are the two options and I think that, you know, Basecamp ended up being a real poster child for the economic advantage because I mean DHH, you know, obviously outspoken, and the economic advantage was really, really, really clear. They're also at a scale which is like not the scale that we're targeting, right? That the scale we're looking at is a much larger scale. And so the economic argument is actually even more compelling when you're at that larger scale. I love it when, you know, the VCs that passed on us because they felt there was no market then would send me like the DHH blog post. It's like, why are you sending this to me? I should be sending this to you. Like, I know this. We just knew the economics of it and we knew we couldn't predict exactly what the trends would look like but believed that there would be folks that were born on the public cloud that would outgrow the economics of the public cloud and want to go on prem.
Economics aside, what does it take to build one of these things? And I saw one of these things. We'll put in a picture of it. It's like a proper like, you know, like my 9 ft tall rack. It's big. It feels like you're putting like, I don't know, like 16 or 32 of those Dell things in terms of size just to get a sense.
Yeah. Yeah. We would have 32 compute sleds in there. That's right.
And what does it take, what did it take to actually build it? What did you need to design in terms of hardware and then software?
Yeah. So well, and we knew this too, that going into the company we knew we were taking a clean sheet of paper, right? And so we were deliberately like, no, we're going to start with a problem. We're not going to build it out of Dell, HP, Super Micro. You're going to start with a problem and how do you best solve the problem? And as it turns out, like there were a whole bunch of, there's a lot of technical debt that had been accrued by this kind of PC ecosystem. So I mean, you know, God, where do you start? Uh, just on the environmentals, like on power, right? The fact that you got AC power in each of these Dell, HP, Super Micro.
Yeah. So if you like put 16, you have like 16 separate AC...
Times two because you have two power supplies per 1U chassis. Two power supplies. By the way, there are two fans sitting on those power supplies and those fans are actually what wear out if you go to the, like in terms of like the whirring fans. It's not just coming from the computer, it's coming from the power supplies because those power supplies are dense. They're packed with stuff. So, they've got to overcome a huge amount of static pressure. So, like that's not the way anyone does it at scale. The way people do it at scale is you've got AC bus bar, you've got a power shelf that is much more efficient and that rectifies from AC to DC and then you run DC up and down and then you blind mate into that. So we knew we were going to do that.
That's a little electronics engineering right there.
Yeah. Yeah. The power engineering for sure and we knew we were going to do that. We also knew that by taking a clean sheet of paper that we would have opportunity made available to us that we weren't necessarily thinking of and that manifested pretty early. So we blind mate into power, which is to say that when you feed a sled in, that power connector you don't see it, it's at the back, you lock the sled in, blind mates into power. And we had assumed that we were going to do what Facebook and Google and others have done, Amazon done, and had networking out the front in the cold aisle. But as we were, you know, taking a clean sheet of paper, talking to some connectivity vendors, they asked us like, why are you, wait a minute, you guys are like taking a clean sheet of paper, why are you putting cabling in the front? Like why wouldn't you also blind mate in the network and the networking connection? And we were like, can you do that? They're like, oh, you can definitely do that. Like, well, why don't the hyperscalers do that? It's like, oh, they would all tell you that if they could start over today, they would blind mate the networking and they're just too afraid to do it at this point, which is like, I mean, that was like catnip for us, you know, like they're too afraid to do it. Like, okay, we got to. And one of the very early, holy God, we're going to bet the company decisions was blind mating networking because if blind mating networking doesn't work, you've got nothing. You don't have a product.
And so what is the difference in blind mating networking versus...
It means there is no cabling in the system at all. So when you've got a sled, you are blind mating into a cabled backplane. So that it's cabled in the factory. So the operator...
So when the box comes in, that's why I didn't see any cables. It's inside. It runs inside.
It runs down the back. And so...
Versus when I look at the pictures of a data center of let's say Google, you see they're very neatly organized. It's like I love organization. So it's like beautiful, but it's cables everywhere and you can see.
And in particular, so because there's no cabling, there's also no miscabling, right? So every computer is not actually on just one network. It actually needs to be on three. It's on a presence detect network. It is on a service processor network. And then it's on that high-speed network that you really care about, like the actual network. In any facility, you need another network for power, environmentals, and so on. It's very easy to have miscabling that's got to go to a different router. It's like there's a bunch of just complexity that we eliminate because we do. And then part of that decision came out of an arguably earlier bet the company decision, which was we did our own switch. So we also did, in addition to doing our own compute sled, we did our own switch.
And last time you told me about this and in our deep dive we did a little bit that like at first you said we did our own switch and I was like, yeah, okay, cool, you did your own switch. And then you told me that actually like that is a second computer to build. Can you tell me why? And it's funny because when we went through Sand Hill initially raising money for the company, nobody asked us.
Sand Hill Road. Exactly. And we were definitely, so people would be like, I've got a technical question for you and you're like, oh God, here comes switch. It's the switch question. But then some other random asked questions like, all right, that's not a very good question. But nobody was asking us about the switch. And we were concerned about the switch because we'd already come to the conclusion in order to make this thing really work, we had to do our own switch. And the reason you have to do our own switch, if we didn't do our own switch, it would be a third-party integration nightmare and we wouldn't be able to actually solve the problem that we're trying to solve, which is when this thing shows up in your data center, we want this thing to come out of the crate. We want you to wheel it up. We want you to put in power and networking and go. We do not want you to have to cable anything. It should be the level of operator involvement should be really minimal. So, we'd already come to the conclusion that in order to make this thing operable and manageable, we need to do our own switch.
And so you're saying that like buying, because a switch to me sounds like a somewhat simple component and you're going to tell me why it's not.
Oh yeah, it's definitely not. No, but that attitude is very important. If you want to go build your own switch, I encourage you to have that attitude as long as you possibly can because otherwise you won't go do it.
So what does your switch, what is switch being obviously the networking switch? What does your networking switch do or that made it so important for you to build it as opposed to like going to one of the many suppliers and saying, you know, let's get your...
So if you actually go to the actual...
Switching silicon is coming from like, it was like one and a half providers.
It's all Broadcom and so what you're actually talking about is Broadcom silicon. Um, what we discovered is this actually interesting piece of actually Intel silicon from a company they had bought called Barefoot and we found Intel Tofino which allowed us to have true programmable networking. So we use Intel Tofino. Intel later killed Tofino. So complicated relationship with Intel over this. Uh, we fortunately have procured enough Tofino to be able to take, we bought ourselves the time we need to kind of design our next-gen switch, but that programmability was very, very important for us, um, and that we were not going to get from Broadcom. It's a very proprietary company. We were not going to get a bunch of the things that we needed in building that switch, we were not going to get out of Broadcom. So it ended up being very important. We were concerned, I mean again another one of these kind of bet the company decisions, very, very concerned about having our own switch, integrating our own. And what we found is that was a win in so many dimensions. So many dimensions that we did not anticipate. And as now you can't imagine the company without having to sometimes do stuff and you might get some wins.
That I absolutely... well I think also like whenever you're deliberating something big like that, the fact that it is big kind of forces you to really deliberate and then once you commit to it, to taking that big risk, you often see unexpected dividends.
Like well as long as we're going to do this, as long as we are taking a clean sheet of paper, as long as we're doing our own switch, we can blind mate the networking. If we were not doing our own switch, we really couldn't blind mate the networking. We really needed to be able to own both sides of that in order to be able to do our own switch or blind mate.
A lot of us, you know, listeners, viewers are software engineers, so we don't know as much about hardware. Obviously, we know how the things work, but can you tell me a bit on what it actually means to design or build a computer? Because, you know, I'll give you the novice approach, which is obviously going to be wrong. But the novice approach is like, oh, here's a processor, here's a few chips, here's a mainboard, I'll just put it on there and I'm done. But when I was in your lab at Oxide, uh, you told me that one of the first engineers turned out to be a radio frequency engineer. You told me how this is great because of all the FDA approvals and all these things and I was like, okay, this is way more involved than I ever imagined.
Yeah, it's very involved.
How do you build a new computer?
First of all, it would be a lot easier if it were all slower, right? The problem is it's very fast. It's high speed. So the connection to memory via now DDR5, double data rate memory 5, is ridiculously high throughput, is very, from a signal integrity perspective, really complicated. These boards, by the way, ultimately this is all analog. We think of it as digital and it is digital, but digital is like a lie that we allow ourselves to tell ourselves. It is actually like you are talking about signals that are racing through a substrate and with a PCIe or DDR5, those signals are very complicated to lay out, that's complicated. The actual like how does the computer start, like this computer is like a 787, right? Or you know, a 747 used to be my favorite jet to kind of pick on, but now the 747 is retired so I got to pick something else and I'm not going to pick another boring aircraft. I don't think an A380, I guess, right? I should pick an Airbus. But you think about like the, okay, an Airbus doesn't just come by itself, like it needs an airport. It needs like a runway. It needs all the infrastructure to feed it. Well, so too for a microprocessor. It doesn't, like the power sequencing for those things is very complicated. It needs another surround that manages the power distribution network, that actually manages its power-on sequencing, that manages all of its environmentals, that manage its connection to memory, to I/O. So it is just fractally complicated to the point that people often just take reference designs and iterate on them. They don't actually really innovate on this stuff because it takes so long.
And you told me this was really interesting last time, that as I understand reference design means, correct me if I'm wrong, that you're an electronics engineer or hardware engineer and you want to build new hardware and you take an existing reference that has been tested, measured out, like it doesn't create accidental like all sorts of radio frequency things and then you implement that. But you told me that this is not what you did. You also told me that it's pretty hard to find electronics engineers who are used to not doing reference design but who are brave enough to like...
Who are brave. Yes. I would say that in computer design in particular, the high-speed designs are so hard. People got very accustomed to taking the reference designs and it was harder to find folks that were willing to take a clean sheet of paper and we ultimately found them. I mean, and we've got a double E team that is extraordinary.
And double E is electronics engineering, right?
And yeah, and absolutely fearless. Um, and in part because like they're actually, but they didn't spend their careers at Dell and HPE. Like they're coming from like GE Medical where they worked on CT systems.
Wow. How did that happen?
