Hello, you Ferret folks. Today we're here at Oxide Computer Company, and I have with me Brian. Hey, who are you and what do you do at Oxide?
Hey, I'm Bryan Cantrill. I'm the CTO of Oxide Computer Company. We're here in our office here in Emeryville, our office and lab, our playground here in Emeryville. We're a computer company. We're a modern computer company. We are a rack-scale computer company. So this is the Oxide rack behind you. And what we have done — our observation was, if you look at those folks deploying compute at scale, Amazon, Google, hyperscalers, Meta, they all built their own computers. And I, along with my co-founder, were at a public cloud company, Joyent, that had been purchased by Samsung.
It was interesting. And when Samsung bought the company, they were really trying to deploy at Samsung scale, and we were deployed on commodity gear. We were deployed on Dell, Supermicro, HP, some Arista, and trying to match all that stuff can be very difficult. And when we hit scale, everything broke.
And to be fair, everything broke in hardware and software. But the difference is, with the software we could actually go fix it, and we fixed a bunch of our software systems. But then the problems you're left with are those problems that are at that hardware-software boundary, and it was pretty frustrating. And you look at how do these other folks do it, and you realize they've actually done their own machines. And so, yeah, what makes — if you want to show to the audience, right? So one of these, this is an Oxide sled. And so this doesn't look like a normal COTS computer, right?
No, it doesn't. It looks like a blade.
It looks like a blade, right? And actually, if you look at the back, it even looks more like a blade. Take off the cap tape there. So yeah, it blind-mates into power first of all. So like everybody running at scale, we run a DC bus bar up and down the rack. So you've got an actual power shelf that contains rectifiers. Those rectifiers then take you from AC to DC. You run DC on the bus bar, DC 48 volt — yeah, 54 volt, yeah, okay, 54. And that is the way that everybody at scale runs with the DC. You can't buy a DC bus-bar-based rack from Dell, HP, Supermicro, and they'll tell you that nobody wants it.
Right, exactly. It's like, don't think nobody wants it. Well, the fact that this exists tells me otherwise.
Exactly, it definitely does. And you know, one of the things I really appreciated, because we kind of accreted the server architecture that we have — this traditional one-U architecture has accreted over time. And until we kind of took a clean sheet of paper, you really don't appreciate just how many things are broken with it. So yeah, I mean, one of the things that you commented on is just like the noise. It's so much quieter. And it's off camera right here, but there's one rack running right now over to my right side. You can hear it, but it's not a tiny noise. It's a very sort of almost like wind blowing noise, which is exactly what it is, right? That's exactly what it is. And you know what's funny is like we didn't design this thing to be acoustically pleasant. It just turned out like that.
It just turned out like that. And one of the things you appreciate is like when you look at the acoustic unpleasantness and — yes, you've got AC that's coming from those small fans in the back, a bunch also coming from those fans on the power supplies. You've got those AC power supplies and it's all like 40-millimeter fans. Those are tiny fans and they are — those AC power supplies, you take them apart, they're crammed, right? And so there's a high static pressure that they have to overcome. Those fans are working hard and it's hot. And of course, like that fan is the thing that blows out on AC power supply. So you have two of them. So now we've got two AC power supplies in every one of these servers, all these power cords, and it's just like the whole thing's a mess. And that's just the beginning. A DC bus bar is just like to me the most basic thing.
So speaking of — well, you say basic. The way a computer is booted is you usually start with what's known as a basic input output system, aka the BIOS. Now, in the early 2000s, that was replaced by UEFI.
UEFI, yes, Intel's gift to the world. Yeah.
And while that works perfectly fine for your average laptop or your average desktop, when you get to this scale, it doesn't make sense. Why is that?
Because it's giving you this kind of optionality that you actually don't want, right? Like actually, when you have something of this scale — and we have co-designed our host operating system with our hardware — you don't need that optionality of booting kind of — I don't need to boot DOS on this thing, right? I don't — you don't want DOS on 16 different machines. Okay, that would be kind of fun. But we actually don't need any of that. But we have preserved all of this kind of ancient optionality in the BIOS. A big problem that we had with the BIOS is that the BIOS has to boot the system in order to boot the system. So one of the things that the BIOS has to do is it has to like power — because it needs to find like, how do I boot this thing? I need to actually do I/O to pull a boot image off of somewhere. Well, it's like I/O as we know, like everything's complicated, right? It's just like, well, you can't just do I/O. Like we actually have to bring up PCIe engines, you bring up all the CPUs. So you're doing all this work to boot the system, and then you find the image that you want to boot, and now you have to like, okay, we have to pretend like we were never here. So it then tries to actually set what we call sending the machine backwards, where it makes the machine look like it has not been booted when it executes that first operating system instruction. But in reality, an entire city has been constructed and plowed under, and that operating system can actually see the artifacts of that over time. There's something called system management mode, SMM.
Ah, yes, yeah, right. And SMM, what some people refer to as ring minus two, if I remember correctly.
That's right, ring minus two. And that kind of platform initialization layer can stuff whatever it wants into SMM. I still find it hilarious that what I think it was HP tried putting like a day calendar in there when you try to boot the BIOS. I was just like, why are you doing this? And it's like they're doing it because they wanted to add value to their hardware without controlling the system software. So the way to do that is to like jam that software into the system software they do control, which is SMM. But from the perspective of actually running this thing as a server, it's like that's just a problem for me. I don't want to have ring minus two. So for us, SMM is empty. Because the other thing is like, why do you end up in SMM for any reason, right? I mean, if you look at the architecture manual, it's like it can go into SMM for like any reason, can stay there for any length of time, unspecified, actually. So yeah.
So for us, if we do have something in SMM mode — if you ever hit SMM mode, we panic the system because under no condition should we enter SMM. So if we enter system management mode, we bring the system down and take a crash dump. We haven't done that — it would be pretty wild if we ever saw that happen, right? Because it would mean that they — but we have not seen that happen. And we wanted to do that to make sure that if something were to errantly leave in our system management mode — but we don't use system management mode at all. We also didn't want to have a BIOS at all.
Yep. So how are you getting around that?
Yeah, so that was tough. In fact, this is something that we didn't really appreciate at the time. AMD didn't think we could pull this off. Apparently, Google tried this and failed. And if Google has tried something and failed, does this be impossible for humanity?
Well, oftentimes what they do is they succeed and then claim that they failed and then just cancel the product.
Exactly, that's right. So, but it was tough and it required us to work very closely with AMD. I think AMD didn't think we could pull it off. It's far — I would even say not a trivial — it's a very complicated problem.
