Thank you. It's really great to be here at Jane Street, or sit back at Jane Street. I gave a talk here in 2018, actually, that I really appreciated the opportunity to give. I actually referred to that talk somewhat recently, because there's a little bit that I regret about that talk. I generally live without regrets. Yes, was it supposed to be a joke? But I guess it was. I guess that's obvious that I live without regrets, given some of the things that people think. I should also say there was a long period of my career that predates really internet video. That's when I was really loose. What you're actually seeing is a much more buttoned-down, wiser, older person.
In that talk in 2018, after the video was over, there was a really great question. The talk was about all of the accreted and accrued complexity we had in our systems, and I don't remember who the questioner was, but I still feel guilty because the questioner asked, "Is there anything that is going to resolve this complexity explosion? It seems, again, this is in 2018, things are just hopelessly more complicated." And I remember just being like, "We're just—it's basically we should all just jump in a lake." I mean, it was very... So I would like to apologize to that questioner because I actually think that that's a lazy answer. And we actually can improve things. We can actually make better systems. And that's something of the topic we're going to talk about today.
And I really—I hate to do this because God, this dude really gets my goat. You know, you would think like—so this is an excerpt from an essay that Marc Andreessen, venture capitalist, wrote in 2011 on why software is eating the world. This thing—so this is written in 2011, so this is now, you know, like almost 15 years ago. This thing has been trolling me for a decade and a half, like this thing won't stop trolling me. And the thing that just drives me absolutely crazy about these VC think pieces is that they're not exclusively wrong. And they're—but they're definitely not right. And venture capitalists seem to have this unique capability, at least for me, to say things that I both emphatically agree with and emphatically disagree with, often in like the same sentence.
And it's actually—which is kind of interesting. It gives you this like kind of crazy emotional reaction. It's honestly kind of the emotional reaction you get when you're raising money in general for a company, because you will get both like absolute stern rejection and total embrace in the same moments from different people. So your brain crosswires itself. And this is what these think pieces are able to do, and this one definitely does this to me. And in this regard, I'm going to do a quick aside. So I want to talk a little bit about the thermal grill illusion, because you'll see where I'm going with this metaphor here. So if you've ever been out to the Exploratorium in San Francisco—I'm sure this is true for other science museums as well, but I know that this is where I discovered this particular phenomenon, where I saw this phenomenon—there's a great exhibit kind of off to the side of the Exploratorium, and there are two coils, and one coil contains— is warm, not hot, warm, about 40 degrees C. 40 degrees C is only slightly more elevated than your body temperature; it feels warm, not hot. And on the other coil, it's cool, not cold, but cool. And the coils then intertwine.
And when you grab the combined—it's a wild phenomenon. When you put your hand on the interspersed warm and cool coils, your brain explodes, and it immediately—because your brain, it is wild, your brain is like, "Get your hand off this thing." Like it feels like you've just touched a hot stove. And this is what's called the thermal grill illusion. It was actually discovered by a physician, Torsten Thunberg, in 1896. I really wanted to get like the original paper, even though it's in Swedish. He's got like a diagram of this thing. And man, I spent so much goddamn time today trying to convince an LLM to like get this thing for me. And it's like, you know, they're like, "Oh, well, if you want this, you should do like an interlibrary loan." Because it's like, I know you trained on this thing. Like, don't just—just give it to me. Like, you already stole it. You've already done the wrong thing. Like, I'm just asking. And anyway, it was—and then it's like it feels like super weird to be trying to talk an LLM into like arguable crimes, because it—I definitely didn't feel guilty at all. I felt annoyed. I'm just like, I just like, come on, give it to me. But I'm not going to do an interlibrary loan from like the Royal Academy of Sweden. Thank you very little.
But discovered in 1896, and so this kind of wild phenomenon—this actually excerpt, which Gemini tried to offer up as the original 1896 paper, by the way. I'm like, don't think that one's from 1896, this like much-lauded Gemini 3 Pro model or whatever. But so this is actually from a much more recent paper in 2019 trying to study this effect, because it's actually still unknown what causes this. And this is where they've tried to study it much more precisely with these kind of interspersed—you've got hot and cold plates, and they're trying to look at the actual like neurological reaction, because they actually don't understand it. So I think like this is obviously a much more scientific way to do it. I think you could just have me read VC think pieces, and you would get the exact same reaction. Like whatever is going on in the brain is the exact same thing, where the brain is like, yes, but also emphatically no, and just like, get away, get it away from me. But unlike with this, I actually am now just like mesmerized by these VC think pieces, and I keep having to go back to it. So I've been back to this essay so many goddamn times. I've spent more time getting trolled by it than Andre ever spent writing it, for sure. So I mean, I guess jokes on me, but so we're going to go through it a little bit.
And so, you know, this line at the top here is kind of the famous line from it, that seeing more and more Silicon Valley-style entrepreneurial technology companies that are going to be invading and overturning established industry structures. Again, this is in 2011. So when he has predictions over the next 10 years, you can kind of hold them to it, which we're going to do here in a little bit. So this is actually the thing that I think is actually kind of like most interesting from the piece in hindsight now, from again 2011, because this is looking back to 2000—2000, much more recent than when this piece was written than 2011 is for us today here in late 2025—and looking at the cost on LoudCloud, which was the failed startup that he had with Ben Horowitz, where just to host a basic internet application was 150k a month. And now that same infrastructure in Amazon in 2011 is $1,500 a month, to which many people with AWS bills are going to be like, "Wow, how quaint. You were able to do anything for only $1,500 a month in 2011." It feels like AWS bills are actually quite a bit higher today for a bunch of reasons. But this is a huge delta, right? Right. And I think that this is something that Andre points out that certainly was revolutionary, is that cloud computing, elastic infrastructure made available—infrastructure that could be software-defined, made available to anyone with a very low cost of entry—and that did allow for an explosion of kind of software-as-a-service kind of offering. So that is all not wrong. This is basically the best part of the piece as far as I'm concerned. That's definitely right.
Then we get to these, and I like—if we're going to talk about this piece, we have to talk about some specific paragraphs. I'm sorry. So, photography, of course, was eaten by software long ago. And this is like one of these you're like, you know, you're like, okay, so like, do words mean anything now? Like, what does that mean in 2011 to say that photography was eaten by software long ago? I don't even know what that actually means. But so yes, okay, we have mobile phones for sure, and we have cameras. Software-powered camera is like—that's—we're beginning to stretch things here, because there's a lot of hardware obviously involved in that camera, much more so, by the way, in 2025 than there was in 2011. I mean, if you look at the quality—and I actually went back to some of the photos that I took on a camera phone in 2011, and they look like a Kodak Chrome, a Kodak 110 camera that I used to have as a kid in the 80s. It's like they're pretty grainy snapshots. And I mean, the photography on phones right now is extraordinary. But it's not just because of software. It's also not just because of hardware, right? That if you look at the history of computational photography since 2011, which, by the way, it wasn't eaten by software long ago, there was a lot of software-hardware co-design as part of that. And who was the beneficiary of that? Well, first of all, we were the beneficiary of that. When you go take a photo of your cats or your kids or what have you, you're able to whip your camera out very quickly. We're all walking around with cameras all the time. I think that is wildly revolutionary, something that was inconceivable, say, a century ago. And I think it's going to have long-term—I think there's so few photos of—if you know, if you're a Gen Xer, you're just—and especially if you're—I'm the firstborn, okay, so I actually got a baby book. My sister, a Gen Xer as well, is just like, did I even exist before like my college graduation? It's just like nothing. I mean, just like no baby book, nothing. And now we document our lives so thoroughly in part because of this revolution, which is pretty interesting. Okay, okay. So Marc, and we're going to like—okay, we're—you know, you're—and then okay, so what are the companies that are these companies that we should look to as the software companies that are going to be eating the world in this coming revolution of computational photography? Oh, there it's Shutterfly, Snapfish, and Flickr. And again, if you're me and you're like immunocompromised when you read this stuff, you immediately are like spending, you know, 35 minutes in a Shutterfly rabbit hole figuring out like which PE shop bought Shutterfly to chop it up. You're like, where am I right now? It's like, why am I? But of course, like Shutterfly, Snapfish, and Flickr are all—I mean, they're not even like trivia questions. I mean, these are companies that are, yes, they exist or they did exist, but they are not the—they were not the innovators. They were not the leaders. I mean, this paragraph is basically like wrong. Sorry. Then I'm not done. I'm so sorry. You're just like, are we going to do this the whole time? Like, not the whole time. But you know, Henry did tell me like I got—you know—that I can maybe—maybe—well, so that now I just educated you about Snapfish and Flickr. We're going to learn more about defunct companies here. So today's largest direct marketing platform is a software company, Google. This is like one of these things where you're just like, you just get to make up words. I guess that like if a company writes software, that's a software company. It's like, it's actually okay. All right. Which is whatever, fine. Obviously, software extremely important, like that's no one's disputing that. So now it's been joined by Groupon, LivingSocial, Foursquare, and others. Who are the others? They got to be just like—I mean, who—sorry, I mean like above—these are above the fold. Who's below the fold for Groupon, LivingSocial, and Foursquare? I mean, this is a cemetery. And they were using software to eat the retail marketing industry. It's like definitely like eating was taking place. I'm not sure who ate whom or what ate what. I think the creditors ate at least some of those companies. And I think we've been generating over $700 million revenue in top line and bottom line. A lot of education, a lot of economic education that can be had here. After being in business for only two years. I mean, that's—well, there's an irony in that sentence for sure. But again, these companies were easy come, easy go kind of companies, right? They were companies that didn't exist and then exploded and then like broadly disappeared. By the way, there's—and I do, I guess, have some self-control because I didn't snip the paragraph on gaming, which points to Zynga and Rovio as the—so at the expense, by the way, of Electronic Arts and Nintendo. And it's like, well, you would have actually been much better off buying Electronic Arts and being an LP in a16z. But the—oh, my apologies to any LPs in a16z. And then, all right, so what's next? So this is the past, where we, you know, these titans of industry: Groupon, LivingSocial, Foursquare, Rovio, Zynga, and so on. And don't forget like Snapfish and, you know, the Snapfish, Flickr, and Shutterfly. Okay, so that's how we got to the current titans of industry. What is to the future? And this is where you just get to like, okay, are we going to revisit this one? That healthcare and education are the next—that's where the next Zynga, Rovio, and Foursquare is going to come from, healthcare and education. I would just be curious. I'm like, no, like, just what happened here? I mean, is there one of these? I actually don't even know that there is. Actually, and this is where I do, again, I've got some self-control. My venture capital firm is backing aggressive startups in both these. I'm like, oh god, I would love to go find—but you got to go find their zeros. It's like, you know, it can be a little bit. So these industries, I would say you have not seen it. Now, which is not to say that like software is not very important in healthcare, obviously, and education. In education, I actually think the most important educational product that was developed is actually like the Chromebook. Honestly, because the Chromebook put the—and we don't really talk about the Chromebook, I think, very much, but if you got kids, you know, your kids have got Chromebooks. Chromebooks are like pretty ubiquitous. They are amazingly cheap. They're very actually surprisingly robust, considering the way I've seen them jammed into backpacks and so on. And they are actually—I mean, one kind of tragedy of the Chromebook is that it's so effectively secure that the kids don't really have the ability to kind of take it apart, because the thing is actually built pretty well in that regard. But the Chromebook really revolutionized education, and certainly during the pandemic, Chromebooks were very load-bearing, right? A lot of kids learning during the pandemic from Chromebooks. So I think in some regards it is actually a hardware-software product that was actually much more revolutionary in education. And then healthcare—I mean, this is what I just have—like healthcare, of course, has been revolutionized over and over and over again by software, software-hardware systems, right? I mean, if you—I mean, god forbid if you have to present to an emergency room tonight with abdominal pain, you will likely go into a CT, right? You will—computed tomography revolutionized healthcare, as did MRI, as did NMR, as did ultrasound. Very, very important technologies. I guess I'm not sure if that qualifies for having been eaten like photography was eaten, but I don't know. I guess we just kind of make things up when we're a VC think piece. All right. So, what's the problem here? So the problem, other than my own me being immunocompromised around VC think pieces, is the problem is that it is extremely reductive. And it's reductive for like understandable reasons. It's like, I get it. Like you're a VC. It's like bacteria. You kind of do what VCs do. And in particular, you conflate innovation with new company formation. This kind of drives me nuts. If you—I mean, you all work for an established company. You know, there is a lot of innovation that happens at established companies. News flash. And yes, there's innovation that happens in startups, but actually when you are new company formation, you got a whole bunch of other things that you're trying to do at the same time. It's actually very hard to be truly innovative in terms of certainly technologically innovative with new company formation. A lot of our deepest innovation actually comes from established companies. So like the fundamental false dichotomy is this kind of conflation of innovation with new company formation, and also innovation with like economic disruption, right? The only real innovation is actually something that is like economically disruptive, that's going to like, you know, assuage regulators or, you know, what have you. It's like that's—if it doesn't move fast and break things, it's not innovative. It's like, well, that's obviously. And our most important innovations actually don't actually do that. They abide by the constraints of the system and do really interesting things. But anyway, he doesn't see that, but of course, like whatever, he's a venture capitalist. Like seriously, what do you expect? That's fine. The thing that then you also get this kind of conflation with software consumption versus software development. Like just because you use a spreadsheet does not mean that you are now a software company, right? So there's that—that's a little bit ridiculous. And then of course the conflation between software development and software companies. Like because software is very core to what you do, does that make you a software company? I mean, I'm not even sure it matters, right? Like what does it mean to even be a software company? I mean, to me, what it means to be a software company is that your product is software. And that's actually like pretty limited and not honestly that interesting to me. Like the true folly here, though, is this idea that software and hardware are totally disjoint. And that essay, I think, over and over and over again kind of implicitly denigrates hardware. Hardware is something that is—it is low margin. It is commodity. There's like a bunch of kind of ways that hardware is actually denigrated, unless it's changing your life with the phone, in which case software ate it, apparently. But there's this kind of idea that like of the primacy of software, that software is what is our true value in innovation, and the physicality of it is actually irrelevant. And I think that that is really problematic. I think it was very wrong when it was written in 2011. Right? So in 2011, if you could go back to a time machine in 2011, like you're probably going to go like buy some Nvidia, some TSMC, some AMD, right? I mean, some Apple, companies that are very much in the physical world, but companies for whom software is extremely important. And like so the—and no one's disputing that, by the way, that software is very important. We know that software is important, extremely important, but it's the false dichotomy that is the problem with Andreessen's piece. All right, so this is going to require us, and I'm going to apologize in advance for this. We do need to think about just for a second of like what even is software, and be careful on this one. I'm kind of like—I'll tell you that I was long ago—I was interviewing Arthur Whitney. And you may have heard of Arthur Whitney. Arthur Whitney is a really interesting technologist. Worked for Morgan Stanley. Developed a language called K. Kx Systems. Wild stuff. So I was actually interviewing Arthur. Apparently Arthur has given no interviews since this interview in 2009, which was kind of wild. And I'm wondering like maybe that affected him more than it affected me. But Arthur is the kind of guy—so Arthur in particular, among other things. So Arthur was taught APL by Ken Iverson. He was like—so when he was like 11. And at the time I had a four-year-old, and he wanted to teach my four-year-old K, which is—some would say that's like—it's kind of child abuse. I would—I mean, I knew—like the difference is like I actually know my four-year-old, and like, good luck with that. First of all, any four-year-old is a challenge no matter what. I'm like, "This kid is—I don't think this kid's going to be very interested in K, as it turns out." But I admired Arthur's optimism. He's like, "Let's teach him K." I'm like, "Let's actually just get him to put on his [__] shoes." So, you know, the different goals there. But in talking to Arthur, you know, Arthur again is so bright and thinks about things so differently, it kind of causes your brain to blow up. And at one point I did—we were having this conversation like, what is software? Because to me this is like—and I feel very lucky that we are kind of where we are at this kind of time in human history, because software is that important. I mean, software has this extraordinary staying power. I mean, I disagree obviously with the false dichotomy of Andreessen, but software really is marvelous. I mean, it is amazing because it has properties of both information and of machine, and you can look at it either way because it's actually truly both. And there's so much that comes from that. The cost of goods sold of software is zero. It's knowledge, knowledge that we run as machine, and that leads to all sorts of like wild things. So I was curious what Arthur thought about this, like what is—and what is your analog for software? And I think that there's like lots of analogs to be had. I think ultimately the answer is like software kind of defies it because it is so different. But I was asking his analog and thought that, you know, sometimes people will say like software is like a bridge, you know, it's—you were building a bridge, like a big civil engineering engagement. Some maybe it's like a biological system, like I don't know. So I asked him this question like, what is the analog for software for you? And he kind of pauses and says, poetry. And I was like, you know, kind of—and the way—I mean, he exudes such craft in the way—I mean, for Arthur, the beauty of code is how few characters it contains. I'm just putting it out there. I'm not casting judgment on it. I'm saying that I'm just like—just letting you know like that is Arthur's view, and there is a kind of aesthetic to that that you have to admire. And he's definitely like lived this. I mean, so he's very consistent in that regard. But this whole thing like really had me thinking of like what even—I'm asking this question about the analog for software, like what is software? Because you think like, what is software? Well, software is the instructions that you run on a machine. Like, okay, so does the—so if I get rid of the hardware, does the software still exist? It's like, well, I think so, right? I mean, I think we would acknowledge that the software exists in a vacuum. Does software have to be Turing complete? Like, I don't think so. If you say that software has to be Turing complete to be software, there are lots of things that look and feel like software that you're not going to acknowledge as software. And ultimately this—maybe it's the mutability. Like software is mutable, hardware is not mutable. The problem is we've got software that we actually have in the mask ROM on a microcontroller that is immutable, that is definitely software, and we also have an FPGA that is hardware that is mutable. Is an FPGA hardware or software? And I think any kind of like discussion on this will ultimately get you to the conclusion that like Euclid was writing software. I don't know how you can—it's very hard to have a definition of software that admits the things that we all agree is software without letting you sneak in the back from several thousand years ago with a GCD algorithm. Right? And software. I mean, this is kind of the amazing thing about software is this kind of timelessness of software. And I think actually to go back at the top in terms of like why was my answer to that question kind of fundamentally wrong, because in software when we do innovate, we can actually—that innovation can carry in perpetuity. There is software that you deal with in some capacity that will survive you. That's kind of wild. That's not true of anything else. The software will long survive the hardware that it's on. Like that's pretty wild. Like that's pretty interesting that the software will long survive the hardware. When I was—I remember during Y2K, after the Y2K rollover, I was actually in New York City talking to a very—to the CIO of a big bank, and we were talking about Y2K, what Y2K meant for the bank. They had many more Y2K issues than we had at Sun. At Sun, our Y2K issues, about a third of them—this is actually—as dumb as it sounds—a third of them were like leap year problems. And you're like, look, I'm like a Gen Z or a millennial, so I like—I barely remember the year 2000, but like wasn't that like a number of digits in the year problems? Like, yeah, yeah, that was a number of digits in the year problem. But it turns out there was another totally different Y2K problem. So every four years is a leap year, right? Except for every hundred. Except for every 400. And a rem. So 2000—if like if you're dumb, it's like, you know, the old like bell curve meme. If you're just like an idiot that like every four years is a leap year, you're fine. Your software is good. And if you're like a brainiac, you're like, well, brainiac, your software is actually broken because you've encoded amazingly. There's a decent amount of software that did this, that encoded the rules one and two, but not rule three. And the way they did this is because they took an offset from 1900, because there's a lot of software that starts as an offset 1900. So if you take that offset and you mod that by—you're like, oh, great, okay, this, aha, I'm clever, it's not a leap year. It's like, no, it is a leap year, just like, can you do me be like the dummy over there. So those were our problems. There was actually a bug in the at command. You would run the at command is—yes. Sorry, we're here, I guess. The at command is a Unix command that allows you to run a job of the future. And if you did at plus—there, at is very strange. You can give it the syntax of at plus and then give it like n days, or you can give it a unit of time. It's also like, where are we? Like this is Unix, like this is like weird Unix. You get this is like the B-sides of Unix. Some super weird Unix. And so you could give it at plus a number month. It's like, what does that mean? So he's so like, well, if you're on the, you know, the 15th of the month, then you want to do at plus one month, it'll run on the 15th of the month. It's like, that's great. I hate to be that guy, but there are days that exist in this month that don't exist in that month. So like, what do you do? Like, you know, what kind of sociopath is going to run something on at plus one month when it's the 30th or the 31st of the month? And so in particular, we had a bug where if you were targeting February 29th, 2000—remember, 2000, the 2000 is a leap year, right? So February 29th very much does exist—that software would run that on March 1st. By the time we discovered this bug—this is only with this plus month syntax—by the time we discovered this bug, four of these days had already passed. There were only so many left in 1999. But people were so gripped by fear over Y2K. It really is like this is where I know I feel like I'm down in the Gen X home telling you tales of the way it used to be. But people were truly gripped by fear. Whenever you see people like—whenever there's a mass hysteria, and there's like lots of mass hysteria about lots of things right now, it's always useful to go back some of the past mass hysteria. And people were very, very afraid that the world was going to stop on January 1st, 2000, which would have been super entertaining because it would have started with New Zealand. Part of me kind of wanted this because you would have this thing where like New Zealand bursts into flames, like the—the no one has heard from New Zealand. There's just like a thermonuclear explosion, and then you're like, okay, now the rest of the world, there's like this doom that's going around the globe, and Hawaii is like, we might live, but we only have—but that didn't happen. So it actually—it turns out like nothing really happened because we spent a lot of time praying for photography. So I'm at the bank, like, where are we? I'm at the bank. I'm asking, and we're talking Y2K, and the CIO says, "You know, it's funny. Actually, 45% of our Y2K problems at this very large bank came from a single hardware platform." So you can imagine I was at the edge of my seat. That guy could have done anything. He could have been like, "And I will tell you that next week." I mean, I would have been—I was at the edge of my seat. Like, what platform? The IBM 1401. And that was actually like a true brain-bending moment in terms—it's like IBM 1401. So IBM 1401, in case all the numbers blur together, which would be very reasonable. The IBM 1401 is a computer so old you barely call it a computer. It's an accounting machine. It is a stored-program computer. You should go to the Computer History Museum in Mountain View. You can go see an IBM 1401. They've got one that's restored. You should go watch—go to the CHM if you ever get out to California. Like the reason to go to California—like, look, I know I'm in a room of nerds. I'm not going to—the reason to go to California is to go to the computer museum in Mountain View and look at the card punch they have on the IBM 1401. That thing is space age, and it was made in 1949. It is an absolute mechanical marvel. So that's how old the 1401 is. 1401, that's the—that's the punch from the 1401. The 1401 dates from the 50s. This is a machine that's basically in the 50s, not the 60s, early 60s. They did a transistorized 1401. But you're just like, how, what? So this is a computer that was old in the early 60s, and how are you running software in production? Well, as you can imagine, it was software that was running on an emulated hardware platform that was running on an emulated hardware platform that's running on an emulated hardware platform, and so on. So on the one hand, like that is amazing, and that is this amazing power of software, but it does get to like, what's the software and what's the hardware in that system? It's running on virtual hardware that's running on virtual hardware that's running on virtual hardware. It's like, and does it matter? I mean, that's the other thing. It's like, is all of this like a distinction without a difference? I think like a lot of this is—we know that hardware is important, and we can talk about what that is. We know that software is important, but I think what's actually important is not thinking about how they're different, but thinking about how we use them together to develop a system. So I think it is—it's kind of a waste of time to try to really distinguish those. So at Oxide, we very much set out with the idea of doing hardware-software co-design. So what we wanted to do—we—and this came from our own experiences trying to run on commodity machines and having no end of problems. And the problems that were in our own software, you can go fix those. The problems that we couldn't fix were the ones that were deeper in the stack. And we—I had a storage product at Sun back in the day. We had written a bunch of software. Every single piece of firmware that we did not control from many different vendors at one time or another was completely debilitating to us in like increasingly comical ways. It was almost like the gods wanted to like really make a point that like the way these systems can fail because they start operating at such cross purposes, and those were the problems that we couldn't do anything about. So when we started Oxide, we felt strongly that we needed to do our own board design. Actually, what I felt really strongly about is getting rid of the BMC. The BMC is the baseboard management controller. It's the computer within a computer on a server, and it's this kind of like garbage that—it's the thing that has like the VGA port out the side, and it's nothing but problems. Also, I've got resentment because it has my initials, the BMC. So certainly back at Sun, the BMC caused no end of problems, and I would have these subject lines that would give me a fight-or-flight reaction because that, you know, BMC induces panic, or it's like, what? No, no, no. So no, I'm kind of like my Michael Bolton there, but I wanted to get rid of the BMC and in particular scale it down, make it something that is much more comprehensible and software that we could control. We wanted to do our own board design, our own rack design. We did our own switch design, and importantly, all of our own software from the lowest layers all the way up. Well, with an asterisk on it. All of the software that we could do. What's amazing is for all the software we've done, there's still software that's out of our reach that we rely on other people to do. And that software is still very, very proprietary, which is a huge problem because what we have found is—and I thought this going in, but have absolutely seen it now having an actual like shipped working product and so on—the ability to actually co-engineer, co-design the software and the hardware allows you to do some really unique things that are really important. So we went into this with the being—that was kind of the belief. We knew we had to do our own compute sled. If I could go back in time and slap myself—I mean, I do it over several things. One of them would be over the fact that I thought we were going to tweak reference designs. Like I said that several times, like, oh, we will just tweak a design. You do not tweak a design. By the time we were getting rid of the BMC and replacing it with our own service processor, you are designing your own board. But like, this should be a possible thing, right? It's like, we're not making our own silicon. We're taking like—we're taking these components, and yes, it's a complicated PCB, and yes, the layout is complicated. Yes, yes, yes. But like, this is a solvable problem, right? And when you go do that, you are able to actually go do a bunch of very important things. One of the things that we did in terms of our own software—so we did our own switch, which we felt that we needed to do. We were—I was a little bit concerned when we first raised that a venture capitalist would ask us like, what are you doing about the switch? Because it's like, well, we kind of think we need to do our own switch, which I know feels like very ambitious now. We want to do not just our own compute sled design, but our actual—our own switch design. Fortunately, I overestimated VCs, who never asked us that. So, and actually, I came horn—I, you know, I would have this like fight-or-flight reaction when someone says, I have a technical question for you. And I used to think like, oh god, like here it comes, here comes the technical question. And if you—if any of you were to say that, I would have exactly that reaction. It's like, okay, here it comes, here comes a really tough question. But that—no, no. When a VC says, I've got a technical question, it's like, I got a question that I think is pretty smart, and I would like you to validate that it's a smart question for me, please, which is a much easier task. Honestly, it's a much easier task. Well, it's mixed. It's a much easier task if it's a good question. When it's a moronic question, you really have to be like, okay, let's pretend that this is a really insightful question. So there was that. But what I was really braced for is like, oh my god, they're going to ask us about the switch. And they never did. It's like, okay, great. In fact, they would also say things like, you know, technically we know you can do this. And I'm like, I am not explaining what we're doing very well because technically this is—I am so far off the end of my skis on this one. Like, are you out of your mind? Like, the amount that we're taking on—like, no, no, like, sorry, there's like—this is loaded with technical risk. But that's obviously the things you're thinking, not saying to you, like, hey, let me stop you there, pal. There is lots of risk in this thing. You're like, okay, am I—you want me to put money into it? Like, yes, yes, put it on the bonfire. You don't want to do that. So like, okay, but I should know like, oh boy, there's a but coming. And of course, the but was like, we don't think there's a market there. It's like, it's like, you've got it exactly backwards. It's technically so challenging that I'm not sure we will ever get a machine that actually works. But if we can, the market is huge, right? And that's like—you can't really explain that. So we didn't know we were going to do—we had not confided. Nobody asked us about the switch until, of course, as you might imagine, someone not unlike yourself, coming from a company that had deployed a lot of compute. A terrific technologist said, "Bryan, you did not preface this without—got a technical question for you," which is of course like, why would you? "Bryan, what are you doing about the switch?" Like, damn it, I almost made it. And we were seeking an angel investment from him. I'm like, look, I know you're not going to want to invest in the company when I say this, but we think we need to do our own switch. I, you know, I'm sorry to—we have died. I'm sorry to tell you this. And he's like, okay, if you're doing your own switch, I'm in. I'm like, okay. And as he said, we did not stand on our own two feet until we kicked Juniper out. And I'm like, oh, okay, interesting. So we've done all this now, and our like—we—it was really hard to do our own switch, but now that we've done it and we've got it working, there are so many things that we can kind of uniquely do. It's really uniquely powerful. And I think like the biggest challenge of doing a lot of this stuff was getting over the fear of doing it, and like once you tack into it, it was all a bunch of things. And we—so it's kind of worthwhile like looking back on these things. And you know, in terms of the things we've done, we—so we eliminated the BMC. Yay. We did our own operating system, appropriately enough, called Hubris. Also, the debugger for Hubris is appropriately enough called Humility. I did the debugger. I think that you could observe that there's actually a lot more humility in Hubris and a lot more hubris in Humility, but that gets complicated. We eliminated the switch operating systems. We did our own switch. We eliminated the UEFI BIOS. That's actually—for all of the things we did, actually the riskiest thing that we did was the elimination of the UEFI BIOS. And so the BIOS is the basic input output system. It's very old, dates from CP/M. UEFI is a stowaway from Itanium. Amazingly enough, it's like it managed to leap the species barrier, and now we've got UEFI. It's like the Wuhan bat or what have you. So the—and this UEFI BIOS is this bootloader that exists at the very lowest layers of all x86 systems, and sadly ARM systems as well, because ARM—there was this moment in history, and if you were like alive and awake during this, you had the same frustration, where ARM is like, we need to be like x86, and you're like, where are you going with that? We're going to adopt a UEFI BIOS. Like, no, no, not you, ARM. That's the whole reason we like ARM is you don't have a BIOS. Like, we need to be successful like they are. You're like, oh god, so they have a UEFI BIOS. Like, oh god. And the problem with this layer of software—it is this very, very lowest layer of software, very much software, and it does—it needs to boot the system effectively, but it's a very complicated system that it needs to go boot. It needs to go find an image to go boot. How does it do that? Well, it actually needs to like bring up the system to boot the system. So there is this—there very much this catch-22, and there's an entire world of complexity in these things, and but it's not written in a very modern way. That's putting it euphemistically. One of my colleagues, Josh Coombs, calls UEFI MS-DOS circa 2099, which is kind of what it is. It's like futuristic DOS. It's very weird about the way the software is structured, but you have this very like—this deep kind of platform enablement layer, and then it finds the thing that wants to go boot, and then it's like, okay, I got—we got to like put everything back. So it basically sends the system backwards, and so when the operating system boots, the operating system actually is on a system that has already booted. And this stuff has real problems. That there's a bunch of—I mean, you've got all the problems in terms of like boot time and so on, but like this software has vulnerabilities in it, real vulnerabilities, deep stuff. And if you are able to break in to the BIOS, you control the brain stem of the system. You control everything. So the—we wanted to get rid of this layer entirely, and that was—I mean, it's a sad state of the server industry when it came along. People were like, impossible. You—
Absolutely can't do it. Like, there's no way you can do it. Apparently Google had tried and failed, and so anything that Google has tried and failed is like tautologically impossible for humanity. A little bit annoying. But this is what we want. And so we actually eliminated this, and what we did we don't feel is very revolutionary from an abstraction perspective. We actually control the first instruction and then we boot the system. That's what we do. So there is no bootloader because we are actually in that SPI payload that is actually that when you come out of reset, that's what you actually pull. We're in that payload, and then there's enough in that payload to pull the rest of the system in, but it's not booting something else. And we are holistically booting all the way up. We never send the system backwards. What this has meant is that we have to do a bunch of very, very, very low-level initialization.
So we bring up the PCIe engines. We bring up obviously multiple cores. We do a bunch of things. We don't have to do DIMM training, fortunately. So DDR, double data rate memory, needs to be trained. This is this very like we're now into like the true dark arts. So DDR is this highly, highly, highly parallel noisy interface that is very high speed, and it needs to be effectively tuned every time you boot. It needs to be trained to figure out what the right parameters are so you don't lose data. That seems nice. It's like, okay, you do your training. Sounds good. Sounds important. That is done by on AMD. That's actually done by there are hidden cores on the AMD package, and that's done by what's called the PSP on AMD that actually does the DIMM training. So we don't have to do that. When you come out of when you execute first instruction, DIMMs are trained, but nothing else is brought up. So we engage in that lowest layer of platform enablement.
It's like, well, doesn't that make you like very welded onto AMD? I mean, yeah, by design we have this like different approach because right now in the server industry people want a commoditized world, so the world becomes commoditized to serve them. It's like you effectively have garbage to build on because you've selected for garbage. If you want to have something that actually works, you need to integrate it much more tightly together. And that means like making choices. So, we made the choice in 2019 that it's like, no, I think it's going to be AMD, not Intel. And there were people who were like, no way. It's like, yeah, this one's easy. Actually, this one is easier than you might think. In 2019, it was very clear that the house was on fire and Intel and that Intel was going to be in the situation that they are now in if they didn't course correct, which they broadly haven't. But so we are very tightly mated onto the parts that we select, which has been really important for us.
So it's like it's kind of reasonable, like okay, so we've got this whole thing working, which is great. We've got it, we've shipped it, we've got our next generation sled that is working, we brought up, we're shipping that, it's great, we've got systems in the field. How do we feel about all this? How do we feel about having taken all this on? And the thing that's kind of amazing is you look back, we don't regret a single one of those decisions. And in fact, the decision to not have a BIOS, which I kind of felt and everyone at Oxide felt like we need to have our own software at this layer. It will be a longer, slower path, but it's important. I think what we learned is actually it was a faster path. It ultimately was faster for us to get up by having that software ourselves. Now it was really hard, but ultimately it was a huge win to control that layer.
And as we look up and down the stack, there's not really a decision that we regret. In fact, the decisions that we regret are where we didn't go yet further, where we've got software that we still don't control that we want to go control because we can in terms of the hardware-software co-design, where we can find those pieces that we can design. We've got both the hardware and the software we can design together. We can do really extraordinary things. Now, I think that there's a bunch of we can talk about all the great functionality that we have been able to build, which is great. I think I'm just too much of a failure junkie in that I'm a bit of a scholar on how things fail. It's mesmerizing, right? Because when you look at how things fail, it tells you how they work. So I think pathology is really important and interesting in computing systems. So for me, like the times that I have felt this the most strongly of like, yes, this is emphatically the right path, are when it doesn't work.
Well, maybe shortly after we get it working, but there was actually never any regret when it wasn't working. It was more just like, I hope we don't die. But when we've had these problems, it's like it just unequivocally I want to talk about a couple of these. And I'll hit on just a couple of these. So one was actually way back in the day when we had our first bringing up our first board. As you can imagine, it was our first board. So there were a lot of things, a lot of things in motion. The CPU would not come out of reset, and the CPU would do this very vexing thing where it would power on, seemingly we would give it, it would initiate its power on sequence which says like, give me, you know, I need to set these rails to this value, great, and then after 1.25 seconds it would bounce and it was like, actually, let's do it again. You're like, okay, what happened? What? You didn't like the first one? Like, nope, I guess I didn't. You're like, okay.
And this was, you know, our assumption was like, okay, the power is marginal. Power's bad. We're spending tons of time making the power, and actually AMD is like, actually, we've never seen power this good before. Like your power margins are very, very good. Like at this point it's definitely not your power margins. And we went down so many blind alleys. And one of the challenges when you are kind of going your own way with respect to hardware-software design is like, I mean obviously like we own, we've got lots of differences between us and a reference design, and the state of the industry is basically like, what are the differences that it must be that you're like, is that what we feel like first principle is linking to you? So in particular there is a pin on the CPU called KB reset. It's active low. So it's KB reset low, and you're like, is the K... Yes. It's keyboard. Keyboard. Like you're on a server CPU. It will never see a keyboard. It's like, we've got a pin on that dedicated to the keyboard? Yes. Yes.
Now AMD says like, that can be a no connect. Like, great. Can it be like a [__] off? Because that's how I feel about it. But it's a no connect. It's a no connect. So we leave it as a no connect. And AMD's like, ah, what about KB reset L? And like, you told us that's a no connect. They're like, yeah, but like this, like the CPU is angry. The gods are angry. We, the gods demand a sacrifice. Can we go sacrifice KB reset L? It's like, it really should not be KB reset L. Desperation will do remarkable things for you. Like, okay, let's see what we can do. Can we, we need to tie KB reset L so it's not a no connect. Doesn't float. We actually need to tie it down. So this is a photo of that happening.
And this is where I mean, I, you know, I've got such a marvel, Eric Austin and our team. This is a photo that he took. This is very much his hands. For any Jenzers, this is a dime. It's too hard to explain. But this is running that from that pin all the way out and tying that down. Of course, didn't change anything. All right. Awesome. That was fun. Actually, it was kind of a relief. Honestly, it would have been really angering if that had been the problem. So I guess it's good that's a problem. All right. So what was the problem? And the problem and several weeks of debugging this, these were some long weeks because these are like, we are not going to live. We actually need to resolve this problem. The problem was an actual firmware bug in software that we did not develop.
So the regulator, the voltage regulator had, there's actually a protocol that the CPU speaks to the regulator which is kind of crazy, right? Like the absolute like lowest layer of the system and it's something that you would recognize even if you're doing software in a much higher layer where there is a protocol and the CPU is requesting different voltage levels at different times. This is how, I mean this is truly a marvel. If you look at the draw on a CPU as workloads are fluctuating, it is a, you will see the current move a lot as it will request more effectively and it will adjust its own voltage accordingly to actually put more work, more power to work, and that's a consequence of phones and laptops, right? Phones and laptops, we really care about the power management of those things. So we have that in the server space anyway, it's amazing. So there's a bunch of amazing stuff in there. Very rich protocol. That protocol says like, 'Hey, here's the voltage level that I need.' As it turns out, there is, that's the VOUT packet. There is a VOUT complete packet that the regulator needs to send back to the CPU saying, 'Hey, I did it.'
And the tool for verifying power, AMD's got this really cool tool called SDE that you plug into the socket and it verifies all your power, which is great. We were using that to verify power. Power is awesome. The problem is that thing didn't care at all about the completion packet. The CPU kind of understandably did. So the CPU would be like, 'Yo, can you set the voltage to this level, please?' And it would do it. And it would never say that it had done it. And as a result, CPU is like, 'All right, I don't know what the hell's going on over there, but I'm, you know, okay, I'll ask again.' I mean, it's like it thinks that we're the four-year-old that won't put on our shoes. Be like, 'Okay, I'll try it again.'
And the actual problem was we had a firmware bug and a firmware bug that actually had already been fixed by Renesas. The Renesas FAE was like, 'God, you really should have contacted me much sooner.' It's like, 'Well, we wanted to make sure that it wasn't that was us.' But this was like a very eye-opening moment because you've got this and a firmware bundle that we don't have access to, but it's not something that can be dynamically updated in the system. You actually, that comes with the part. You've got a finite number of those that can be flashed onto the part and then you're out of slots, you blow the slots. So it's in non-volatile RAM and you think like, wow, all that software, like there's sophisticated software at these very lowest layers of the stack. And it was for all of my desire to start a company that would not get bit by firmware bugs, it was a firmware bug that bit us first, which I think is kind of amusing. But that was kind of, that was a wild problem and also a relief that we had done so many other layers that we are, if we're down to the voltage regulator software, like, okay, those we'll accept, I guess, at some level.
Another one that was absolutely brutal. We had a NIC, and so the NIC is the network interface card. NIC choices are bad just in general. Like it's just, it's not a great landscape. And we kind of famously, we've got something called RFDs, requests for discussions, like kind of RFCs that we are, some are public, some are not. But our RFD on our NIC selection, we called the four NICs of the apocalypse. Because it felt like we were choosing between death, war, pestilence, famine, disease, I guess. And Broadcom, by the way, is war. Broadcom's always war. Like, you know, Broadcom is war. I can't remember Mellanox split into. And then like Intel was like pestilence, I think. And Chelsio was famine, and we're like, you know what? We're going with famine. And there are a lot of reasons why we were, I mean, that was a lot of things that were valuable about that.
But so we had this Chelsio NIC, a Telco NIC, and that NIC would, we would transiently fail to train all of its PCIe lanes. So on some small subset of boxes, it wouldn't, and then you could bounce it and it would train. But some of these would train all the time and some of them would train much less, much more sporadically. And we ended up doing a bunch of very low-level debugging over a long period of time. But it required us to be able to modify that lowest layer of boot software over and over and over again and with a lot of debuggability to figure out what was going on. And what we actually discovered is that if you reset this thing a second time, it would always function correctly.
What we call the double PUR'd. As it turns out, this is, so they isolated this, this has been a bug for them for 19 years and assuredly there was something out there that does a double PUR that needed to be accommodated. So every system on the planet will issue two resets to every NIC to accommodate God knows what. It would be actually interesting to know like what broken device everybody's accommodating. But the brokenness in this NIC was able to hide out under that bug. So this NIC had an unrelated issue that they never located because everybody does this double reset. Well, except for us because it's a reset. You need one of them. It's like, well, long story, but better send us a second one. So it would again, it would, and this was also an eye-opener in terms of how long, I love the very long shadow that these things cast.
So the, and the last one, and this one is definitely germane because you all at Jane Street were the first ones to actually see in this. So the IBC is an intermediate bus converter and you had a sled, sled 19, famous to us, that would, that every like, not so long, every like 24 to 48 hours its drives would reset.
And which, and then, oh, so when the drives, which by the way, like the operating system is not, that's not really the way it works. Like you're not allowed to just like reset things arbitrarily. And so in particular the drive is like, all right, well I guess I'm waiting to be initialized now. I was just reset. The operating system meanwhile has I/Os in flight and so on. And at some point the operating system is like, what happened to all of our I/O devices? And the operating system was like, we have lost all of our I/O devices. And the operating system would bounce and it would bounce, everything would come back up, be fine. But we were seeing this like pretty persistently and assumed that like we've got a drive issue, but we're seeing it on all drives at once. So all 10 drives in the sled were all resetting simultaneously. Like this got to be like a drive issue. And actually what we thought we were, I mean the nice thing about doing all this software yourself is whenever we have a problem anywhere, it doesn't even enter our mind that the problem might be due to somebody else. We're always like, 'Nope, we definitely own this one.'
And in particular thought that this might be an interaction with the firmware on the part with respect to the way we're bringing up PCIe engines. So we were very concerned about that. Well, as it turns out, that's actually not what it was. What it actually was was the bus converter is a big part at the back of the sled that converts from 54 volts DC down to 12 volts. And on a small number, 3% of our sleds, occasionally, every day or so, maybe once a week, that voltage would dip from 12 to 8 and go back up. And it would go down for about a millisecond and a half. So it's like 2 milliseconds down and 3 milliseconds up. And amazingly, this is just like this kind of the amazing bit about regulators and physics and so on, that from 12 to 8 you are within the input threshold for all the regulators on, like everyone else is like, all right, weird but cool, like don't know why my input voltage is dropping, but like I'm with you, my job is to regulate 3.3 or my job is to regulate 10, and my, okay, that's fine, I'm cool with it.
The drives, however, like, so again we're going from 12 to just below 8. And the drives are like, yo, we operate on 12, like I'm not cool with it at all. And so that's why the drives are all reset. So the drives all bounce, but everything else in the system was fine. This was another one of these that was like super chilling. So we, and we wanted, before we got your sled back, we wanted to actually take some iterations on it. So one of the things that we did is we actually modified our service processor firmware to use the regulators, the hot swap controllers that are actually on those shark fins on the drives because they are witnesses to a crime and they have a little facility in that regulator that allows them to record minimum maximum values. And so they would see the voltage drop and like, okay, so we actually know conclusively that this is the issue.
So we got that back, that was on 3% of sleds. We were, as you can imagine, working with our IBC vendor to understand why this is happening. Of course, we heard what I swear we at Oxide will never say, which is like, never seen this before. This is like a Dellism, right? If you have Dell, like, we've never seen this before. And you're like, what? Why are you saying that? Like, what is that? Like, it's like, how are you not saying that to care less about my problem? It just feels like that is the opening sentence in a paragraph where you close my case, right? I mean, it just, it's, and you won't hear that out of us. You may hear from us that we've never seen this before, but you won't hear, you're the only customer seeing this problem. Like we, so they had never seen this before. We were the only customer seeing the problem.
And what, they couldn't reproduce it. We sent these units back to them. They couldn't find it. And what we were able to do is actually test these things in the factory. We were able to validate them in the factory. So we weren't able to ship any more of those sleds out. They actually, we just learned just this past week that like actually they debugged it. And there is a, there's a circuit issue on the input voltage, that circuit is not totally robust and in particular that it's just close to the margins. And if you have parts, this is one of the things that's very frustrating about like actual hardware, like the stuff that you hold in your hand, indisputably hardware. We can argue about an FPGA, but like a capacitor or resistor, these are definitely hardware. The thing that's frustrating about those is like there's variability to those things, which is super frustrating if you're coming from software. We don't have really that kind of, like some software either works or it doesn't. You don't have that kind of variability.
But if that variability could add up and then you would have an IBC that was vulnerable to thinking that there was a drop on its input that wasn't there. So it would start to regulate down because it would think its input is dropped and then it realizes like, [__], put it back. And within that, the drives reset. So I, but this was a chilling example because I think to myself, God, how many times have, like if you just saw a drives reset, your first thought is not that the bus converter is broken. Like that thing's job is like super straightforward. Not to take anything away from the complexity of that, but it's also like it regulates that voltage for, and it's going to have an uptime of 100.0% you would expect. So the idea that you would have a transient in a bus converter, it would again get eye-opening and eye-opening about like if we had not done all of our own software and kind of thought of this whole holistically, you just have a ghost.
And that's what we actually have in other systems. Systems where you aren't doing that hardware-software co-design, you just have ghosts, like bad juju, voodoo happens, whatever it is, poltergeists, reboot it. I'm trying to get the, like it's like everything is like Linux audio, right? You know, you're just like, please God, you know, work. Like what sacrifices do I need to make for my Linux audio to work? And like when we don't actually build these things together, that's what the world is. So it's really dangerous to think about. And Moore's folly really is a problem to think about like these things as being like, no, no, the primacy of software. It's like, no, we actually need to co-design these things together, holistic design. And when we design things holistically, we're able to solve really novel problems. And I actually think it's actually, we shouldn't think of Moore's folly, although again, if you're like me and you get trolled by VCs, you probably will, but really Alan Kay put this best in 1983: people who are really serious about software do their own hardware. And that is the way we should think about hardware.
Hardware is when we collectively are really serious about our own software systems, we want to co-design that with the hardware. And what's on what side of that divide is actually less interesting than it is to actually build these things together to deliver a high-performing, robust system that is able to enable us to do things way, way, way up the stack at a great distance from that co-design. With that, thank you very much.
And sorry, I know I've gone a bit over. I'm not sure if you are... So, yeah, we've got time for questions. If people just want to raise their hands, we can pass around a mic or I think this box should work, too.
