SE Radio 709: Bryan Cantrill on the Data Center Control Plane

Awesome. Thanks for having me. It's great to be here.

Yeah, I mean, so at Joyent — so Joyent was a cloud computing pioneer. We competed with the likes of AWS and then later GCP and Azure, and we, I mean, we were operating at a scale, right? We had a bunch of machines, a bunch of DCs, but ultimately, you know, we were a VC-backed company and, you know, a small company by the standards of — certainly by Samsung standards. And so when Samsung bought the company — and the reason, by the way, that Samsung bought Joyent is Samsung's cloud bill was, let's just say it was extremely large. They were spending an enormous amount of money every year on the public cloud, and they realized that in order to secure their fate economically, they had to be running on their own infrastructure. It just did not make sense, and there was not really a product that Samsung could go buy that would give them that on-prem cloud in that regard. Like the state of the market was really no different. And so they went looking for a company and found Joyent. And when we were on the inside of Samsung, we learned about Samsung scale. And Samsung loves to talk about Samsung scale. And I got to tell you, it is more than just chest-thumping. Like Samsung scale really is — I mean, just the sheer number of devices, the number of customers, just this absolute size. They really wanted to take us out to levels of scale certainly that we had not seen. The reason for buying Joyent was to be able to stand up on their own infrastructure. So we were going to go buy — we did go buy a bunch of hardware, and I remember just thinking, God, I hope Dell is somehow magically better. I hope the problems that we have seen in the small — you know, I just remember hoping, and hope was of course a terrible strategy, and it was a terrible strategy here too. And the problems that we saw at the large — when you scale out, the problems that you see kind of once or twice, you now see all of the time, and they become absolutely debilitating. And we saw a whole series of really debilitating problems. I mean, it's in many ways like comically debilitating in terms of showing just how bad the state of the art is. And we had — I mean, it should be said, we had great software and great software expertise, and we were controlling our own system software, but even controlling your own system software, your own host OS, your own control plane, which is what we had at Joyent, ultimately you're pretty limited. I mean, you've got the problems that you can obviously solve, the ones that are in your own software, but the problems that are beneath you, the problems that are in the hardware platform, the problems that are in the componentry beneath you, the problems that are in the firmware, those problems become unresolvable. And they are deeply, deeply frustrating, and we just saw a bunch of them. Again, they were comical in retrospect. And I'll give you like a couple of concrete examples just to give you an idea of what kind of what you're looking at. One of our data centers had really pathological I/O latency. We had a very database-heavy workload, and this was kind of right at the period where you were still deploying on rotating media, on hard drives. So this is like — so an all-flash array did not make economic sense when we did this in 2016. This would probably be interesting to know, like when was the kind of the last time that actual hard drives made sense, because I feel this was close to it. So we had a bunch of pathological I/O problems, but we had one data center in which the outliers were actually quite a bit worse, and there was so much going on in that system. It took us a long time to figure out like why, and because when you're seeing worse I/O, I mean, naturally you want to understand like what's the workload doing? Trying to take a first-principles approach. What's the workload doing? This is a very intensive database workload to support the object storage system that we had built called Manta, and the metadata tier was stored in — we were using Postgres for that, and that was just getting absolutely slaughtered and ultimately very I/O bound with these kind of pathological I/O latencies. And as we were trying to like peel away the layers to figure out what was going on, and I finally had this thing, it's like, okay, we are seeing at the device layer, at the disk layer, we are seeing pathological outliers in this data center that we're not seeing anywhere else. And that does not make any sense. And the thought occurred to me, I'm like, well, maybe we do — do we have like a different rev of firmware on our HGST drives? HGST, now part of WD, Western Digital, were the drives we had everywhere. So maybe we had a firmware bug. This would not be the first time in my life at all that I would have a drive firmware issue. And I went to go pull the firmware rev, and I'm like, Toshiba makes hard drives? So I had no idea that Toshiba even made hard drives, let alone that they were in our data center. I'm like, what is this? And as it turns out, and this is, you know, part of the challenge when you don't have an integrated system, which not to pick on them, but Dell doesn't. And what Dell would routinely do is just make substitutes, and they make substitutes that they, you know, it's kind of like you're going to like, I don't know, Instacart or whatever, and they're out of the thing that you want. So, you know, someone makes a substitute, and like sometimes that's okay, but it's really not okay in a data center, and you really want to develop and validate an end-to-end integrated system. And in this case, like Toshiba doesn't — I mean, Toshiba does make hard drives, but they basically were not competitive, and they were not competitive in part for the reasons that we were discovering, they had really serious firmware issues. So these were drives that would just simply stop acknowledging any reads for on the order of 2,700 milliseconds. Long time. 2.7 seconds. And that was a drive firmware issue, but it was highlighted like a much deeper issue, which was the simple lack of control that we had over our own destiny. And it's an example among many where Dell is making a decision that lowers the cost of what they are providing you marginally, but it is then giving you a system that they shouldn't have any confidence in because it's not one that they've actually designed. And they leave it to the customer, the end user, to make these discoveries. And these things happen up and down the stack. And for every — whether it's — and not just to pick on Dell, because it's true for HPE, it's true for Supermicro, it's true for your switch vendors, it's true for storage vendors, where the one that is left actually integrating these things and trying to make the whole thing work is the end user sitting in their data center. There's not a product that they can buy that gives them elastic infrastructure, a cloud in their own DC. The product that you buy is the public cloud. Like when you go in the public cloud, you don't worry about this stuff because it's AWS's issue or it's GCP's issue, and they are the ones that get this to ground. And they — and this was kind of, you know, the eye-opening moment, not a surprise, they are not Dell customers, they are not HP customers, they are not Supermicro customers. They have designed their own machines, and to varying degrees depending on which one you're looking at. But they've taken the clean sheet of paper, and the frustration that we had kind of at Joyent, beginning to wonder, and then Samsung, and kind of wondering what was next, is that what they built was not available for purchase in the data center. You could only rent it in the public cloud. And our big belief is that public cloud computing is a really important revolution in infrastructure. Doesn't feel like a deep thought, but cloud computing is a really important revolution. It shouldn't only be available to rent. You should be able to actually buy it. And there are a bunch of reasons for doing that. One, and the one we saw at Samsung, is economics, which I think is still the dominant reason, where it just does not make sense to rent all of your compute in perpetuity. But there are other reasons too. There's security, there's risk management, there's latency. There are a bunch of reasons why one might want to own one's own infrastructure. But that was very much the — so the genesis for Oxide was coming out of this very painful experience, and a painful experience that — because I mean, a long answer to your question about like what was it like to be at Samsung scale, those are the kinds of things that we — I mean, in our other data centers we didn't have Toshiba drives, we only had the HGST drives, but it's only when you get to this larger scale that you begin to see some of these pathologies. But these pathologies then are really debilitating in terms of those who are trying to develop a service on top of them. So it was very educational in that regard, and you're very grateful for the experience at Samsung in terms of opening our eyes to the challenge of running at that kind of scale.

Yeah, at the smaller scale you see fewer of them, right? You just see — it's like, okay, we — you know, what you might see is like, that's weird, we kind of saw this in one machine versus seeing it in 100 or a thousand or 10,000. So you just see them less frequently. As a result, they are less debilitating. I think that it's when you go to that larger scale, those things that were unusual now become routine and they become debilitating. So it really is in many regards a function of scale. And then I think it was also — it was a little bit dispiriting that kind of the substrate we were building on really had not improved. And if you buy a computer server, buy an x86 server, there is a very low layer of firmware, the BIOS, the basic input output system, the UEFI BIOS. And this is like an abstraction layer that has existed since the 80s and hasn't really meaningfully improved. But beyond that, like this lowest layer of platform enablement software is really only impeding the operability of the system. You look at the baseboard management controller, which is the kind of the computer within the computer. There is an element in the machine that needs to handle environmentals, that needs to handle operating the fans and so on, and that traditionally is the baseboard management controller, and that architecturally just hasn't improved in the last two decades. And you know, that's — it's a proprietary piece of silicon generally from a company that no one's ever heard of called ASPEED, which has to be — it's written all in caps, so I guess it needs to be screamed. ASPEED has a proprietary part. There is a root password, infamously. The root password is encoded effectively in silicon. And for anyone who kind of goes deep into these things, like, oh my god, are you kidding me? When we first started Oxide, the Wi-Fi password was a fraction of the ASPEED root password for the BMC. It's kind of like a little BMC humor. But those things — it was just dispiriting that the state of the art was still basically personal computers running in the data center. And that's part of what was the motivation for doing something new.

Oh man, you're going to have like some fraction of your listeners, maybe a big fraction, are like, yeah, like what are the problems? That's a good question. And then you're going to have the people that actually deal with these things, like their heads already hit the desk being like, what are the problems? Like what are the non-problems? Like what works actually — that's like a shorter answer. I mean, there are so many problems, and a lot of it is just like — I mean, there are problems just architecturally. These things are just so — I mean, the problems spread to the horizon, so you can kind of start wherever you want. But I mean, as like a really concrete example. Okay, so the BMCs, the computer within the computer, that needs to be on its own network. So you now have like not one network, you got two networks. And that network, by the way, that's the network that you're going to log into to like reset the machine when it's otherwise unresponsive. So going into the BMC, you are able to control the entire machine. Well, it's like, all right, so now I've got a second network that I need to manage. What is running on the BMC? What's running some ancient, ancient version of Linux that you got — it's like, well, how do I patch that? How do I like manage the vulnerabilities with that? Because if someone is able to root your BMC, they control the system. So it's like, this is not — and now you've got to go deal with all of the operational hair around that. How do you upgrade that system? Updating the BMC is — I mean, it's like you've got this like second shadow bad infrastructure that you have to go manage. Generally not open source. There's something called OpenBMC which people use to varying degrees, but you're generally stuck with the proprietary BMC. So you're generally stuck with iLO from HPE or iDRAC from Dell or the Supermicro BMC, the HPE BMC. And it is just excruciating pain. This is assuming, by the way, that everything is behaving correctly. The problem is that these things often don't behave correctly, and then the consequence of them not behaving correctly is really dire because it's at that lowest layer of the system. So I mean, I'll give you a concrete example a customer of theirs reported to me. So I won't disclose the vendor, but let's just say that a well-known vendor had an issue with their temperature sensors were broken and the thing would always read basically the wrong value. So it was the BMC that had to like invent its own different kind of thermal control loop, and it would index on the actual inrush current. They look at that, at the current that's going into the CPU, to adjust the fan speed. That's an interesting idea. That doesn't work because that's actually not the temperature. So that software would crank the fans whenever you had an inrush of current. And this customer had a workload that would spike the current. And when it would spike the current, the fans would kick up, and then they would slowly degrade over time. Well, this workload was spiking the current faster than the fans would degrade, but not fast enough to actually heat up the part. And ultimately, over a very long time, in a very painful investigation, this customer determined that like my fans are cranked in my data center for no reason. We're blowing cold air, and it's like this is on the order of like 100 watts a server of energy that you shouldn't be spending. And that comes down to this kind of broken software-hardware interface at the lowest layer that has real meaningful consequence in terms of hundreds of kilowatts across a data center. So this stuff has very, very, very real consequence, and it's such a shadowy world. Part of the reason that your listeners that have dealt with this, their heads will hit the desk, is because it is really aggravating to deal with problems at this layer. You feel powerless. You don't control or really see the software that's on them. It's generally proprietary. You're relying on your vendor. Your vendor is telling you that like, boy, I don't know, you're the only customer seeing this. I mean, the number of times I have heard that, and I have pledged that we're not going to say that at Oxide because it's such an unactionable thing to say, like you're the only customer saying this. It's like it feels like, are you blaming me for my problem? Feels like you're blaming me for my problem. And what you begin to realize is that to a degree these folks are speaking their own truth because the folks that are running at real scale, at hyperscale, those folks aren't Dell, HP, Supermicro customers. They have actually done their own thing. So it's like, yeah, Dell's not seeing that problem because they're not running at the same scale. But when you do run — you only have to run at modest scale before these things just become overwhelming in terms of the headwind that they present to people that want to deploy infrastructure.

No, no, no. I would think — I think anyone who deploys any number of servers, especially now, especially if your experience is only in the cloud, you're going to be like, what the hell is this? I mean, just again, just to get this thing working at all, it's so hairy and so congealed, right? It's not designed. I mean, nobody who is setting up a rack of servers is going to think to themselves like, yes, this is the right way to go do it. This all makes sense. I mean, it's a bag of bolts. It's a bunch of parts that you're putting together. And so even at the smallest scales, that stuff is painful. Just architecturally, it's painful at the small scale. But at least you can get it working. I think the stuff that then becomes debilitating at larger scale are the things that are worse than just like, I can't — like this thing is a mess to get working. It's like the fan issue where you are now seeing this over, you know, hundreds of machines or thousands of machines. So it is painful at more or less all levels of scale. There is no level at which the PC, which is really what this is — this is the personal computer architecture from the 1980s — and there is really no level of scale where that's the right thing to go deploy, especially if what you are trying to run is elastic infrastructure, a cloud. Because the other thing is like we've kind of been talking a lot about that hardware layer, like hardware is just the start. Like you actually got to go put software on that and actually run that as elastic infrastructure. So you need a hypervisor, yes, but you need a lot more than that. You need a distributed database. You need web endpoints. You need a CLI. You need all the stuff that you need to actually go run an actual service of compute or networking or storage. I mean, and even for compute, there's a ton of work to be done. And compute is by far, I would say, the simplest of the three. When you look at like network services, storage services, there's a whole bunch of stuff that you need to go build in terms of distributed systems to actually offer that as a cloud. So I mean, it is painful at more or less every level if you are trying to deploy cloud computing on-prem.

So control plane is the thing that is everything between your API request and that infrastructure actually being acted upon. So you go say, hey, I want to provision a VM. Okay, great. We've got a whole bunch of things we're going to provision with that. We're going to provision a VM. We're going to get some storage that's going to go along with that. That's got a network storage service that's going to come out of. We got a virtual network that we're going to either create or attach to. We got a whole bunch of things we need to go do for that. For all of these things, there are metadata components that we need to keep track of this thing beyond the actual infrastructure that we create. And then we need to go actually like act on the actual compute elements, the host OS, what have you, the switches, what have you, and actually go create these underlying things and then connect them. And there's of course the challenge of just getting that working is a big challenge. But getting that working robustly, getting that working is, you know, when you go to provision a VM, all the steps that need to happen and what happens if one of those steps fails along the way. What happens if, you know, one thing we're very mindful of is these kind of — you get these long tails of like generally our VM provisioning happened within this time, but we get these long tails where it takes much longer. What's going on? Where in this process are we actually spending time? And there's a whole lot of complexity that you need to go deal with, that whole distributed system that itself needs to be reliable and available. So you need to be able to — what happens if you pull a sled or if a sled fails? How does the system deal with that? How does the system deal with getting another sled added to the system? Like how do you actually grow this distributed system? And then how do you update it? How do you actually go from one version to the next? And all of that has to happen across an air gap where this is going to run as part of the computer. So it is fractally complicated. There is a lot of complexity here in the software system, and all of that we call the control plane, and what exists at AWS, at GCP, at Azure. You are hitting an endpoint that's provisioning an EC2 instance for you. There is an AWS control plane that is doing all of this and has some of these similar aspects and certainly some of these similar challenges.

Yeah, I mean, a little bit. I mean, it kind of like — vSphere, yes; VMware, no. So it's like VMware ESX is kind of a key building block upon which you can build something that is a more meaningful distributed system. When it's just like a machine that you're provisioning VMs on, it's like, okay, well, that's actually — you as the human might be the control plane. Like that's a much easier problem. But when you got tens, hundreds, thousands of machines and you need to do it robustly, you need something to coordinate that activity, and you know, you need to pick which sled you land on, and you need to be able to move these things. You need to be able to update that whole system. That's when you're getting into a control plane. So, you know, some of these things have kind of edged into a control plane. Certainly VMware, now Broadcom, has delivered something that's kind of cloud-ish. I think that for folks that are truly born on the cloud, it still feels somewhat like you're going backwards in time when you look at these kind of on-prem offerings. But it's got these aspects to it for sure. And some of these other things, when you're just looking at KVM or just looking at Proxmox, you kind of need to connect it to other broader things to turn it into something that really looks like manageable infrastructure. And then many of those projects are really — they're either proprietary products like vSphere, or you are really dealing with open source projects that are not necessarily aimed at the same level of scale. You know, you look at, again, a Proxmox, or you look at an OpenStack, and you know, OpenStack is just a lot of things, right? I mean, OpenStack has got so many — OpenStack was kind of a free-for-all for every infrastructure vendor. And I, you know, there was a time people were like, aren't you worried about all these companies together that, you know, are coming together for OpenStack? I'm like, haven't you ever worked for like a company? Like companies don't get along, by the way. It's like having multiple companies work together on a thing. That's bad news, not good news. And I think, you know, one of the things that OpenStack has definitely struggled with — well, actually there's so many different kind of vendor elements in there that it's very much not a product, it's a project that you're trying to run. But that very much is in — I mean, that's similar certainly in spirit.

That's right. Yep. And I mean, in order to do that in a modern way, it's not just like a user-friendly way. You really need to have a CLI and a web UI and an API. Those all need to be drawn from the same kind of single ground truth. Like you don't want to have any of those be an afterthought for the other. You want to have the same way of generating all of those different endpoints, entries into the system.

Yeah, so we built more or less everything in-house. I mean, what you have is — and I think, you know, over time we've gotten slightly better tools. For example, at Joyent, when we were building a cloud at Joyent, there wasn't really a good distributed database, so we were using Postgres as our database for metadata, and there were a lot of challenges. And Postgres is not a distributed database; it's running with a primary-secondary architecture, and there's a bunch of issues there, many of which we discovered the hard way. When we were coming to Oxide, you had much better options to pick from. In terms of distributed databases, there was a period that now seems maybe potentially brief in hindsight, but of really high-quality open-source distributed databases. So there were really some good ones to pick from. We've built on CockroachDB, on CRDB. So that was a really important component that we had at Oxide that we didn't have at Joyent. So we were — I wouldn't say we were rolling our own distributed database; we were just using Postgres and dealing with an enormous amount of pain there in terms of the surround on top of that. You know, a control plane is much more than a database, obviously, and you've got to deal with — there's a whole bunch of software that you need to go write to be able to transform these kind of API requests into something that is reliable infrastructure, right? And there's a lot to that, especially when networking gets in the mix, when storage gets in the mix. There are a whole bunch of like complicated steps that need to be done. At Joyent, we — in part because the history of the company — and like, look, this just is not going to sound good, but it just is what it is, and I'm just going to own it — we did it all in Node at Joyent, which I know sounds really — right now just sounds like you built it with Tinker Toys. You okay? Did you think it was — you built the skyscraper with Tinker Toys? It's like, well, okay, we actually had greater aspirations for the Tinker Toys once upon a time. But let's just say that that experiment did ultimately end in a predictable fashion, and we decided that maybe Node was not going to be the best decision long term. Joyent was the company behind Node.js back in the day. Ryan Dahl worked for Joyent, and then we landed that in a foundation in about 2015, something like that, and began to consider our world beyond Node. A big tool that we had in the arsenal when we started Oxide is Rust. So indeed the name of the company is a tip of the hat to the language that we were pretty sure we were going to be building a lot of stuff in, namely Rust. And Rust has been huge for us, a very important revolution in programming languages. I think what has been surprising is the sheer number of layers at which we use Rust in terms of — we've done our own embedded firmware in Rust. We've done — in the host operating system, which is still largely in C, but very big components are in Rust. The hypervisor, Propolis, is all in Rust, and then of course the control plane and distributed system on that is all in Rust. So that was a very important thing that we very much did not need to build ourselves. We were able to really leverage a terrific community. We were able to use — and we've done this at Joyent as well, but at Oxide we've used illumos as a host OS component, which our variant is called Helios. We've used Bhyve as that kind of in-kernel hypervisor component. We've made use of a bunch of different open source components to build this thing, which has been really, really important for us, and open source components that didn't exist even like five years prior, as part of why we felt that 2019 was the right time to start the company when we started Oxide.

Oh boy, yeah. Again, I kind of had higher hopes. Oops. In 2010, I would say, when we sat on this, the problem that we had just writ large: JavaScript is really designed to allow as many people on Earth to write a program as possible, which is good. I mean, that's like a laudable goal. The problem is it's much more difficult to write really rigorous software. We actually want to be able to write rigorous software. And it's actually okay if it's a little harder to write rigorous software. That's actually okay if it leads to more rigorous artifacts. But in JavaScript, I mean, a concrete example: there's nothing to prevent you from referencing a property that doesn't actually exist in JavaScript. So if you fat-finger a property name, you are relying on something to tell you, by the way, I think you've misspelled this, because there is no type definition for this thing. And I don't know that you've got one that's spelled correctly, one that's spelled incorrectly that's often undefined. And then when you actually go — so you've got this typo that is lurking in what you want to be rigorous software, and if you don't execute that code, like you won't know it's there. And then you do execute that code, and now you've got an undefined object, and now that's either going to be an exception or it can — again, depends on how that's handled. It can be really difficult to determine the origin of that error, of that programming. And that is a programmer error. And one of the big challenges that we had with Node is that programmer errors and operational errors — like, you know, I'm out of disk space is an operational error — those get conflated, and it becomes really hard. And in fact, I think the language wanted to make it easier to just kind of drive on in the event of all errors, and it's like, actually not what you want to do if you're trying to build a reliable, robust system. So we had no end of issues. We've got a lot of experience developing rigorous systems, again coming out of operating systems development and so on, and we brought some of that rigor, if strangely, to JavaScript. So one of the things that we did is we brought a lot of post-mortem diagnosability and observability to Node. And so if one of our Node processes died in production, we would actually get a core dump from that process, a core dump that we could actually meaningfully process. So we did a bunch of kind of wild stuff. I mean, actually wild stuff where we could actually make sense of the JavaScript objects in a binary core dump and things that we thought were really important, and this is the rest of the world just looks at this being like, what the hell is this? I mean, it's so out of step with it. The problem is that we were trying to bridge two disconnected cultures: of one, developing really rigorous software and really designing it for production diagnosability, and the other, really designing software to run in the browser and for anyone to be able to like, you know, kind of liven up a web page, right, is kind of the origin of LiveScript and then JavaScript. And we were kind of the only ones sitting at the intersection of that. And you begin — when you are the only ones sitting at that kind of intersection and you just are — you're kind of fighting a community all the time. And we just realized that there were so many things...

that the community wanted to do that we felt are like, no, no, this is going to make software less diagnosable. It's going to make it less robust. And then you realize like, we're the only voice in the room because we have got desires for this language that it doesn't have for itself. And this is when you realize you're in a bad relationship with software. It's time to actually move on. And in fact, actually several years after we'd already kind of broken up with Node and it was like it was a bit of an acrimonious breakup, there was a famous slash infamous fork of Node called IOJS. And this was viewed because the community thought that Joyent was not being an appropriate steward of Node.js and was not allowing more things to come into Node. And of course, the reason that we of course felt that we were being a careful steward and we were actively resisting those things that would cut against its fitness for a production system. But the way the community saw it, and I think we knew before the fork that's like, this is not working and we need to get this thing out of our hands and we're the wrong hands for this. This needs to be in a foundation. So we kind of gone through that breakup and maybe it was two years after that that a friend of mine who was running the Node Summit, actually it's unfortunately now passed away, Charles Beeder, but Charles, venture capitalist, great guy, and Charles was running Node Summit and came to me in 2017. He's like, I really want you to keynote Node Summit. And I'm like, Charles, I'm not going to do that. I've got nothing nice to say. I'm the last person you want to keynote. He's like, oh, if you have nothing nice to say, you should definitely keynote. You're like, oh God, okay, here we go. He's like, no, I really want you to talk about like, you should talk about the Joyent breakup with Node.js. I'm like, oh man. And that led to a talk that I'm really happy that I gave because it was a very important talk for me personally called Platform as a Reflection of Values and really looking at the values that we had for Node and the values that Node had for itself and they didn't line up. Like there's nobody in the Node community who's like, I don't want rigor. I hate rigor. It's just that if they had to choose between rigor and making the language approachable, they would choose approachability every single time. They would never choose rigor. And that was a big eye opener. I was, if you watch this talk, because I knew that there's like the audience was going to be filled with people who had been a part of the fork in 2014, I think was the fork, the IOJS fork. I set a little bit of a trap for the audience and the trap I said, you know, I kind of talked about the values that we had and the aspirations we had for Node, the aspirations that Node had for itself and how they were different. And I'm like, look, in hindsight, like a fracture was inevitable. And in 2014, there was finally a fracture. And do people know what happened in 2014? And you could listen to that talk. Everyone almost says in unison like, IOJS. I'm like, oh right, IOJS, right. That's actually not what I was thinking of. And I go to the next slide and it's a tweet from a guy named TJ Holowaychuk who was the most prolific contributor to Node. And it was his tweet also in 2014 before the fork, before the IOJS fork, explaining that he was leaving Node and that he was going to go. And if you turn the volume all the way up, you can hear the audience gasp and it's just delicious because the community had never really confronted why TJ left. And I went through a couple of folks, Felix, a bunch of other folks, early Node folks that were there in 2010 were leaving in 2014. And they were going to go primarily and they were going to go because they were sick of the same things that we were sick of. They had hit the same things that we had hit and they were frustrated. I really do believe this, that platforms do reflect their own values and when you are making a software decision, you should select values that align with the values that you have for that software. That's way more important than other things that people look at. I think people look at, for example, quote unquote community size way too frequently. Community size is like, eh, maybe it can be fine. I've been in very large communities, Node. I've been in super small open source communities like illumos and DTrace and a bunch of others. There are strengths and weaknesses to both approaches just as like there's a strength to being in a big city versus a small town. Me personally, I'll take the small community more or less every time because the small community is almost always self-selecting based on values. And just for the same reason that I like working at small companies or small teams, there's a lot of value to be had in a small community. It's not to say that large communities don't have value, but again, long answer to your question of kind of where did things go south with Joyent and Node. They went south because the values that we had and the values the community had didn't line up. And that was a very educational experience as you might imagine.

Yeah, I mean, well, so the value for Go, I mean, I understand why people moved from Node to Go. Go to me was kind of a lateral move. There were a bunch of things that I, Go was still garbage collected, which I didn't like. Go also is very strange in terms of there are these kind of like these autocratic kind of decisions that are very bizarre. I mean, generics is kind of a famous one, right, where Go kind of as a point of principle didn't have generics even though Go itself actually the innards of Go did have generics. It's just that you, a Go user, weren't allowed to have them. And I just think that the arguments against generics were kind of disingenuous and indeed like they ended up adopting generics. And then there's like some super weird stuff around like they're very anti-assertion, which is like, what, how is someone against assertions? It doesn't even make any sense. But I was like, oh, nope. Okay, there's a whole screed on it. Nope, we're against assertions. And then, you know, against versioning, there was another thing like, you know, Rob Pike has kind of famously been like, you should always just run on the latest commit. And you're like, does that make sense? I mean, we actually built it. And so there are a bunch of things like that you're just like, okay, this is just exhausting. And I mean, there's some things about Go that are great and plenty of other things that I just, I'm not a fan of. I think that the, and like Go cares a lot about like compile time. Compile time is super important for Go, right? It's very quick compile time. I'm like, okay, but that's like, compile time is not like, it's not unimportant. It doesn't have zero importance, but I've got other things that are like lots more important than that. What I really care about is I want a high performing artifact. And I wanted garbage collection out of my life. I got to tell you, garbage collection to me is an embodiment of this like larger problem of where do you put cognitive load in the software development process? And what garbage collection is saying to me, it is right for plenty of other people and the software that they want to develop. But for me and the software that I want to develop, infrastructure software, I don't want garbage collection because I can solve the memory allocation problem. I know when I'm like done with something or not. I mean, it's like whether that's in C with, I mean, it's actually like, it's really not that hard to not leak memory in a C-based system and you can give yourself a lot of tooling that allows you to diagnose where memory leaks are coming from. So it's like that is a solvable problem. There are other challenges with that. But like when you are developing a really sophisticated system that has garbage collection, is using garbage collection, you spend as much time trying to dork with the garbage collector to convince it to collect the thing that you know is garbage. You're like, I've got this thing. I know it's garbage. Now I need to use these like tips and tricks to get the garbage collector. I mean, it's like it feels like every Java performance issue goes to like minus XXX, use the other garbage collector. Whatever one you're using, use a different one and using a different approach. It's like you're in the worst of all worlds where the reason that garbage collection is helpful is because the programmer doesn't have to think at all about this problem, but now you're actually dealing with these long pauses in production. You're dealing with all these other issues where actually you need to think a lot about it and it's kind of, it's witchcraft. It's this black box that you can't see into. So it's like, what problem have we solved exactly? So the fact that Go had garbage collection, it's like, no, I do not want, like, and then you get all the other like weird fatwas and, you know, everything else. I'm like, no, thank you. Go was a no, thank you for me. I get why people like it or use it, but that was not going to be it. I'm like, I want C, but there are things I didn't like about C too. I was looking for something that was going to give me the deterministic kind of artifact that I got out of C. But I wanted library support. In C is tough because there's, it's all convention. You know, there's a bunch of other things that are just thorny. And I remember thinking vividly in 2018, I'm like, well, it's Rust or bust. I'm going to go into Rust. And when I did what a lot of people were doing at that time and people have been doing since of, you know, really getting into Rust and really learning it, appreciating the difference in the model for sure, the ownership model people talk about is also obviously very important. It was the error handling that blew me away and the idea of like algebraic types. I never really had algebraic types and the ability to have, and for error handling is one of these really, you really appreciate these things where it's like, how do you deal with a function that can either succeed and return something or it can fail? And the way C deals with that is bad with these kind of sentinels for errors and, you know, does negative one mean failure? Does zero mean failure? Some C functions, zero means failure. Traditionally in Unix, zero means success. And like, what if you want to return a file descriptor? You know, it's like, oh, and then it's like, okay, then it'll be like zero through positive N will be a valid result. Negative numbers will be, and like, was it negative one and I said error, no, or is it a negative number that did, I mean, it's like, and that's all convention, right? People do all those different things and it's all convention and it's easy to get wrong, easy to have bugs, can't be statically checked and so on. Then what Go says is like, well, you're going to have like two return values and then you're going to have to like just like constantly check all of these all the time, which is also kind of gross. JavaScript is like, hey, let's toss an exception if we don't like something. If we see an error, we'll throw an exception. There are a bunch of reasons I don't like that. And you look at what Rust does where it's like, no, no, no. We're going to have these algebraic types, which is to say this thing can be a this thing or a that thing, but it has to be one of these. And by the way, you don't get to process this thing until you conditionally match on one of these things. You're going to have to have a pattern match on this thing to determine if it's a this or a that. And if in the result type that you, the result is a generic where it's like, it's going to be either the thing that you want to return, it's going to be an OK that contains the thing you want to return, or it's going to be an error that contains your error. And it forces your code to deal with that. And what that does is it shifts the cognitive load from the person that is operating this thing in production to the actual developer that is in development. And that to me is like, I love that shift. And that shift to me is really important. And that's what I was missing. That's what Rust gives you. Rust forces you to think about your code as you write it, but as a result, you have an artifact that is much more supportable, much more sustainable, and much faster.

Yeah, absolutely. And I just think that like, why also like, if it's software, if it's again, if it's infrastructure software, I mean, the kind of the question that you should have when you're writing software is how long is the software going to live? How many people are going to use this software? And if you are writing an operating system, the answer for this thing that you're going to write, it's going to live for a long time. Like if we just look at plenty of aspects of the system that have been around for decades, it's going to live for a long time and many, many, many people are going to use it. Why would we not expect people writing that software to have more cognitive load when they're writing it to give us something that's going to be a better artifact? Now, conversely, you're like, hey, I kind of don't care about this and like, I don't know. I'm just like, I want to see if this whole thing works. I've got, I like, I'm just stringing this together. I don't like, no software, like we'll be lucky if it survives until tonight. But then like, who cares? Yeah. Yeah. Garbage collect, you know, if you're prototyping something, whatever. And this is why you really do get like different choices, different technology choices depending on the way that you want to solve the problem at hand. And for the software that I want to write, I do like that cognitive load that is upfront. And although I think, I think the thing that is really wild, that is the twist that I don't think anyone really saw coming is that in a, in an LLM age, the cognitive load up front almost needs an asterisk on it because so much of that can be assisted by an LLM. And now, I mean, I would like to believe, and maybe this is me being optimistic, that in the LLM age, we will see, I mean, Rust is a great fit for the LLM age because the LLM itself can get a lot of feedback about whether the software that's written is correct or not, much more so than you can for other environments.