Thank you so much, Nell. It's great to be here at RustConf. And I'm just going to apologize in advance. There are going to be a lot of matrimonial metaphors here. So, I've been married for 23 years, so I've got a good marriage. We got a good thing going on. We got a good marriage going on with Rust as well. So, I'm just going to apologize in advance. And like any good matrimonial story, it starts with a bad breakup way back in the day.
So, you know, I'm blessed with terrific colleagues at Oxide, and one of my colleagues you'll be hearing about a couple of times during this presentation, Rain. Rain, you know, we were taking one of our one-on-ones, and Rain's like, you know, I've been meaning to ask you something. You seem like a sharp guy. How did a guy like you end up in Node.js?
And I'm like, you know, Rain, I really appreciate... and it just, you know, a question asked with such kindness. There was no, you know, like, no, I'm not accusing you of anything. No, no, I really appreciate that. And, you know, it started off in 2010. I was very enthusiastic about Node. It didn't go very well with Node. It was a bad relationship. And by the time we broke up with Node in kind of the 2014-2015 time frame.
And around 2017, a friend of mine, actually sadly now deceased, Charles Beore, was running the Node Summit and he asked me to give a keynote at Node Summit. I'm like, Charles, bad idea. I've got nothing nice to say about Node. Charles is like, that's even better. I'm like, that's... Oh, come on. But it actually, as I thought about it, I'm like, you know what? This is going to be an opportunity to really reflect on what happened with Node.
And what happened with Node, because I mean, like many a breakup, it's no one's real fault per se. In that what we had with Node was a values mismatch. The values that I had and the values that Node had for itself were different values. And causing me to reflect on that, I gave this talk, and this is going to be a... I'm going to apologize for this as well, a bit of a clip show, and then I'm going to refer you to a bunch of other talks, but other content, not just from me, but from other folks at Oxide.
And this talk, Platform as a Reflection of Values, ended up being one of the most important talks that I feel I've ever given because it allowed me to process what had happened and why. And as I was thinking, like, what is important to me and what was important to us, I was at Joyent at the time, and why did this cause this fracture? Because for us at Joyent, things like rigor were really, really important.
And Node, if you need to be told, like, it is... it's JavaScript at the end of the day. It's kind of, you know, it's the bell curve meme, right? It's like, you know, we wanted so much more for JavaScript, but the real reality is it is JavaScript. And their values are... it's not that rigor is unimportant, it's that there are other values that are more important. And in particular for JavaScript, the approachability of the language was much more important than the rigor. And when those things came in tension, they would pick the approachability every time.
So in 2017, I was beginning to think about what was next. And what was next was clearly Node. And in this talk I mention both Go and Rust. I had not done anything with Rust, but looking like, could Rust be what was next? And in 2018 I really started diving into Rust. And this talk I gave only... and I say in the talk I'd been implementing in Rust for only like 40 hours total. So I was only a couple weeks into Rust, but already discovering that like, this is really, really interesting.
And as I got deeper and deeper into Rust, I did fall in love with Rust. And this is a blog post I wrote at the end of 2018. And I was loving so much about Rust. In particular, one of the things when I was kind of coming to it is, again, this is in 2018. I feel like if you came to Rust like before non-lexical lifetimes, I feel like there should be like a sticker you can put on your badge for like, I was here before non-lexical lifetimes.
Just because there was... the relationship with the compiler was a lot more complicated before non-lexical lifetimes. There was a lot more like screaming and, you know, kind of tears streaming down the face. Not really sobbing, not really screaming. But this is right before non-lexical lifetimes. And yes, of course, there was the borrow checker and the ownership model, but what I realized is the ownership model had prevented me from really diving into this thing, and there was so much to love beyond just the ownership model.
The ownership model is terrific and is rightly thought of as the epicenter of Rust, but oh my god, the error handling. I mean, like, look, okay, I've lived in the filth of C and JavaScript of my entire career. So like, algebraic types are like clean water to me. And I'm like, I can... I pour, I take the water out of the faucet, haven't done anything to it, haven't treated it, I drink it and I'm not sick. And people from like OCaml or what have you are just like, are you okay? God, people are so sad. And I know it is sad, but you know what? Clean water is amazing.
Okay, so I love clean water and the error handling is extraordinary in Rust. And of course it's like, these are all the things we know. And I was trying to show a beacon to my fellow C programmers that, hey, there's a lot more here than just the ownership model, the borrowing. Like, don't let the borrow checker prevent you from getting into all of this amazing stuff. I thought, and the community in particular was terrific.
I say that last point of the performance. I had a program that was written a C/JavaScript hybrid. Yeah, I know. Okay, fine. But the performance-critical aspect is in C. It's the fact it'll be a C program with kind of a Node front end on it. And I was rewriting it in Rust, and by the time... and I was struggling with the borrow checker. I made the mistake, of course, of trying to implement a doubly-linked list in Rust, even though the blog entry says, no, no, don't do this. You don't mean to do this.
I'm like, yes, yes, I want to do this. No, no, actually I don't want to do this. Then I got to the point where like, I just want this thing to work and then be done with it. And I'm like, I know that I'm making decisions that are going to make this thing perform poorly. And then I ran it and it outperformed my C. I'm like, what? Why? And I got a follow-up blog post explaining why. The reason why, by the way, at the core of it is because in Rust, if you want to use a balanced tree structure, you're using a B-tree.
I've been using an AVL tree. A B-tree is actually a better structure to use than an AVL tree. A B-tree is really, really hard to use in C. There's not a good B-tree library in C because of the ownership model. Because in C it's unclear who owns what. That interface gets very messy. It's very hard to have a component that is an actual B-tree. In Rust, obviously, that's what we use all the time, every day. It's a better structure and it outperforms my C. This was like a wakeup call.
So I mean, I knew two things in 2018 because Joyent had been bought by Samsung in 2016. I was beginning to plan what was next, and I knew two things. I wanted to start a company with Steve Tuck, who I'd known for many years at Joyent. He and I had decided many years earlier, whatever we do next, we're going to do it together. We were very excited about that, and I had made another very important discovery. I want to do it in Rust.
And we knew we wanted to do something in Rust before we knew what we wanted to do. And we had a bunch of absolutely atrocious ideas. Ideas that I won't bore you with or entertain you with, really, because they're all very bad. They were all small ideas. And as I was thinking about, like, we actually need... this language is extraordinary. We need an idea that really benefits the language. We need an idea worthy of the language. We need to go bigger.
And what we ended up doing is, as we were kind of thinking about it, like, what can we do? What would we do if we were unafraid of failing? I think often subscribed to Eleanor Roosevelt, but I think it's Eleanor Roosevelt, like Mark Twain and Ben Franklin, that get a lot of quotes subscribed to them. So I'm not sure if that's... but what would you do if you were unafraid of failing? And what we would do if we were unafraid of failing is take a galactic swing and actually start a computer company.
Rack-scale machine, doing our own computers, our own switch, importantly, all of our own software end to end. This is the actual... this is verbatim our pitch deck in 2019. If you look at our pitch deck in 2019, we've actually done what we said we were going to do. I know most companies don't, but we actually did do what we said we were going to go do. And so we were beginning to germinate this idea of we should actually start a computer company, and we should do all of this software in Rust.
And now that's a lot of interesting opportunity. And I gave this talk in 2018 on rewriting the operating system in Rust. This is a talk, this is an InfoQ talk. It's on YouTube. They promote it a lot. This is actually the most viewed talk I've ever given, and comments are open on YouTube. And this was an eye-opener for me of getting into the real, like, wild of Gen Pop YouTube. And holy mackerel, the comments on this thing even to this day.
It's like, one thing I definitely learned is like, Rust is a bit of a hot button. And I definitely had some people with some very strong opinions. And I was scrolling the comments with my kids, and my oldest was like, 'Dad, you've got haters. This is great.' And I'm like, 'This doesn't feel great. This feels less than great.' But that talk was about really beginning the germination of these ideas, but we needed a name.
So we knew we wanted to start a computer company. We need a name. My kids were hard at work. Susanna, she's now 14, age six at the time, suggested Updown. Like, I don't know that Updown is a good... in hindsight, like, Updown maybe is not a bad name. I mean, I feel like I would say it's like, Updown is not a great name for a computer company. It feels like we're not exactly up-down. Actually, on our worst days we're down-down. So it's like we're... and I guess it could be worse.
I mean, at least the up's in there, but okay. My 11-year-old, he was kind of in a mythology kick. He had a bunch of mythology ideas. Love the mythology ideas, but moving right along. My 14-year-old suggested the Michael Scott Paper Company. A tip of the hat to the 14-year-old. The kids are okay. But as we were going back and forth on this, I'm like, 'Hey, what about a kind of a Rust-themed name? What about Oxide Systems?'
And I immediately followed up by apologizing, like, 'I'm sorry, I'm just on yet another Rust high.' So in 2019, though, we did start the company. We started Oxide Computer Company, raised money in the fall of 2019, started in December of 2019. And putting Rust at the epicenter was extremely deliberate because we knew that the Rust values around rigor, around performance, would attract the technologists that were attracted to the problem.
And we had plenty of people that came to Oxide because they were Rust-first and got excited about building a computer, because we're all excited about computers, of course. And then plenty of people who were excited about the computer and then got excited about Rust. So Rust was a very important cultural beacon for us. But we weren't doing it by fiat. So when we started the company, we had a little bit of trepidation in that, like, what I didn't want is for people to choose the wrong tool for the job.
And in particular, I definitely wondered about where we would use Rust at different elements of the stack. We knew we wanted to use it at the lowest layers of the stack and felt like it was probably a fit everywhere, but we wanted to allow people to come to their own decisions. What we discovered really quickly, and I in particular, I wanted people to consider Rust first. And what we discovered really quickly is that Rust fits in lots and lots and lots of places, and in fact all the places, which was really amazing to discover.
So one of the first things we're going to go back... there we go. One of the first kind of like learning to live together with Rust, one of the first things we wanted to go do as we were looking at what our host operating system was going to be on that computer. We were looking to... and you've got to have an operating system that actually runs on the CPU itself. And we were looking at a bunch of different operating systems potentially, but also looking at illumos.
I know people assume that illumos was a foregone conclusion for us. Yes, we had a lot of experience with illumos, but it was not a foregone conclusion. We really wanted to explore all these different options. There's an RFD. We have these things called requests for discussion, very much inspired by IETF RFCs. We have an RFD 26 that goes into all of this. If you look at RFD 26, one of the top criteria for the operating system is like, we think that we're going to be rewriting a bunch of this operating system in Rust.
So we want to make sure that we're picking a host OS that is going to be really amenable to in-kernel Rust. And I don't have to tell you that there are other operating systems that make us a little bit nervous with their various opinions and so on. So we were really looking at illumos because we knew that illumos would be hospitable to it. But we needed to add support for Rust for illumos and Rust.
And this is always eye-opening, right, because you can learn a lot about a community by the way it treats small communities. And illumos is a small community, no question. What was really interesting, and we've seen this in other communities, right? When we... I mean, I don't want to name names, but I guess I am. When we did the Go port to illumos, Go invented the notion of second-class ports so they could put us in it. And you're like, well, this is very uplifting. I'm really glad, thank you.
I'm really glad that I worked very hard to get your dumbass model to work on a non-Linux system. Very much a Linux... Go is very, very bound tightly to Linux. That was a very frustrating experience. The experience with... there were a lot of burrs along the way. One of the things that was definitely frustrating about Rustup is that you need that entire crate ecosystem to work. And of course there is like one crate that we've got a very simple one-line fix to.
And the maintainer of the crate is like, okay, this is perfect. That's going to be in the next version. What we didn't realize is that the whole community didn't want the next version for a bunch of other reasons. Like, okay, I've just stepped into like a marital disagreement, what have you. And fortunately, we got past all that and the community has been very supportive. And that was a good... like, okay, this is good. Folks are interested in these other systems, which is really important.
And then we get the kind of the protracted Rust honeymoon. This is a talk given by my colleagues Dave and Adam Leventhal. This is during COVID. Those are self-haircuts that they both gave themselves. I think they both gave themselves very good self-haircuts. I've got some self-haircuts from COVID that are not as good. I think these guys look sharp. But this is where we are beginning to use Rust at the top layers of the stack.
And in particular, when you're starting a computer company with a small number of people, there's so many things to go do. And what Dave and Adam did is they kind of started at the far end of the API. What do we want the API for this machine to be? Because we want this to be effectively an elastic compute, elastic computer. We want it to be API-driven. They went to that kind of far end and began to use Rust at that API endpoint and developed something called Dropshot, which they described in this talk.
And Dropshot used the obviously incredible powerful proc macros to allow you to decorate your server such that we can generate the OpenAPI spec from the code. The code ends up being the contract. And this was, I think, for Adam in particular, who's coming in being like, I don't know, I think like, you know, Java is kind of the way these things are done, but has got a very API-centric disposition. As he got into this, he's like, oh, this is amazing. This is such the right fit.
We also were using it downstack. So this is a terrific talk by my colleague Cliff Biffle. We did our own embedded operating system, which we came to, by the way, reluctantly. We wanted to use other things that were out there, but it was very important for us to be all Rust. So a bit of a tip of the hat to our own hubris. We called the operating system Hubris, a microkernel-based operating system. The debugger, of course, is called Humility.
The irony is that there's a lot more hubris in Humility and a lot more humility in Hubris. But, you know, I digress. Cliff gave a terrific talk at the Open Source Firmware Conference on Hubris and Humility. And now we're at the point where things are just clicking, just all over the place. This is very, very clear. I write a blog post in 2020, Rust after the honeymoon, and we are just discovering... I mean, there's so much to love here.
And this is where I'm getting into like the stuff that is harder to find. I do like string continuations. String continuations are like something that is like impossible to Google. And you don't necessarily know that it exists, but if you put a... you've got a string and you put a backslash as the last character, you can continue that string and all leading whitespace will be elided. So you can have a multi... and if you're an 80-column purist, as I am, this is extremely important to maintain God's intent of living within 80 columns.
So there's a bunch of like little stuff like this, but really powerful stuff, important stuff, and we were just... it was great. So 2021, we're nest-building. This is a talk that Adam gave with our colleague Ben Nekron on doing USDT probes and DTrace that allows you to instrument your Rust code with DTrace. Have DTrace probes that have zero probe effect when disabled, and you can go actually enable those in production.
So we're doing all sorts of really neat things. Then we ship the product 2023. We actually shipped this thing with Arian's tweet of kind of how it started with like quite literally a cocktail napkin, and then the actual machine. So things are going great. We got a whole brood. We've done things from the bottom of the stack to the top of the stack, the lowest layers. We've done several different operating systems, many different debuggers. Rust is working great for us. So it's terrific.
And then marital tribulations. You know, in every... you can really judge the strength of a marriage by its ability to endure tribulations. And this, I think, is the most serious tribulation we have had with Rust is around cancellation safety in async Rust. And my colleague Rain gave a phenomenal talk at RustConf last year. If you haven't seen this talk, it is a must-watch talk. Honestly, one of the best talks I've ever seen.
Rain's talk is so good for many reasons. One, it's technical. It's terrific. Also, Rain is able to express her disappointment with great affection, with great kindness. It is both kind and firm. Kind in that, like, I know we didn't mean to make something that was this difficult to use, but firm with the, like, what we have made in terms of async cancellation safety is a real problem for Rust.
So when you cancel a future, the ability to actually clean that thing up is borderline impossible. We've got an RFD. I point people to obviously Rain's terrific talk, our RFDs 397 and 400 that talk about the problem and the solution. But the solution is heuristic. I would also say, by the way, that there are... I mean, we did see this a little bit when people would... there'd be people like, go, well, of course, yeah, I mean, obviously this is something that you need to know about.
If you are cancelling a future, it's like... and the cancellation of futures can happen in ways that are pretty surprising, honestly. So you can get some really surprising behavior. We hit this, and the RFD gets into this, but we hit this in bugs starting in 2022 that we couldn't understand. And we hit this in particular with the mutexes behave very badly with respect to the... token mutexes behave very badly with respect to cancellations.
And so you just have like a mutex that effectively is no longer protecting a critical section. You end up with programs in state that they can't possibly be in. And again, Rain's talk goes into all those details, but there's a little bit of like, well, this is just something you need to be aware of. And like a great Rust programmer will know not to do this. And you're like, you know who you sound like? You sound like a C programmer that I'm trying to get to use Rust.
And they're explaining to me that like memory safety is not an issue for them because they understand how to use memory safely. And the problem with... and you know, Rain refers to this a couple times in her talk, the problem with cancellation safety is it requires non-local reasoning, and that's the exception in Rust. I mean, the good news is, and you know, I don't want to overly fixate on this because Rust has been so great. This is what stands out in contrast. This was... we were just disappointed. We were a little let down.
It's like they're like, oh, Rust, you were supposed to protect us from this. But okay, so we have this kind of crisis. We have a way out of the crisis. It's not wholly satisfying, but we're able to kind of fix these problems and convince ourselves that like, okay, we can actually... we can make the relationship work. Like, we're still married. When the... it's like, why would you say that? Like, we're still married. It's like, well, no, I mean, we are, right? But we began to discover like, okay, these are the reasons we fell in love in the first place.
My colleague again, Adam Leventhal, gave a great talk on Progenitor. Progenitor is what we use to actually generate all of the clients for our APIs, and we were able to do all sorts of incredible things, and in particular able to do some things around the versioning of APIs that was really powerful and really important. Okay, so things are going well. We do hit like a brief sickness. This is what we call a state map.
This is a link to a podcast episode we had with Dave Pico and my colleague Eliza Weisman, where we were talking about a problem that we had hit in Tokio that was very counterintuitive. And this is another one of these async-related problems. So there is a little bit of a theme here. But okay, so we've got that problem. We kind of fixed that. That's just a bug.
My colleague Dave Pico gave an extraordinary talk on how we update the system. Very much uses Dropshot and the adding versioning to Dropshot, which is this REST services library that we use all over the place at Oxide. And Dave used the versioning of that to really be able to allow us to update the system because one of the things we need to do in an Oxide rack is an Oxide rack is potentially running behind an air gap.
We have a distributed system that we need to update. We need to update that system while the system is live. Which means that when you update the system, it's in this kind of hybrid state. And the versioning that we put in Dropshot, which itself very much depends on the flexibility that we have in Rust, allows us to do this. So there's this very clear through line from our language choice to being able to do something that from an Oxide customer perspective, you're like, oh, wow, that's amazing.
That you can actually update this distributed system over an air gap without operator intervention, ideally when it works, when it's up. Systems not up, down systems or down systems. So, you know, with that caveat. So we do hit a recent and mysterious illness. My colleague Sean Klein gave a great talk on this. We call this future lock. This is where you have effectively implicit and surprising dependencies that prevent your program from running.
Sean again gave a great talk on this. One of the things I love about this talk is that this is like the entire Oxide cinematic universe. The talk within the talk within the talk. Sean references also Rain's talk. And the thing that was really interesting to us about future lock, the thing that I love about future lock, this is a problem that we saw once. So we didn't see... we didn't have to see this problem, unlike with the cancellation, the async cancellation safety issue.
Where we saw many different bugs with many different manifestations before we could put it together and figure out what happened. With future lock, we saw it exactly once. We debugged it. We were able to then turn that into something that could reproduce it, and you could talk about the problem. And we felt that it needed a new kind of name because it's not exactly... it has deadlock characteristics, but it's definitely not a deadlock. So a great talk to check out. You can also check out RFD 609 on that.
We kind of... where we are now, we're in a pretty great spot with Rust. And now really beginning to look about what we can go do beyond the foundation that we've got. So this is my colleague Will Chandler at Oxide, and he's demonstrating something called Hon, which is a post-mortem debugger for Tokio, for async programs that allow you to get into complete async state.
So we can see, I mean, it's one of the things that's a little bit frustrating about a program that uses async is that the program ceases to make progress. And you know that, like, in that core dump, if you were to take a core dump of that process, you have all of this state, like you have all the futures, all the tasks are in there, but the debugging support to actually navigate all of that is often not present. And so Hon, we actually just open-sourced that recently. You can go check that out.
But this is one of the... and we have demos every Friday, and this is one of these demos where like someone is demoing penicillin. People are like, 'Hey, thank you, God. When can I get the penicillin? Stop the drug trial.' So this is really interesting. So I mean, Rust has been... again, I don't want to fixate on the reason that some of these divots have been so outsized for us is because Rust has been so good in so many different dimensions.
The artifacts are terrific. I was talking with Dave as we were kind of thinking about this keynote, and Dave's like, you know what's amazing? We used to just have ambient core dumps all the time in Joyent. We were running... when you're running in a dynamic language, you've got all sorts of problems that only exist in production. And this is like... this is the real challenge. I think that the most important thing that Rust did is Rust has shifted the cognitive load from running a system in production to the developer in development.
A developer, a software engineer is forced to think about all of these problems that historically have only emerged in production. So I mean, GC is a great example of that. GC is great for the developer. Oh, I get to have as much as I want. It's terrible for the person that needs to actually run that as a service. And what you've done is you've shifted the cognitive load onto that user in production. And Rust actually goes the other way, shifts that cognitive load onto the developer.
And over and over again we see developers forced to deal with that complexity up front, but the result is an artifact that is a really persistent and endurable artifact. So I think that that's been the huge win of Rust. I would say conversely, our biggest technical challenges are when we have these things that... this emergent complexity in production. So as we look to the future of Rust, what do we want out of Rust? Well, one, don't screw it up.
I don't want to... not to phrase this too bluntly, but sometimes I think when you've got a very, very good thing going, and maybe in a marriage, you've got a very good thing going, you lose track of what a great thing you've got. Rust has lightning in a bottle. What we have is extraordinary as a community, as a language, and it's very important that we not screw it up. So that, like, hey, please don't screw it up.
How could we screw it up? Well, one of the ways that things get a little bit screwed up is when we do things that do accommodate that developer in development, but then result in complexity, emergent complexity in production. And those are the things that we need to be very, very wary of. If someone's like, oh, I've got an idea, this will make this much easier to develop, I think the question should always be, okay, but does it make it easier to develop at a cost of making it more difficult to debug in production?
Because if we... and the beauty of Rust is let's split that false dichotomy. We can have nice things. Let's find a way to have that robustness in production while still having a great experience in development. That's what Rust has managed to do. We love it for that. We are very much looking forward to growing old together with Rust. Thank you very much.
