The Complexity of Simplicity

Here we go.

All right. Good morning. It is great to be with you. I'm very excited, I have to tell you, to be presenting on this topic, on the topic of complexity. I think complexity, if you'll forgive the analogy, this is kind of like there's a joke that comedians famously tell to themselves privately called the aristocrats joke, and a horrifically vulgar joke that, and the objective is to kind of make it as vulgar and inappropriate as possible. I promise you I'm not going to do that this morning, but it, you know what, maybe I will. Actually, you know, I got to say it's good to be in Europe a little bit. You know, we can let our hair down. Maybe. Okay. You know what? This is okay. Noted. Noted. I appreciate that. I appreciate that audience feedback. Maybe we will. Um, the, I wasn't expecting that. Um, but the, you know, comedians kind of famously tell this joke to one another. Um, and I kind of feel the same way about this topic of complexity. Complexity is a bit of an evergreen, and many, many, many software engineers over history have presented their thoughts on complexity. Um, I have not actually, and I was really glad when Kim asked me to present on this specifically because I have not had the opportunity to present on this, and I got some thoughts actually. I got a lot of thoughts. So I'm really excited to be here.

Um, I'm going to give you a quick teaser. So, um, this is kind of, maybe amazingly, this is still the most viral tweet I've ever had. Uh, yes, still Twitter in my heart. So, we're, this is, I don't want, I don't know what happened to Twitter after 2022. So, I will, um, but back in the day, um, I had this tweet, um, asking about a conference. Should we have a conference called In Retrospect in which presenters reflect back on past work and revisit it? Now, there's actually, you may think, I mean, I've shot my mouth off a lot. Um, there's actually very little I want to revisit, by the way, which you may find quite surprising. Um, there's actually like the things I want to revisit are actually like very inside baseball and small. Um, so no, I will not be retracting any remarks made about Larry Ellison in 2011. Although I am finding that I am get, I have got an influx of, this is, I'm not making this up. In the last like two weeks I have had an influx of journalists wanting to get my take on Larry Ellison. And so look, if you don't know, I somewhat famously gave some rather unvarnished opinions about Larry Ellison and Oracle. I thought I was just speaking a truth that we all know. Namely that you should not anthropomorphize Larry Ellison. Doesn't feel like a deep thought to me. Um, but I did this in 2011. If you go to any Hacker News thread about Oracle, you'll only get a couple comments in before you'll see a reference to a talk that I gave, minute 33 if you want to go look at this talk. Um, I will not be retracting any of that. So when we talk about In Retrospect, like don't worry, I'm not going to retract any of the good stuff. Um, but I did want to, I wondered about kind of revisiting this stuff. Um, and this is a teaser because I'm going to get to in later, we'll talk about what specifically I wanted to go revisit. And it's actually the thing I really wanted to revisit was actually not even recorded. So this is a teaser. Um, we'll get to this in a little bit. Um, but it is kind of a lead-in to this topic in terms of complexity and where it comes from.

The essence of what we do in software systems is the creation of abstraction. That's what makes this kind of magical. You know, we build these castles of the mind out of abstraction. I mean, none of this, like at some level, like none of this actually exists. Uh, when I was first, I was at Sun Microsystems, a now defunct computer company. I have to clarify that. My daughter thought it was a brewery apparently at some point. So, I want to clarify that I spent 14 years at Sun Microsystems. Um, and when I first came out to work at Sun, I worked with an engineer named Jeff Bonwick, who went on to invent ZFS. And I was in Jeff's office, we were debugging a problem, and Jeff says, 'Does it bother you that none of this actually exists?' And I'm like, 'Come again.' And I'm like 22 or whatever, and I'm like, 'Have I made a horrific career mistake by working with this individual? Like, are you on drugs?' Um, he's like, 'No, no, no. None of this exists, right? This is like your name. Your name doesn't exist. We just made it up.' And at this point, I'm really beginning to second guess my career decisions. But as you think about it, it's like actually there's a level at which none of this does exist, right? That this is all abstraction. This is all purely synthetic. We've made it all up. And the thing that is remarkable about it is it has this property of being information that we made up, but it's also machine, and it has these engineered properties. And this is where software is unlike anything we have ever done. And we're still like grappling with what that means. What does it mean to have information that functions as machine? It's got this duality. And you can see it as both that it's really interesting but also vexing. Um, and the abstraction is at the core of it. The abstractions that we create, layer after layer after layer of abstractions. These abstractions are what allow us to function at all. Without these abstractions, uh, we're just alone on the machine, right? The abstractions are really the core of software. And the abstractions themselves are qualitative. There are good abstractions and bad abstractions, but there is an aesthetic involved in those abstractions. It's not something you can really quantify. It's like, no, this API just sucks. I'm telling you it sucks, or it's great. This API is beautiful. What's the difference between these two things? Like, well, it's a lot of things, right? It's complicated. Uh, and the good abstractions isolate us from the gory details beneath us, and there are many, many, many gory details beneath us.

So I started a computer company in 2019. Uh, I joke, but through more of an observation, that I keep waiting for the day that I understand how computers work, um, because it like wasn't yesterday, don't think it's going to be today, unlikely to be tomorrow, because as it turns out, like when you actually build the entire thing, you learn how much complexity it turns out like, uh, it's actually analog, unfortunately. Digital is a lie, which is very upsetting. Um, but there's abstraction after abstraction after abstraction, and there are many great abstractions that have allowed us to build. So those are the properties of those good abstractions. The good abstractions actually allow us to not understand how it works. There's something glorious in not having to know how an operating system works or how a computer actually works. And part of your education and my education was ripping those abstractions apart and learning about the mechanics underneath. And I actually view the primary role of a technical education in computer engineering or computer science, software engineering, is giving you humility that anything works at all. That once you understand how complicated, this is what a former colleague of mine called the miracle of boot. Once you understand how extraordinarily complicated it is for a machine to work at all, you're humbled. You're like, how does any of this work? And you realize it works because of all these abstractions. These abstractions are very, very important. They're not all good. That's the problem. There are bad abstractions, bad abstractions that leak, where you actually like, well, I would love to not have to know how this thing works, unfortunately it's broken, so I actually do have to understand how it works, or this is an API that is porous, where the implementation details shine through in a way that convolutes me with the thing I'm trying to build on top of, and this is where complexity blossoms, and you end up with this entanglement that you did not intend.

Um, and if you haven't read it recently, um, I would really recommend reading Fred Brooks's 1986 essay, No Silver Bullet. Um, and so paraphrasing Aristotle or borrowing from Aristotle, Fred Brooks calls this accidental complexity. Accidental complexity is the complexity that we induced in the problem with that kind of entanglement. It actually didn't have to be this complicated. It could actually be simpler. And you could actually, if you could eliminate this accidental complexity, you could be reduced to the essential complexity. The essential complexity is the stuff that's just endemic to the problem. This is when someone's like, well, how complicated could this be? It's like, oh, glad you asked. Like, here are the, like, how complicated, like, why is this such a big deal? Like, yeah, like, we, I provisioned a VM, and like, yeah, the VM is now, like, yeah, I'm logged into the VM. Like, why are you celebrating? It's like, well, let me explain all the things that have to happen to go provision a VM and all of the steps. And by the time, you know, you've like, okay, God, mercy. Great. I'll celebrate with you. Sounds great. Um, because there's a lot of endemic complexity when you go spin up a VM on a cloud. And we want to reduce ourselves to just that essential complexity. We don't want to have the accidental complexity.

Now I think one of the things that is a bit of a paradox, and this is what Brooks doesn't really identify, is that in our domain where abstractions are the coin of the realm, abstractions are everything. When you have accidental complexity at one layer of the stack, it can explode, and that accidental complexity can become the essential complexity of someone trying to build on top of you. And I think this is really important because I think that we don't realize how contagious complexity can be when it explodes that way. So when we have accidental complexity that we don't manage, it doesn't merely accrue. It's nonlinear and it's qualitative, right? It's hard to like put a number on it. So we really need to kind of be mindful of where that complexity comes from and the outsized impact that complexity can have at different layers of the stack. And you can tame complexity within existing systems. We can talk about that. There's a lot of things you can go do, but it's uphill. Um, most changes in complexity happen actually with new systems. That's where we get most deltas in complexity.

So I want to taxonomize this a little bit. So you can, I'm freewheeling a little bit here, but I'd love to get your take on kind of how I've taxonomized this. I don't want to be too pat about this, but as you kind of think about different systems and how they're built and what they represent in terms of complexity, some systems are what I'm going to call emergent. And I'm not trying to be pejorative with this. Um, I'm going to differentiate emergent from engineered. And again, I'm trying to be careful about not trying to impose my own kind of value system, but an emergent technology is one that emerges in reaction to technologies around it. And this is still like very important. I don't want to dismiss the importance of that, versus an engineered system is more, let's start with a clean sheet of paper. What does it look like if we start with a clean sheet of paper and solve the problem in the abstract, more from first principles? Trying to take on that essential complexity from first principles. But again, that's not in any way to dismiss the importance. Many of our systems are emergent systems. You could also view that as more iterative systems. Very, very important. We're going to get into lots of details about an emergent system versus an engineered system. And then the actual resulting system, the abstractions that that resulting system provides, has their own kind of spectrum from the simple to the complex. So let's treat this, got a little grid here. Uh, yes, I've got a bit of a value judgment. I do value simplicity over complexity. So we're going to make, and okay, look, I do value engineered systems over emergent systems. I guess I'm going, fine. Let me just do. All right. Big reveal. It is a value judgment all along. No, it's really not. I actually really, but I am going to put, sorry, simple and engineered. I'm going to put that in my Gartner magic quadrant. I shouldn't even say that. Uh, but we're going to be up and to the right on simple and engineered. And I want to explore each of these four quadrants because they are different. And again, I don't want to be too pat. We're going to talk about a lot of things that you could reasonably put in any of the quadrants. So, but I want to label them. And um, let's actually, I want to actually start. We're going to kind of move around a little bit randomly here, but I want to start up here in the upper left when we've got a system that is engineered, but it's complex. The abstractions that it offers are complex.

There's, uh, you know, Paul Erdős, the mathematician, would talk about proofs that are what he called in the book, Paul Erdős, weird dude. Uh, Paul Erdős, I love the fact that Paul Erdős lived, I think, 93, uh, was a child prodigy, uh, wanted to do math until he like died, which he basically did. So he basically, I mean, it's harder to do more math in your life than Paul Erdős just on the outset. The guy also did amphetamines his entire life. So it's like, okay, that's like, I guess that's an added kicker. The guy's on stimulants his entire life. And Paul Erdős, I do like, by the way, that Paul Erdős is actually only the second most prolific mathematician in history. Leonhard Euler, man, that guy is just off the hook, that dude. Um, anyway, but Paul Erdős did a lot of math. And Paul Erdős, um, he resented God, who he called the supreme fascist. Again, idiosyncratic guy. Um, maybe like, maybe a little less on the amphetamines, but he believed that the supreme fascist had a book of the most elegant proofs. And when he would look at a proof that he felt was too complicated, he would say, 'That's not in the book. You've proved it true. You've proved the theorem, but that's not in the book.' These constructed systems are not in the book. No one's going to look at these constructed systems and say, 'Wow, that's tight. That's elegant.' Now, that's not, again, not being pejorative, because they've got a very high degree of essential complexity. These are solving, the good ones anyway, are solving really complicated problems.

They are often serving larger functionality. So they're doing something, the software system is part of something larger. It is an airline reservation system, right? To take some of the earliest, you look at the earliest examples of computing. It's always interesting to me where we use computing first, right? SABRE being one of the very earliest computing systems. And it turns out like the reservation system is super, super complicated. There's a lot of essential complexity or complexity that's become essential over time. But these are, so these are big systems, constructed engineered systems. They're big. They're big in terms of scope, what they do. They're big in terms of people. They're big in terms of time. This is software in the large. This is, you've got program managers. You've got roadmaps. You've got, this is the stuff of The Mythical Man-Month. Fred Brooks's terrific book, iconic book about the development of OS/360 at IBM. For these systems, complexity is an absolute killer. Accidental complexity in particular, right? In Brooks's '86 piece on No Silver Bullet, talking about essential complexity versus accidental complexity, is really pointed right at these systems, saying that you've got to focus on the essential complexity and eliminate all that accidental complexity because it kills you. And these are the systems that are afflicted by scope creep, right? Or second system syndrome. Second system syndrome is where you've developed a system and it's kind of regrettable. You want to replace it. Like it works, but we're going to replace it. And everyone's like, 'Okay, great, we're going to replace it.' Okay, train is leaving the station. Everyone's got to be on it. So with the second system syndrome, you're like, 'I've got so many ideas of what we need to go do in this second system.' And the second system just becomes bloated. Uh, and this is where you, very much related to scope creep.

Now, bad news. The most foundational software components that we have, operating systems, compilers, databases, the stuff that we like, we literally can't get out of bed without. I mean, literally in our case, right, those of us relying on computers to wake us up in the morning, which I think is most of us at this point, uh, literally can't get out of bed without these foundational technologies. These are big engineered constructed systems. And in the beginning they were all constructed because, and this is kind of amazing. This is where you think like, I don't think of myself as a living fossil, but I guess I am, because you think about like in my lifetime, in, you know, if you wanted, and not just in my lifetime, but in like my adult lifetime, right, if you wanted a database in 1990, you had to write it or buy it from someone who wrote it. Like those were your choices, and it was really big and complicated. So if you look at the thinking on software engineering really through the 1990s, it's all reflecting back on these systems because these were the only systems we built. This is largely because we live in a pre-open source era. Open source is a really important development in the history of humanity because it allowed us to get out from underneath building constructed systems all of the time because these systems are really, really hard. And if you look at the, I mean, it's kind of again interesting to go do, it's kind of like the aristocrats joke. You can go find any pioneer and look what they had to say about complexity. Every computer science pioneer weighed in on complexity. Dijkstra weighed in on it. Seemingly Dijkstra wanted to go one for one. So anyone, anytime someone else weighed in on complexity, he would weigh in on complexity yet again. Um, complexity was the absolute killer of these systems, and these folks knew it. So this is a Brian Kernighan quote, but there are many. Um, but 'Controlling complexity is the essence of computer programming,' says Brian Kernighan. Um, this was the top of mind for software engineering certainly when I was coming up. And the reason that this is top of mind is because these constructed complicated systems, these constructed engineered systems, have a really basic problem because their abstractions are complicated and there's not really a goal of having simplicity in the abstraction. It's kind of a non-goal for these systems. And when simplicity in the abstraction is a non-goal, you don't know what to say no to. What do you say no to? What do you say is inconsistent with the philosophy of this system?

And this is like, I have very carefully, maybe I shouldn't be quite so careful given some of the folks here, but I've been very careful to like not mention Kubernetes as part of this. Like, is Kubernetes a constructed system? But this is, Kubernetes has historically struggled with what to say no to. Fair. Okay. That's, you know, you never know. The feels, I'm trying again, trying to be maximally generous here. Maybe too maximally generous. Um, because simplicity in abstraction is a bit of a non-goal, and that's again, that's a little bit, majority of the structure, it's very important when we focus, if in the great words of Steve Jobs. Steve Jobs, I got a complicated relationship with, like Paul Graham. It's like, you know, Paul Graham, the guy can say things that I strongly agree with and strongly disagree with like in the same sentence. I can, it's anyway, Steve Jobs is kind of the same way. But if you look at Jobs at the top of his game, it's really terrific. And if you look at his 1997 Worldwide Developer Conference keynote, which I swear was like shot on a Handycam or whatever. You can find this on YouTube. It's grainy. Um, Jobs, uh, it is, first of all, you're reminded like what Jobs, that kind of like his apex look like, but Jobs is very direct about what he needs to go do with Apple, namely they need to stop doing a bunch of things. He's returned to Apple, Apple has bought NeXT, and the line that he had that definitely stuck with me from that keynote is, 'Focus is saying no. People think focus is saying yes, but that's actually not focus. Focus is the art of saying no.' And these systems don't know how to focus themselves because they don't know how to say no. Which is why they spread to the horizon. And when constructed systems are done well, there is someone at the helm or a group at the helm or unifying principles that allow them to know what to say no to and give them permission to say no. When they don't, when they say yes to everything, these things are a disaster because that's constructed systems.

There is a counterbalance to constructed systems. Maybe folks in this room, you're like, 'Jesus Christ, this is so complicated. This constructed system, this big mofo is so complicated. Does it need to be this complicated? Can't we do this with much less?' And these are the rebellious systems. So these are what I would call emergent systems. They are emerging often in reaction to these constructed systems and asking like, 'Wait a minute, this is like, you guys have way overthought this. This can be way simpler.' And you have now created what you think is essential complexity. That's actually complexity that you've created and fed back on itself. So you're in something like a complexity death spiral. We can do this much, much, much more simply, say the rebellious systems. And some of our most important systems have come from these acts of complexity rebellion, from saying, 'No, this is way too complicated. We can be way simpler.' And of course the canonical example of this is Unix versus Multics. Unix, the name is a pun on Multics. They are castrating Multics. That's where the name comes from, right? And they are eliminating apparently lots of things from Multics. Like, 'I thought I needed that.' 'No, you don't.' It's like, 'Oh, no, wait a minute, I think I need that.' All right. So Unix was minimal with respect to Multics, and they were right, broadly speaking. And this is a trend we've seen over and over and over again. C versus PL/I, that's, I think that's a bit of a stretch. RISC versus CISC, for sure, right? CISC, complex instruction set architecture. RISC, the reduced instruction set architecture. Instead of having, x86 has an actual instruction that does that, does binary encoded decimals, it's insane. RISC says, actually, let's reduce our instruction set to something much simpler, and then we can make the CPU actually much faster. And this was extraordinary when Dave Patterson at Berkeley developing RISC, they were able to do with a very, very, very small team what much larger teams have been unable to do because they rebelled against that constructed system of the CISC microprocessors. So, a really extraordinary, Rails versus J2EE, right? If you remember, I mean, I think it was one of the first YouTube videos I watched was David Heinemeier Hansson demonstrating deploying on Rails. I think it was what, 2006, something like that, 2007 maybe. But it's actually so funny because that YouTube video, which I felt was watched by kind of everybody, he had this like 15, famously 15 minutes to set up an MVC application using Rails at a time when J2EE was definitely not 15 minutes. 15 minutes to like, you know, download the jar or whatever. Um, and I felt that video was watched by everybody. You go today and it's only got like 400,000 views on it. You're like, oh my god, the planet was small in 2006. Um, but Rails very much at that, PDF versus PostScript, right? So, and we'll talk about PostScript itself, that PostScript is a different kind of revolution. Certainly, microservices versus monoliths, there's some asterisks on that one. We'll talk about that in just a second. Node, I was really active in the Node community starting 2010. Node very much defined itself as giving you asynchrony without the complexity of Twisted and some of these other asynchronous event processing things in Python and Ruby, which were super complicated. And Node's like, yo, it can be way simpler. Let's do JavaScript on the server side, which sounds like such an innocent idea. As it turns out, that kind of ended in tragedy, but it felt right at the time, certainly with respect to like EventMachine and Twisted Python. Um, I would stand by that. Um, NoSQL versus SQL, right? Right on the tin. You know, I say that these things define themselves by what they aren't. And that's not a way. Again, I'm not trying to be pejorative, but like it's a little hard to, well, NoSQL doesn't define itself by what it is. And I'm like, pretty sure it does. It's right on the tin, right? And a lot of these things, ditto RISC, ditto Unix. I mean, in the name, they are defining what they're not. They are a reaction to this constructed system. Did it tell us, right, I wasn't sure who I should put here to kind of like what apple cart we're tipping over, so hopefully Ubuntu, Rocky is fair, but same thing, right? Defining what you're not. We actually don't need all this garbage. Turn it off. Let's power all that stuff off. We don't need it. We can make this a lot simpler. And these systems are enormously important, but there are risks associated with them. Risk number one, first of all, many of these, I mean, these are some of the most successful, these technologies are some of the most successful technologies in computing. And sometimes people draw the conclusion that like a rebellion is always successful. It's like, for sure it isn't. Most aren't actually. And there are many reasons why a rebellious system will in fact fail. Number one among them is that you throw out like, we don't need any of that stuff. It turns out like we did actually need some of it actually. So you actually threw out, you're discarding what you view as accidental complexity, but hey, there was some essential complexity that you also threw out. And by the way, this thing is now actually not solved your problem. And there are a couple of examples of this, or worse, like creates another problem. I mean, two, I'm going to pick on maybe kind of favorite whipping boys here, but microservices for sure, right? There was kind of a church of microservices for a while. That church has not been in session for a while. Um, because they were like, oh, we're going to break up the monolith. And you're like, okay, we're going to break up the monolith. We're going to turn this into a bunch of, it's like, you know, it's the Unix philosophy. We're going to turn these into a bunch of small things. Like, okay, but like you've got a lot of the same problems, right? And how do I now upgrade a system? Remember once I was with, I was on a panel, I mean, Uber among other things, Uber, the company was just on a microservices bender. And there was a moment like at the height, you remember, remember this, like when they're like, you know, we deploy like 13 new microservices a second. And they were, you know, whatever it was, I mean, they were loved how many new, they had microservices spreading to the horizon. And I was once with a technologist, I'm kind of like trying to ask her, it's like, surely this has like, there's complexity. And they're like, no, it's great. But it can't be like, that's not. And I kind of like blink twice if you're here against your will. Um, because like it didn't make sense. Like there's a bunch of us like that does not, like I see what you're trying to do, but so much of that was just Conway's law, right? Where people are like, actually, I just want to like, I just want the three of us to not have to go to that stupid architecture review board. I just want us to like ship this thing into production. Like it's like, okay, that's just Conway's law. That's fine. Um, but what we ended up with was distributed monoliths. And we realized that you really can't be too dogmatic about microservices. Another one is unikernels. An idea that you may not have heard of for a reason. Um, but there was a, and unikernels are an idea that's been around for a long time. It's been, it's actually been trolling me my entire career. Um, the original exokernel paper in 1995 trolled the living hell out of me, and it just doesn't seem to be going away, or I keep leaving it for dead, put it that way. In 2016, there was a big resurgence in this idea. What is the idea of unikernels? The idea of a unikernel is like, oh, this thing is so complicated, this thing, the operating system, get rid of it all. And again, I've done OS development my entire career, like, easy. It's like, no, no, we don't need any of this, and everything should be in the kernel. We're going to have a unikernel. It's like, so you actually have no protection boundaries between an application and the kernel. Like, yeah, man, we're going to get rid of those protection boundaries. Like, they've been keeping us down. It's like, okay, how much kernel development have you done, dude? Because like what you are now doing is you're saying that every single application is going to be developing an in-kernel component without any sort of protection between those components. It's like, and by the way, you're now the operating system. So if you screw up, like the system doesn't have any liveness, system resets, and like there are lots of reasons why you wouldn't want to do it this way. But unikernels had the idea of like, no, no, the complexity of operating systems is bad. It's all bad. It's all accidental complexity. It's like, it for sure is not. But no, it's all, it is all accidental complexity. We're going to eliminate all of it. Like that is going to be dead on arrival. And the, and there, and if you're a unikernel advocate, maybe you're among these people. They're very sad and mad at me because yes, I wrote a blog entry called 'Unikernels Are Unfit for Production' because they are. And this is identified, it's like, you killed unikernels. I'm like, I did not kill unikernels. I just, I, yes, I may have delivered a eulogy for unikernels, and maybe it was a little bit like our aristocrats joke. Maybe it was an off-color eulogy. All right, I just want to make sure it was in the actual ground. But the reason I thought unikernels, and the reason that unikernels are unfit for production is because they're undebuggable. They're undebuggable. You're asking everyone to be a kernel developer. And that is a bad idea. Not because people shouldn't, shouldn't expand their horizons to develop at ring zero, but because it's really hard and the consequences are high. And that protection boundary that we have between processes, between an application and the kernel, that is like the canonical great abstraction. That is the abstraction that allows us to actually hide true essential complexity. So, uh, this is when these systems fail, it is because they have overshot the mark and because they have been too reductive and they've eliminated things that we need.

Okay, so we got rebellious, we got constructed. What are an emergent complex system? I'm calling this accreted. This one maybe a little more value judgment on this, but this is a lot of what we deal with, right? Accreted systems are the ones that like you just wouldn't design it this way. Like if you were designing it from a clean sheet of paper, you definitely wouldn't land here. And everyone knows that, like no one's being like, there are, there are constraints around it. Often accreted systems were born somewhere else. And accreted system actually started off life as a constructed system or rebellious one. Like if you're the person, like there definitely, I mean, we were talking yesterday, someone was trying to recall a database that was really annoying to deal with, and someone else suggested like talk about CouchDB. I'm like, oh, CouchDB, got it. And not thought about CouchDB in a minute. CouchDB started off as kind of a rebellious system. It's very much if you are supporting CouchDB today, you are definitely viewing it as an accreted system, not, not a rebellious one. So many of these systems started off somewhere else, and they all kind, that everything kind of falls into accreted, right? Accreted is kind of at the bottom. And again, no one would design these systems this way. You know, I think there are systems that are born accreted, which you kind of have to feel sad for because like you're born accreted, you're like, and these are systems that even their proponents don't like. They're like, oh yeah, I know it's terrible. It is terrible. So like, I mean, no, you ACPI, UEFI, the BMC, base management controller, IPMI, any acronym with intelligent in it, like that's definitely a red flag. These are things that came out of the womb accreted. I mean, born losers at some level. I mean, really kind of sad for these things. But that's not most things. Most things start somewhere else and land, land in accreted.

And these things are problematic. All of this accretion is problematic. On the one hand, like the accretion is glorious because it is the fact that software is both information and machine allows us to accumulate all this stuff without, there's actually no cost of goods sold of it, right? It is just information. So that's part of why this accretion grows. And the accretion is great at some level, but also horrific because all of the complexity down there in that accretion becomes actually the essential complexity that you're building on top of. The UEFI becomes the complexity that now constrains what someone else might go do, and it ossifies us in this web of complexity. And so, you know, we often call these things technical debt is another kind of, another name for them. And in terms of the failure modes, it's a little hard to like failure modes. Like, I mean, especially if you're born accreted, like aren't you born a failure mode? It's like, yeah, at some level. But it's not surprising that like these projects, like, you know, morale is low on these projects. If you go to like the things, especially that are born accreted, it's like, oh, what a surprise. It's a bug-infested death march to nowhere. It's not surprising. The, you know, it's like written by prisoners or whatever. So you, the everyone's looking for another job. The, and the thing is, the thing is, I know, I know, no, the thing is like folks that were working on those systems, like actually, that's funny. I was actually incarcerated at the time, and that was like a condition of my release that I go work on ACPI. So no, I'm not, not even disputing it. And I was actually like, thanks to cruel and unusual punishment, ultimately I was able to get a stay on that. But the real problem with these accreted systems is that they work well enough. This is the other problem is like these things are not, they may be broken from an abstraction point of view. They've got complicated abstractions, but they actually function, and actually ripping them out is not simple. Like there's a reason that rebellion is around constructed systems and not accreted ones. So it's easy, you know, with all these, you got, you got the complexity coming out of constructed systems, even the complexity coming out of like rebellions that kind of fizzle. So you've got those all kind of accreting, and then of course we've all dealt, we many of us deal with, with accreted technologies, right? And it's easy to become overwhelmed with despair about complexity, like, oh my god, the universe is just so complicated, and it's getting worse and worse and worse. And, you know, again, because that's all the work we do is in these systems, we can become overwhelmed with despair.

And so, and this is where we get to kind of that aside I promised you at the beginning. So, um, this is a talk I gave in 2018. That tweet actually was from June of 2018, and I did not have this talk in mind when I was tweeting it out. This is a talk I tried to get a very like contrarian pose here of me, you know, a kind of a WTF look. Um, this is a talk that I gave at Jane Street very generously had me out to give a talk on debugging pathological performance. And the talk is, I loved the opportunity, very much stand by the talk. Okay. So, but part of this talk is focusing on all of the performance artifacts created by these very complicated systems and going into some systems that you would think you'd be pretty basic, but then looking at their systems' performance and realizing that they were actually inducing very pathological systems.

All right, so the video ends and there was a Q&A session, and this is the part I want to take back, but again, it wasn't recorded. There was a Q&A session, and one of the questions was, 'God, this is like so complicated, and this just seems to get like worse and worse and worse.' And it was kind of a question coming out of this complexity despair, and like, do you think this is like, is this going to get better basically? And I fell into a trap that I, that I would like to not fall into as a senior technologist, um, that as senior technologist shouldn't fall into, but I kind of jumped into the despair pit along, along with the questioner, and I'm like, 'Yeah, I just don't think it's going to get any better. I think it's just going to get worse basically. I mean, we should all just die.' I mean, it's basically the, I, the, like, I, like that was like the whole answer to the question as I recall it. I was just like, 'Nope, just going to get worse. Next question. I see another hand up over there.' I mean, it was like there was no kind of nuance to it. And there's a real irony to that. And I reflected back on that moment only a couple months later through the summer of 2018 because my life changed a lot in 2018. And the universe kind of reminded me that no, no, no, no, no. Hey dude, it doesn't always get worse, and sometimes it takes a big leap for the better. Oh, fight me.

So I, in particular, I, like a lot of people in 2018, started exploring Rust, and Rust was a watershed moment for me. I went in with despair. I'm like, it's got to be, I hope it's Rust, because if it's not, I'm going back to C. I'm very, I'm burned out from Go. I've been burned out from, I'm definitely burned out on Node. Node and I are having a divorce. It's good. JavaScript on the server turns out is exactly what you would think. Awful. And I went into Rust pretty skeptical, honestly, and it came back realizing that, wow, there were so many things that I viewed to be essential complexity that were actually accidental complexity. If you have not learned Rust and you are unfamiliar, the most important thing about Rust to someone who's new to Rust is the way it handles errors, right? That the sum types, as it turns out, are life-changing from an error handling perspective. And much of what we do in system software is handling errors. And I'm like, wow, why have I been resisting this because of my concerns about the borrow checker when algebraic types and my ability to revolutionize error checking was kind of, was waiting for me the entire time. And it's not the only language to do, I mean, this is borrowing it from OCaml, but I, and this blog entry goes into a bunch of the details of the things that I found where Rust was. And I'm, trust me, I'm the last person that would have thought that a programming language was going to change my life. I'm not a PL person. I'm a systems person. I take a very pragmatic view on programming languages. Programming languages are there to do a job. They're not there for these kind of, these other accoutrements, like not interested. But Rust totally changed my thinking in that regard.

And coming back here, you're like, oh yeah, right, right, there's another quadrant, right? The revolutionary systems. And the revolutionary systems are the ones that are engineered and deliberate, but part of the revolution is in the abstraction. We are going to deliberately change the abstraction, and we're going to do it thoughtfully. We're not going to do it, we're not a reaction to, we're not throwing out something. We are actually doing it constructively, not reactively, and the abstraction is part of that differentiator. And these are extraordinary systems. And if you look, and we'll talk about some examples here, there are lots and lots and lots of examples of these. These are, if you look, there are some commonalities across these revolutionary systems. One, they are small, focused teams working often for an extended period of time. They tend to be, part of what makes their contribution so important is that these teams tend to have a real reverence for history. I mean, part of why Rust is so extraordinary is Rust has deliberately tried to learn everything from every other language. And whatever language you're coming from, Rust probably borrowed some element that you like from it. And that is it. It took the good stuff and not the bad stuff. And this is true of a lot of these revolutionary systems. The CDC 6600. This is, I've done these in chronological order. And yes, I put some of my own technologies on there. It's like, hey, if you, this is what you get for writing the slide. You like, you want to do, you want to put your technology on there, write your own goddamn slide. I'm running the slide, I get to put my technologies on there. I feel I can defend it. So there you go. The Control Data 6600, extraordinary machine developed at CDC. Very famous memo written by Thomas J. Watson Jr., CEO of IBM, to his executive staff at IBM, indignant that Control Data has developed a machine faster than the best IBM machine at the time with, I think it was 31 people, including the janitor. And there's this great memorandum that, and if you are, and by the way, when we started Oxide, one of our earliest engineers, Josh Coons, printed that out and brought it to work on like, you know, on like the fourth day of Oxide. And it's very inspiring to read because when, if you are endeavoring to build a revolutionary system, there is a lot of mountain ahead of you, and that can be really overwhelming. It is that there's a solace in that history of knowing that there are people that have come before you who did exactly what you are endeavoring to do, that climbed maybe not that mountain but a similar one. The odds felt similarly long with a similarly small group of people, and indeed that those are the only way those mountains get summited. So that I love the CDC 6600.

System R from IBM, this is the first system to have the idea of like a query language. Databases should be relational. Unbelievable system that actually ultimately failed in part because it didn't run on Unix, which I think is just very ironic. TCP/IP for sure, and showing us that there could actually be simpler abstractions where there were a bunch of problems about the network. You don't want your application to have to worry about the health of the network. And before TCP/IP, like applications definitely didn't worry about the health of the network, and the network collapsed, right? The great internet collapse in 1986. TCP/IP very much a reaction to that. Obviously the Macintosh and insanely great, right? A team that Steve Levy has got a great book on the development of Macintosh. AS/400 at IBM, Project Silverlake. AS/400 was an integrated hardware-software machine, revolutionary, an explosively large business for IBM. Arguably the most successful computing product ever introduced is the AS/400. I made the huge mistake, by the way, of, we were with venture capitalists in a partner meeting, and they're like, 'What is the best historical analog for Oxide?' And I'm like, 'The AS/400.' And looking at the blank looks that I'm getting now on the AS/400, I similarly blank looks in the partnership. That part of my brain is like, 'Nice going, jackass. Like AS/400. Now you get to hold a history lesson on the AS/400. Like, hey, hey, next time say iPhone. Just say iPhone. Don't, don't overthink it, right? People know what the iPhone is. People don't know what the AS/400 is. Well, they should know what the AS/400 is.' And that, and you're like, 'And that's why they're going to pass on you, you idiot.' Like that conversation is going on in my head. But the AS/400 is a very revolutionary system. PostScript, unbelievable how important PostScript is, right? This idea that you should have a Turing-complete programming language on your printer. It was unbelievable. PDF was actually a, PostScript actually gave us too much, and part of the reason that PDF is a rebellious system is it's like, actually, let's prune this back to something that actually just focuses on, we don't need to solve all problems, which PostScript very much is doing. PostScript is amazing. NeXT, at NeXT, Steve Jobs. If you haven't, Steve Jobs and the NeXT Big Thing by Randall Stiles, excellent book on Steve Jobs and the history of NeXT. Walter Isaacson did a terrible job describing it. Spends like six pages on NeXT in his biography of Steve Jobs. The work at NeXT is actually, I think that is the epicenter of Steve Jobs's career. It's not the most commercially successful, but I think it actually tells us what would happen later at Apple. And NeXTSTEP is the operating system they developed there. Very revolutionary. I think Python, NFS developed at Sun. TeamWare and BitKeeper, would get very much a reaction to BitKeeper, but the bring-over, modify, merge model of software development, very revolutionary. I actually grew up with TeamWare at Sun. So this is when colleagues were talking about the problems they would have with Perforce or Subversion, I literally didn't have a way of relating to them because I just didn't have the complexity that they had to deal with because I was living in the future, as it turns out, with this bring-over, modify, merge model where everybody has a complete snapshot of the tree all the time. And that, of course, is what we do with Git today. Plan 9 from Bell Labs, definitely took kind of the file abstraction to its extreme. HTTP for sure. Java developed at Sun. Disco, which was started at Stanford, that became VMware. I think, of course, now that VMware has been eaten by private equity, it can be hard to remember how extraordinarily innovative early VMware was. And they showed us that we could actually virtualize x86 at a time when many people thought we couldn't. SQLite in the early 2000s. SQLite having the idea like, actually, I think a database should be a library. And boy, how many systems have been built by SQLite because it's so reduced the complexity that we have to go deal with. I can actually, I can, there's so much actual accidental complexity I can just eliminate. I don't even have to talk to a socket over this thing. It's right here. LLVM, very, very important. Actually, LLVM, load-bearing for Rust for sure. Chris Lattner and crew developing this starting at the University of Illinois and ultimately ending up at Apple. Very, very important development because it allowed us, I mean, LLVM removed all of the complexity from code generation or abstracted it in a way that GCC definitely did not. And so LLVM honestly allowed for the explosion of languages that came after it, very much including Rust. x86-64 from AMD, the iPhone obviously, ZFS was its son. And I, you know, this one I had very much a ringside seat for. And it was wild to watch. I mean, as long as I knew Jeff, Jeff Bonwick, he wanted to go solve the problem. And to watch Jeff and Matt together and then growing a large team around them, blow up all the abstractions related to volume management, file systems, and turning it into something. When you look at the architecture of ZFS, you're like, it's so simple. Like, why am I dealing with any of this complexity on any other system? It's like, well, because it required someone to go to a clean sheet of paper. DTrace, I developed at Sun. I stand by it. Again, my slide, my technology, I don't care. I stand by it. S3, very important development at Amazon, right? S3 is what leads us into the cloud era. And EC2 developed a couple years later. Go, I'm going to put in the revolutionary category. I think it kind of, it's right on the cusp of revolutionary rebellion. And I'm trying to not throw up my mouth when I write Go on a slide, so I'm practicing that. TypeScript, actually, TypeScript is another one of these things. Super important development, right? So, and developed by Microsoft in 2012, and TypeScript allowed us to gradually bring clean water to JavaScript. Really, exactly. Really, really important. It's like, but I don't have to drink out of the outhouse. It's like, no, you don't have to drink out of the outhouse. It's like, well, I want to drink out of the outhouse. Like, well, then we'll go into the next one. We'll go to the next house. This person's gonna... All right. They want to have, fine. Docker, I would for sure put in this category. I think Docker plopped into the accreted systems pretty quickly. It may have set a land speed record from revolutionary to accreted. But it definitely started out as revolutionary, right? There was that, there was that like 12 months of real revolution for Docker. I absolutely put Rust in that category, and I put Oxide in that category, honestly. So Oxide, we are a computer company. Come see our hardware over here with Travis and me. We sell hardware and software together, and all ACPI, UEFI, the BMC, IPMI, those don't exist in our system because we took a clean sheet of paper. And if you know computing, you're like, how does that not exist? It's like, because you take a clean sheet of paper, because no one should have those. You shouldn't have those things in the system. They are an artifact of a history of iterative broken architecture, and we took a clean sheet of paper to Oxide.

So now, in terms, there are risks too. One of the most straightforward risks is revolutionary systems are so revolutionary, you actually don't know when to ship. And this is a challenge because, and this has been a challenge for every one of those things. Every one of those things shipped late. First of all, took longer than their creators thought they were going to take. I mean, including DTrace, to put my own self out there. Very much including Oxide. And like, when do you draw the line and ship the thing? Because you do actually have to ship it. You actually have to ship it. And it can be really tough when you've got a revolutionary system because you are innovating in the abstraction. It takes a while for adoption because you are trying to upgrade the most calcified software of all, the software in our minds, right? You're changing the way people think about the system. And especially if you're coming at a clear blue sky and people aren't ready for it, there may be a lot of resistance to it. And you certainly see that with something like Rust where you're like, okay, no, like you can't do it this way. Like you can't do a borrow checker like that. No, you can't. It's like, actually you can, but like that's so we get kind of natural resistance to that stuff. This can really slow adoption. And of course then you also have like revolutionary systems that like the like adoption is slow because this thing kind of sucks, right? You also get the like, I think it's revolutionary. It's like, yeah, you're crazy. It's actually not revolutionary. Like I get you spent like, you know, I get you spent many years of your life on Perl 6. I understand that. It is not as revolutionary. And I get there are a lot of apocalypses or whatever the hell you're calling them, but like this is, this sounds crazy. It's, this is not, we're not using this thing to pick on one revolutionary system that failed. There are of course many of them.

The biggest challenge for a revolutionary system is to stay funded. Because you, it can take a while, and almost every one of those systems took a while to get results. But also the way they built them, and I, you could do easily, and I won't, but we could do another presentation on how to build a revolutionary system because there are some tricks to staying funded. And one of those things is to hit milestones early. You got to hit the first milestone. And it's, we did this with DTrace, remember vividly hitting the first milestone where everyone was like, oh my god, okay, this is, this really changes things. And we viewed that in the first, was a project that ran for almost four years, but we had six months, I felt we had six months to prove out that core thesis, and we did that. But we really had to focus in that first six months of proving that out. And when you do these revolutionary systems, you have an extraordinary amount of essential complexity that you've got to deal with. If you don't deal with the essential complexity, you will have a system that doesn't work. And the art is figuring out where that essential complexity lies.

I would say that, you know, just as the origin of rebellious systems is in constructed systems, one of the things that's interesting about revolutionary systems is their origins are often in accreted systems. It's like, yeah, I've had to deal with ACPI and the BMC and UEFI, right? I've had to deal with that just like you have. And when you are mired in accreted systems, that is when revolutionary ideas begin to form. So that is a solace that we can take when we are in these extremely complicated accreted systems.

So as you're drowning in complexity, and if I could go back to 2018 and answer that question again, is it going to get better? The answer is absolutely. Not universally, not universally, but absolutely it will get better. And we will, and over time, we, this will continue to happen. We will continue to see revolutionary technologies that change our idea of what the abstraction can be. And you saw that list on the earlier slide. These are the most, some of the most important technologies we've ever developed. So we can do it. We will do it. And you can have that solace when you are working on a constructed system, you can know that there's a rebellious system, maybe like Talos, that a rebellious system that you may be developing when you're working on an accreted one, maybe there's a revolution on the other way, on the other end. And if it's not already obvious, a quote that actually, this is from Jeff Bonwick sitting in my office sighing when he was right in the middle of ZFS. He says, you know, it's very complicated to make things simple. And it is very complicated to make things simple. And I would observe it's very simple to make things complicated. With that, thank you very much.