Hey, good morning everybody. I'm Bryan Cantrill, I'm the CTO of Oxide Computer Company, and I am thrilled to be back at P99 Comp. I think I have presented at every P99 Comp, so I feel like I'm a bit of a mainstay, and P99 Conf has been a mainstay for me and for us as well. This is such a terrific conference, and I love the fact that it's all online, in part because I have pre-recorded this and I can heckle myself along with you in the audience. So looking forward to that.
And as I was wondering, like, what am I going to talk about this year, we had an amazing anniversary pass, namely software that I had written, or I had written with other folks, passed its 21st birthday. It could legally buy itself a drink, and it seemed like it would be a good opportunity to reflect on DTrace at 21, because we can look back on now this fully grown software system, and a lot has changed personally in my life. 21 years is a long time. Before we integrated DTrace on September 3rd, 2003, I wasn't married. I got married the next month. Now I've got three kids, my oldest of which is 20 years old. So a lot has changed, but a lot has not changed in DTrace, and it's a great opportunity to kind of look back.
DTrace has been, we feel very lucky about DTrace. DTrace has been able to be not just useful to us, but useful to broader folks and influential. It was open sourced in 2005. It found its way into a bunch of other systems: Mac, Windows, Linux. It's influenced many more, and it is now, like now, the time is right to kind of look back on it because we've been using this thing more or less continuously for its entire life, and we've got the benefit of hindsight now.
And as I was thinking about this talk, I would love to give you a talk on all the things we did wrong in DTrace, but I don't think it's really accurate. I think we, again, we feel very lucky in this regard, but I don't think there's much that we did wrong in DTrace per se. We got some things certainly very, very right, in many cases more right than we realized at the time. And on the things that we missed, it's more that we figured it out. In some cases we figured it out really quickly, in other cases we figured it out over a longer period of time, but we figured it out. So I don't know that we did anything wrong.
And then we got really lucky along the way. And there are some things that were just some big, big, big lucky breaks. And whenever you're working on something, there are going to be things out of your control. Some of those will break against you, and certainly we had some of those. But then some big ones may break in your favor, and we had some really tremendous things that were really lucky for DTrace. So I want to kind of reflect on all that, talk about some of the things we got right, talk about some of the things that we figured out and how we figured them out, and talk about some of the ways in which we got lucky.
First of all, one thing that we definitely got right, and this was at the outset, this is very much a principle of DTrace, is the focus on production systems. DTrace is designed to instrument systems in production, and our big belief, which I think has proven to be really correct, is that production systems were the interesting systems. We had already, we knew this from our own experience, that the pathologies that we saw in production weren't necessarily mimicked in systems in development.
And in the cloud era, actually, the line has blurred, and we think of all of our systems really rightfully as being production systems at some level or another. And it's really important to be able to be on those production systems and understand the pathologies there. And the reason is pretty simple, because if you have a production system that is misbehaving and you try to reproduce it on another system, what you are going to be reproducing are symptoms. You don't actually know what the underlying cause is.
And one of the things that was our experience, but then definitely we saw to be true over and over and over again, is that, of course, disjoint problems can manifest the same symptoms. So when you're reproducing the symptoms, you don't actually know that you're reproducing the same problem. And boy, so many times over we had seen folks who had tried to reproduce a problem on a development system, debugged the problem on that development system, thought that it fixed the problem with the development system, deployed it into production, and it didn't make a difference because they had debugged a different problem.
So our focus really from the outset was on production systems, and that guided a lot in DTrace. In order to be able to instrument a production system, it has to be absolutely safe above all else. And we knew this from our own experience, that if you have a system that is chancy, that is, say, I may run this and the application may crash, the system may crash, the application that I'm instrumenting may crash, that is to say, I may do harm with this tool, boy, you only have to do harm on one production system before you will simply not use it again.
And we knew that we needed to earn the trust of users of DTrace by being absolutely safe above all else. So, you know, I have said that systems reflect their values, and a value for DTrace above all else was safety. Even if DTrace itself were being misused, we didn't want the system's safety, the underlying system, to be affected.
In addition to being absolutely safe, we felt that DTrace had to be absolutely available where you needed it, which is to say on this production system. We didn't want to rely on recompiling things. We didn't want to rely on the presence of a compiler. We didn't want to rely on people downloading binaries to run it. We didn't want to require people to download symbol tables or to download any, to take any kind of auxiliary action in preparation for instrumenting this system. Very, very important for us that it was completely available.
And these two things have kind of an implication, namely that DTrace is very tightly integrated with the underlying operating system. And there are drawbacks to that approach, but it's got a great strength in that you can actually have it on a production system. You have DTrace. We wanted DTrace to be available for you, for the users of systems, on their worst days, on their stormiest days, not just the sunny ones, on those production systems and not just in development. So that was something we got really, really right, and don't regret that focus at all.
Another thing that we got really right is dynamic instrumentation. So when we were first conceiving of DTrace in the late 90s and early 2000s, instrumentation usually required changing the application in some way, shape, or form. Maybe you had to change the application to recompile it with different flags. Maybe you had to change the application to actually indicate where it should be instrumented. And we feel very strongly that the instrumentation should be completely dynamic, that you should be able to walk up to software that you had never seen before and you should be able to instrument it. You should know what it's doing. You should be able to ask questions of that software that you couldn't otherwise see.
And part of this came from that production constraint, because the production constraint, in order to be able to have DTrace available in production, it is very important that DTrace have zero probe effect when it's disabled. Part of having zero probe effect when it's disabled is that it is going to actually dynamically discover the things that it needs to instrument and dynamically change that program text on the fly to be instrumented.
So we cared much more about that disabled probe effect than the enabled probe effect. I think up until that point, honestly, much focus had been on minimizing the enabled probe effect of instrumentation, and that's important, but it's not nearly as important as a zero disabled probe effect to allow it to be dynamic, to allow it to be in production.
We also felt very strongly that you should be able to instrument arbitrary contexts in the operating system, and this is harder than it might sound because, in particular, you want to instrument the code path in the operating system that is descheduling a thread. So a thread has, it needs to block, it's blocking on a synchronization variable, or it's being taken off CPU. You want to be able to have arbitrary instrumentation in that context. Problem is that context is very, very delicate. You've effectively hijacked this thread that can't really do very much. In particular, you obviously can't block in that context because you are already blocking.
So that very much guided the way we developed DTrace, and in particular, DTrace is designed to be lock-free, wait-free in terms of its implementation. It's designed to, in terms of that enabled probe effect, we are very constrained when we are in what we call probe context, and that is by design because we want to be able to instrument arbitrary context in the system.
Because we also wanted to replace a bunch of different existing tools, in particular a tool called lockstat that had been developed by Jeff Bonwick to instrument the synchronization primitives. We wanted to use lockstat's underlying approach. We wanted that to... We knew we wanted to have CPU-based profiling as a source of instrumentation, and then of course we knew we wanted to dynamically instrument the running program text, certainly the operating system kernel.
That forced us into another really good decision, which is to separate out the methodology for instrumenting the system from the underlying framework that actually consumes that data. So that was another really good decision.
Another thing that we got right from the early onset, and this is one of those things that, you know, as I get older, I appreciate more. As a younger engineer, I didn't think about it that much because this just felt like, of course, the right way to do it, but was our organizational approach, actually. So we've been thinking about DTrace for long before we started. We actually started, wrote the first lines of code in November of 2001, but I'd been thinking about it since 1995, 1996.
And we had been thinking about it together, I along with Mike Shapiro at Sun. He and I had been thinking about what we wanted to go do. And Mike has got a very, what to us became a famous page in his notebook where he had all of the things that we would need to go do before we got to DTrace. And we did a lot of that work from 1997, 1998, 1999 into 2000.
So we had been doing a lot of this like incremental work with this vision of what we could go do with DTrace, but we knew it couldn't be done as a side project, that we had to focus. And I bought myself some time. So instead of trying to lay out an entire path for something that we would go do, we basically negotiated for six months of focus time.
And I mean, it helped that I was doing, I did this in exchange for helping a microprocessor that was in crisis, a microprocessor that you haven't heard of called Cheetah, cruelly named microprocessor, definitely not the fastest animal on earth. But in return for kind of helping them out during a moment of crisis, I was able to get six months of focused time.
And in that six months of focused time, we were dedicated to making the case for why this was an important technology. So we were very, we were prototype intensive, and we started small. After that six months, we had presented our work to a broader group, and it was pretty clear that it was compelling. It didn't take much to show to other people that it was compelling because we had those proof points, and we were able to add a third engineer and then stay focused for an additional two years at that point.
The team was always really, really small. So I'm a big believer in very small teams. We had three people on DTrace, that's it. There were not additional folks on DTrace. And importantly, we were not in an office. So this was the, I mean, we had an office that we went to occasionally, but the office was mainly a waste of time. There were many, many, many days where it's like, boy, I got way more work done on Caltrain going to and from Menlo Park from San Francisco than I did actually in the office.
Love Caltrain, by the way. Got so much... DTrace owes a debt of gratitude. Something we got lucky about, I guess, was Caltrain. When they introduced the baby bullets, they actually impeded DTrace's progress because the train was so much faster. But we did a ton of work on DTrace on our own at home.
And I think there's, especially in this, you know, kind of the current day, this kind of return to the office mandate because that's what's needed to foster innovation. And I'm just telling you from my own experience, it was actually the focus time at home that was necessary for that innovation. And of course we were collaborating very tightly, but you can collaborate at a distance. So that organizational approach was a big win, something we got right.
Something we figured out, and this maybe we figured this out really, really quickly. To be fair, we figured this out like in the first weeks of DTrace, that we realized that we wanted to have expressive power when we instrumented the system. And one of the things that we just figured out very quickly is actually we need our own language. We need a domain-specific language here.
And I actually think that I'm maybe a DSL maximalist. I think that people should reach for DSLs earlier rather than later. I think, I mean, you want to get the timing right, but I think by the time one reaches for a DSL, it's almost always the right choice. And I think that you can make your DSL easy to learn by borrowing constructs and concepts that exist elsewhere.
And we were heavily inspired by AWK. Talk about a language that's in adulthood. I mean, AWK is deep in middle age, fortunately without a midlife crisis, I'd like to believe. I don't know what AWK's midlife crisis would look like. But we were heavily inspired by a terrific language developed at AT&T Bell Labs by Aho, Weinberger, and Kernighan. Of course, that's where the awk name comes from.
And AWK used to have an outsized influence. It had a very explicit influence on us because we were able to borrow a lot of AWK concepts to make D, which is what we called our language, easy to learn. We dubbed our language D. The intermediate form is called DIF.
I would just like to say for the record that Walter Bright's D was being done literally concurrently and in secret to our D. And they're both letters of the alphabet, so you can't trademark a letter of the alphabet. So sorry to Walter Bright and any D aficionados. I hope I haven't caused much confusion.
The good news about D is you can't do very much in it. It is very, very domain-specific. In particular, one of the things that we figured out early that was very important was to have not just a domain-specific language, but a domain-specific virtual machine. So this was kind of a surprise to us, an early surprise. We kind of assumed that we would transpile DIF to native instructions, SPARC at the time, later x86.
But it became really quickly clear that we, and as kind of an intermediate step, we had this virtual machine that we were interpreting. It became very clear that actually the VM, the domain-specific VM, was the actual destination. That actually interpreting this intermediate form in the kernel was a galactic, galactic win because it allowed us to better adhere to that safety constraint.
Because it was very easy to have very important runtime checks around things like memory safety when you're in an operating system and you are going to execute something in the context of the kernel that is user-directed. A stray load can have side effects. It's very important that you validate that that load is not going to have side effects before you actually do it. By having our own VM, we were able to do that really easily.
And then a bunch of things started falling out of that. In particular, DIF is Turing incomplete. It is not a Turing complete language. There are no backwards branches in DIF. So that makes D, the D in DTrace, and Walter Bright's D will never compete with one another because you really can't write a complete program in D.
And that is by design. We need to know that a DTrace program will halt because it's going to be executing in arbitrary context in the kernel. We need to know that it's going to halt without solving the halting problem. And the way we did that is with no backwards branches. And that can be, you know, that has consequences for users, but we believe that those consequences are more than worth the safety guarantee that you get in return.
One thing that we got really right, frankly, we continue to get right, we use DTrace ourselves. We still use it. We use it on a daily basis. We were heavy users of DTrace at the outset. I started from the various, from the earliest time I could, using DTrace to solve actual problems, solve them in the operating system, solve them in the larger system.
This was really, really important, and I can actually remember vividly whenever I'm having kind of a down day, I'm like, you know what, just need to remember, it doesn't feel like that long ago when whenever I would use DTrace, I would find an underlying bug and then I would also find a bug in DTrace. And I just felt like, God, am I ever going to use the system and not find a bug in the system itself? Am I ever going to use the system and use only to find underlying bugs?
And of course, it's been years and years and years since we, I know we use DTrace all the time and we don't find bugs in DTrace. DTrace basically, I wouldn't say it's bug-free. I'm not going to quite go Don Knuth on you. And I'm certainly, we're not going to offer, not going to offer doubling bounties for finding bugs in DTrace, but DTrace basically works. It's basically done and we don't find bugs in it.
And that is a consequence of the fact that we used it ourselves. We also, because we used it ourselves, because we used it to actually find real problems in production, when we needed something, we would add it. And the features in DTrace are there because they serve a very concrete need.
And I think this is honestly unappreciated because there are lots of features of DTrace that people who do not use D or have not used DTrace, frankly, don't appreciate because they've never needed it. They seem very arcane things like thread-local variables, anonymous tracing, the ability to instrument during boot, speculative tracing, post-mortem tracing.
So the ability to actually have a DTrace-based kind of flight data recorder that you get, that you can get the output of when the operating system panics or you otherwise force a crash, the ability to instrument arbitrary instructions, USDT. There are a lot of things like this that are the kinds of features that you don't need until you need it, and when you need it, you really need it.
So I think that this is a consequence of the fact that we used it ourselves, and there is, there's nothing, or very little anyway, that's kind of extraneous in DTrace. There are features that are used much less frequently, much, much less frequently than others, but those features exist because someone needed them at some point to really debug a problem.
Another thing that we figured out was statically defined tracing. I talked about the importance of dynamic tracing, and certainly that is the genesis of DTrace. The origin is dynamic, but statically defined tracing is really, really important too. And our tagline for DTrace was that it offers concise answers to arbitrary questions.
It's really hard to know what question to ask when you are dealing with only the implementation, and with DTrace's ability to instrument the system arbitrarily, there was a lot of implementation coming at people. So we really wanted to have the ability to have a statically defined instrumentation points where we could actually take points of semantic relevance, be they I/O scheduling, CPU scheduling, and so on, instrument the system that way in a statically defined way that still had basically zero probe effect when disabled.
And then someone could go ask questions that were relevant to them, like what packets am I sending out and to whom, what I/O am I doing, when am I being descheduled, without having to know how the actual kernel was implemented. So this is really important and required a lot of subtlety.
Actually required this thing called CTF, the Compact C Type Format, that we use to assure that we had symbol information in a running system. It required things called structured translators that allow us to, you can easily use DTrace to know the path names of the files that something is writing to, but that actually requires a lot of sophistication under the hood. So this is something that we figured out that was really important.
Another thing that was very important that we figured out was application-level instrumentation. So we started with a focus on kernel-level instrumentation, but one of the things that we figured out pretty quickly is that kernel-level instrumentation is essential because you can see these systemic effects and you can understand what the system-wide effect of an errant application is.
But you really need application-level instrumentation to get that all the way home, especially when you want to go instrument the application in terms of its semantics. I'm looking at a database. I would like to know what queries it's doing. I'd like to use DTrace to know the queries that Postgres is running or MySQL is running or Cockroach is running. I'd like to know that. I would like to actually be able to instrument it in terms of Rust, for example.
So user-level SDT, user-level statically defined tracing, has been huge for us, and we continue to use that all the time. And in particular, it's USDT that's been essential for us at Oxide. I linked the crate there that Ben and Adam Leventhal developed, but that's been really important for us to instrument the system in its native fashion, so we can actually understand this in terms of its own semantics.
Another thing we got right was writing our own documentation. This came out of having done it wrong on previous projects, and this is nothing against doc writers because I think doc writers have the hardest job on the planet, where you need to not just know how to write, but you actually have to really understand the underlying technology. And I think it is really, really hard to write good technical documentation.
My belief is that for at least my own technologies, that you need the technology itself to write that documentation. It can be brutally hard, but I think really, really important. This is something that we really got right, that documentation is still very much available today. And by the way, in an LLM-based world, canonical docs are very helpful for training a model, as it turns out.
One of the little surprises, so what the reason that LLMs can be effective for helping us write things is because there is canonical documentation upon which they can train.
Another thing that we figured out is writing actually rigorous papers on what we had done. Adam and I went to a conference, a debug conference, no longer, that's now defunct, but that inspired us to write a canonical paper on DTrace, which we did in the USENIX Annual Technical Conference 2004, A Dynamic Instrumentation of Production Systems.
I deliberately didn't want DTrace to be in the title. I don't know, maybe that's something we did wrong. Maybe I should have put DTrace in the title. But that paper was really important to us because it allowed us to rigorously describe it. We wrote a broader paper on ACM Queue in 2006 called Hidden in Plain Sight.
Hidden in Plain Sight made the case for software observability. It was only a couple years later when people were talking about observability, and I'm like, this is great that so many people were talking about observability, but I think people are actually, I think the actual the wellspring of that is this paper I wrote, and I think that that's the case. Prove me wrong, but certainly it doesn't need to be the case.
But I find it really inspiring that the fact that people saw what we had done with software observability and wanted to apply it elsewhere to their own problems, which is great. We think that there's tremendous value in rigorously describing your own ideas.
And then the dimension in which we got really, really, really lucky was open source. So, you know, a lot of what I've described was in our control, but there's something big that was not in our control. We worked for Sun Microsystems at the time. DTrace was born in an entirely proprietary system.
And there had been conversations, certainly in 2003, there had been a lot of conversations, conversations as early as 1999, 1997 about open sourcing the system, and the urgency was growing around it. But ultimately we were not in control of whether the system was open sourced or not. Whether the system was open sourced or not was effectively an executive-level decision at Sun, and how it was open sourced was not our decision either.
We got lucky because it was open source. Sun did see the need to open source this technology. Speaking for myself personally, and certainly by the time we integrated DTrace in 2003, I already had made the decision that I didn't want to live without this thing. One of the great things about DTrace is once you've used it to solve problems, you're like, I ain't going back. I'm not going back to the old world. I want to be able to use this thing to solve problems.
And the idea, and again, I remember vividly having this thought in 2005 when we were finally going to open source all this stuff, and there was talk about we want to open source something to show that we're serious about it. It's very hard to open source something that's proprietary. We had a lot of covenants we had to go resolve and so on, but DTrace was clean. It had been implemented entirely by Sun. We could open source it ahead of the system.
And it was a real honor to me that Sun chose to open source DTrace first in January of 2005. And from my perspective, like, this is terrific because this assures that I'm going to be able to use this for the rest of my career. Of course, I couldn't have predicted the wild path my career would take, but we got lucky because it was ported to other systems.
DTrace is not easy to port. It is very tightly integrated with the operating system for a bunch of very good reasons, and it requires a really in-depth technologist, a veteran technologist, to port it to another system. And it's been ported to a bunch of other systems by terrific folks. I call out in particular the late John Birrell here, who ported FreeBSD. That was the first port, really, really heroic.
Ported to macOS. Steve Petersen and crew at Apple did an unbelievable job porting it to the Mac. It was ported to Cygwin, to Linux. There is a Linux port of DTrace that was done by Kris Van Heusen's team at Oracle. It's been ported to Windows, and really grateful for all of these technologists and their very hard work importing it to their own systems.
And they've had to take different liberties with their own systems. I mean, in particular, the DTrace port to Linux uses now eBPF as a backend, and that's obviously a very different choice than what we made in illumos, which is where illumos and Helios, it's where we use it at Oxide. But every technologist that's working on porting DTrace to their system has that shared goal of dynamic instrumentation in production, and we're really grateful for it.
And finally, we got lucky because DTrace endures. I feel extraordinarily lucky and blessed to still be using DTrace every day. We use it on our thorniest problems. We have some, had some gnarly problems at Oxide, and we had a week that we call Data Corruption Week. Speaks for itself. And DTrace was instrumental in debugging the underlying problem there.
And it's really gratifying. It's gratifying to use it. It's gratifying to bring new people into DTrace. Right now, the technologists that have been brought into DTrace are still younger than DTrace, but that will not always be the case. And I'm sure out there there's someone younger than 21 who has been using DTrace. But to new people, especially, new languages, new systems, we feel lucky that it's open source.
Being open source has granted DTrace eternal life, and we're really grateful to it. And we're still grateful and lucky to be working together. And, you know, that's not completely true. And, you know, over the years, obviously, folks have passed away or moved on or, you know, relationships come and go. But some of those relationships have really endured.
I feel, again, extremely lucky to be working with my colleague on DTrace, Adam Leventhal. We still work together at Oxide. We've got a podcast, Oxide and Friends. Would love for you to join us there. But we feel very, very lucky that DTrace has endured, and extraordinarily lucky that 21 years after we integrated it into the system, I get to give a talk on looking back on DTrace at 21. Thank you so much, and enjoy the rest of P99 Conf.
