Thanks. Yeah, apparently some of you use containers, as we saw from the Chain Guard intro. And you know, I spent 10 years working in containers. I joined Docker in 2015, and it seemed kind of weird. I felt, you know, it's funny, we just started with a reflective talk. I've been feeling a bit reflective after leaving. I ended up, I started at Docker as a software engineer back in 2015 and accidentally ended up becoming a CTO at some point for some reason. It was a bit of an accident. And so I decided to leave just before 10 years. And I don't know who's worked in one company for 10 years before. Yeah, a few of you. It's kind of a weird thing when you leave because it's like, what was all that all about? Was that a horrible mistake? Was I there for 10 years? Like, and you know, containers are kind of everywhere. This is a picture from down the road. I live in the middle of nowhere in Suffolk, and even there there are containers everywhere in the sort of farm derelict areas that you get between the fields.
So why did containers get so big? What happened? What did we get out of that? What do we get out of that decade? Was it worthwhile? Should we have stopped this long ago and given up and done something else, or did we actually get something good out of this? So definitely feeling in a bit of a reflective mood. So I'm going to talk about some of the things I kind of learned along the way that maybe were surprising. One of the things that happened that you have to do when you're CTO is deal with the random people who call up the company and say, "We need to talk to someone about something." And so, a few years ago, back in 2020, the Federal Trade Commission came and said, "Well, Broadcom wants to buy VMware. We want to know if you're a competitor of VMware, and we've heard that containers and VMs are very similar, and we want to know if the fact that you're here means that VMware are in a big competitive market." And it's like, that's kind of weird because, you know, years ago when containers started, like everyone was always asking, oh, containers and VMs, they're the same, they're different, what's the difference and things like that. But people kind of gave up after that after a bit because it was kind of obvious that they were kind of different and weren't really in the same market. So I kind of had to explain to the FTC what a container was and why they were there.
And you know what, and I remember it brought me back to all this stuff. This is Jesse's blog from like 2015 or 20 or something like that about like what's the difference and things like that. And right at the beginning I looked at the slogan for the first DockerCon, "containers are the new virtualization," and we very much told this kind of story. But really, you know, when I was talking to the FTC, I was like, no, we're obviously not competing with VMware. I mean, we don't sell to the same customers as VMware. The people who buy VMware, I don't even, you know, I don't even know who they are. I don't even talk to them most of the time. And so I kind of told the FTC a kind of different story. I said, well, you know, back in the day, when VMs became successful, back in the day when people wrote these great books that you can now find on the Internet Archive, people bought VMs because they ended up having a lot of computers and they were hard to manage their computers. And most of their computers just had one application on, and this was kind of inefficient even back then, like 20, 25 years ago. But that wasn't really, you know, it was hard to manage these computers because you had to go, I remember I used to have a job doing this, you had to go and you had to install the operating system and configure it all by hand, and eventually we started inventing things like configuration management and things like that. But it was a terrible kind of thing to do. The processes were terrible.
Most VMs apparently had like 15% utilization because they were running one application. And this VMware came along and said, look, you can consolidate all these servers into one thing, you can manage it with more automation. It was especially for Windows systems as well, it was where the situation was kind of terrible. But the story for containers though was different. It wasn't that you had lots of computers and therefore you install containers on them. The actual containers came along when you had lots of applications and you needed to manage lots and lots of applications, and this was, you know, kind of a later thing. And so it was solving a different problem. It was a kind of follow-on problem after you had, you know, you got used to having too many computers, and then you kept hiring more software engineers because software was taking over the world, and you ended up with like even more applications, like you had thousands and thousands of these applications, and that got hard to manage. And so you kind of needed something to actually make it easier to manage applications and think about applications. And that's why, you know, that's when containers came in as like, ah, we can, each container is an application. We can manage them, we can automate them, we can script them and so on.
And so that was kind of really the kind of origin story I told the FTC, was like this was about managing a completely different thing, totally different market. Most people run their containers on VMs anyway. We're not competing with VMware. The people who are competing with them are actually people like AWS and the cloud providers who are providing a different kind of virtualization layer. And so it doesn't matter if Broadcom buys VMware. I mean, they're going to screw the customers, but that's a different issue. I did tell them that, you know, they're like, yeah, well, but I mean, that's not because there's no competition from containers, but it's just like that's going to happen anyway because that's what Broadcom does. But so they kind of agreed that was all fine and they could buy them and screw the customers, and so now here we are.
I mean, I think like this diagram, does anyone remember this? Anyone work at Monzo? This was a diagram that Monzo produced a few years ago about how many microservices they had and how they were all connected together. I think we kind of went a bit overboard with this kind of thing, like encouraging this kind of thing, like just because you can doesn't mean you should. So we, yeah, we kind of went a bit too far. We said you could make tiny little services. We'll talk about that a bit more in a minute.
So anyway, that's what I told the FTC, and that was kind of interesting. It was actually kind of interesting though, because I think a lot of people also wanted containers. And I mean, when you talk to people, they didn't really talk about like, well, we have lots of applications that are difficult to manage. They actually said, when you actually went to talk to enterprises, they actually had a totally different story, which was actually kind of weird, which was that, well, we want everyone, we want our developers to move to the cloud. We've heard the cloud is great. We've actually got all these applications in VMs, and we don't want to just move the VMs to the cloud. Even though at the time Microsoft was going around to people saying, "Oh, we'll just lift and shift your VMs into Azure. You don't have to do anything. It's straightforward. We'll take your data center that you own and we'll turn it into an Azure data center and you'll never know." But a lot of companies didn't really want that. They actually wanted containers in order to modernize things, and so they used to just have this sort of stick and carrot thing of like, well, if you want to move to the cloud, you want this nice Docker technology, well then you have to use the Docker technology, you have to do some work. And a lot of it seemed to be actually about, can we force our developers to actually do some work, which was a bit of a weird thing. And that was a kind of different, kind of weird thing that went on as well, that it was like, do you want some nice new tools? Then you have to do some work. You can't just have those.
One of the things that was kind of really, I thought about a lot when I was at Docker, was like when containers were introduced, we kind of had a bit of a sort, I think when you introduce some kind of new technology, you have a sort of change budget of like, people are going to adopt something new, but they're not going to change everything. But you can force them to change a small number of things. You have a sort of change budget with new technologies. And one of the things that made containers really easy for people to adopt was you didn't have to change your code at all. You could just take your code, put it in a container, deploy it, and it just worked. And it was great. Like really easy. No changes really needed. But we did persuade people to change one thing, and that was we persuaded them that it was impossible to update containers at runtime. You had to rebuild and redeploy them. That wasn't true. Obviously, you can actually update containers at runtime. And you know, it just works perfectly well. You just go in, you log in, you could run SSH in the container, you log in, or you run an FTP server and you update your PHP code and you update it dynamically. It works perfectly well. But we just told everyone it was totally impossible. It did not work. But weirdly, no one built these tools for FTPing into containers and updating this stuff at runtime, because they didn't, you know, we told them it was impossible. And so we got this one item of change budget, and it was like, oh well, you have to redeploy stuff, which was actually really cool, because that actually led to all sorts of really useful things. You know, like being able to understand what's in production because it's the thing you built and it's, you know, it's got a hash and you can identify it's exactly the same one, and then you can tell things like, well, what vulnerabilities are there running in this thing that's running? Like what's there? And that was something that you just couldn't do in the old days when we used to FTP into production and change our PHP code on the fly. And it led to things like Chainguard, who can tell you what vulnerabilities there are in production, tell you you should buy something to fix the problem. And so it was kind of amazing that we got one thing done, and it was a really cool thing.
And it kind of led to the one also useful bit of technology that weirdly VMs never got, which was why was there never any kind of workable kind of public registry of VMs that you could install? There were lots of weird sort of private ones, like you know, you could get your AMIs in AWS and things like that, but containers really had this thing of having like Docker Hub originally and more registries later where you could just get stuff that was useful and you could run it. And I spent a lot of time thinking about while I was doing it, like why did that work and why did it not ever work anywhere else? And so I spent a bunch of time thinking about what's the kind of, why are the things on Docker more useful than other things in other ecosystems that tried, you know, failed to do the same thing.
And one of the things that really turned out to be important is that if you want things to be sharable and reusable, it's no use if they're kind of finished, finalized objects. Like the container or VM that is configured to run in my infrastructure with my secrets in is not something I want to share with anyone ever. But some vague nginx image on Docker Hub that doesn't have a configuration and can be used to do lots of different things is really useful to share. And so sharing and community turns out you need some sort of way of having things that are useful to a broader audience and they're not configured. And so this kind of thing where we separated out configuration from code actually made the sharing thing really work. And with VMs, it actually sort of eventually happened that we invented things like cloud-init which cut out some of the configuration, and you could have a generic image and a cloud-init config. And that actually made VM images more sharable. But I think somehow accidentally Docker kind of fell into working out that this is the kind of thing that worked, and it did mean that configuration was still totally separate, and then we had to do something about that. But I'll talk about that in a minute. So this is, you know, kind of the immutability thing was that kind of thing that we did.
Another thing that we managed to kind of put through the change budget in Docker was that we persuaded everyone in the world to use Go as a programming language. I did a talk about this sometime during the pandemic, which was quite fun to research, because I went and interviewed, like, so how did that happen? And Solomon said, "Well, we saw the people at YouTube were using Go, and we thought if they could use it in production, then we could." So I went and interviewed the people at YouTube, and they were like, "Well, it really didn't work very well. But we did have these people at Google who would fix things for us, because like we had these four-core machines, and YouTube was growing really quickly, and Go only worked on two CPUs at a time, and so we basically had to go and get them to fix it before YouTube crashed." And internally it was kind of chaos, but they just managed to get it to work in time with some heroic software engineering, whereas the people at Docker looked and said, "Looks like it's working really well for them. We should use that." Docker had been a kind of Python shop before that. So we managed to introduce Go, which is still very popular as a language for microservices, I think. It's kind of losing out to Rust as a systems programming language now, but it had a good run.
Kubernetes came along. I think as I was saying, we separated out the configuration piece from the container images, and that was successful in terms of being able to do sharing, but also how do you deploy configuration? I remember when Kubernetes was kind of new, and I went around to all the Kubernetes meetups people were having in London and seeing why people were excited about this new thing. And for the first year or so, everyone was talking about orchestration and schedulers and stuff like that, which we've totally forgotten about later. No one cares about schedulers anymore except the people doing AI nowadays. But there were whole conference tracks on schedulers for a while, but really there were some weird obsessive people and no one cared about them. But when you actually talked to people who were actually using Kubernetes, they all were writing deployment scripts, and that was the thing that was like the number one use case, was like, well, we want to write some code to deploy our code because we've read the Continuous Delivery book. It was great. It was the best book. I remember reading it too. I was like, "This is amazing. How do we do this?" And they were like, "Well, we're going to use Kubernetes to do this." Okay, that sounds cool.
At Docker, we had this thing called Docker Swarm. It was terrible. And the reason it was terrible, it took me a while to realize this, was that you couldn't write a deployment script in Docker Swarm. And it turned out that that's all people wanted to do with these tools, because our security department had decided that that would be insecure, because if you could modify the state of the system from inside, bad things would happen. So they said, no, you can't do that. So no one could do the only thing that they wanted to do, which was write deployment scripts. And Kubernetes actually was great for that. And the things that we invented for Kubernetes were things like GitOps and all sorts of deployment tools that were amazing. And they kind of put together that, well, I've got my configuration for my applications, which are unconfigured containers, and I turn them into configured containers, and I can deploy them, and I can update this stuff really easily, and I can do continuous deployment because I read the book, and it was amazing. And so that was great at Docker.
There were some terrible things at Docker. One of which was Docker Swarm, but also our commercial product, because no one understood deployment because you couldn't do it. Our commercial product, you had to paste YAML into a text box on a website to deploy something. It was awful. We sold that to Mirantis. I hope they fixed it now. But it took a while internally at Docker for people to actually, until we actually adopted Kubernetes ourselves when we got rid of our commercial product and started using it, then we could start doing deployment properly internally. So it was really interesting that kind of lack of a deployment culture fed into so many things. And it was kind of weird, and I should have really noticed and tried to fix that earlier, but you know, sometimes you can't fix things.
Another thing that I remember people were asking for many years was, and this is the new V&A storehouse, you should go, it's in East London, it's amazing, databases. People said, "Well, when are people ever going to run databases in containers?" And that was the question for years and years and years. And then suddenly it turned out the answer was no. Why are we running databases at all? We don't want to do that. We've just realized that the only thing you can do if you run a database is lose all your data. Either it works or you lose all the data. And this sounds terrible. We're just going to tell Amazon to run our database or someone else, anyone except us. And we're not going to run a database in a container or anywhere or on any hardware. We hate databases because we just realized it's the scariest thing ever because we do not want to lose our data. And I think this is partly because in containers, like it's so easy, like everything's all software and volumes and storage is all software, and it's very easy to sort of just redeploy and delete the things, and it's like they go away and you've lost all your data, and it's like, oh yeah, software, it makes it so easy to lose all your data. It's amazing. So I kind of think that maybe we pushed that from that sort of psychological lack of safety of data in containers, but it's really hard with a software environment to make data safe if everything's very ephemeral and quick and easy to do, and you can keep automating everything. You can automate deleting your data so easily.
So I kind of think that we probably caused that. So now no one knows how to run databases anymore except the cloud providers, and it's kind of sad. And I think we have to kind of reclaim our data at some point because we've realized that we don't like other people having all the data and us not having any data. And so I think probably we'll all get used to running, you know, keeping your own data and keeping it safe, I think is probably a good idea. Yeah.
Gent DevOps, my memories of the early days of going to DevOps conferences in Belgium, best place to go to them other than London, obviously. We kind of, like all this focus on deployment and the whole complexity of Kubernetes and everything, we kind of forgot about DevOps and what it used to be. I think that's our fault too, you know, through this process. You know, we got obsessed with deployment and all those things you could do with Kubernetes, and that was really hard, and we didn't help the developers understand it. We just put it into a silo and said, we're going to call this DevOps. Why do we? I don't know. And it's not going to be what DevOps used to be, which was, remember, the developers and the operators working together and helping each other and sharing knowledge. That, sorry, I really, I mean, I thought these times in Belgium were great. Like I used to be a sysadmin who built computers from scratch, and I discovered Puppet, and then I discovered, you know, all this stuff was amazing. This was how I got into this in the first place. But I was also a developer, and I was like, I like writing code. Like these DevOps people, they're my people, because I've done both of these. I do both, I like doing both these things. I kind of want to bring the things I do together in my mind, but we lost all that. Sorry.
We didn't really change developer environments as much with containers as we thought. I think somehow Docker, we convinced ourselves that because developers love Docker, we'd kind of change things, but you know, you still had a big pile of code, and most people didn't use containers to manage their developer environments. They used them to run some services on the side while they were developing, like I want to run Redis and Postgres, so I'll run them in containers because that's convenient. And there was all these nice stories that we used to tell about reproducible developer environments. And I think we kind of showed people that those things were possible, but we didn't make them nice. And now we've got like most of the people who want reproducible developer environments, things like that, are becoming Nix obsessives, which is great. It doesn't make it any easier for the rest of the people either, but it's definitely reproducible. I don't know. So I think we didn't change that as much as we thought. I mean, we did replace some of the... Vagrant, I remember hating Vagrant so much. It was better than Vagrant. But actually a lot of that was because I think a lot of that was around the fact that the tools for Python and Ruby and things actually got better for managing version management and package management and things. I mean, they're much better than they were a decade ago, and that actually made it easier to kind of get things okay.
I think containers really helped people to some extent with this whole thing about like let's build our applications out of components that are open source, and like 95% of the contents of everyone's applications is now open source components that are great because they're good quality. They might have CVEs in them sometimes or whatever, but they have less problems about maintenance, but they're better than the position where you were in before, you had to write all the code yourself. Now we're trying to replace that with AI that writes all the code itself in low quality again. But I mean, that's, you know, I think a lot of the time, like a lot of the container ecosystem stuff was actually about distributing this code and making it easier to use these components, and like there's so many of them, and they're all on Docker Hub, and you can use them, and they're amazing, and you can actually build new cool things with them. And I think containers kind of helped in that process, it was happening anyway. I mean, I think GitHub kind of started that off, and it became the sort of standard way of building applications, and I think we kind of helped that.
This is, sorry, I just put, this is Moxie. She's a very cute dog. So where are we now? You know, where did we get to? Is it all nice and soft and hardy? We started off with virtualization was introduced because hardware was only being used at 15%. I looked up the 2024 Datadog report to see on the state of cloud costs, and it told me that 83% of container costs are associated with idle resources. So what's happened in 10 years is we've gone from about 15% of capacity was used in VMs to now we know exactly 17% of capacity is used because we've got way better monitoring, and so we have much more accurate numbers that tell us that the situation is exactly the same as it was a decade ago. So we haven't really, like the compute we're using is 10 times cheaper, but we waste way, we scaled up how fast we can waste it much more effectively. And you know, we've got pockets of efficient applications, I think, but like the vast majority of things are still not terribly efficient.
The "Choose Boring Technology" essay was written in 2015, so 10 years ago as well. I reread it recently because boring technology has become really fashionable recently. It took 10 years for that to have its maximum impact. I think the boring technologies back then were things like MySQL and Postgres and PHP. So I asked ChatGPT, you know, are containers and Docker, are they a boring technology now? It told me that Docker was mostly boring. So I think that's, you know, we trust our AI, so it's like, well, listen, we managed to make a new boring technology over 10 years, which is mostly boring apparently, wasn't convinced it was totally boring.
And so we kind of, you know, what have we learned? I mean, summarizing, you know, what we've done over these 10 years, like immutability is great, we did some immutability stuff, we lied to people and told them they had to do it, and they did it, and it was great. Continuous delivery, we built this gigantic infrastructure for doing it, and it's great. Like, we were in a much better state than we were because I remember reading the Continuous Delivery book and thinking, how are we ever going to do that? Reproducibility, you know, Docker didn't really do it, to show people it might be possible, and we kind of, it's a sort of teaser, and now the Nix people are doing it. Nix is 10 years older than Docker. Not many people realize it's 20 years old. But they've nearly got it working now, and we'll all be using it soon. Microservices, I think we went a bit overboard, like it was a terrible idea. I think there's been a reaction. No one has the Monzo diagram anymore. And containers are mostly boring, which is kind of good, and we can get on and just kind of use them and things.
So I kind of felt that was kind of, you know, 10 years, we did some things, we got some things out of it, it was not too bad, we made some things a bit boring, was good. Maybe, you know, maybe there were alternate timelines in which things happened differently, and but you know, sorry about DevOps. But hey, some casualties along the way, you know. So, thanks very much.
