Hi, well, nice to be here. Great to see so many people here. I thought it was just going to be me and YouTube, but you're all the hardcore people who are not going to leave until everything's done, which is great. Hope you've been having a great KubeCon. It's been a really, really nice experience for me.
A bunch of people ask me why I'm giving a talk about object storage. It's, you know, it's like everyone else, I've submitted talks to KubeCon for a while and they've all been rejected and sometimes given the keynote and sometimes I haven't. But anyway, this time I thought, well, I'm just going to submit a fun talk that I want to do because that is unexpected and is about some of the things that have been really interesting me and that I think that, you know, maybe other people can learn about too and understand about. So, you know, object storage, I think is actually one of the most exciting kind of areas in cloud native right now, and I'll explain why.
Anyway, I'm Justin Cormack. I'm the CTO at Docker. I've been working at Docker for a long time. We use a lot of object storage at Docker. We have this thing called Docker Hub and it uses a very large amount of object storage, so it's something that, you know, we care about. But we also, I think there's a lot of other lessons from object storage. I think it can teach us a lot about what cloud native infrastructure looks like and why, you know, it's one of those things that actually is really kind of foundational and kind of interesting in the way the user experience and the way you can use it.
Amazon S3 was my first experience of the cloud. I think it was for many people, it was the first cloud service that Amazon launched back in 2006. And I remember thinking this thing's kind of cool. And then bringing it in as shadow IT into a company I was working in and being told off for not using official Amazon servers and using my own private account. But it's like, well, get used to provisioning this stuff, we're going to need more of this. And there's a whole lot of new stuff going on that I'm going to talk about. Like, it's actually like this year has been actually like the most exciting year for object storage since 2006, I'd say. So there's actually an awful lot going on.
Like most of my talks, I'm going to start with a bit of history. I like understanding things through the view of how we got here. And the brief for Amazon S3 was malloc for the internet was what Jeff Bezos asked for. And it's kind of weird, but it's kind of catchy as well. And it's kind of nice they didn't announce this at the time. It was published in some Amazon post much more recently.
But the idea was just, you know, you can allocate something and store it on the internet and it's there, and then later maybe you can free it again or something. But it's like that kind of idea of like, it's a really, really simple primitive that you can build other stuff on. It's not the core of anything, but it scales out and it, you know, it works for everyone. It's like just basic infrastructure that absolutely every application will use was the idea. And that's kind of starting to, you know, it's really starting to pan out.
This was the announcement. The page is still up. It's a very short announcement compared to the announcements Amazon makes nowadays. And it kind of hints that this was, you know, something they used internally for running their websites. And I think for many, many years that was one of the biggest use cases of S3 was people running websites on it. It kind of made sense. But again, we're going to talk about the kind of applications people are building on object storage.
Now, just, you know, what is an object store exactly? It's roughly, it's a key-value store with an HTTP API. You have names which look a bit like file paths but have weirdly different properties, and then you can store things in them up to sort of typically about five terabytes or so. And there's some listing functionality that's, you know, mostly people have copied the Amazon implementation interface, not entirely. Azure has their own thing and Oracle has a couple of different things, you know, but it's become a kind of de facto standard.
There's lots of open source implementations, like MinIO is the best known. They've got a stand over there, and they're doing lots of interesting things. So they're, you know, they're one of the largest open source implementations that was actually implemented, I think, by some of the people who worked on the Amazon implementation.
There's some other constraints. It's not a file system. You can only update objects all at once, so it's very restrictive. But that's kind of, you know, why it's restrictive is like actually because the kind of capabilities and design requirements are just kind of very, very, very... I mean, they're kind of, you know, there are 11 nines in that durability statement. That's a lot of, you know, that's a lot of engineering work has to go into something that's going to have 11 nines of durability.
You know, I think that you've got to really think about like how simple can you make the service, how much can you verify that it's not got bugs in, and all those things. And I know that, you know, the implementations out there have, you know, have done a lot of work. And that simplicity and scaling, I mean, 280 trillion objects stored in Amazon S3 alone, let alone all the other providers who have these storage interfaces like this, is something that's really different from the kind of traditional sort of file system use cases.
And this is really, I think, what, you know, cloud native is about: this kind of infinite scale out, never have to worry about something, you know, running out of space, not working. Like you just, you can just put things in it. It's cheap and it's simple. And, you know, you can write a really simple program that uses it, and that's really quite, really quite powerful. And I think that having simple primitives allows you to really invest the effort in making them absolutely 100% reliable, or 99.999999% reliable.
And, you know, that gives you the assurance that you can safely build applications on this, and so many people have done that. Next part, we're going to talk about how to use object stores and like the ways that people use them, and what's weird about them, what's normal about them.
The performance characteristics are kind of odd if you're used to kind of, you know, traditional file system things like that. There's a lot of latency to an object store. Some of it's just going over the network, but a lot of it's also to do with that reliability thing and availability. Like the data has to be replicated out to multiple locations every time you write. And when you read, you've got to find, you know, there's obviously a huge amount of data. You've got different storage tiers and you've got different, you know, different amounts of caching and different amounts of, you know, kind of scale in terms of like hotspots and so on.
So there's a lot of work that's going behind the scenes, which gives you latency just kind of in addition to the network latency. There's, yeah, as I said, there's dynamic usage-based scaling, so that if you, you know, if you're using a particular part of an object store more, then more servers will be there, you know, scaling out to provide the bandwidth.
The total throughput in general is like, is extremely high. I mean, again, using Amazon example, you can get 5 Gbits on a single connection to S3, but you can run as many connections as you like, and you can saturate, you know, 100 Gig Ethernet connection on your cloud machine to a bucket. And then you can, but you can also have like thousands of machines accessing at the same time. So it's a really, you know, there's really high scale, high parallelism, high throughput, but you've got this high latency.
And particularly, we'll talk about this with some of the workload characteristics of things people are building, thinking about like, you know, the latency and the fact that you don't want to do lots of, with high latency, you don't want to do lots of tiny sequential writes. You want to do, you know, reasonably large size writes to kind of overcome that latency.
Cost is something that's kind of complicated. Depends who you know, depends who your cloud provider is, what their model is, or whether you're self-hosting with or using, you know, self-hosting, right, MinIO. You've got different cost characteristics, but roughly you do have to think about bandwidth. Egress bandwidth, you know, if you're serving from an object store to the internet, can be very large. But there's Cloudflare R2 is an interesting service, for example, because they don't charge egress bandwidth. So for a lot of applications, that's really, you know, it's a lot of the cost.
Storage costs depend again on, you know, the latency and the tiered storage models that the cloud offers, because object stores are generally replicated across availability zones. You effectively get between replication, in between availability zone replication for free. We'll talk about again a bit later with some examples.
And, you know, but you've got slightly weird cost models that you have to kind of bear in mind with how you're using it. But generally, object storage you can generally treat as being pretty cheap. In general, it's not usually the dominating cost for most applications. So it's pretty exciting.
I want to talk a whole bunch about concurrent updates because, you know, when S3 was originally launched, you know, the example use case was websites, and that was kind of relatively straightforward. Usually for updating a website, there's a single deployment pipeline, so you didn't have to worry about multiple updates at the same time, concurrency. So it was a relatively straightforward kind of process. You just, you know, write some, write new, update the website kind of page by page, and that would be okay mostly.
So people weren't very concerned about concurrent updates, but as people started building more applications on top of object storage, they got more issues around this. One thing that we use heavily in, you know, in the OCI registries is content-addressed storage. If every piece of storage is just named by the hash of the content, concurrency becomes much easier because either the thing is already there and you've got hash ABCDE and that's written and that's there.
There's some complicated issues with the registry stuff about permissions, but that's another piece. But either it's there or you need to write it. There are even ways, at least on AWS, you can actually generate signed URLs that only give you the ability to write to a particular location with content that matches the hash, which is kind of cool. So you can actually just say, like, you can write the content with this SHA-256 hash to this file name that has this SHA-256 hash, and here's a signed URL. It doesn't give you any other permissions, so you can't write anything else.
We don't actually use that on OCI for complicated reasons. But so that solves some concurrency problems with a lot of the base storage for registries, but it doesn't deal with things like actually updating tags concurrently.
Then a lot of the strategy that people used for concurrent updates was to use a database to manage concurrency on top, which is effectively what we do on Docker Hub at the moment. But a lot of people use DynamoDB. There was a sort of standard process of using DynamoDB on Amazon, for example, as a way of managing concurrency and dealing with locking, which was kind of hacky, a bit worked.
But recently, there's a new primitive been added to a lot of the object storage implementations. Cloud had it for some time. MinIO, I got them, thank you for not accepting my pull request but fixing the feature anyway earlier this year. And then Amazon in August added support for it. So it's generally quite widely supported now, which is basically put with if-none-match, which effectively is like create this object if it doesn't already exist.
Which gives you a kind of mutual exclusion concurrency primitive that you can use to actually have multiple applications trying to write something, and if they either succeed or they have to back off and retry, you know, retry another object. And so you can really get that. It's a really good concurrency control primitive that's now becoming very widely supported, and you don't have to kind of use a database. So you can just use the object store directly.
And a whole bunch of people are using this for things I'll talk about in a minute. The pattern that you can do once you have this primitive is really just the simplest way of using it is just create ordered numbered files: 0000000000, 0000000001, 0000000002, and each time an application comes along, it tries to put the next file in that sequence. It either succeeds automatically, or someone else has got in there first and it has to, you know, potentially re-read the data that's been written by the other process and adjust what it's done and retry.
This was originally actually where I got quite excited about this whole building applications on object store with the Delta Lake paper. So Delta Lake is a Linux Foundation project that's basically the basis of Databricks' services, and their paper is really nice, very well written, really readable, highly recommend reading it. It talks about, you know, using these concurrency primitives on object storage.
And it was one of the, I read it a few years ago, and it was like one of the first things that actually got me really... It's actually quite an old paper, actually, I didn't see it at the time, but it got me excited about like, we can build database applications on object storage, and like, this is kind of cool.
There's a nice blog post by Phil Eaton on, like, which runs through basically building the basis of this in 500 lines of Go code, just to kind of show you how it's done as well, which is kind of nice. And this ordered sequence of files is basically, you know, in database technology, is a log, which is the basic primitive from which databases are built.
There's this great Jepsen article from a long time ago from LinkedIn about what you should know about logs and thinking about the log and thinking about that as a primitive. And essentially, you know, a log is, you know, it's just an ordered set of, you know, an ordered set of files that has the changes that you're making to your database, and you can build any data storage system from the log. It's how things are built internally.
Yeah, there's lots of things you might want to build on top in terms of indexes and all the rest of it in terms of databases, but it's the fundamental storage primitive. And so this has really given us a way to have, you know, a way of thinking about having logs on object storage and using that as a primitive to build all sorts of, you know, applications that use data, databases, whatever you want to call them.
So I'm going to talk about a bunch of the things that are being built that I think are really exciting, worth learning from. I say some of them are open source and some of them are not. I actually think there's surprisingly few things being built in open source using this model, so I'm kind of encouraging people to think about building more things. But so some of the examples are not open source, but some of them are.
As I said, Delta Lake was the first one I came across and the first one that was kind of published, but actually all of the analytics databases, Snowflake and so on, are built on object stores. And really, particularly this year, there's been the whole, the kind of open format war in that area. It's been kind of like, I think, hasn't got much notice outside the analytics community, but everyone has decided that open standards are really important here and everything is going to be interoperable.
Maybe one of the formats will win, or maybe everyone will support everything. They're still kind of arguing, but all analytics databases are now essentially being rebuilt on a fully open backend that's built on object storage, more or less. I mean, these things, Delta Lake again has the paper, but the other ones are kind of similar. They're more or less Parquet files, which are basically indexable columnar data format.
Plus information about schema changes in the database, just set in a log, with a whole bunch of sort of compaction and things like that on the way. There's a whole lot of nice properties on these things. For example, because this is just an immutable log on object storage, if you don't do compaction, you don't delete stuff, you can query the state of your database at any point in time.
So there's some really nice examples where, you know, tools where, techniques where people are actually building kind of Git-like database systems that you can like checkpoint and branch and things like that, because you can basically fork a different log and start doing, you know, take part of the timeline and history of checkpoint at one particular point in time and then go and branch it.
And so there's a whole lot of nice properties that these things have that are kind of fun for, you know, as a developer and so on. So those are kind of interesting byproducts. So these are really exciting, and it's a really interesting area where basically openness and standards are starting to win in a way that means that there's a really exciting community of people building on these open standards that is really interesting.
Another one that I came across recently, and they're working with the SEF project, is for virtual disks. So this is again, it's an open source project, and they're building, they basically want to build scale-out disks. So they have a local SSD cache, and then they send the changes, the changed blocks to S3 as a log in batches.
And essentially, what this enables them to do is, you know, just send file system writes anywhere on the block device, you know, as a stream to S3 with a local cache. So you can effectively have, you know, a pretty much an infinite, if you want, virtual disk. And if you look at the performance of it, it's really good. It's actually much cheaper than, you know, using Amazon EBS.
Because you've actually got a lot of, you know, a lot of performance out of this, and it's very cheap and it's very, very high performance. And it's really quite, you know, it's quite a simple model. They're reworking this for some other use cases, and there's a bunch of work going on here. But, and they kind of, it's, you know, as I said, part of the SEF project.
But I think it's a really interesting one where, you know, there's a lot of use cases where having, you know, having a virtual disk is a kind of useful abstraction for building something on, but having it, you know, in a way that actually gives you a backup onto S3 every time you have sync, and you can potentially, you know, duplicate it down to another machine really easily or whatever you want because it's available in the object store.
So we're doing some work internally with quite a similar model to this, and this code's been really kind of interesting. So that's a nice disk format one. This is just a very simple one, like how do you do leader election, which is again a kind of useful primitive, just using the if-none-match as a mutual exclusion, build a log and do leader election on it.
So you can, again, one of the things you notice with these things is like you don't have to build a lot of the distributed systems primitives because you've already got this, you know, consistent backend that you can use, and you've got this, and now you've got a concurrency primitive, you suddenly got the ability to really build things on top of that really easily.
SlateDB is a kind of embedded database, I think SQLite, but backed by an object store rather than a local disk. It's very new. They've been working on it, you know, only for a few months as far as I remember. But it kind of, it's kind of an interesting case where, you know, it's got the high write latency that you have with object storage, but depending on how often you chunk your writes and so on, what latency you're prepared to put up with, and the cost.
But it gives you a very simple kind of primitive, you know, building an embedded database to use in your applications that will persist, you know, will persist reliably in, you know, in your object store forever, and you can use that. So that's, this is another, you know, fun project for which they, you know, they've got a lot of example use cases here.
We're going to some of the non-open source ones. Turbopuffer is a vector database on object storage. Their customer was one of their first customers. It scales out extremely well. And this little story here is kind of amusing where they basically just say, like, you know, again, because the object store is doing all the work of the reliability, you're basically just taking ephemeral compute in front of it.
And so you end up with a very different kind of reliability story. I mean, like, their high availability story is just, yeah, we've got pods, we've got lots of pods, that's our redundancy. And like that, because you don't have to worry about all the, you know, normally when you're writing a database, you spend a lot of your time working on, you know, persistence and all those pieces, but all those things are done for you if you use an object store.
And so, you know, but again, you've got this high latency but high throughput performance characteristics, which you kind of have to work around a little bit. Another interesting non-open source one is WarpStream. That's an Apache Kafka implementation. They were recently acquired by Confluent, and as part of their 'you can run Confluent everywhere' thing, Confluent are using it as the bring your own cloud, bring your own, effectively, you know, bring your own object storage bucket implementation.
There's an interesting article, I think, there by Jack Vanlightly talking about, you know, the fact that, you know, for a long time Confluent didn't want to run on-prem, you know, in the customer's data center solution because they really invested heavily in having a, you know, large multi-tenant solution. But WarpStream was simple enough because of the operational simplicity of it just being an object storage with ephemeral compute that they felt that that was something that they could reliably support, you know, on the customer's premises.
And that real operational simplicity is really, really, you know, important from the point of view of if you're going to run stuff that you have less observability and control over because it's at the customer's sites, then you've got to be really confident that it's absolutely reliable.
It uses the fact, one of the selling points of WarpStream when it was an independent company was that they, again, you get the free replication between availability zones because the object store is replicated between availability zones, so you don't have to pay cross-network traffic there. And so their pitch before they were acquired was it was just very, very cheap to run as long as you didn't mind a bit of extra latency, was the cheapest possible way to run Kafka.
Again, I think, you know, it's not open source, but I think, you know, these things are kind of interesting from the point of view of things. Another not open source one, BitDrift from Matt Klein. Matt Klein, who did Envoy, again, they've got a bring your own bucket model for how to host the service. And again, he talks about this blob store architecture and how every modern database is being built with blob store first architecture is like, you know, his point.
So again, like this is, you know, he goes through, you know, these advantages of this, you know, again, there may be local storage and RAM as a cache, but in some of these architectures to overcome the latency, but the underlying storage is the object store.
So wrapping this up, that, you know, the things that could be built, I think, you know, the reason I titled this 'Object Storage Is All You Need' is I'm kind of really quite, you know, having looked at these kind of architectures and the benefits of them, I think that there's a huge advantage to using these kind of simple cloud native primitives as the basis for building all sorts of applications, and you can really simplify what you do if you can offload, you know, the difficult bits to something as simple as an object store.
I, at one point, I was going to, I was thinking of a while ago back of like taking the CNCF landscape and marking off which things are, you know, which things are not annoying to operate because they've actually got a Raft implementation in them, and like that's, you don't really want to have to do those things if you can avoid it. And building your persistence layer on object store means you don't have to do that.
There's this, we're starting to see sort of next generation of object stores that are lower latency. Object store was built on hard drives, and people are rebuilding those on SSD, so that cuts down some of the latency. Some of it's again network latency. Amazon launched their object store like that recently, and one of the other cloud providers is launching one soon. So I think we're going to see somewhat lower latency options, which, you know, gives you some more choice.
We're just seeing, you know, lots and lots of databases and applications being built directly on object storage, and, you know, I think this is really, this is, you know, it's a really attractive option if you're trying to build something that has data, which is pretty much everything.
And, you know, as I said, like it's very cloud native, it's a very cloud native way of thinking about it. Use these simple primitives, offload the hard things to the cloud or to a system that's dedicated to doing that. Even if you, you know, even if you run MinIO yourself, you're still off, you know, it's still one point which you can offload to.
If you're building something, let me know. I've been spending a bunch of time talking to people who are building in this space to find out what people are doing and talk to them, and I'm really happy to help out if you, you know, and I kind of, I'm interested in, you know, building a community of, you know, people who are interested because I think that it's something that, you know, there's a whole lot of applications that can be built and there's a whole lot of, you know, it's a whole different way of thinking about building applications, interactive applications, and it'd be really interesting if people are interested.
So get in touch and, you know, let me know, let me know if you're building something or you've got ideas or you want any help. So thanks very much. Hope you enjoy the rest of your stay in KubeCon Salt Lake City. And happy to, got three minutes for questions. So, but otherwise I'll be around very briefly, although I have to go to the airport pretty quickly. So, but you can always email me and get in touch.
