Spin Up a Whole Production Database for 1 Cent | Reynold Xin, Databricks

Databases are viewed as this clunky, expensive, heavyweight thing. The term infrastructure reminds you of PG&E and utility companies, right? They're not nimble. And as a result, they in the past actually inhibit innovation because maybe you can write code very quickly, but if you can't actually provision a database very quickly, and you can run a database very cheaply, you might decide not to run an experiment at all. So we want to enable experimentation at scale that is very low cost. For example, maybe if something – if you don't know if something is going to take off or not – it better not cost you a fortune. It should cost you maybe a dollar or two in order to run an experiment, maybe even less. But once it takes off, you should not require to re-architect everything because you wouldn't have time to do that. We think infrastructure in general and databases in particular – database is one of the most important pieces of infrastructure – needs to enable rapid experimentation at scale cheaply, but at the same time, for any of these experiments that actually takes off, you should be able to scale to mission critical on exactly the same set of infrastructure without having to do any re-architecture. So that's what we're doing with the lake base, and there are many things that go into it and many benefits that come out of it. For example, you can provision a database in less than a second. So now your engineers don't have to wait. Everything is auto scale; it can scale down to zero automatically if there's no load. So if your experiment fails, you don't really pay anything. And because of auto scale, if your experiment actually becomes successful, you can provision it to a much larger resource pool. So you can now support, for example, millions of transactions per second. As a matter of fact, we even talked about we will support multi-cloud disaster recovery, which is probably the industry's first managed multi-cloud disaster recovery. So you can really go to mission critical even if the whole cloud was down.

Yeah, from a database side of things, it's one of the things we think: databases are a fleet-wide problem. You should not think about a database as a single instance. You should always think about databases as, hey, you have a collection of databases and fleets, and how do you actually manage them? Each of them, we think auto scaling is going to be extremely important here. Because most databases in the past, or probably even now, are done in a way that you provision one and it's up and running 24/7. It never shuts down. And that's where a lot of the cost comes from. Because you have to provision for the peak, and more often than not, for something that is not super successful, you provision it but then you forget to shut it down, and before you know it, it racks up hundreds of dollars per month, and you have a thousand of them. Auto scaling takes away that issue. Because if there's no traffic, it's not up. It doesn't cost you anything. And if there is a spike in traffic, it auto scales to have more resources so it can handle that load. We think that's a very critical part of it. The other thing is that a lot of the infrastructure spend that people put together by agents is largely because of experimentation and CICD cost.

Lake Base also solves the problem with this branching. Basically, Lake Base has this thing called branching. It's actually really unique to Lake Base. What it does is it takes a complete clone of your database – logical. So it doesn't actually really copy, it doesn't really clone the database. It just marks a copy-on-write clone of the underlying storage layer. And it's basically a pointer saying, 'Hey, this is the state of the database.' And if you do updates, it will now start recording updates. But with this copy-on-write branching, you can actually clone a production database or a test database in less than a second without paying for extra storage. So this becomes a great way to run CICD by combining all the scaling and this branching, because every time you create a new pull request to, for example, your application code – this is actually very common in our model of customers now – they would create a branch of their database. And they'd run the entire CICD on that pull request and that branch. And then when CICD is done, it's shut down. It scales down to zero. They often delete the database. It incurs maybe 1 cent of storage cost because of that pull request trying to insert another row or two. And then it incurs very little compute because it just runs for the duration of that little CICD pipeline. And then it's gone. This is not something you could – by the way, it has perfect isolation because it's an actual new Postgres instance that's running just for this. So this is something that's impossible to do outside of Lake Base right now, because if you actually want to do a clone of the database, you have to clone the data outside of Lake Base. And as a result, it becomes very expensive. It takes a long time. But the cloning of your production database could take down a production database itself. That's another horror story that happens all the time. So Lake Base just solves all of these problems and really makes it much easier to control the cost and have much lower TCO while enabling all of this possible experimentation and innovation.

That most of our empires is not a natural physical copy, right? It just appears a logical copy. A logical independent copy is isolated, so you can run all your experiments on it. It has no impact on the rest of the other PRs or your production system.