Databricks Lakebase (OLTP) Technical Deep Dive Chat + Demo w/ @Databricks Cofounder, Reynold Xin

Close enough. Close. Look, we were just talking about my name and how difficult it can be. So, I think we're even.

Yeah, hope I pronounced it correctly. Said that couldn't have said it better myself. Well, thanks for having me here. From a broader industry point of view, I was talking about this at Data and AI Summit. We felt like databases, OLTP databases, really haven't changed that much in the last 20, 30 years. If you look at analytical databases, they have evolved a lot. We are now probably two or three orders of magnitude faster than we were in the 90s and maybe even early 2000s. That's because we built on a lot of foundational technologies that are both cloud-based and also just hardcore database engineering. But OLTP database, more or less, if you crack open a MySQL, Postgres, or Oracle today, there are a lot of changes but they look more or less the same as what they were back in the 90s. So we felt that it's the right time to think about how we can disrupt and build next generational databases that are cloud native and also agent native. One thing that was pretty interesting, we actually didn't know about this until the Neon acquisition, was that 80% of the databases on Neon were actually created and managed by AI agents. This is an incredible stat because a year ago that number was 30%. So it went from 30% to 80%. And if you extrapolate that line, at some point 99% of the databases will be created by AI agents. Historically, databases were operated like provisioned and operated by humans. With the LLM trend and agent coding, we will see that the actual persona might still have a human somewhere supervising all those agents, but really the persona is changing from humans to agents. That shift in persona comes with a lot of new requirements that historically simply weren't even part of the requirements list for databases, or maybe they weren't even in the top 100. An example would be, you want to parallelize a lot of different agents to work on a task and maybe have each agent run some version of an experiment and you want to compare which one does best. At that point, you want to be able to snapshot your data and then provide a different instance of the database that's fully isolated and sandboxed just for that one agent, because the agents could be making mistakes. You don't want the agent to be impacting your actual data. And every one of these operations needs to be pretty fast because the agents are operating at pretty fast speed. So there are a lot of new requirements coming in now that historically were simply not a concern for database system developers. We love that when new requirements come in, it gives an opportunity for us to rethink the fundamentals and think about how we should be architecting databases. Then we can build the databases for the future. The databases of the past were built for human operators. In the future, they will be built for human operators supervising a lot of different agents doing it. It's intellectually challenging, but ultimately it comes down to we think we can build a new era of databases that are just much better than what was done in the past, especially given the new persona.

No, absolutely. Databricks Apps has been the number one, I think maybe the fastest growing product in the history of Databricks. It used to be Databricks SQL, but I think Apps actually took that. It's still early on, so it's a little hard to predict exactly how it will go in the future, but it's been growing super fast. The number one feedback we've been getting from customers is, can I just get a provisioned database together with every app I provision? So we're definitely seeing that trend.

The truth is, a lot of the requirements for agents, once we can support them, ends up helping humans a lot also. But I would say for humans it's a much nicer to have, whereas for a lot of agents it becomes kind of a necessity. Let's take creating snapshots and branches of your databases quickly. That's actually an incredibly useful feature even just for human coding. One thing we see a lot of customers doing now is that for every Git branch, they would automatically create a snapshot, like a branch of the data for the database. Developers really love it because it makes it much easier to test against high fidelity data.

Yeah. So Lakebase has been building on this technology we actually acquired from Neon. The interesting thing is we have been working together actually long before the acquisition, and that led to the acquisition itself because we realized how great the tech was. Earlier we talked about how analytics databases look very different today from 20 years ago. One of those changes was the lake house. The lake house has this very important property: separate storage from compute. So you can store your data in massive volume in cloud object stores like S3 or Azure Blob Store, and then you can launch compute instances that are very ephemeral to process the data in object stores. There are many advantages here. One is it enables elastic scaling. You can dump as much data as possible, and when you have a big job to run, you launch more compute resources; when you don't, you shut them down. It's much more cost-effective, and the cloud object stores also tend to be probably the cheapest durable storage medium for data. The thing that the Neon team pulled off and built is that they have a similar architecture, but it's actually harder to accomplish in OLTP because in analytics, a 100 millisecond query is considered pretty fast, so tail latency spikes from object stores are fine. But in OLTP, you want to be able to process queries in less than a second or less than a millisecond. A second is way too slow, even 100 milliseconds is often considered too slow. Also, Postgres has become the lingua franca of OLTP databases. If you look at all the database usage trends, Stack Overflow, the DB-Engines ranking, Postgres is going up super fast. It has a large open source community, and thanks to that, its usage and adoption is growing like crazy. But Postgres fundamentally is a coupled storage and compute instance. If you go to the vast majority of Postgres providers, they give you a box. The box has storage in it, has compute in it. You cannot independently scale it. If you need more storage, you now need more compute, and even that move itself is very difficult because it's considered fixed capacity provisioning. The Neon team figured out a really interesting way to implement separation of storage from compute for Postgres specifically. They built this storage service. There are actually two key services that sit on top of cloud object stores like S3: a safekeeper and a page server. The safekeeper is basically a replicated write-ahead log. The way databases work is instead of modifying the data in place, they write to a write-ahead log first. The safekeeper is a distributed replicated write-ahead log service. The page server is a service that takes all the actual Postgres pages. Databases, in the case of Postgres, separate all the storage into different small granular pages. The page server is a distributed storage system for all the pages. It ultimately stores everything in S3, but because we don't want to pay for the latency hit of going to S3, it's a distributed farm that caches all the data as much as we want to provide much lower latency access. They made extremely minimal changes to Postgres to change the underlying storage layer. They found some very narrow ways in Postgres so you can change Postgres to, instead of writing to local disk for both the write-ahead log and the data pages, use these two services on top of S3. There are a lot of benefits made possible with this architecture. One is the storage cost is super low, and storage is automatically durable, as durable as the cloud object storage which is often either multi-AZ or multi-region. In the past, when you get a fixed capacity Postgres instance, that storage is only as durable as the storage on that box. If that box goes away, you lose all data. So people come up with backup solutions or use EBS, which is super expensive compared to S3. Another benefit is it makes super fast autoscaling possible. On top of this architecture, the Neon team built an autoscaling service that provisions compute nodes, Postgres nodes, in less than 500 milliseconds. Because you can do this, I think it changes the paradigm for both agents and humans. The agent is not just a marketing term here, but it changes the paradigm of databases. While there are one class of databases that are extremely latency sensitive and you simply cannot afford a tail latency spike at all, I think most applications, especially internal applications that a lot of enterprises build, it's okay every once in a while to have a latency spike of a few hundred milliseconds. You don't want it to be seconds, but 100 milliseconds is fine, because that's roughly human perception time, 300 milliseconds. If we can provision and acquire compute resources in hundreds of milliseconds, we can scale the database itself entirely down to zero without paying for it when there's no traffic to your service, which happens all the time. You might have a lot of traffic for an internal app in a specific time zone during 9 to 5, but past that hours it drops down to approximately zero for many companies. For many companies, maybe the lunch hours also drop down, or in some cases you have spikes at 9 AM because everybody gets into work and starts looking up some app, but then at 10 AM it goes down. It doesn't go down to zero, but it goes down. So they built this autoscaling architecture that makes the database itself adjust its resources dynamically based on the load and can also go all the way down to zero. This is super cool because I think it's probably the first time that you can have an OLTP database be so responsive and so elastic.

Yeah, it's a great question. At a very high level, there are some similarities, especially if you think about Aurora also having some attempt at separate storage from compute for OLTP. Aurora might be the most popular offering out there. But if you look deep enough and think about the technical trade-offs, it's actually very different, and that leads to very different benefits for the end user. One is that Aurora started more than a decade ago, and at the time the object stores were much worse. So they had to build a completely proprietary storage engine that is not open at all and only Aurora services would have access to. Whereas on the Lakebase side, Neon is built on just cloud data lake, built on S3, built on Azure Blob Store. This might seem like, what's the big deal? A proprietary storage system versus a more open storage system. There's actually a massive difference. One is, aside from engineering being more complicated to make it work for object stores, what's the benefit? Why should the user care? One is that cloud object stores are far cheaper than EBS, just because of the way it was done. EBS is considered more of a high-end network storage, whereas cloud object stores are more of a data lake, and data lakes are fundamentally cheap. You have much better unit economics. The second thing is because it's built on cloud object stores, the data stored on it is also open spec and open source. They're literally just Postgres pages. So we're kind of recreating the interesting lakehouse paradigm here, except for OLTP, which is you have the actual source of data stored in cloud object store which can be very high throughput in open format. On top of it, we could not just build an OLTP database with Postgres being able to access that, but we can actually build, we haven't done all of it yet, this is still in the works, but it enables the possibility of having, for example, Spark directly querying all the data, or DuckDB querying the data, provided there's the right adapter for it. Because it's an open format, and it's honestly just Postgres pages, we could be creating, either us or the community could be creating those adapters. So it just opens a lot of use cases. Let me give you an example. One very simple one is there used to be this giant divide between analytics, like lakehouse, data warehouse, data lakes, and OLTP. When your OLTP becomes successful enough, you want to start applying analytics on that data. How do you do that? You start trying to build pipelines to get data out and you learn complicated concepts like CDC and all that. But if the actual OLTP data is in open format on top of a data lake, which is basically the lakehouse now, they're no different from lakehouse data. Just go read them directly. You can read them using Spark. You can read them in massively parallel fashion. If you have a petabyte of data, go read them, it's not an issue. I think that just changes the scaling and also makes a lot of operations simpler. Every large enterprise operates tens of thousands, if not millions, of databases. They are destroying database instances because the storage was not shared. By having all of that storage in one data lake, basically the lakehouse here, you can now enable workloads that plow through all of them. For example, if you want to do a global scan, let's say hypothetically you found one of your databases got compromised and there's a table that got created with a ransomware note. Hopefully in the case of Lakebase you don't have to worry about getting your data back because we have time travel and snapshots, but at the very least you want to know what's going on with your other databases. Did anything else get compromised? How do you do that across maybe a million different instances if they are destroyed? It's incredibly difficult, time consuming, laborious job. But if all of them are actually in the lakehouse, you could write a single Spark query to scan all of them, or in Databricks SQL if you like SQL. I think that is going to be revolutionary to how we operate and manage databases as well.

Yeah, exactly. And then because of the autoscaling from 0 to 4, this is sort of approximate some sort of memory, but you can actually get it to super small, you can have it scale to zero. So if there's no usage after like 5 minutes, this whole thing just disappears, but then if it needs to come back, it will come back in hundreds of milliseconds. So the compute is disappearing. You're still paying for the storage. You're paying whatever you have in storage. If you have no compute, you're not really paying anything more for compute at that moment.

Yeah. Yeah. Yeah. But yeah. So this is basically the entire experience. It's super simple. It's a full-blown Postgres. We didn't really show it, but it's a full-blown Postgres. You could use the built-in SQL editor, but the vast majority of people will actually be connecting to it from their applications.

Yeah, so to clarify, it's a little bit confusing. We announced Lakebase public preview at Data and AI Summit, and that is more of a fixed provision Lakebase. You provision whatever resources you want and that's it. You could reprovision or provision it down, it would take a while, because it's still separated storage from compute, it's the same Neon storage technology under the hood. But what I have shown here is an improved version upon that, in beta right now, that actually has basically all of Neon's technology. So it's not just the storage part, but the entire autoscaling, the developer experience, all of that. So now we have snapshotting, branching, backup, restore, all of those are now super simple. It's just in the UI or you can use command line, all that works. It's much snappier. We actually see this to be the future and eventually replace the current provisioned Lakebase.

Yeah, exactly.