Databricks Lakebase (OLTP) Technical Deep Dive Chat + Demo w/ @Databricks Cofounder, Reynold Xin

Close enough. Close. Look, we were just talking about my name and how difficult it can be. So, I think we're even.

Yeah, hope I pronounced it correctly. Said that couldn't have said it better myself. Well, thanks for having me here. From a broader industry point of view, I was talking about this at Data and AI Summit. We felt like databases, OLTP databases, really haven't changed that much in the last 20, 30 years. If you look at analytical databases, they have evolved a lot. We are now probably two or three orders of magnitude faster than we were in the 90s and maybe even early 2000s. That's because we built on a lot of foundational technologies that are both cloud-based and also just hardcore database engineering. But OLTP database, more or less, if you crack open a MySQL, Postgres, or Oracle today, there are a lot of changes but they look more or less the same as what they were back in the 90s. So we felt that it's the right time to think about how we can disrupt and build next generational databases that are cloud native and also agent native. One thing that was pretty interesting, we actually didn't know about this until the Neon acquisition, was that 80% of the databases on Neon were actually created and managed by AI agents. This is an incredible stat because a year ago that number was 30%. So it went from 30% to 80%. And if you extrapolate that line, at some point 99% of the databases will be created by AI agents. Historically, databases were operated like provisioned and operated by humans. With the LLM trend and agent coding, we will see that the actual persona might still have a human somewhere supervising all those agents, but really the persona is changing from humans to agents. That shift in persona comes with a lot of new requirements that historically simply weren't even part of the requirements list for databases, or maybe they weren't even in the top 100. An example would be, you want to parallelize a lot of different agents to work on a task and maybe have each agent run some version of an experiment and you want to compare which one does best. At that point, you want to be able to snapshot your data and then provide a different instance of the database that's fully isolated and sandboxed just for that one agent, because the agents could be making mistakes. You don't want the agent to be impacting your actual data. And every one of these operations needs to be pretty fast because the agents are operating at pretty fast speed. So there are a lot of new requirements coming in now that historically were simply not a concern for database system developers. We love that when new requirements come in, it gives an opportunity for us to rethink the fundamentals and think about how we should be architecting databases. Then we can build the databases for the future. The databases of the past were built for human operators. In the future, they will be built for human operators supervising a lot of different agents doing it. It's intellectually challenging, but ultimately it comes down to we think we can build a new era of databases that are just much better than what was done in the past, especially given the new persona.

No, absolutely. Databricks Apps has been the number one, I think maybe the fastest growing product in the history of Databricks. It used to be Databricks SQL, but I think Apps actually took that. It's still early on, so it's a little hard to predict exactly how it will go in the future, but it's been growing super fast. The number one feedback we've been getting from customers is, can I just get a provisioned database together with every app I provision? So we're definitely seeing that trend.

The truth is, a lot of the requirements for agents, once we can support them, ends up helping humans a lot also. But I would say for humans it's a much nicer to have, whereas for a lot of agents it becomes kind of a necessity. Let's take creating snapshots and branches of your databases quickly. That's actually an incredibly useful feature even just for human coding. One thing we see a lot of customers doing now is that for every Git branch, they would automatically create a snapshot, like a branch of the data for the database. Developers really love it because it makes it much easier to test against high fidelity data.

Yeah. So Lakebase has been building on this technology we actually acquired from Neon. The interesting thing is we have been working together actually long before the acquisition, and that led to the acquisition itself because we realized how great the tech was. Earlier we talked about how analytics databases look very different today from 20 years ago. One of those changes was the lake house. The lake house has this very important property: separate storage from compute. So you can store your data in massive volume in cloud object stores like S3 or Azure Blob Store, and then you can launch compute instances that are very ephemeral to process the data in object stores. There are many advantages here. One is it enables elastic scaling. You can dump as much data as possible, and when you have a big job to run, you launch more compute resources; when you don't, you shut them down. It's much more cost-effective, and the cloud object stores also tend to be probably the cheapest durable storage medium for data. The thing that the Neon team pulled off and built is that they have a similar architecture, but it's actually harder to accomplish in OLTP because in analytics, a 100 millisecond query is considered pretty fast, so tail latency spikes from object stores are fine. But in OLTP, you want to be able to process queries in less than a second or less than a millisecond. A second is way too slow, even 100 milliseconds is often considered too slow. Also, Postgres has become the lingua franca of OLTP databases. If you look at all the database usage trends, Stack Overflow, the DB-Engines ranking, Postgres is going up super fast. It has a large open source community, and thanks to that, its usage and adoption is growing like crazy. But Postgres fundamentally is a coupled storage and compute instance. If you go to the vast majority of Postgres providers, they give you a box. The box has storage in it, has compute in it. You cannot independently scale it. If you need more storage, you now need more compute, and even that move itself is very difficult because it's considered fixed capacity provisioning. The Neon team figured out a really interesting way to implement separation of storage from compute for Postgres specifically. They built this storage service. There are actually two key services that sit on top of cloud object stores like S3: a safekeeper and a page server. The safekeeper is basically a replicated write-ahead log. The way databases work is instead of modifying the data in place, they write to a write-ahead log first. The safekeeper is a distributed replicated write-ahead log service. The page server is a service that takes all the actual Postgres pages. Databases, in the case of Postgres, separate all the storage into different small granular pages. The page server is a distributed storage system for all the pages. It ultimately stores everything in S3, but because we don't want to pay for the latency hit of going to S3, it's a distributed farm that caches all the data as much as we want to provide much lower latency access. They made extremely minimal changes to Postgres to change the underlying storage layer. They found some very narrow ways in Postgres so you can change Postgres to, instead of writing to local disk for both the write-ahead log and the data pages, use these two services on top of S3. There are a lot of benefits made possible with this architecture. One is the storage cost is super low, and storage is automatically durable, as durable as the cloud object storage which is often either multi-AZ or multi-region. In the past, when you get a fixed capacity Postgres instance, that storage is only as durable as the storage on that box. If that box goes away, you lose all data. So people come up with backup solutions or use EBS, which is super expensive compared to S3. Another benefit is it makes super fast autoscaling possible. On top of this architecture, the Neon team built an autoscaling service that provisions compute nodes, Postgres nodes, in less than 500 milliseconds. Because you can do this, I think it changes the paradigm for both agents and humans. The agent is not just a marketing term here, but it changes the paradigm of databases. While there are one class of databases that are extremely latency sensitive and you simply cannot afford a tail latency spike at all, I think most applications, especially internal applications that a lot of enterprises build, it's okay every once in a while to have a latency spike of a few hundred milliseconds. You don't want it to be seconds, but 100 milliseconds is fine, because that's roughly human perception time, 300 milliseconds. If we can provision and acquire compute resources in hundreds of milliseconds, we can scale the database itself entirely down to zero without paying for it when there's no traffic to your service, which happens all the time. You might have a lot of traffic for an internal app in a specific time zone during 9 to 5, but past that hours it drops down to approximately zero for many companies. For many companies, maybe the lunch hours also drop down, or in some cases you have spikes at 9 AM because everybody gets into work and starts looking up some app, but then at 10 AM it goes down. It doesn't go down to zero, but it goes down. So they built this autoscaling architecture that makes the database itself adjust its resources dynamically based on the load and can also go all the way down to zero. This is super cool because I think it's probably the first time that you can have an OLTP database be so responsive and so elastic.