Introducing Lakehouse//RT and Reyden — Reynold Xin, Co–founder and Chief Architect

All right. Thank you, Ali. I'm super excited coming on stage today to talk to you about this new effort. And honestly, it's probably the largest single innovation we've done since our introduction of Lakehouse. And as Ali already mentioned, with the Lakehouse, we were able to actually unify data science, data engineering, as well as data warehousing. And our data warehousing business has taken off exponentially since its introduction five years ago. It was recognized by Gartner and Forrester both as a leader in data warehousing. I think over 60% of the Fortune 500 uses the Lakehouse for warehousing. And I joke that we actually scan exabytes of workload depending on which time zone you are before you have breakfast.

And over the course of the last few years, we worked fairly hard to improve performance. At every single Data and AI Summit, I think me or Shawn somebody else would come up on stage and introduce to you some massive performance improvements we've done. But the reality is the warehouses or the Lakehouses hit a wall at about a second. And what do I mean by hitting a wall? It doesn't mean you can never get a response from the Lakehouse or warehouse in less than a second for a query. It just means if you have a very stringent workload that you want to have very tight SLAs in the range of milliseconds or hundreds of milliseconds, it would be very, very difficult to accomplish that because you might actually get latency spikes.

And as a result, I think for many organizations that actually require workloads that have very stringent SLAs, they started setting up a separate serving stack. They would copy their data instead of the data warehouse or the Lakehouse and copy a portion of it that's required for serving into a separate stack. And this caused a lot of problems. It's actually a data pipeline to maintain for every data set. These serving stacks tend to perform super well on very simple queries, but doesn't actually work well for more general workloads. It's a governance nightmare. You now have to worry about, 'Hey, did I forget to actually secure my data set in this specific platform, but I actually did in my underlying warehouse?'

So, if the whole theme here is unification, what if you never had to move your data? What if you never had to copy it? And we asked ourselves that question a couple years ago. And now, it turned out it's actually fairly fundamental. It's very, very difficult to tackle this problem. As a matter of fact, Michael Stonebraker, who's a Turing Award winner, and some of you might know him as the original creator of Postgres, wrote a whole academic paper about it called 'One Size Fits All: An Idea Whose Time Has Come and Gone.' And in this paper, Stonebraker actually argued it's very, very difficult to engineer a single database engine that's capable of running a wide variety of workloads. And ultimately, it's because the increase in complexity makes it very, very difficult to actually engineer the system.

And the fundamental reason for that is, if you look at how database engineering were done in the last four decades, or even five decades, every database engineering team started working approximately this way, Databricks included. We'd go targets of, 'Hey, here's the new workload we might want to study and then understand and then actually build a system for it.' And the way to do it is to study all the latest and the coolest academic papers about the specific type of workloads. And some of the papers actually go back forty, fifty years. Some of the papers were published in the last few years. You try to study them, understand, 'Hey, in my real life apply this.' And you spend a lot of time implementing it, and at some point you try to roll it out to test how well it works with real-world workloads. And inevitably, this happens. The new technique you implement, it could be a new algorithm, could be a new data structure, it works remarkably well for a subset of the queries, and it works terribly for a subset of the queries. So, the database ends up, 'Hey, we're doing better here and actually regressing.' And many of the algorithms and data structures actually work really, really well for very, very low latency to guarantee maybe a millisecond response time. Tend to actually backfire when you have larger data sets and vice versa. So, this is one of the biggest problems.

So, we two years ago decided, 'Can we break the status quo? How can we challenge it and do the best we can?' And now we actually went back to the drawing board with an amazing engineering team. We flipped the way how database engineering were done. Instead of starting with ideas and techniques, we went back to the workloads. Luckily on Databricks we have collected over zetabytes of data where we scanned traces from all of this data, which is in the range of quadrillion traces. I have to learn how to pronounce that word. And all coming from trillions of queries that were executed. And based on all of these traces, we were able to actually build a machine learning model that gives you very high fidelity estimation of what might happen in practice given a specific new technique. And based on this machine learning model, we can do two things. The first is we can select the right algorithm to dispatch very quickly at runtime. But it's not an LM because LMs have too high latency. We can actually pick the right algorithm to dispatch at runtime. But even more importantly, we can decide and predict what algorithms to even implement before it hits production. Because there are millions of algorithms and data structures out there. It would be virtually impossible to actually find them all. Knowing what to implement is one of the biggest advantage one can get.

Right. So, the resulting engine of that is Raiden, which you all know is a reference to Mortal Kombat. And the resulting engine is actually pretty remarkable. And you might have actually used some of the other serving stacks out there. And if you have, you run into many of the issues you're talking about. For example, a system can't run queries longer than five seconds. Almost all of them require you to copy the data into a very specific format. It's very engine specific. They can't run complex joins and all of that. But Raiden can actually do it all, right? And the way Raiden will show up is going to come in the form of a new SQL warehouse. And this is kind of abstract. So, let me give you maybe a quick demo for you to help understand what Raiden actually does.

All right, can I get the demo on stage? Okay. So, here I have a very simple setup. Honestly, it's a very simple query. I have the New York taxi dataset. You don't really need to understand what it does. But for those of you that actually know what the dataset is, it's fairly small. And this runs... First of all, I ran this query in Lakehouse today. It finished in about a second. And on the right-hand side, I have the Raiden setup, which we call Raiden RT. And let's try to run this. Okay. I don't know if you can see this number.

Let's try to run it one more time. Yep. Did we hardcode it 1.007? Oh, no, we didn't. But that's just one query. All right? What's the fun of running just one query? Here I want to show you a demo of what will happen in practice when you start hammering the system. And we've built a simulator simulating real dashboard application workloads hitting against the Raiden engine. And I'm going to show you starting with just one active session here. So, let's imagine this is an actual dashboard running. We're just going to load this dashboard. It loads fairly quickly, all right? It's probably what we expect. But what happens if you scale up to ten, one hundred, or why don't we try one thousand? So, now we're going to simulate 1,000 active agents hitting the same engine at the same time. Each of the agents, by the way, is generating more than one query. The dashboard actually takes about eight queries for it to load. And they will all hit exactly the same moment. Let's see what happens. Done. All right. So, we executed 8,000 queries at 6,000 queries per second with a tail latency of 37 milliseconds. This is incredible. None of existing systems can do this. That's a very simple demo.

But when I told you, 'Hey, none of the existing systems can do that,' let's put it into the perspective of existing systems. So, here I want to show you a benchmark measuring latency versus throughput. Throughput here means queries per second. Right? So, we have the Y axis, which is the P90, 90% tail latency, and the X axis, the number of queries executing. And we're running a very simple query, TPC-H query six, which is a canonical query of scanning a little bit of data, running some aggregation. And let's see what happens when we test the different systems out there. So, for a competitive vendor, using their latest warehouse, we were able to actually push the warehouse. It started fairly low latency, but as you start pushing up about 160 queries per second, latency started climbing up and hitting a wall. So, the same vendor actually announced a fairly innovative offering last year called Interactive Warehouse. It's so interactive, they can't actually read the same data with the general warehouse. The only way to actually have it work is to copy the data into a separate warehouse. And Interactive Warehouse does do much better than their latest generation general warehouse. You're able to push like 300 queries per second before the latency starts spiking up. And we also tested through an open source query engine that's known for being able to run very simple queries very, very fast with the yellow color. And it's actually running much better on this specific workload compared with the Gen2 and the Interactive Warehouse. It's able to push all the way to close to actually I think 15,000 queries per second. That's pretty good, right? But then it just started crashing right after. It doesn't actually time out, it just crashes. So, how does Raiden do? Right? How does Raiden do? On the same workload, Raiden could actually run all the way to 12,000 queries per second while keeping the tail latency still below a second.

There's one other benchmark I want to show you, which is what if we run all of TPC-H, which is a standard data warehousing benchmark. As you can see on the screen, when the dataset is small, actually all systems run reasonably well. I mean, it's like a few seconds running 22 queries. Some are a little bit faster than others, but they all run pretty well. But as you scale up, for example, 100 GB, you start seeing the yellow system's run time started spiking. And this is actually I'm trying to point out the very specific part of my talk earlier, which is a specialized serving engine doesn't actually work very well on the more complex joins, which will show up in the full TPC-H queries. And of course, when we scaled up to actually bigger amount of data, both systems actually started crashing. The Interactive Warehouse started crashing because it cannot run any queries longer than five seconds. And the yellow system couldn't actually complete because it started running out of memory. And Raiden was actually able to complete all queries at all scale factors.

You might have benchmark fatigue at this point, so many benchmarks. Who cares about synthetic benchmarks? One of the things that we're really excited about in the process of developing Photon is the design partners we've been working with. Saw which I've shown their logo on the screen here. I want to share with you some results from these design partners. The first one is Ambers. Ambers is a global leading data and AI platform for the energy sector and they tested Photon with 11 of their most representative production queries. And what they found compared with the existing serving stack is that with Photon, the same extremely low latency queries remain sub-100 milliseconds, whereas the longer running queries can now actually dramatically shrink their run time and across the board on average they're actually getting a 16 times speedup. So, Paul from Ambers was super excited to see that they're able to now execute queries in tens of milliseconds, but also shrink the run time of the longest running queries by almost 100x. And Meta also tested Photon and they found very similar things. They were able to have their typical queries come back in tens of milliseconds and the most important thing is you don't need a separate system on the side just for your serving. So, what Photon does is it can give us the best low latency possible at extremely high concurrency. It can actually run real world workloads really, really well and it can also handle the complex CDL EDW workloads super well. And today very excited to announce the first product offering powered by the Photon engine. It's called the Lakehouse RT. A Lakehouse RT is a new warehouse that's showing up starting with read-only workloads. It will support millisecond performance and massive concurrency, but the most important thing is directly on the lake governed by Unity Catalog on the same Delta or Iceberg data.

So, what can you do differently with Lakehouse RT where RT stands for real time? Challenge your conventional wisdom. If you have a separate serving stack, test it out, and you might be able to collapse the two platforms into one. And that would give you millisecond performance, massive concurrency directly against your data lake in open formats. And beta starting today. Try to talk to your account team to get early access. And there's also two talks at this conference starting soon today on Raiden and Lakehouse RT that you can learn more about. And that's it.