The Value of Datatabricks' Lakeflow, Lakebase, and More - w/⁨ @Databricks Cofounder, Reynold Xin

The vision of LakeFlow is really to simplify data engineering. If you think about data, everything starts with data engineering. There's typically this persona, data engineer, that tries to get all the data into a central repository, which in our case would be the lakehouse. If you use a proprietary data warehouse, it could be the data warehouse. Then they need to transform this data through maybe what we call medallion architecture, from bronze to silver to gold. This is an extremely complicated, super costly process. LakeFlow aims to make that simple for data engineers. It has Flow Connect to help you get data in from a variety of different systems. There's workflows and there's also a Spark declarative pipeline to help you with the transformation. We're also working on more to make data engineering not just simpler for data engineers but also visible for non-data engineers.

Yeah, exactly. So they don't have to be blocked on having...

So then people don't have to block always on you.

Yeah, Lakebase is our take on OLTP databases, which is the other side of the giant data estate that we haven't touched in the past. We want to make databases far easier to use and make them much easier to manage, and also make them great for this new persona that we call agents, in addition to humans. We want to actually make databases integrate much better with the lakehouse. When we talk to a lot of customers, they talk about how they have their OLTP database but it's in a different silo from their data infrastructure for analytics. But we always end up finding that the OLTP database is actually one of the most important, maybe the most important, data sources for the data lake. How do we actually combine them? We think we have everything it takes to actually solve this problem for everybody. That's why we're excited about Lakebase.

Yeah, ultimately, I think we have all these technical personas, data engineering teams, data science teams, but if I want to democratize data, I want to actually enable every individual, whether they consider themselves an analyst or not, to be able to get insights out of data. One of the most important tools, or toolkit, every enterprise has is their BI tool. AIBI is our take on BI. It's actually free because we actually think we can monetize enough just based on the level below that we don't have to worry about charging you with user-based pricing or some other seat-based pricing. It's completely free and it gives you not just what traditional dashboards can do. It's an independent dashboard product. It's gone from pretty crappy in the past.