Conversation with Databricks CEO Ali Ghodsi on Lakewatch, Genie Code, IPO, and What’s Next

I'm excited. Yeah.

Yeah, honestly, our customers dragged us into this. Like they started doing it themselves. Some of the biggest banks and some of the largest organizations on the planet were starting to build what we refer to here as open security lakehouse. They were already doing it. And you know, we were more kind of working with them trying to understand why are you doing it there? And they were telling us that the existing SIMs can't store those large organizations can't even store longitudinal data like run out of space. And then secondly, they were complaining about the costs. It's very expensive to get all the data in there, you know, because they charge on how much you ingest. So they hated that. And one of them actually ripped out one of the sort of main vendors. And so that's when we started studying sort of closer to what are they doing? And then about two years ago, we were starting to see sort of the AI attackers get more and more sophisticated and LLMs being used. So that's when we kind of said, hey, this is the time. There's a secular shift coming. There's a new trend with AI and our customers are already starting to store their data in these lakehouses. Why not combine the two? So why not have an open security lakehouse and then make it really, really easy to unleash the agents on this? So really the attackers are using agents and they're looking at any of the data and everything is automated. So let's stop fighting the agents with manual humans that are sitting in a SOC and ingesting data and just looking at parts of the data. Let's just look at all the data and also use agents. So let's fight agents with agents.

Those are great. We use Claude Code at Databricks by the way. We're big fans. But this is really Genie Code is really focused on how do we actually focus on data work. So if you want agents that actually are not just writing code for you because that Claude Code can do really well for you. But how do you make an agent that actually really understands the data. So the code is not that important. The data is the important thing. The code is a means towards that data. That's your end. So and you know we were running benchmarks and we published some of these results and we tried many many of these different ones. We were only getting like about 30% accuracy on our benchmarks that we were running on some of the frontier agentic systems that were out there. So with Genie Code we've been able to actually already climb up to 70%. Actually I think it's close to 78%. And of course, you know, benchmarks, your mileage might vary and blah blah blah blah and so on and we're going to come up with new benchmarks and we're going to climb them and so on. But we just think that it's having all the context on the data is really important and in Databricks since you already have all your data there, there are things we can do that are unique to us. And that's two things. One is you ask a question from Genie Code or from Genie for that matter. We might give you an answer right now, but we can continue in the background after we've given you an answer next day, this night, next week, next, you know, six months to look at all the data that you have and start looking and say, 'Oh, you know, the answer I gave you two months ago is actually I have an even better answer now,' you know, and we can keep improving it because that all the data is there, you know, so that's different from an agent that you know just has access to certain data or has to use MCP servers to get that data and it's you know, slow. So that way we can keep improving the results. So that's one important aspect. The second important aspect is that we have Unity Catalog. Unity Catalog really lets you audit and really control access of the agents on your sensitive data, you know. So you really want that to be fine grained because more than ever now, more than ever because you know these agents go off and they're you know useless if you don't give them access and if you just give them access they can do all kinds of stuff. So modulating that is really really important. So that's where Unity Catalog comes in and part of Unity AI. So for us those two aspects was what led us to double down on Genie Code and Genie. Why the relaunch or the splash as you refer to it? Because you know it's like really improvement in you know it's a step-wise improvement that's significantly different from let's say before when we move to the agentic loop essentially and really using research techniques like DSPy and others that we've published before we can really improve the quality of the data work.

I think they're wrong actually. I think that the chat bots are going to kind of as we see them today going to die, you know, and we actually see it with MCP apps and so on. I think that watch any sci-fi movie. How do you interact with the AI? Not just blob of text that you have to type and read. That's not on any movie. Future will look like Minority Report. Okay. You know, it's going to be interactive.

You know, yeah, exactly. And you know, a picture says more than a thousand words. That saying is not going to become incorrect in a bunch of years. That saying will remain true. We still if you go see what happens in boardrooms and in meetings, people look at charts, bar plots, they look at visualizations and they interact with them. And so that's the best way to interact. So I actually think that BI is kind of the future maybe of sort of the chat interface. I think the chat interface in many ways is very frustrating because you're just looking at a prompt and I don't know what to ask now. Oh, you could ask it. Let me show you a cool demo. Okay, sure. But I didn't know. I'm just dumbfounded. I'm staring at the screen. I don't know what to type here. Well, there's like three options you can click on. Sure. But, you know, I think having a much more richer experience where you're interacting with the visualization so that you can actually understand the data. That's the future. Now, is that exactly BI? No, it's not BI. That's why we called it AI BI. AI and BI is going to blend and you know today or in the past BI human did all the thinking. You look at the dashboard and you do the thinking and say ah something is going on there. That's me thinking. I want to understand more about what's going on there. So you ask and then you go you know you click on it you do a drill down you know you do a drill in and so on. In the future what's going to happen is that you know a lot of the thinking will be done by the AI for you in AI BI it will tell you hey you want to pay attention to this by the way you know if you click on it so it's like pointing you towards that it opens up it shows you the report it creates a report on the fly on what are the things you should be looking at and it's visual so it's not a pre-planned dashboard of the drill in or drill down but rather it's like it's creating it for you on the fly that's actually by the way Genie can already do that for you today can create your dashboard on the fly and now you can actually interact with it. So that's great. And then dashboards today also have another limitation. So I'm talking about both limitations of the chatbot and of the both of the AI and the BI and how they're going to blend. Another limitation we have in dashboards today is that well they're great but the logic is drill downs cross filtering some reporting and that's it. It's all you can do. If you want something more custom well that's a much more complicated story.