Okay, this is fun. Hi everybody. Thanks for coming to this summit. Ali, should we kick off with telling some stories about Databricks early days or Berkeley earlier?
Yeah, whatever. Okay, so you and I go back to 2009, 2010. You got to Berkeley and there was an email going out. I think it started with Jan or Scott Shenker sending this email to the Mesos group. It's like, here's this visitor. He's going to be joining the team. And Matei and Ben and I were like, who is this Ali guy? And we were very insecure and who was going to be first author still. So, we were like, he's And then he comes in and he's jacked. We were like, okay. I hope we don't have to arm wrestle for this first author. So, that was some of my formative memories of you being injected into our research project. And then you went on to like do dominant resource fairness DRF paper and just become like a mentor and then basically one of my co-advisors. So, that's what I remember. What do you remember?
Yeah, I remember a lot of fun stories. You know, it was amazing years. 2009, 10, 11, 12 was kind of, you know, UC Berkeley was at the frontier of all these new algorithms for the new computer. The new computer was the data center. And we were doing all this amazing research. And I remember like, you know, there was a lot of skepticism around, you know, AI, machine learning at the time we called it, and data centers and all of that. So, yeah, it was best years for me, you know, most formative years. My favorite memory is probably when we had started Databricks. And actually it was this big debate: should we run the company out of Berkeley or should we run it out of San Francisco? And like half of the founders wanted to move the company to San Francisco and half of us wanted to keep it here in Berkeley. And I remember we brought in the Do you remember this?
I do. Yeah, I think I was on the Berkeley side.
Yes, I was on the Berkeley side, too. And then you I think called the mayor of Berkeley.
Yeah, first I think we met with the chancellor. And then he said you should talk to the mayor and he introduced us and so we with the mayor and brought him into our office, which was not a very nice office because Berkeley doesn't have great office space.
And he brought a lot of people with him. I think it was like us as a researcher I was like, oh, these are all politicians. And they came in and we pitched them on algorithms, advanced algorithms for scheduling and I think the mayor just fell asleep and just dozed out through the whole talk and then woke up at the end and came up to me or you and shook our hands, I think, yeah. And then he said, I hope that one day I will have a data brick in every home.
No, it hasn't happened. I mean, maybe Nvidia's most recent rack, you know, it could be where we go someday.
So those days when we were here at Berkeley, they were the most formative for me, too, and there was something magic happening in the air or the water. What do you think it was and do you think that it's still here? Has it changed since then?
Yeah, I mean, I live in Berkeley still. Yeah, I think actually the city is special and I think the university is special. Actually before I came to Berkeley, I did a bunch of postdocs. I actually lived like more or less in nine different countries and visited nine universities and lived like in different places of the world and I think Berkeley and UC Berkeley specifically is very special and the thing that's special about UC Berkeley is that there's this kind of attitude that you can change anything, you can question anything. I mean, academia has that in general, but UC Berkeley specially has that. And I think I actually It was kind of a little bit of a culture shock for me when I came to the research group because the kind of things we were working on I was like, oh, that's not research. We're supposed to do that. And we're No, we're doing it. It's cool. It will have impact. And they're like, well, that doesn't make any sense. This is like a weird you're cheating here or something. It's like, no, we can do anything. Like this is we're just here to change the world. So, I think that that's the kind of spirit of Berkeley. And I think that's university has it, but the city has it, and its history, and you know. And I think that's what makes it special. And sometimes it goes a little bit extreme. Like, you know, there's research that has super high impact, and then there's some research that, you know, is further out. Like, I remember there was like research on delay-tolerant networks. And it was like, oh, but you know, obviously the internet is broken if we want to do interplanetary communication. And we have to redesign a new internet for interplanetary communication, because eventually we're going to colonize all of the universe. So, that was one of the research projects here in networking. So, you know, I think it's like the aperture is really wide, and it helps people to kind of broaden that.
Yeah, I know that in the early days, we were operating systems and distributed systems research group, and we took deep inspiration from BSD, and I remember we went through a couple naming iterations with what became Mesos in the end. And one of the contenders I put forward was the Berkeley cloud distribution, BCD, or Berkeley cluster distribution, named as inspired by BSD, Berkeley system.
BDAS, Berkeley distribution of something.
No, no, no, no. There was BDAS, which was badass. Yeah, badass. That was like the official nomenclature. You have to pronounce it badass, the badass stack. But no, this was what I put forward for the name of Mesos. I said we should call it BCD.
And I don't know you guys I guess you might not remember. Matei, I'm pretty sure Matei and Ben both said, no, no, no, no. Like, who are we to like be so lofty as to name ourselves like only one edit distance from BSD. And I was like, well, we're going to like build the more impactful operating system. That's who we are. So, yeah. I did not win that vote. I think Mesos was pretty good.
So, that's the Berkeley spirit. Yeah. Question norms. That's continued at Databricks. We most recently announced or we announce many things every week, but Omnigant is a big open source framework trying to play at the harness or that is putting forward a new kind of router for harnesses is how to think of one way to think of it. Also, we talk about open eval which you can talk more about later. So, how has that played forward from your perspective as the leader of Databricks for so long now? What we carried from Berkeley in terms of open innovation and that maybe sky's the limit mentality still carrying into Databricks strategy.
Yeah, I think open for us has been always super important. So, we always want to open source standards, open source interfaces because it creates communities and those communities become very powerful. Actually, Databricks would not have existed as a company if it wasn't for the Spark ecosystem because the Spark ecosystem just was way bigger than anything any of us would have done. And it's so it's both good for the community, it's good for the world, but it's also good business idea as well as Databricks has proven out. So, we just continue doing that. You know, and we do it in things that we think are the most useful and will have the biggest impact. So, a bunch of years ago we did MLflow. And MLflow actually people don't know, but it's 40 million downloads still a month every month. So, you know, billions of downloads a year. And what was that? It was like in the ML era, machine learning era. So, say 10 years ago, people were obsessed by the best models, but it was just very hard for machine learning people to iterate on the different models. So, they needed sort of a git version control kind of a way to be able to iterate and keep track of all the different experiments. So, that was MLflow. Omnigant which you just mentioned recently, that came from you know, there's just so many harnesses out there. And unlike large language models, large language models, people switch between them quickly. They're stochastic. Every time you ask them a question, they give you a different answer. And we can see it in the Databricks data because we serve all the different models. We serve actually proprietary models and open source models. So, we have like data on all of that. And we see that people quickly switch between them. So, when there's a new proprietary model or a new open source model, people quickly within days are moving to it. We've never seen anything like it. You know, anything else in software is sticky. You stick with it, you know, and there's a decay and people slowly churn and migrate, but it takes a while. You don't just instantaneously everybody just flip-flop. With LLMs, that is the case. But what we saw with harnesses was that people are building harnesses on top of them and the harnesses are kind of like old software in that they lock you in and you get used to the interface and the memory and key sort of shortcuts and all of that. So, we wanted to create something that lets you actually switch between these and have it be completely open. And so, the idea was a meta harness. So, harness of harnesses. So, it will work with your Claude Code, it will work with your Cursor, it will work with any actually of them OpenCode, Pi, you name it. So, that way you can get the multiplexing, you get the community. And the project that we just launched it this summer. And, you know, I think just in the first couple of weeks we had a thousand forks on GitHub. So, it's really taken off. And over a hundred of people contributed immediately within weeks. So, you know, we're very excited and I think we're going to continue putting out open source projects like this. I think open sources actually have a huge boon in the next bunch of years thanks to vibe coding or AI because before it was like, okay, it takes huge amount of effort for a Linus Torvalds to build a whole operating system and compete with, say, Windows. And at some point someone has to pay his salary, so they get hired by companies and so on. But today, since the cost of writing software is going down, I actually think it's going to be much more palatable. We're going to see many, many more open source projects out there that become standards.
I have a theory. It's not really here nor there, but it feels like we're going to get a GitHub version where instead of pull requests containing code, they're going to just contain the prompts. And then the PR reviewers will decide will like run those prompts. And I think we're like moving towards that in things like OpenClaw and probably Spark in that the work on the committers is now much more about deciding how much of the code was written by a human or how deeply like how many hours did a human spend on this pull request. I don't know. Gets me thinking about all the big different the big ways that the world you and I grew up in and we started Databricks in is like fundamentally going to be different going forward. Any reflections on kind of the paradigm shifts as they play out for Databricks or I don't know, the future of software dev and live coding?
Yeah, I do think we're going more and more towards, you know, if you can specify what the program should do and you kind of have the testing foundation and harnesses, then the implementation actually doesn't matter. The AI can figure out the fastest, the best implementation of it. And you might be rewriting it many, many times, but it really requires you to have this really sort of airtight way of specifying the code. And I think it's moving that way.
Yeah, that reminds me of evals. So, we at Databricks recently announced this effort where we internally scraped a bunch of our own coding issues and tasks. We like built a dataset, very large dataset. How many over 1,000?
