Today I want to delve into two main points: one is why Databricks is so amazing, and the other is why you are so amazing—who are their best people and where do they put them? Of course, you might not see this publicly. Their number one priority is to win compute and storage. Just a relatively successful startup versus an incredibly successful startup—what bridges the gap? One major difference between Databricks and incredibly successful startups is that they are willing to expand. I think for any startup, a very important reason for its early growth is...
Okay, welcome, Reynold!
Hey, class representative, I've received many messages from other people, and I've also been following your videos online. Wow, really!
I made it! Hahaha, thank you so much!
Let me introduce Reynold and your company to the viewers. You are the co-founder of Databricks. You've probably all heard of Databricks; it's a pre-IPO company that hasn't gone public yet. Your last funding round was over a year ago, when the valuation was 43 billion. And it's still in a period of rapid growth, especially recently, the momentum has been very strong. So, I won't guess what the valuation will be when you go public, but in short, it's a relatively rare company right now. The scale itself is huge, the development speed is very fast, and the ceiling is very high. Because you are doing data in the AI era, there is actually an even bigger ceiling. In my opinion, do you think it's appropriate for me to summarize Databricks like this?
It's quite appropriate. It's just that I haven't talked about what we specifically do yet.
Yes, yes. Actually, what you specifically do is a core question for me today, but we'll talk about it later. And let me introduce you. You are Reynold. You graduated from UC Berkeley in 2013 with a PhD in Computer Science. At that time, I guess you did a lot of Spark-related work while you were in school. You were also part of the core Spark team, including being the release manager of Spark 2.0, and for a long time you were the top contributor, right? It seems that recently you've been too busy.
Last year or this year, I haven't really written much code for open-source Spark since 2017, but anyway, I probably wrote a lot before, and I was always at the top, the number one position. Around the beginning of this year or the end of last year, other new people gradually surpassed me. I think this is a good thing. It proves that Spark itself is still a very live and... a project needs fresh blood, right? It's impossible to say, 'This is from 10 years ago.'
Okay, and after you co-founded Databricks, you've always been at the forefront of technology, as you said. You've also created and managed many important business lines. On the Databricks website, I saw DataFrames, and in your interview, I saw DatabricksSQL. How to put it... no, it's Spark's DataFrames, and DatabricksSQL is all about popularizing this technology and making it more usable. That's a very important point. Oh, is that appropriate?
For me, I was mainly responsible for data-related systems, specifically product development and R&D. And of course, ultimately, we haven't talked about what Databricks actually does, right? But the ultimate goal is to design and create better Databricks products. Because the use cases for data AI are so complex, it's extremely difficult to actually put them into production. How can we simplify the complexity of data AI workloads for various enterprises as much as possible?
That statement itself is very informative. I really want to go deep... We're digressing here because what we're seeing now is that everyone wants to use AI, but many people are like this: you hire someone to do AI, but then you realize you need to hire a data engineer, and even the basic data isn't clean, and your computation is too expensive. So, or your data simply can't realize its value, and you can't use AI effectively. It's definitely difficult, but I think I'll put that issue aside for now. I want to talk about another problem with AI: AI is too easy to demo, especially after the LLM program came out. For AI, it's probably the easiest thing to succeed with a demo, but the hardest to go live. The biggest reason for this is that we recently interviewed Glean, who said that many companies initially optimize time to demo, which is definitely wrong because it's too easy to get a demo. But if you start with the difficult things, you'll find that they're also very difficult. Even things like security, privacy, and infrastructure are very difficult to implement, and they change very quickly, making them especially challenging. Okay, so based on my understanding of you, and because I've been following your content for a long time... Including your two interviews with Professor Xu, and your keynote speech below, I was truly impressed. You explained the technology and business very clearly and specifically. You never talk about empty talk; you always start from very concrete technical details. So, before this interview, I specifically spent an hour talking with Professor Xu and discussed some questions that we felt were very worthy of in-depth exploration. We felt there were so many questions worth exploring. Your preparation was very thorough, haha.
Yes, because this opportunity is quite rare, so...
Today I want to delve into two main points: one is why Databricks is so amazing, and the other is why you are so amazing, haha. Actually, Databricks is a very new generation company. In 2013, among the cohort companies we hear about now, the most successful ones are probably Databricks, Snowflake, and OpenAI, right? And this achievement is actually very rare. By 'very rare,' I mean that most people hadn't even heard of this company at the time, let alone had such a high valuation. And there are definitely many factors behind such great success. Of course, luck and everything else are very important. Today I want to hear what factors you think are involved and how much weight each factor carries. But you can answer this question now, or you can wait. I'll ask you another question first, and then I'll come back to this one.
Okay, then I'll ask you another question first.
I've already asked a lot of questions. After I finish asking this question, you can start your presentation. The question is, what exactly does Databricks do? It has two layers. One is, what does Databricks do? I've actually asked a lot about this, including myself, who works in data. I asked my colleagues, data engineers, and infrastructure engineers at my company. Their understanding might be managing Spark. And now, Databricks has acquired Tabular, so it's probably closely related to data lakes. And of course, if you go to the Databricks website, you'll see it's about data plus AI. But I feel like I don't fully understand what kind of story, or what kind of value proposition, or what kind of problem it solves. I understand the big picture, but I don't understand the smaller details.
This issue is actually quite complex because what we do is constantly changing. This company has experienced very rapid growth in its product line. However, if you're looking at it from a high-level perspective, you can think of us as a data plus AI platform for enterprises. There are several keywords here: data, AI, platform, and enterprise. Let me put it the other way around. As for 'enterprise,' we're not a consumer-facing (B2C) company; we're a business-to-business (B2B) company. Our clients include some SMEs, but most are larger enterprises, like Fortune 2000. The 'platform' refers to the fact that we don't sell solutions directly; we sell a platform. Our clients can develop or implement their desired solutions on this platform, whether they do it themselves or seek help from third-party services, consulting firms like Accenture (the largest), or smaller boutique consulting firms. The data and AI platform has two layers. First, every enterprise generates massive amounts of data. How to manage and process this data is a significant challenge for every enterprise. Previously, you might say, 'I have a data warehouse to handle my BI workload, a streaming system for real-time streaming, a dedicated machine learning system, and so on.' Basically, any existing enterprise—those that have been around for a decade or two, many even hundreds—will have numerous data-related systems. Then, with the advent of AI, even more systems like Vector Database and Pinecone have been added, making things incredibly complex. One reason for the complexity of data and AI is the sheer number of different systems. Each system has its own maintenance, usage, and security features. Managing all this data within a large enterprise is extremely complex, let alone using it effectively. So, we aim to provide a comprehensive solution that centralizes all your data management in one place, eliminating concerns about processing and maintaining complex systems. This might seem cumbersome, but beyond data management, simply managing data itself isn't valuable. You need data to generate value, such as reports, past events, and future predictions – this is BI and AI. Ultimately, everything serves these purposes. This broad market can be segmented into many different applications. Essentially, we address the biggest problems our clients face, regardless of their specific data or AI-related needs, aiming to gradually solve them within this system. Initially, our focus was on data science. For example, a data scientist (like myself, with a data science background) who wants to interactively analyze large amounts of data. Our first product was a data science notebook, and our second was a data engineering pipeline. How do you prepare your data and create an operationalized workflow, rather than a one-off process? This is a data engineering question. We have a data engineering product, and many people think it's about managing Spark. But the real issue isn't managing Spark; it's about a product for data scientists, a product for data engineers, a product for machine learning engineers, a product for data warehousing, a product for BI, and various other product lines. Gradually, we've expanded and perfected our product line. To reiterate, for all data and AI-related issues in enterprises, we hope to provide a platform solution that gradually simplifies the complexity of data and AI-related systems.
Regarding your second question, there are two competitions: one with the cloud provider and one with Excel. What does this mean? It means that the process of transforming a lot of data into a solution—assuming it's not true big data, or rather, many things can be done in Excel, especially scalable Excel. This is the second point. On the other end, all cloud providers are telling the same story: you use our cloud, we're a platform that provides various hosting solutions. How do you view this competition?
The difference is significant. Excel is arguably the most successful data platform in history—bar none, in terms of user numbers, it's absolutely number one. We don't really have a competitive relationship with Excel; it's more of a partnership. Many people might think that these things are competitive, regardless of the context. They might think that two companies, because their products seem similar at a high level, should have a strong competitive relationship. But for me, my definition of competition is very limited. It's about whether customers consider both our product and the other when deciding to spend money. If they don't, then it's not a competitive product. Excel almost never appears in this process. It's more about whether I'll have Excel, and also Databricks. Sometimes I download data from Databricks in Excel format, such as CSC or Excel. This is additional processing when the data volume isn't too large. Of course, many of our clients are sensitive and don't allow data to be exported to Excel. This is because anything done on the platform can be audited, but once it's moved off the platform, the audit can't know where the data goes. So, platforms like GDPR and DSR require that if a client in the EU requests the deletion of their data, you need to completely remove that data from your company within 30 days. If you have many different Excel spreadsheet demos on many employees' computers, you simply can't delete that data. So, we don't really compete with Excel. In fact, we've spent a lot of time figuring out how to make Databricks work better with Excel, within the client's permission. For example, a feature we were working on the other day was that we wanted a new BI tool where clients could directly drag and drop Excel files into it. Then we can instantly visualize it.
Your second aspect is that the three major cloud providers (CSPs) are indeed our biggest competitors. And from this, I heard from Databricks for the first time today that cloud providers are competitors. Previously, Databricks people were very careful in saying that cloud providers are both partners and competitors.
This is very obvious. Because we give cloud providers a lot of love for us. Databricks now launches 25 million VMs daily across the three cloud providers – that's 25 million. This is a very, very large volume. We may have internal data for one cloud provider, but I don't have internal data for other cloud providers either. But I personally guess that we might be the company with the largest workload among cloud providers. A big reason for this is that many of our workloads are data engineering. Data engineering includes very computationally heavy tasks. For example, the simplest step of converting JSON from JSON format to PartK format requires burning many CPU cycles. So cloud providers really, really love us. They also hope to get closer to us because we can bring them benefits. For CSPs, what they care about most is compute and storage.
Let me challenge this a little. If we look at Microsoft's early story, one point is that when PCs and software appeared simultaneously, Microsoft dominated the operating system. This ultimately led to all PC manufacturers becoming commodities in the entire value chain. All the value was captured by Microsoft and the remaining software, leaving PC manufacturers with very little profit. Cloud providers, I guess they see this today too. If they only care about compute and storage, they will eventually become commodities. Compute and storage have no added value, and their profit margins will be very low, right? Because you can do these things yourself.
Yes, that's right. In the very long term, but I estimate that in the next 10 years, even by 2024... Everyone thinks the cloud is the future, but pre-premium IT spending is still much higher than cloud computing. So, for cloud providers, it's simple: their number one priority is to win compute and storage. And because the scale is so large, there are many things you can see, such as who their best people are, who they put in charge, and where they put their best people. Of course, you might not see this publicly. It's about how they manage the cloud, how much time their CEO or SVP spends focusing on each different department, and how much interaction exists between departments. You can see this.
I agree with what you said. I think in the future, generally speaking, everything will be gradually, in the full extent of time, commoditized, and it starts from the bottom up. Of course, there's a time factor involved, because optimizing for the future too early is meaningless. But today, in the areas of cooperation and competition I just mentioned, have you felt the competitive pressure? Or do you think they're really starting to compete with Databricks, or are they still fighting for market share?
