Thank you, Mr. Ion Stoica, for agreeing to participate in the interview with us. I read several things about where you started. You left Romania for the United States after you finished your studies. And you had as influences some of the best computer science teachers in Romania, who influenced you a lot. And your father influenced you in the area of mathematics. I would like to know what that meant for your career.
It meant a lot. I had as a teacher and I had a pretty close relationship with Mr. Nicolae Țăpuș, who still teaches now and was the head of the Faculty of Automation and Computers. And Ranadă Moisa and Denși were the ones who created the first computer that was compatible with the IBM PC personal computer. And after that, another great influence on me was Mrs. Irina Atanasiu, who helped with the software for the first IBM PC compatible computer. So this happened right before the revolution. They helped me and others who were getting computer science books that weren't in Romania and lending them. And that's where it started. It was a very important part of my education before I went to the United States. After that, I went to the United States in the fall of '93 for my doctorate. I ended up at a smaller college, Old Dominion University, after two or so years I transferred to Carnegie Mellon University in Pittsburgh, a larger university. I finished my doctorate in 2000 in computer networks. Back then, computer networks were very hot, so to speak. It basically coincided with the development of the internet during that period. During my doctorate, companies like Amazon, Google, Yahoo were created before that, and many others. So it was a period of ever. And after that, I ended up at Berkeley after spending a few months at MIT. Since then, I've been a professor at Berkeley practically since 2000, 2001, where I worked in various areas of computer research. After that, I did peer-to-peer networks. After that, I worked in big data, in processing large volumes of data, and more recently in artificial intelligence and systems and cloud computing.
At Databricks, you also had a student, Matei Zaharia, together with whom and based on whose idea you founded Spark and Databricks. Can you give us two or three very brief and understandable examples to tell him about how Databricks works?
Yes, so once again, as context, in 2006, 2007, there was another system before that was called Hadoop. This system did data processing in a very scalable manner. Why was this needed? Because after all the companies, the new companies of services that used the internet, starting with Google and others, they had a large volume of data from users. So they wanted to look at this data to understand how to improve their products, for example, for ads to present to each user the best advertisement, what products, how they had to improve their products so that users spend as much time as possible on those sites. So that was the problem. Google before developed an internal system called MapReduce for processing this data, very large volume of data. How do you process it to understand certain user trends and use these trends to develop better applications or improve applications? So that was the major problem. In a short time, almost all companies, even classic companies like General Electric, or car companies like Ford, General Motors, had this problem: they have as much data as possible and how to use this data. The technology at that time to process the data was databases like Oracle, but for that volume of data they were very expensive and could not be scaled. In databases, you have more restrictive types of operations. In the case of this, when we had user data, we needed much more sophisticated operations. Even then, people started trying to figure out what was then machine learning or artificial intelligence. What did that mean? If I look at the forum, if I have your data, I can predict what you like. So that's it, and then I can show you ads about products that I think you're going to buy. So already then there was a bit of machine learning and artificial intelligence. So now it was Hadoop, and in 2009 we started developing a series of systems to scale data processing, and generally scale software. But before that, there was another system, Mesos, to scale applications that run on multiple machines, and then there was Spark, Databricks, and all these systems. How they work is that instead of using a single computer, they use a lot of computers. They take the data and partition it, part on one machine, part on another machine, and so on. When I talk of several machines, there can be 100, 1000, 10000 machines. Each computer processes the data it has and then sends it so that the information can be agreed upon, the results, and then to have a common result. A very classic example is to say how many times a word appears in a data set, for example, in a book. You want to discover what is the most used word. So you can put a chapter on each machine, and each machine looks in the text and counts how many times each word appears. For example, the word 'is' can appear 200 times in one part, 150 times in another, and so on. After that, the results from the first processing for each computer say for each word how much it appears in that chapter. So once again, a machine processes a chapter, and after that these results are sent further, and another process, another computer, finally counts how many times a word appears in the whole book. For example, if 'is' appears 150 times in one chapter, 100 times in another, and 200 times in another, then it appears 450 times in the whole book. So that's what it comes down to. Spark made the processing of this data much faster than the systems before it, like Hadoop. It made the processing faster by keeping all this data in the computer's memory and not on the disk like it was before. There were different problems that had to be solved to do this while maintaining the reliability of the system. The problem when you have a lot of computers is not only how you process the distributed data and how you synthesize the final result, but also what happens when the machines crash. A machine crashes, it doesn't work anymore, you can't lose the data from there. So you have to have techniques to replicate, and in memory instead of on disk, the techniques are much more complicated. This was Spark. So the process could process data 10 times, even 100 times faster than previous systems.
I would like to know how you managed to identify problems in creating a business. How you managed to do this from the academic area. Did you see a problem that is universal for the current moment, which was then the future for you, a processing of a very large amount of data?
Yes. So at Berkeley, as in other universities in the United States, but especially Berkeley and Stanford, they are very close to Silicon Valley. In research, it is very integrated with industry. For example, in Berkeley, it has a long tradition where there are a lot of projects. A project is about five years, and each project is a group of professors and a group of students who work on solving certain problems. This tradition started a long time ago in Berkeley, probably since the 1970s or 80s. A lot of technologies have been developed: microprocessor technologies, technologies to save data efficiently on disk called RAID, even the technology that was at Google's base to use computers that were not supercomputers but computers that you could get at a normal store to make very powerful distributed systems. So it was in this framework. When I was there, basically in 2006, 2007, there was a new laboratory that I started here, and that was generally to focus on data processing and distributed computing systems. Within that laboratory, the laboratory is funded largely by industry, by companies like Google, Yahoo, Amazon, and so on. So we were working quite closely. Although it is funded by industry, the laboratory receives gifts or grants from the industry. Everything the lab develops is open source, and we don't even have patents because we want everything we develop to be able to be used by anyone. So this is the framework in which research happens. We have many meetings twice a year, basically retreats or conferences with those who sponsor these labs, and we talk about various problems that they have, what results we have obtained since the previous meeting. Many of the sponsors who were working on these problems were actually the authors of the previous system that I told you about. So I knew the problems from interactions with these people. There were two sources of problems for me. One within the lab that I was in, which had a lot of researchers from various areas: from systems, from computer networks, from machine learning, databases, and so on. When there are people in various fields, innovation happens more easily. There were certain students who wanted to process data much faster because there was a contest that Netflix had launched at the time, in which they gave some anonymized data and asked who can develop an algorithm that can predict what users who use Netflix want to watch. Those people came to us, who were in systems, and to Matei, that they needed something like this. I initially told them to develop a system that was then of computing, but it was too slow. Then Matei started to create a system, it was Spark, the very first version, to solve that problem. There was also another part that I had experience with at a previous company that was Conviva, which was in the video distribution area. They had this problem to process a lot of data in real time. This was another problem, and this was very difficult with the computing systems of the time. So I saw it practically from the lab and from the industry. I heard from the industry, from my own experience, this problem: to process the data as quickly as possible, as interactively as possible, in real time. So that was the genesis, the problem that Spark tried to solve.
A very close collaboration between academia and business is essential for identifying problems. I would like to ask you if you see anything that you have done, you and Matei and the other co-founders of Databricks, and if there is anything outside that has helped you and contributed to this success that was independent of you and why you succeeded.
Obviously. So we started when I started with Matei the development of Spark in 2009, around the second half of 2009. It became open source in 2010, and we started the company Databricks in 2013. An interesting thing is that at the beginning, we didn't think of starting a company. In fact, I told you it was open source, so anyone can use it for free. But what happened is that when it started to become more and more popular, bigger companies started to use it. One question that they had is: what happens when Matei and other students who worked on the project will graduate? Who will maintain the project? Who will continue the development? For example, in the case of the system that I told you about, which was the previous system, there were already two companies, Hortonworks and Cloudera, that were very successful at the time, and I knew the founders very well. We talked quite often with those people. Actually, at the beginning, we wanted to ask one of these companies to take over Spark and continue the development so that Spark as a system would be successful. But they didn't want to because they had internal projects that competed with Spark, the new systems. Then we decided it was a good time. I decided to create this company. It was a quite good time because I could take a sabbatical. In the United States, every five years you can take a sabbatical to do research or go into industry. Matei graduated, and the other founders from then were basically at a degree, or some even left school to create the company. We created this company in 2013. You asked me what were the factors. I wanted to say this because actually we created this company because we wanted the system that we created to be successful and have a future. For that, it has to be an entity that supports the continuous development of the project. The decisions that I made then, when I look back, I was CEO for the first almost three years of the company. After that, I went back to Berkeley, and Ali Ghodsi, who worked with me at Berkeley, became CEO until now and is still CEO, a very, very good CEO. One decision that we made in the first was to focus on cloud. In 2013, still most of the data was in the data centers that the companies owned, called on-prem. For example, the other companies that I told you about, Cloudera and Hortonworks, who were making products for processing data at scale, were focusing more on on-prem because there was a lot more data there, and big companies in particular used on-prem because of security and other reasons. Actually, the company at the time which was the biggest in this big data cloud market was actually wanted to start in the cloud, that's why it's called Cloud, but after that it pivoted to on-prem and they were very successful. We decided to go and focus only on cloud. It was a pretty hard decision because a lot of people were telling us, which was true at the time, that a lot of data is on on-prem and that's where the business is. The reason why we focused on cloud: there were a few reasons. One quite practical reason is developing software for cloud is easier. In cloud, if I develop a software that someone uses, I have quite a lot of control when to upgrade it, when to update it. For example, when you use Facebook or Google Gmail, you don't even know when the software is updated. All the users basically use the same version. On-prem, the user has to download software, which means that you as a software developer do not have control when the software is updated. You have to manage different versions of the software, which can be three years, four years older. When a security problem appears, you have to patch the old versions; it is much harder. On-prem, each individual data center, each company is a little different, so the heterogeneity is much greater. It is much harder to test. After that, the visibility: if it is in the cloud, I have visibility practically what is happening all the time. If a problem happens, I see it immediately, I can fix it. On-prem, I don't even know because I don't have access to the user's data center. So it is much, much more difficult. We thought that we want to create a new database processing application, processing large amounts of data for data scientists. That's what we wanted to do. So it was a decision. The reason why we made this decision, I told you, it was much easier to develop software. After that, I thought, like others, that cloud will be the future, it will grow much faster. Actually, it was curious that when you asked people who said to make the product work on-prem, when you asked them, they said because the on-prem market is much bigger. But when you asked which market grows faster, cloud or on-prem, practically everyone said cloud grows faster. So in a way, I said, OK, if this grows faster, we will bet on this, and we will bet on the fact that it will grow so fast that we will grow with the market. After that, we focused on data scientists. Data scientists are people who are specialists who develop for processing this data and to understand what is happening with the data or to understand what are the trends. Data scientist was a profession that was at the beginning then, but we came from academia and it was quite clear that we saw that practically all universities are starting to have a data scientist discipline, to create a data scientist discipline and to award degrees for data scientists. So once again, we bet on the fact that there will be more and more data scientists, although back then there weren't very many. So that's kind of how the bets that we made back then would come. Another bet that we made was that the success of open source is necessary for the success of the company. That's quite not very clear because if it's open source, everyone can use it, and then what's your advantage? You put a lot of resources in open source, everyone can use it, other people can contribute to open source, and so on. The stake there was that because we're going to do it on the cloud and because we're actually the people behind who created Spark, we're going to be able to make the best products based on Spark. So that was it. But that was once again very difficult because at the beginning, there were many other companies that had sales earnings from Spark that were bigger than ours, like the company that created Spark people. One reason for that was because many companies took over Spark, and we helped because we wanted Spark to win. We said Spark wins the market for data processing as open source software, and after that we make the best products for Spark. The reason why other companies had bigger sales is because they sold it on-prem, and on-prem the contracts were much bigger than in cloud. From the beginning, they were 100 times bigger, the contracts that I saw on-prem than in cloud, which in cloud were smaller companies at the beginning.
So basically, there were four key decisions that you made: setting up the company so that the students could continue the work of developing the project, choosing cloud, going to data scientists, and using open source, which you continued with the best specialists. I would now like to move a little towards the future of Databricks. You have raised in the last 12 years, I think 15, you raised 19 billion dollars of funding, and most recently you raised 12 billion in a round that took your funding to 62 billion. I would like to know how you want to use this amount and what is the future of Databricks. What do you want to happen with the product? Are you going to continue to develop it, or do you want to do other types of products or a spinoff?
At that time, actually, Databricks has a lot more products than when we started, obviously, and even a lot more systems that were open source. But Databricks, when we started, the vision and what we wanted, or the problem that we wanted to solve, is to make it as easy as possible for companies that have a lot of data to use this data to improve their business, to improve the products that they have, and so on. So that was the vision. Spark was to solve this problem at the beginning. But as users become more sophisticated, they need many other applications, other tools. Databricks has always tried to develop these applications to make it easier and more efficient so that the user can get to understand what's going on in that data and use that understanding to improve the products of that company, to launch other products, and so on. So as I told you, I started with Spark for data scientists. Then I started with for data engineers to use Spark to process the data before they pre-process the data before data scientists use it, because this data comes from users, has errors, you have to identify the errors, fix them in the data, and so on. There is data that is not there that should be because it was lost on the network, and so on. After that, people have data in a lot of systems. They have data in their databases, not only using what they see from the users who use the product, but there is also other data that they have from the sales people, for example Salesforce, but also a lot of other systems. In a way, when we started processing large amounts of data, big data, all the systems started as I told you were different from databases. Databases are very fast but also expensive at the same time and not as scalable. On the other hand, Hadoop and Spark and these systems scaled a lot, but neither was Spark for certain processing as fast as databases. After that, the guarantees for Spark data were not as good as a database. What's happened in the last few years is basically a unification. Through the products that Databricks has continued to do, it is basically to unify and make it practically no longer practical to be able to process together the data from databases and with the data that was being collected from the user, and even in the way that you use them, practically to be for the user to still be able to see, even though the data is scattered in a lot of systems, a user sees like a single system which is very fast and which guarantees the properties of the data. So that unification of databases with these systems for processing large amounts of data, big data systems, and after that the next thing is artificial intelligence. Data is fundamentally important to artificial intelligence. As you know now, a big problem with models no longer grow in capabilities so big, so much in the last six months because there is no more data. They use all the data that has been produced. All the companies that are Databricks customers use, I want to use artificial intelligence once again to know their business better, as he said, to improve their products. But artificial intelligence is very dependent on data. Databricks products already have access to all the data that a client has. So it is very natural for clients to use Databricks for artificial intelligence because Databricks already has access and manages all the data that they have.
I would like to move on a little bit to another topic. I would like to ask you about the IPO, but I think that this is a topic that is talked about more. The CEO even said, Ali Ghodsi, that the IPO could take place this year, but because of unfavorable market conditions, what would be the favorable market conditions?
So there are conditions not only external but also internal. The company has certain plans that it has new products. In a way, as a private company, an unlisted private company, you have more flexibility how to raise funds and how to spend. So if you want to bet on another major product, you can do it easier. There are certain conditions that make it easier to do this as a private company than listed. So I wanted to say that in fact it is not only that the market is favorable, but also what are the internal plans. So this is a favorable one. Once again, you want it to be ideal to be at the beginning of a period of prosperity, not that you want when you go out on the market after that to continue for quite a long time as a period in which the economy is doing well, that then stocks will grow. The most problematic is when you go out when a period after that an economic crisis begins. But we will see how for now still the geopolitical conditions are quite problematic at least, and we will see what happens. The prospects are equally dark.
