Amr Awadallah - Making AI Work in Enterprise - 12/25/2024

It's going well. How are you doing?

Yeah, or something like that. Just you know how it is, kids get bugs, all that fun stuff. So yeah, what have you been up to?

That's awesome. Well, thanks for joining. I think the last time we hung out was a couple months ago in Australia. It was fun with you for a bit.

I think it's far away from everybody. Yeah, if you live there it's not far, but outside of Australia and New Zealand, it's a really long trip to get there.

Yes, that's correct.

It's up and coming. So of course Silicon Valley is special, even within the US Silicon Valley is special. So I would put Egypt on par with some European countries like Spain, Italy, and so on in terms of the technological scene. But yeah, they still have a long way to go.

Yes, so the primary reason is family, I'll be honest. My mom is back there and it's very important to go back and see her. But then also, whenever I'm there I try to either do something on the AI side or something on the entrepreneurship side. So actually this time there is a conference called the Youth Summit. I'm giving an entrepreneurial talk at that conference.

Yeah, I think it's important to give back. I mean, if you look at the Valley, I think what made Silicon Valley special to be honest is there's so many people who give back. They don't just make their money by having their company succeed and then go buy an island in Hawaii and retire. No, they actually stay engaged and giving back. And you know, one of the most famous stories was Steve Jobs when he called up, I can't remember, was it HP, one of the HP guys, and Steve Jobs was only 13 years old and they still answered the phone and they gave him an internship and they helped him build the first computer. So I think that culture of giving back is very, very important in my opinion.

Yeah, in tech, I mean, it goes back to my childhood essentially. So ever since I was young, I always had the curiosity of, like many of us engineers, the curiosity of figuring out how things work. It's like I would break open everything my dad buys. Like he would bring a new TV, a new washing machine, whatever, I would disassemble it and then reassemble it. And that was the beginning of me getting excited about that. And then my dad, I have to give him credit, when I was very young, I had an Atari, and mainly for games as you know, but he got this nice attachment that I was able to put on top of it that had a keyboard and it had BASIC programming language. And that was the beginning of me now, okay, we can get this thing to do our thing. And I think that's what put me on that trajectory. Though I would have to say also that my dad always told me when I was young, when you grow up you're going to be a professor like me. And my dad was a professor of Economics at Cairo University. So that was my goal in life was to be a professor. I had no inclinations of being an entrepreneur. And then I give credit Stanford for flipping me. Like Stanford, from day one, they give you talks, they invite people from industry, they teach you how to do a business plan, a pitch, everything. And that flipped me. So Stanford flipped me.

Yeah, yeah.

What kind of professor were you planning on becoming?

Okay, yeah. Computer Engineering. And I was on the track to a tenured professor at Cairo University and I had to give that up. Like I told them I'm not coming back and I gave up my scholarship to be a professor.

It's Stanford. Like, again, first they teach you how to create a business, right? So it demystifies it. Like, if you don't know how to make a business, then it feels hard, it feels like impossible, it feels like insurmountable. So the number one thing they do is they give you lots of education insights about how do you do a business plan, how do you do a pitch, how do you do the incorporation, cap table, how equity and dilution and how that works. All of these things that makes it much more tractable. And also on the business side, they teach you marketing, positioning statements, branding, how to go to market, sales. So you start seeing, wow, that actually is not hard, I can do this. And when you combine that with them also inviting a lot of entrepreneurs from the industry to come speak on campus, you get to see, you get to like the founder of HP or the founder of Intel or the founder of Google, and they're very humble people. They're all like, they don't have wings or halos on their head, they're just normal people like me and you. They just have the grit and the motivation to make something happen. So once you see that, or once I personally saw that, that made it for me a lot more clear why I wanted to do that. And another thing also I'll highlight is that being a professor is great, and I think one of the cool things of being a professor is that you teach a lot of students and that's kind of your legacy, is how many people you taught. But creating a company is a different type of legacy. If you can create a company that succeeds over time, then that can survive you and your students and your students' students and your kids' kids and kids, right? It can keep on going for a very long time. And that was something important for me. Like I want to make a mark on the world where I know one day I'm gonna die, I know my kids are gonna die, and I want to leave something that will keep going. So that was another reason why this entrepreneurship thing is more attractive.

Yeah, so actually I dropped out for my first one from Stanford. I was doing my PhD, I was three years in, and I dropped out and I created this company called VivaSmart. That was back in 1999. And within one year, we were very fortunate that Yahoo acquired us. So I ended up at Yahoo because of that. The company was about five, six people. We got acquired for about $9 million, which back then was a big deal. Today $9 million acquisition is like peanuts in Silicon Valley. And we hadn't really raised that much funding. It was a great, great first experience. And I ended up at Yahoo because of that. And then Yahoo was nice enough to let me go back and finish the PhD actually. So I did finish the PhD while I was working at Yahoo.

So yes, you're kind of like walking through the timeline step by step. So excellent job on the homework there. But yes, at Yahoo, at the beginning I was working on integrating my company VivaSmart into Yahoo Shopping. So VivaSmart, the startup that got acquired, what we had was a crawler that can crawl the web and extract prices and specs and product images and so on. And you need that to be able to build an online shopping experience. And back then that was hard stuff. Today that's easy, you can do that in a blink of an eye. But back then that was the beginning of the internet, so back then in 1999. So after I finished integrating that into Yahoo Shopping, which took about three years, I started getting a lot more excited by data science and business intelligence and how can we leverage data not just as a passive participant, meaning a dashboard that you look at, but as an active participant that changes the behavior of the products as a function of the data. And I, among a number of other great people, started the A/B testing kind of movement within Yahoo to do A/B testing across all of the features: Yahoo Search, Yahoo News, Yahoo Sports, Yahoo Mail, etc., etc., to make the products easier to use in real time.

Yeah, that was the beginning of the, I would say, the AI renaissance. Like the AI renaissance really started taking off, not GenAI, which has happened within the last five years, but AI as in leveraging data and machine intelligence to optimize products in real time. It really started happening around the early 2000s. And then the problem that we hit at Yahoo, which led to my second company Cloudera being formed, is if you're doing that at the individual basis, I call it the segment of one. Right? So the old way of doing targeting and changing behavior of products was by doing something called segmentation, where you try to classify people: this is a male in the age range of 40 to 50, then they must like this, right? That's kind of the old way of doing things. And that allowed us to crunch down the data and make it fit in a small table, which was easy. But we found out that no, if you do personalization, where you create a segment for every single user, for every single user you know exactly what they care about. It's not about their age, it's not about their gender, it's about what they care about, what they're reading, what they're clicking on, what they're spending time on. Then you're now able to optimize their experience a lot better. But that also meant now an explosion in the amount of data that needs to be processed, right? Because you cannot aggregate it anymore. You cannot do aggregates, just sum up the numbers by the different segments. No, you have to create a segment for every single user. And we're talking about hundreds of millions of users at that scale. And that triggered for me a lot of problems actually in my organization. My team's name was called PIE, Product Intelligence Engineering, or PIE for short. And one of the biggest problems we had, we had this very big query we called the Monster Query, was the name of that query. And the Monster Query essentially, what it was trying to do, it's a very simple query. It's doing a join between all of the impressions, everything you have seen on a page, every single element, not the page itself, and then all the clicks that a given user had made on the clickstream on that page. And you want to join both of these and then you want to dedupe them on a per-cookie basis, meaning on a per-user basis. It's a very simple query, like a select distinct join, like a very simple query. But it's running on massive amounts of data. So we were using Oracle back then in our team, and Oracle just started to like fume. Like it just couldn't finish. It actually reached a point where it just couldn't finish that job. The Monster Query would not finish on Oracle. We tried a number of other vendors that actually did work, but they were very expensive. And I'm like, no way I'm gonna pay that much money to do this query. And then I was lucky that Doug Cutting, Doug Cutting was also at Yahoo, working with the search team on building Hadoop, right? A new way to crawl the web scalably based on the papers from Google of MapReduce and GFS, the Google File System. And then one of the key engineers on that system, in that team, was called Alan Gates. And he said, you know what, that Monster Query, I can do it for you in Hadoop and it will run in minutes. I'm like, no way, you're kidding me, it's not going to happen. And he did that and it ran in minutes. Like the query that wouldn't finish even in days on Oracle, ran in minutes inside of the Hadoop environment. So that was for me the light bulb. Like, okay, there's a big business opportunity here. If that solved my problem, I'm sure many other people start seeing that problem as well. And that's when I left Yahoo actually to found Cloudera. So I was the GM of Cloud.

Yeah, I mean, I see the world as a continuous puzzle that we need to be solving to make better. That's how I see the world, right? And because it's a puzzle, that means you have to keep discovering the different ways that you can make it work better. And that's where the curiosity comes from, always curious about how things work. Like that's something I'm always actively looking at. But when it comes to companies, entrepreneurship, that tends to be driven from, I'm seeing that, like I know that's a problem, I have it personally, but I know that other people are going to have it. And that's why I should go build something to make the problem easier to solve, right? And that really was the genesis of Cloudera. Between myself, one of my co-founders is Jeff Hammerbacher, he was doing the same thing at Facebook and same problems. And we looked at the data we're collecting back then, I'm like, that's the tip of the iceberg. Like we're just at the beginning of what really can be unlocked, which became Big Data later on. And Cloudera played a role in that name, Big Data. Jeff Hammerbacher, he gets credit for the name Data Scientist as well. But to answer your question, if I boil it back, it's the curiosity driven from an innate desire to make the world around us, the place around me, a better place, which hopefully will be around all of us. And if you really want to do that, then you need to understand how things work. So if I were to like peel the layers of the onion, that would be the essence at the heart of it.

It's really hard to know what to take a bet on, like I'll be honest with you. But experiencing the problem yourself, it gives you extra confidence that this is something worth taking a bet on. So if you have not experienced the problem and you're guessing, for example, actually I don't want to make examples because I don't want to finger any other companies, but you're guessing it's going to be important, right? This is going to be important. I think it is, not because I experienced the problem, I know it's a big problem, no, it's because I have a gut feeling this is going to be useful in the future. That is very, very risky, right? To go with something like that. But if you have experienced it yourself, know this is really a problem, I know this problem exists, then the question now is not about whether the problem exists, the variable is this a one-man problem, one-person problem only, or is it going to be a problem that many other people have, right? So the next layer of the onion there is now you need to talk with your peers. In my case, for example, for Cloudera, Jeff Hammerbacher, wow, we actually have the same problem. So that now is evidence that goes beyond us, and that gives you really good confidence that this is something that will be worthwhile. So it's a combination of curiosity again, understanding why problems exist, the root cause for them, and how to solve them, coupled with I just was lucky at the right place, the right time, where I could experience these problems because before others could experience them. Even with Vectara, my new company, which we can talk about in a second, I saw that at Google. Same thing, I saw the beginnings of why these large language models, as they hallucinate, we cannot really depend on them in business. We're going to need something that fixes that. I experienced that firsthand with Google. So again, it was coming from that. So my advice to entrepreneurs always is, if you can do something that is being driven out of a problem that you're very acutely aware of, then your chances of success is going to be way higher.

Thank you for the question. So I'm going to give you two definitions for that. One of them is the business layman definition and the other one is the more technical engineering definition. So the business layman definition is we help businesses build AI agents and AI assistants that are reliable, meaning they don't hallucinate, they don't make up stuff, they have high quality results. Second, they are secure, they don't leak information, resilient to prompt attacks, etc., etc. And third, are explainable and observable. They explain how they came up with their recommendations or their conclusions, which is a must in regulated industries. So in industries like finance, insurance, health, you cannot just have the AI tell you this is the right mortgage, go give it to them. No, you have to explain why did you recommend this mortgage rate for that person. So it has to be explainable. So we have these three things. This is exactly what Vectara does. We give you AI assistants and AI agents that are reliable, that are accurate, that are secure, and explainable. They explain how they made their conclusions. Now the more technical term, it's very simple, is RAG as a service. So we have a retrieval augmented generation pipeline where we control the pipeline beginning to end. We have our models for doing the embeddings, for doing the ranking. We have our own vector database that we built ourselves as part of the platform. The generation, the hallucination detection, and so on. And we merge, we put all these pieces together to create a real-time pipeline that satisfies these needs while allowing you to change data in real time. So the data can be updated at any time and the impact of it will be available in the output right away. Because as many people will tell you, if they start using RAG, the biggest problem with RAG, number one problem by far, is accuracy. Is making sure that the RAG system is giving you an accurate answer. And that problem boils down to two things that you need to fix. The first one is hallucination, so you need to fix hallucination or have something that measures that. And the second one is data quality. As we like to say, garbage in, garbage out. In the same sense that we have ETL for BI systems, we need to have ETL also for AI systems. Because if you're fetching in the wrong data into the RAG pipeline, guess what, your RAG system is going to give you the wrong answer. There's a very funny story on that one. Sorry for the digression. But Google, a few months ago, they had launched their own RAG system for the Google web search. And in Google web search, somebody was searching for, I'm trying to make a pizza but the cheese keeps sliding off the pizza. Like every time I cook the pizza, the cheese slides off the bread. And how can I make the cheese stay on the bread, right? And then the AI responded back is put glue in the middle. Just put glue between the cheese and the bread. And many people at the beginning thought that that was a hallucination, right? And it turned out no, it was not a hallucination. That was a data quality issue. It turned out there was a Reddit post where somebody was asking about this. They said I'm trying to cook a pizza and my cheese keeps sliding off. And then a human sarcastically replied back saying put glue in the middle. And he was being sarcastic. But people liked that response so much, it got so many thumbs up because of how funny it is, right? So the AI, as it was crawling that, the AI is looking, thumb, thumb, thumb, thumb, thumb, that means that must be the right answer. The right answer is you put glue in the middle. So it turned out to be a data quality issue. So but to tie this back, that's why it's very important if you have a RAG pipeline that's working on your data, you need to have mechanisms first for observing the output so you know when a problem happens, because problems will happen, I assure you, and very quickly how to fix that, right? To detect, oh, this wrong answer came because that document was an old document, it's the old version, let's remove it or let's include a timestamp in our metadata so that we're relying on the newer versions versus the older versions, etc., etc. So you need to be able to tie the loop. But the bottom line is you need to solve hallucinations and you need to solve data quality. If you don't solve these, your RAG pipeline will be a failure.

Yeah, it does.

Yeah, yeah.

Well, I can't, I mean, Google is a... I mean, confidentiality and so on. But there's some public information that came out, so I can talk about it from that perspective. Yeah, so Google actually had the precursor of ChatGPT two years before ChatGPT came out. Like they had it inside, it was called Meena. M-E-E-N-A. And I guess Meena is driven from meaning or something. And we were able to chat with that chatbot all the time. That's why there was also some engineer at Google that said I think the AI is sentient and was trying to hire a lawyer for it. So that was the same AI that all of us were playing with. And you can tell when you're working with it, you can tell that it keeps making up stuff. Like it keeps making up things that can be, even though it's very eloquent and it says things perfectly and converses perfectly, every now and then it just makes some really, really wrong factual conclusions. And that clearly made it visible that, okay, if we want to leverage that in business for consumers and doing chatbots like Character AI, whatever, it's okay if you hallucinate every now and then, even though it can actually be dangerous sometimes in that context too. But in a business context, yeah, within a business context, if you're doing a mortgage, if you're doing a financial audit, if you're doing a legal draft, no, you cannot have these hallucinations. So that was the trigger.

So I had, my co-founders actually started before me and I joined them about a year after into their journey. So they started in 2019 and then I joined in 2020.