Lost Tapes Elad Gil on the Eve of the AI Boom

Thanks so much for including me.

Yeah, to your point, I think the bar for what AI or machine learning actually means keeps going up. So in the 80s, it was things like, oh, maybe we can have a plane that can land itself or things like that. And obviously those sorts of things have happened since then. And then I think the first real AI-first company that really worked at scale to some extent is Google, where artificial intelligence kind of pervaded everything that it did early on in terms of thinking about how to rank search results or how to target ads. And so I actually started off between 2004 and 2007 at Google, and one of the things I worked on was the ads targeting machinery, which at the time was sort of the world's biggest semantic machine learning systems alongside search. And I think in the early 2010s, there was a big wave of discoveries around things like AlexNet, CNNs, RNNs, deep learning, GANs, all these different innovations which fundamentally really focused on things like machine vision and voice. And the interesting thing is that a lot of this sort of, you could almost call it like second wave of AI, if Google was the first wave, largely translated into incumbent value. In other words, companies that already existed captured all the value of these models. So for example, Netflix launched a really strong recommendation engine, Amazon launched Echo or Alexa, so you had this voice face, suddenly Facebook had the news feed. But the reality is very few startups that were AI-first actually turned into anything interesting over the last 10 years. And so this first wave of really compelling AI models translated to enormous incumbent value and very little startup value, which I think is a really interesting question in terms of why is that. And now what's happening because of Transformers and these large language models, we're seeing a whole slew of new startups, and some of them are actually getting really interesting traction. And so that will be platforms like OpenAI, maybe tooling like Hugging Face, or specific application areas like Jasper, copy.ai, you know, GitHub has done with Copilot, or Replit has been doing in terms of its AI assistant or AI mode, I think are really cool examples where you're starting to see people really scale using it in ways that you didn't see in the first wave of AI-first. And again, incumbents worked, but for some reason startups didn't in the prior wave.

You know, I think there's an interesting question of whether people were choosing the wrong application areas or whether the technology simply wasn't far enough along. And I'm guessing it's a little bit of both. I think some aspects of the technology have really been advancing in leaps and bounds, and you can clearly see that because of Transformers, which again were only invented about four or five years ago and only really got started being used two, three years ago, right? And so it's a very recent technology innovation. I think some of the things that these models can do are really dramatically superior to what was done in the past, and the things in the past were already quite impressive. I mean, you know, you talk to Alexa or you look at some of the results that you get through Google or other things, it's actually these things work very well in terms of large data sets. But for startups, finally it seems like something, there's this opening, particularly if you start building like a tool or a series of workflows around a core use case. And so I think one of the areas where these things will be really useful and performant from a startup perspective is applications where you have white-collar workers who do highly repetitive tasks and don't have a strong workflow tool overall. And so that would be things like copy.ai for marketing copy, you know, it may be obviously it's the coding side for various different tools, and it may be all sorts of other types of work. Like you could imagine a sort of co-pilot or companion or helper for tools around, you know, physician practices, legal work, summarization of documents, data entry, like there's all sorts of automation you can imagine. And then in parallel, you could imagine all sorts of really interesting applications on the consumer side. You know, do you have a better interface to search if it's a chatbot, or can you create a virtual companion? So I think there's lots of really interesting stuff that's still coming.

Yeah, so 'Attention Is All You Need' came out in 2017, and there were eight authors on the paper, and six of them have gone on to start companies. And of those companies, four are in the AI space or natural language space, and then one actually started off that way and turned into the crypto protocol Near. And so you've, and I'd say four of the five companies started, I believe, in the last 24 months. And so it's this really interesting wave where the people who first came up with and developed this technology, and it was all Google authors at the time, have since left to start some really exciting companies in this area. And I think Google almost had like a Xerox PARC style moment where, you know, just like Steve Jobs went and visited Xerox PARC and he saw the GUI and he saw the mouse and saw all the early things that then turned into core parts of what Apple launched later, and Xerox PARC made nothing off of it, you know, they didn't commercialize the technology. Google came up with this amazing breakthrough and then they kind of sat on it a little bit. The people inside were trying to work on it and launch things, but I think a lot of them got blocked by Google and sort of a degree of AI safetyism and a degree of fear of brand impact and things like that. And OpenAI, I think, saw this technology and they were really smart about saying this is the future, and then they went and they built GPT-1 based off of it. And so I think it's kind of a parallel moment to what happened with Xerox PARC where all the innovation happened in one place but other people ended up commercializing it. And maybe the big difference here is instead of all the researchers from Xerox PARC leaving and doing things, my sense is a lot of them kind of stayed there or joined Apple. I think in this case, a lot of the authors off of this paper decided, you know what, there's a great opportunity, we're building all these really interesting things inside of Google that could be useful outside, and so let's go and start building companies that are really useful here. And they're not doing direct analogs, it's more just they understand the technology is so fundamental that they can build new things, you know, that weren't available or possible before. So it's a really seminal kind of group of people, and it reminds me almost a little bit of like the Traitorous Eight or something, I can't remember the number, the people who all left Shockley Semiconductor to start Fairchild, and then from Fairchild there was people who started Intel and a variety of other things, Sequoia Capital came out of there, a bunch of really seminal places did.

Sure. So with a lot of these Transformer models or large language models, which are a subset, a lot of the emphasis has been on how large these models are and how many billions of parameters do they scale to. And so a lot of the early thinking in the industry is the only way to win is to have these massive models, which means you need a massive amount of compute, which means you need to spend a lot of money on it just for the compute, right? You don't need that many people, but you do need a lot of hardware to train the models. And I think one of the really exciting things that happened more on the image gen side is what Stable Diffusion did or Stability AI, where Emad, the founder, tweeted that they spent $600,000 to train their model, which is nothing, right, in comparison to the hundreds of millions of dollars that we're bringing into some of these companies, or billions in some cases now. And then they went ahead and open-sourced it, and once you open-source something, it's really easy to modify it. And so that means people are very rapidly modifying things like the safety filters. And there's this whole field of, to call it liberal arts or something applied to technology, where there's a strong emphasis on not wanting to launch things because they could be offensive in different ways. And offensive is very broadly defined in terms of anything that offends a specific subgroup of people who are focused on safety versus everybody in the world. There's obviously some things that are really offensive that most people would really want to avoid, but there's lots of other things that may just depend on your political orientation or your specific point of view. And there's a very specific point of view that's represented in all these companies in terms of what safety actually means. And so they made it very easy to strip out that safety layer, which means that suddenly you had an open-source model that anybody could use that could be modified in all sorts of interesting ways, so it could take any type of prompt. And that's created this explosion online of different people trying different things simultaneously. And the real breakthrough that I'm waiting for now on the open-source side is something that can compete or reproduce the level of fidelity that the GPT models have on the language side, because these are all image-based things that have been open-sourced now. And there's some open-source models that are pretty cool on the language side, and EleutherAI and a few others are working on these things, Hugging Face has a bunch of these models, but fundamentally nothing is as performant as GPT-3 and or GPT-3.5, and then of course GPT-4 is supposedly coming soon. So the question is, can you build that in an open-source way, and what's the bottleneck? Is it data quality and cleansing, is it some model structure, is it something else? And so I think once that happens, you know, it really changes things, I think, in terms of accessibility of this technology to everyone.

Yeah, absolutely. I think he points out, and I'm probably going to get this wrong, but he comes from a very specific religious background himself, and he's like, well, why isn't it reflective of my specific ethical framework versus a very specific framework that all these people who went to the same Ivy League institution seem to have? And so it seems like a very narrow sliver of quote-unquote ethics is represented in a lot of these things, and it'll be interesting to see how these things end up globalizing over time and incorporating multiple cultures.

Yeah, absolutely. I think, you know, OpenAI really feels like the juggernaut now in this sort of area. They recently launched a really interesting API called Whisper as well, which deals with voice and text transcription. And so I think they've really been building a lot of really interesting things from GPT to DALL-E to whatever. You know, I think Microsoft provides capital and other aspects of a relationship to them, and obviously some of those things can be found elsewhere, particularly over the last few years where we had such a large capital bubble, right, where so much money was flowing into tech. And I think their choice of Microsoft as a long-term stable partner makes a lot of sense. I mean, Facebook did the same thing, right, in a prior way when they were really thinking of themselves as battling Twitter and Google and sort of everybody else, and they raised a lot of money from Microsoft, and I think that created a stable partner for a while there as well. And so, you know, I think there's a pretty good precedent in terms of Microsoft being a good partner now to a variety of companies that are scaling in a way that didn't exist 20 years ago when they were kind of viewed as the evil empire that would quash anything that they viewed as competitive. You know, and so it feels they've taken a really smart, ecosystem-embracing approach now.

I think it's going to really depend on the type of company you have and the area you're focused on. So if you're an API-first company like an OpenAI, you could imagine that you're charging off of some API usage fee. And then the other thing that platforms tend to do, although it may not happen here, is they tend to forward-integrate into the apps that are working the best on their platform. And I'm not saying OpenAI has any plans to do anything like that, I'm just saying that's a generic pattern that happens often but not always. And so you look at Microsoft and its OS, it forward-integrated into the things that were used most on Microsoft PCs, which was Excel, Word, PowerPoint, and Access, and they basically bought or cloned multiple companies on their platform that were succeeding there and then subsumed them eventually. And Microsoft apparently was doing really shady things like they'd change their OS to break any app that wasn't theirs that was competing with their apps, right? So that's one approach. The other approach is you truly have something that is open and you're just charging on the API side. And to some extent you could argue AWS is more like that, it's kind of like a services layer that anybody can use. Obviously with things like S3 and others they have in some cases kind of integrated a little bit in compute, but in other cases they've left things reasonably open, just build on top of us, you know, they never built Dropbox or whatever it is even if they had a cost advantage there. It's just possible that OpenAI ends up more like AWS, I actually have no idea, you know, I don't know their inner workings or plans. If you're a vertical application like a Jasper or a copy.ai or GitHub Copilot, you can obviously just charge for your service like any other SaaS company. And then I think a lot of the value again in this industry is going to get captured by incumbents. I think a good analogy is mobile, where, you know, when the first real smartphones launched like the iPhone and Android, there were a lot of really interesting mobile-first companies like Instagram and WhatsApp and Uber. And then at the time people were saying things like we're building the mobile CRM company, and you're like, well, mobile CRM is Salesforce on your iPhone, you know, it's not a new standalone startup. And I think the same thing will happen here. And so I think there'll be a lot of startups that will either be a technology hammer looking for a nail, or there'll just be things where the incumbent will just win because it's a pre-existing distribution and it kind of doesn't matter if your product is better in that case.

I don't know exactly what the context was in terms of that statement. I think, you know, Google and a few other companies were perceived as having a really strong position around AI and machine learning-based technologies. And to some extent the Transformer world was one that Google effectively lost, right? They gave it away in some sense. And that doesn't mean they can't come back from it, they're an incredibly talented company with great people, but they created an opening for OpenAI and Cohere and A21 and all these other companies to emerge using technology that was invented in Google to sort of impact the world in a big way. And that's going to be true of applications as well. There's lots of things that Google could be doing with this. I think fundamentally though, if you do assume that at some point there will be really strong open-source models of what Stable Diffusion or Stability AI is doing in the image generation world for everything in this world, and you know, OpenAI just open-sourced Whisper for example, then you really change the nature of the business models. And there are lots of open-source business models, and often what you end up doing is you end up with a hosted version or a SaaS tool or some equivalent. You know, a lot of the software stack that Amazon eventually ended up providing to people was something that consisted of a lot of open-source components that everybody had to stitch together, and they'd rather just go on Amazon, right? You could use a Linux server for free effectively in terms of the software side, but then you had to do a bunch of stuff to make sure it was running all the time, right? And the first versions of AWS just simplified really basic aspects of their platform. So you could imagine something similar happening in the AI world or the machine learning world where eventually maybe a bunch of the stuff is open-source and then there's other services layered on top of it or tooling that you pay for, other things like that. But so far that isn't the case, right? It really seems like, you know, early in a technology curve there's often a technology advantage to being centralized. It's not always true, right? Crypto is a counter-example to that in some cases. But even the most quote-unquote decentralized aspects of crypto end up being centralized if you look at them more closely, right? Like if you look at mining pools for Bitcoin, it aggregates I think up into five to ten players, and you look at the number of code contributions that are accepted from who, it's a small number of developers, and you look at the number of exchanges and it's pretty centralized. And so I'm not saying Bitcoin is centralized, I'm just saying many of these things end up being a little bit more centralized than you think simply because there's scale effects to every business. And on the AI side, I think it's an open question in terms of open versus closed source, but I think Stable Diffusion gives real hope that there will be an open-source version of all this stuff.

Yeah, I can't say about Stability specifically simply because they have thought about it much more deeply than I ever will, and so I think everything I'll say will probably be wrong. But I think in general there's going to be two different aspects to it. One is what are the areas that I want to impact from a platform perspective most, is it image gen, is it language models, is it both? And then secondly, how do I create long-term sustainability? Because I may be able to keep raising money indefinitely up to a point, and then once that point is reached, what do I do? And so I start thinking about the SaaS services that I can wrap around the open-source models, or are there some things that are proprietary that I somehow hold back? But again, that doesn't seem to be too much the direction philosophically. And so maybe it becomes more just, you know, what are the SaaS pieces that I build around this, or other aspects of open-source models that tend to work these days.