Aravind Srinivas : How a Simple AI Search Engine Disrupted Social Media and Went Viral Overnight!

So you know, I got together with my co-founders Dennis and Johnny, and all we wanted to do was build cool products with LLMs. It was a time when it wasn't clear where the value would be created — is it in the model or is it in the product? But one thing was clear: these generative models have transcended from just being research projects to actual user-facing applications. GitHub Copilot was being used by a lot of people, and I was using it myself and I saw a lot of people around me using it. Andrej Karpathy was using it, people were paying for it. So this was a moment unlike any other moment before, where people were having AI companies where they would just keep collecting a lot of data but then it would be a small part of something bigger. But for the first time, AI itself was the thing.

Yeah. So GitHub Copilot, for people who don't know, it assists you in programming — it generates code for you. I mean, you can just call it a fancy autocomplete, it's fine, except it actually worked at a deeper level than before. And one property I wanted for a company was it has to be AI-complete. This was something I took from Larry Page, which is: you want to identify a problem where if you worked on it, you would benefit from the advances made in AI. The product would get better, and because the product gets better, more people use it, and therefore that helps you to create more data for the AI to get better, and that makes the product better — that creates the flywheel. It's not easy to have this property. Most companies don't have this property; that's why they're all struggling to identify where they can use AI. It should be obvious where you should be able to use AI. And there are two products that I feel truly nail this: one is Google Search, where any improvement in AI — semantic understanding, natural language processing — improves the product, and more data makes the embeddings better. Or self-driving cars, where more and more people drive, it's better, more data for you, and that makes the models better, the vision systems better, the behavior cloning better.

Voo, Tesla doesn't matter. Anything that's doing the explicit collection of data. Yeah, and I always wanted my start also to be of this nature. But you know, it wasn't designed to work on consumer search itself. We started off with searching over — the first idea I pitched to the first investor who decided to fund this, Elad Gil, was: 'Hey, you know, would love to disrupt Google, but I don't know how. But one thing I've been thinking is if people stop typing into the search bar and instead just ask about whatever they see visually through a glass.' I always liked the Google Glass version, it was pretty cool. And he said, 'Hey look, focus. You're not going to be able to do this without a lot of money. Identify a wedge right now and create something, and then you can work towards the grand vision.' Which was very good advice. And that's when we decided, okay, how would it look like if we disrupted or created search experiences over things you couldn't search before? And we said, okay, tables, relational databases — you couldn't search over them before, but now you can, because you can have a model that looks at your question, translates it to some SQL query, runs it against the database — you keep scraping it so that the database is up to date — and you execute the query, pull up the records, and give you the answer.

Correct. So you can't ask natural language questions of a table; you have to come up with complicated SQL. Like, 'most recent tweets that were liked by both Elon Musk and Jeff Bezos.' You couldn't ask these questions before because you needed an AI to understand this at a semantic level, convert that into a structured query language, execute it against a database, pull up the records, and render it. But it was suddenly possible with advances — Copilot, code language models were good. And so we decided we would identify this insight and go again — scrape a lot of data, put it into tables, and ask questions by generating SQL queries. The reason we picked SQL was because we felt like the output entropy is lower, it's templatized, there's only a few set of SELECT statements, COUNT, all these things. And that way you don't have as much entropy as in generic Python code. But that insight turned out to be wrong.

Correct. Separate — they're not general-purpose; they train on GitHub and some language. So it's almost like you should consider it was like programming with computers that had very little RAM. So a lot of hard coding — my co-founders and I would just write a lot of templates ourselves: 'this query, this is a SQL; this query, this is a SQL.' We would learn SQL ourselves. This is also why we built this generic question-answering bot, because we didn't know SQL that well ourselves. And then we would do RAG — given the query, we would pull up templates that were similar-looking, and the system would build a dynamic few-shot prompt and write a new query, execute it against the database. And many things would still go wrong — sometimes SQL would be erroneous, it would catch errors, it would do retries. So we built all this into a good search experience over Twitter, which was created with academic accounts before Elon took over. Back then Twitter would allow you to create academic API accounts, and we would create lots of them — generating phone numbers, writing research proposals with GPT. I would call my projects things like 'BrinRank' and create all these fake academic accounts, collect a lot of tweets. Twitter is a gigantic social graph, but we decided to focus on interesting individuals, because the value of the graph is still pretty sparse and concentrated. And then we built this demo where you can ask all these sort of questions — tweets about AI, identifying mutual followers. And we demoed it to a bunch of people — Yann LeCun, Jeff, Andre — and they all liked it, because people like searching about what's going on about them, about people they are interested in.

I think there's something powerful about showing something that was not possible before. There is some element of magic to it, and especially when it's very practical too. You are curious about what's going on in the world, what's the social graph, interesting relationships. I think everyone's curious about themselves. I spoke to Mike Krieger, the co-founder of Instagram, and he told me that even though you can go to your own profile by clicking on your profile icon on Instagram, the most common search is people searching for themselves on Instagram.

So it's funny, right? That's funny. So the reason the first release of Perplexity went really viral is because people would just enter their social media handle on the Perplexity search bar. We released both the Twitter search and the regular Perplexity search a week apart, and we couldn't index the whole of Twitter because we scraped it in a very hacky way. So we implemented a backlink where if your Twitter handle was not on our Twitter index, it would use our regular search to pull up a few of your tweets and give you a summary of your social media profile. And it would come with hilarious things because back then it would hallucinate a little bit too. So people either were spooked — 'Oh, this AI knows so much about me' — or they were like 'Oh, look at this AI saying all sorts of things about me,' and they would just share the screenshots. And that's what led to this initial growth from being completely irrelevant to having some relevance. But we knew that was a one-time thing, not a repetitive query. At least that gave us confidence that there is something to pulling up links and summarizing it. And we decided to focus on that. Obviously we knew this Twitter search thing was not scalable, because Elon was taking over and was going to shut down API access. So it made sense for us to focus more on regular search.

Honestly, the way we thought about it was: let's release this, there's nothing to lose. It's a very new experience, people are going to like it, and maybe some enterprises will talk to us and ask for something of this nature for their internal data, and maybe we could use that to build a business. That was the extent of our ambition. Most companies never set out to do what they actually end up doing — it's almost like accidental. So we put this out and a lot of people started using it. I thought, okay, it's just a fad and the usage will die, but people were using it even during the Christmas vacation. I thought that was a very powerful signal, because there's no need for people hanging out with their family on vacation to come use a product by a completely unknown startup with an obscure name. So I thought there was some signal there. We initially didn't have it conversational — it was just one single query: you type in, you get an answer with a summary and a citation. We launched the conversational version with suggested questions a week after New Year, and then the usage started growing exponentially. And most importantly, a lot of people clicking on the related questions too. So we came up with this vision. Everybody was asking me, 'What is the vision for the company? What's the mission?' I had nothing — it was just 'explore cool search products.' But then I came up with this mission along with my co-founders: it's not just about search or answering questions, it's about knowledge — helping people discover new things and guiding them towards it. Not necessarily giving them the right answer, but guiding them towards it. And so we said we want to be the world's most knowledge-centric company. It was actually inspired by Amazon saying they wanted to be the most customer-centric company on the planet. We want to obsess about knowledge and curiosity. And we felt like that is a mission that's bigger than competing with Google. You never make your mission or your purpose about someone else, because you're probably aiming low if you do that. You want to make your mission about something that's bigger than you and the people you're working with, and that way you're thinking completely outside the box. Sony made it their mission to put Japan on the map, not Sony on the map.

Yeah, except it's not easy for them to serve that mission anymore. And nothing stops other people from adding on to that mission, reinventing that mission too, right? Wikipedia also in a sense does that — it does organize information around the world and makes it accessible and useful in a different way. X does it in a different way. And I'm sure there'll be another company after us that does it even better than us, and that's good for the world.