Since we recorded this Lite Cone episode with Scale AI CEO Alexandr Wang, Meta has agreed to invest over $14 billion in Scale, valuing the company at $29 billion. Alex has also announced he will lead Meta's new AI superintelligence lab. Our conversation you're about to hear covers the history leading up to this investment, from Scale's early days at YC to its integral role in the training of foundational models. Let's get to it.
The AI industry really continues to suffer from a lack of very hard evals and very hard tests that show really like the frontier of model capabilities.
The biggest thing is you just have to really, really, really care. When you interview people or when you interact with people, you can tell people who are just sort of like phone it in versus people who sort of like hang on to their work. It's like so incredibly monumental and forceful and important to them that they do great work.
Very exciting time to see how the frontier of human knowledge expands.
Welcome to another episode of the Lite Cone. Today we have a real treat. It's Alexandr Wang of Scale AI. Jared, you worked with Alexandr way back in the beginning, actually. What was that like? What year was it? Put us in the spot.
Yeah, Alex, I mean, most of what we want to talk about today is like what Scale is doing now because like the current stuff is like so, so awesome and so interesting. Since Scale got started at YC, I thought it just seemed appropriate to start all the way at the start. And it's funny, Diane and I were at MIT last month talking to college students, and like of all the founders, the one that they like most look up to and like want to emulate is actually you. Like everybody wants to be the next Alexandr Wang because everybody knows the story of how you like dropped out of MIT and ended up starting Scale, but they don't know the real story. And so I thought it'd be cool to go back to the beginning and just talk about the real story of how you ended up dropping out of MIT and starting Scale.
So before I went to MIT, I worked at Quora for a year. And so this is 2015 to 2016, or no, sorry, 2014 to 2015 was when I worked as a software engineer. And this was already at a point in the market where ML engineers, as they were called, or like machine learning engineers, made more than software engineers. So that was already like the market state at that point.
I went to these summer camps that were organized by rationalists, the rationality community in San Francisco. And they were for precocious teens, but they were organized by many people who have become pivotal in the AI industry. So one of the organizers is this guy, Paul Christiano, who's the inventor of RLHF actually, and now he's a research director at the US AI Safety Institute. He was at OpenAI for a long time. Greg Brockman came and gave a speech at one point. Eliezer Yudkowsky came and gave a speech at one point. And actually, I was very like, when I was, I don't know, must have been 16, I was exposed to this concept that like potentially the most important thing to work on in my lifetime was AI and AI safety. So something I was exposed to very early on.
So then when I went to MIT, I started MIT when I was 18. I studied AI quite deeply. That was most of what I did in the sort of day job. And then kind of got antsy, applied to YC. And then the idea was kind of like, okay, how could initially it was like, okay, where can you apply sort of like AI to things? And this was in the era of chatbots, which is like crazy to think about actually, that there was like this mini chatbot bubble.
In 2016, which was I guess spurred by Mezi, right? Or some of these apps. And Facebook had a big vision around chatbots. And anyway, there's this little mini chatbot boom. So the initial thing that we wanted to work on was chatbots for doctors, right? Which is like a funny idea because do you guys know anything about doctors?
Like basically no. It was just sort of like, oh, doctors are a thing that sound expensive. And so I think it's like indicative of like, I mean, I don't know, you guys see this all the time, but I feel like most of the times young founders' first 10 ideas are like always, first of all, they're very memetic. So they're probably like, there's a lot of like the same ideas over. There's like a dating app, there's like something for like, you know, social life, you know, there's the same ideas. And then I think that like, I think young people have a very poor sense of alpha, like what are the things that they're actually like going to be uniquely positioned to do. And I think, you know, most young people don't have a sense of self. So it's, you know, it's not clear.
So when we were in YC, we were roommates with another YC company and we were sort of like, we were sort of observing this like chatbot boom that was happening at the time. But it was very clear that like chatbots, if you wanted to build them, and this is funny to say in retrospect, required lots of data and required lots of like human elbow grease to be able to get them to work effectively. And so like just kind of off the cuff at one point was like, oh, like what if you just did that? What if you just did the data and the like language data and the human data, so to speak, for the chatbot companies.
We were also very lost, by the way. I think you probably remember we were quite lost mid-batch, and like many YC companies, I think. And so then we switched to this like concept. I think the initial idea was like API for human tasks or something along those lines. And one night I was just like trolling around for domains. scaleapi.com was available, and then we just bought it. We launched it, I think, a week later.
We Product Hunt. Yeah, I remember the Product Hunt page is still live. I was reading it last night and I remembered the tagline. It was an API for human labor. Like that's my recollection of sort of like the distilled insight that you had was like, what if there is an API, what if you could call a human with like an API?
Yeah, and that was, I mean, I think it was like three days for us to put up the landing page. It launched on Product Hunt. I think this idea captured some amount of imagination of the startup community at the time because it was sort of like this weird form of futurism where you have like humans delegated, like APIs delegate to humans in this interesting way.
It's like an inversion of the... Yes, yeah, humans doing work for the machines instead of the other way around.
Yeah, yeah, yeah. It's funny because the initial phase, you know, we just worked with all these engineers who reached out to us from that Product Hunt, which was a real grab bag of use cases. But then that was enough for us to raise money at the time and to get going. And then a few months after that, it became clear that like self-driving cars was actually the first major application that we needed to focus on. And so there were many very big decisions, I would say, in the first like year or so of the company.
One thing that was curious is at that point there were other solutions that were already the game in town, like Mechanical Turk from Amazon was sort of the thing that people were using. But you ended up capturing this whole other set of people that didn't know about it, and you had a way better API, and you kind of won. It was not clear at that point because you probably were compared a lot with Mechanical Turk.
Yeah, so Mechanical Turk was definitely the sort of like the concept in most people's mind at the time. I mean, it was just, it was kind of one of these things where I think a lot of people had heard about it, but anyone who had used it knew it was just awful. And so it's like whenever you're in a space and that's kind of the like, that's like the thing, it's like people mention a thing but it sucks, that's usually like a pretty good sign. And so that was enough to give us like early confidence.
But then I think the thing that like really, I would say the thing that was actually fundamental to the success of the company was actually focusing on this like seemingly very narrow problem of self-driving cars. I think that, you know, I remember very early on when it was maybe like six months after we were out of YC basically, there was another YC company, Cruise, that had reached out to us on our website, and sort of like in the blink of an eye, they became our largest customer.
And they found you just from your launch or...?
Yeah, just, yeah, I think maybe even Google. It's not even totally obvious, but just vaguely from our launch. And vaguely, it was actually an YC founder that was working at Cruise that reached out to us. So maybe some YC mumbo jumbo.
We're a ketti. Yeah, who knows.
The world works in mysterious ways. But and so they grew very, very large. So then early on we made this decision, and I remember we went to our lead investor at the time, and you know, we had this conversation. It's like, hey, actually, we think we should probably just focus on this self-driving thing. You know, it was actually a very interesting conversation because the reaction was like, oh, that's just like obviously way too small a market. Then like, you know, you're never going to build like a gigantic business that way. And we were like, we think it's probably a much bigger market than you think it is because there's like, you know, all these self-driving companies are getting crazy amounts of funding, and the automotive companies are doing huge programs in self-driving, and it clearly is the future. Like it feels like something that should exist. And so we're like, if we focus on it, we think we can build the business much more quickly.
And it's funny looking back because both things are true. It is both true that it enabled us to build the business to get to scale pretty very quickly, and it is also true that that was not a big enough market to sustain a gigantic business. The story of Scale in many ways is like this progression of like how do you continue, you know, AI is this incredibly dynamic space, lots of things are constantly changing, and a lot of I think what we pride ourselves on at the company is how we've been able to continue building on and contributing to this very fast-moving industry.
When did you become much more aware of the scaling laws? Because, you know, one of the interesting facts that sort of emerged is that you're a little bit the Jensen Huang of data.
I think that in self-driving, scaling laws were not really a thing. And the fundamental, the biggest reason actually was that like one of the biggest problems in self-driving is that your whole algorithm needs to run on the car, and so you're very constrained by the amount of compute you have access to and is available to you. So like a lot of the engineers and a lot of the companies working on self-driving never really thought about scaling laws. They were just all thinking about like, okay, how do you keep grinding these algorithms to be better and better and better that are like small enough to fit onto these cars.
But then we started working with OpenAI in 2019. This was like GPT-2 era. And I would say like GPT-1, GPT was sort of like this curiosity. GPT-2, I remember OpenAI, like they would have a booth at these like large AI conferences, and they would like, you know, their demo would be to allow researchers to like talk to GPT-2. And it was like mildly, like it wasn't like particularly impressive, but it was like kind of cool. It's like kind of this thing. And then I think by GPT-3, it was sort of this like, that's when the scaling laws clearly, you know, felt very real. And that was, I mean, I think GPT-3 was 2020, which is actually like long before the world caught on to what was happening.
Did you know as early as 2020, did you have a strong inkling that this was really going to be like the next big chapter of Scale, or not until ChatGPT took off? Was it 3.5 or was it 4?
I think that like in 2020, I think it was clear that scaling laws were going to be a big thing, but it was still not totally obvious. I remember this like interaction, you know, I got early access to GPT-3, and then it was like in the playground, and then I was like playing with it with a friend of mine, and I told the friend of mine, oh, you can like talk to this model. And during the conversation, my friend got like visibly frustrated and angry at the AI, but in a way that wasn't just like, oh, this is a dumb like toy. It was like, it was in a way that was like somewhat personal. And that's when I realized like, whoa, this is like somehow qualitatively different from anything that had existed before.
I feel like it was passing the Turing test at that point, kind of.
It was like semblances, yeah, semblance. It was like sort of like the glimpses of it potentially passing the Turing test, right? But I think the thing that really caused the recognition of, I would say, generative AI, which is still even the term in some ways, it was really DALL-E. I think that convinced everyone. But I think my personal journey was like GPT-3 was like highly interesting, and so it was like one of many bets at the company. And then in 2022, over the course of DALL-E and then later ChatGPT and, you know, GPT-4, et cetera, and we worked with OpenAI on InstructGPT, which is kind of the precursor to ChatGPT, it became very obvious that that was like the inflection moment for the company and for frankly the world.
That's when we saw it as well with the big shift in companies because it was that 3.5 moment release end of 2022, and we started seeing a bunch of companies and smart people changing directions and pivoting their companies in 2023, and that was that moment.
This dynamic that you referenced, which is kind of the, you know, Scale's the NVIDIA for data kind of thing, I think that became quite obvious. I would say GPT-4 really was the moment where it was like, it was like, wow, this is like, like scaling laws are very real. The need for data will basically, you know, grow to consume, you know, all available information and knowledge that humans have. And so it was like, wow, this is like this astronomically large opportunity.
Yeah, for the first time it was something that you could get to not hallucinate basically ever. You could actually have a zero hallucination experience in limited domains, and which is we're still sort of in that regime even at this point. You know, the classic view is that if it's hallucinating, you're not giving it the correct data in the prompt or context, or you're trying to do too much in one step.
Yeah, I mean, I think like the reasoning paradigm has a lot of legs, and it's actually been interesting this last era of model improvement because the gains are not really coming from pre-training, which is so we're like moving on to a new scaling curve of reasoning and reinforcement learning, but it's like shockingly effective. And I think that, you know, the analogies between like AI and Moore's law are pretty clear, which is like, you know, you'll get on different like technical curves, but like if you zoom way out, it'll just be feel like this like smooth improvement of models.
One of the things that has been popping up with some of the like really big well-known labs is they're getting access to full parameter fine-tunes of the base models, especially the frontier base closed-source models. Is that like a big part of your business, or, you know, something that people are sort of coming to you for, just like these verticalized full parameter fine-tuned like datasets?
Yeah, I think this is going to be a like blueprint for the future, right? So right now, I mean, like the total number of large-scale parameter fine-tuned or reinforcement fine-tuned models is like still pretty small. But if you kind of think about it, like that, like one version of the future is that every firm's core IP is actually their specialized model or their own fine-tuned model. And just in the same way that like, you know, today you would generally think that the IP of most tech companies is their codebase, in the future you would generally think that their specialized IP might be the model that powers all of their internal workflows.
And what are the special things they can add on top? Well, they can add on data and environments that are somehow specific, very, very specific to the day-to-day problems or information or challenges or business problems that they see on a day-to-day level. And that's the kind of like really gritty real-world information that, you know, nobody else will have because nobody else is like doing the exact same business motion as them.
There's a lot of weird tension in that, though. I remember friends of ours from one of the top model companies came by, and they were like, hey, do you think YC and YC companies would give us their evals so we could train against it? And we were like, no, dude, what are you talking about? Why would they do that? Because that's like their moat. And then I guess now that based on this conversation, it's actually, I mean, evals are pretty important as a part of RL cycles. And then even the evals are not really the valuable part. The valuable part is actually the like properly fine-tuned model for your dataset and your set of, you know, sort of problems.
Yeah, it's like these Lego blocks, right? If you have the data and you have the environments, and then you have a base model, you like, you know, can stack those on top of each other, get a fine-tuned model. And obviously the evals are important. This is some of the tension, and this is basically, you know, in a nutshell, the sort of like, does AGI become a Borg that just sort of like swallows the whole economy in like, you know, has one firm, or do you still have a specialized economy? My belief, generally speaking, is that you still do have a specialized economy. Like these models are platforms, but the alpha in the modern world will be determined by, you know, to what degree you're able to sort of like encapsulate your business problems into datasets or environments that are then conducive towards building like, you know, differentiated models or differentiated AI capabilities.
Yeah, that's why asking for evals was so crazy to me because it's like, okay, you get the evals, the base model is way better, and then now all your competitors have exactly the same thing that used to be your advantage.
I think we will undergo a process in AI where we learn what the bright lines are, right? I mean, I think that like it's like very obvious and intuitive to tech companies that they should not give away their codebase and they should not give away their database, like they should not give away their data, they should not give away their codebase. The analogs of that in a, you know, highly AI-fueled economy, I think, will identify over time, but are, yeah, the evals, your data, your environments, etc.
I think you have a very techno-optimistic view of what the future is going to be with how jobs are going to be shaped. Can you talk more about that? Because I think you hinted at it before, it's going to be more specialized. It's not that all these jobs are going to go away, right?
First off, it's undeniably true that we're at the beginning of an era of like a new way of working. Like, you know, this, there's this term that people have used a long time, which is like the future of work. Well, we are like entering the future of work, or certainly the next era. And so work fundamentally will change. But I do think humans own the future, and we are like, we have a lot of agency, actually, and a lot of choice in how this sort of like reformatting of work or how the reformatting of sort of like workflows ends up playing out.
You know, I think you kind of see this play out in coding right now. And I think coding in some ways is really the sort of like case study for other fields and other, you know, other areas of work where sort of the initial phase is this sort of like assistant style thing where, you know, you're kind of doing your work and then the models are kind of like assisting you a little bit here and there. And then you go to a, you know, the sort of like Cursor agent mode kind of thing where you're like synchronously asking the models to like carry out these workflows, and you're sort of like, you're managing like one agent kind of, or you're sort of like, you're kind of like pair programming with a single agent.
And then now with like Codex or other systems, like it's very clear the paradigm is like, oh, you have this like, you have this like swarm of agents that you're going to deploy on like all these various tasks, and you're just going to like sort of like, you know, give all these tasks, and you'll have this sort of like, this cohort of agents that are sort of like, you know, doing this work that you think is appropriate. And that last job has a semantic meaning in the current workforce. It's a manager. You know, you're basically managing this sort of like, this set of agents to do actual work.
And so, and I think that like AGI or, you know, AGI or doomers or whatnot, like they take this view that like, oh, even this job of like managing the agents will just be done by the agents. So like humans will be taken out of the process entirely. But our belief, my personal belief is that, you know, this is, management is very complicated. Management is also like more about like what's the vision that you have and what's the sort of like, what's the like end result you're aiming towards. And those will be fundamentally, I think like, you know, we have a human demand and human desired driven economy. So those will be driven by humans. And so I think the terminal state of the economy is just is large-scale humans manage agents in a nutshell.
I have a funny story where a founder friend of mine is trying to promote one of his, you know, junior employees, but they're really, really smart and they're working on the agent infrastructure. And then he was like, hey, do you want to like, you know, I'm looking for someone who could step into management. You've never managed people before, or do you, you know, if we hired some people under you, like how would you feel about that? And this, you know, mid-20s, really smart, you know, sort of dude, he's just like, he's an engineer, and he's like, why would I do that? Like, just give me like more compute. Like, you know, the model, like, look at what just happened to the model literally like last month. And, you know, I didn't have to do anything. It just started doing things that it couldn't do a month ago. Why would I want to manage people? Like, just give me like, I will just manage more agents for, and it's fine.
Okay, so what are the unique things that humans will do over time?
I mean, I think this like, this like element of vision is very important. This element of like kind of like debugging or sort of like fixing when things go wrong. Like most of a manager's job, speaking as a manager, is just like putting out fires, dealing with problems, dealing with like issues that come up. Like I think intuitively, you know, the idealistic manager job seems like this very cushy job because you're like, oh yeah, all the other people do all the work and I'm just sort of like, I just vaguely supervise. And then the reality is obviously like highly chaotic.
I think people often jump to this like, you know, extreme reality where it's like, oh yeah, these like, you know, you're just going to manage the agents and you're going to sort of like live this like, you know, kind of Victorian life where all your problems are solved. But no, I think it's still going to be pretty complicated. Like getting agents to like coordinate well with one another and like coordinating the workflows and debugging the issues that come up. Like these are still complicated issues. And, you know, having seen what happened in self-driving, which was more or less that like, you know, it's easy to get to 90%, very, very hard to get to 99%. I think that like something similar will happen as with large-scale agent deployments, and that like, you know, final 10% of accuracy will be like, you know, will require a lot of work.
Yeah, even for self-driving cars right now, there's the remote assist for all these super edge cases. So there's still a human at the end managing the car.
Yeah, and the ratio, by the way, I mean, the companies don't publish them, but I think the ratio is something like five cars to one teleoperator, or maybe even less than, maybe three cars per teleoperator. So the ratio is like, you know, much lower than people think. I think that like humans are much more involved even in self-driving cars than I think most people appreciate.
I mean, which if you put it in that perspective, I think it's still very optimistic. It's just the output of getting rides instead of doing. In today's world, if you're an Uber driver, you just do one car. In this world, you can do five cars, right?
Well, you have to believe for this like, for an optimistic version of the future where, you know, unemployment is still low, etc., you just have to believe that humans are like almost insatiable in their desire and their demand, and that like, you know, prices will go down, things will become, you know, the economy will become more efficient, and we'll just like want more. And I think this has been a pretty reliable trend for like the history of humanity is that like, you know, we have somewhat insatiable demand. And so I have, I have like conviction that like, you know, the economy can kind of get as efficient as it needs or as it like can get like hyper, hyper-efficient, and then human demand will just like continue to sort of like fill the bucket.
Yeah, in the 20th century, you know, when you said computer, maybe early 20th century, people didn't think of like a computer as it is today. They thought of a human being that would sit in front of a punch card tabulator, and that was like what a computer was doing. I mean, literally, that was a real person's job. And then, of course, now today, it's like, where are all the computers? Well, they're actually real computers now.
I don't know, it's that was the Apollo mission. It was a bunch of people just crunching numbers with the trajectories of Apollo, and that was it. Because the computer that went on the rocket is actually was a microcontroller with, I think, only like single-digit hertz. It was like very tiny amount of computations. It was just humans doing it.
Totally. And even this, like, I mean, I think the concept of being a programmer is somewhat, is like highly esoteric in the sense that like, oh, you're like writing the instructions for these like machines to just like, you know, just continue to do repetitively. And in some ways, it's like the leverage boost that all humans will get is like similar to the leverage boost that like programmers have had historically for a long time. I think a lot of people in Silicon Valley say this, like the closest thing to alchemy in our world pre-AI, let's say, is programming because you sort of like, you can do something that creates like an infinite, there's these infinite replicas of whatever you build, and they can sort of like run an infinite number of times.
And I think the entire human workforce will soon see that kind, that large of a leverage boost, which is extremely exciting because I think that like programmers have benefited over the past few decades from this like unique perch where they have like, you know, one 10x or 100x engineer can like, can build something like absolutely incredible and like very, very valuable and like very shockingly productive. And all of a sudden, I think like humans in all trades, I think will like gain this like level of leverage.
Alex, I'm curious to return to a point that you made earlier about like how Scale has kept reinventing itself. If you had to like describe the arc of Scale, like what's the story and what were the turning points?
Our initial business was all around, you know, producing data, you know, generating data for various AI applications and primarily self-driving car companies, right?
For the early years, it was really like you're saying, you're really focused on that, yeah?
For the first like three years, fully focused on that. One of the properties of focusing on that business, of building that business, is over time, you know, we had this like obligation to really like get ahead of most of the waves of AI, if that makes sense, because, you know, for AI to be successful in any vertical area, it needed data. And so like our demand for our products would precede a lot of times the actual sort of like evolution of AI into those industries.
So, you know, as an example, we started working with OpenAI on language models in 2019. We started working with the DoD on government AI applications and defense AI applications in 2020. This is like long before, I think, the, you know, recent sort of like drone-fueled, you know, AI craze in the Department of Defense. We started working with enterprises long before there was sort of like this, you know, the recent sort of like larger waves around enterprise AI implementation. So almost sort of systemically or intrinsically, we've had to basically build ahead of the waves of AI.
I think this is actually quite similar to NVIDIA. You know, whenever like Jensen gives his annual presentations about, you know, NVIDIA and its two trends outlook, like he always is so ahead of the trends. And that's because he has to get there on the trend before the trend can even happen. That's, I think, been one way in which our businesses continue to adapt because AI is like this, you know, it's the fastest moving industry, I think, ever in the history of the world. And so, you know, that each turn, each evolution has moved incredibly quickly.
The other thing that happened late 2021, early 2022, we started working on applications. And so we started building out AI-based applications and now much more so agentic workflows and agentic applications for enterprises and government customers. And this was an interesting evolution of our business because historically, like our core business is highly operational. You know, we build this like data foundry. We have all these processes to produce data. It's a very operational process that involves like lots of humans and human experts to be able to produce data with quality control systems in place. That highly operational business. And the success of that business is what created the momentum for us to, you know, sort of dream about building an applications business.
When we went into it, I had studied other businesses that had basically successfully added on very different businesses. And what are sort of like the unique traits or why do some of those work? And one of them that is probably the most interesting, I think, is like the most singular in modern business history is Amazon building AWS. You know, if in 2000 you had written a short story that said that like, you know, this large online retailer would build this like large-scale cloud computing rent-to-server business, like it would seem like nonsensical.
I remember when they launched AWS in 2006, Amazon stock went down because all the analysts thought it was such a terrible idea. It'd never been done before. It just like, it doesn't seem related at all to their core business. It has, it's like this like weird thing. But the sort of like wisdom of that was, I think, twofold. I think like first, and from talking to people who are like there at the, you know, the sort of like the genesis moment of this business, like one thing, probably the most important thing was that they had conviction that the sort of like underlying business model of AWS would basically be this like, this like infinitely large and growing market, like that market would literally grow forever. There would be like this like exponential of the amount of compute that needed to be built up in the world. And if you did that, there was like sufficient cost advantages from economies of scale.
I think like startups, you know, you kind of like, you kind of have to like switch modes at a certain point where like early on you're trying to go for very, very narrow markets, like almost the narrowest markets you can. And then you're just trying to like gain momentum and then sort of like slowly grow out from those hyper narrow markets. And then at some point, if you like have ambitions to be a hundred billion dollar company or more, then you have to sort of like switch gears and say, where are the infinite markets and how do you build towards those infinite markets?
And so this was sort of like the moment where we realized that. And the simple realization was that every business and every organization was just going to have to reformat their entire businesses with AI-driven technology and now obviously like agent-driven technology. And that would just be like, over time, that would swallow the entire economy. And so it was like another one of these like, okay, that's an infinite business to build out AI applications and AI deployments for large enterprises and governments.
I think a lot of people don't realize that you guys are in the middle of this transformation. They still think of Scale as the data labeling company. But like, if you fast forward ten years, do you think most of Scale will actually be the agent business?
