If you watch a movie in Polish language, all the characters get narrated with one single voice. That was the initial trigger point of like, hey, we know this will change. Knowing what's possible across audio space decided to start ElevenLabs, and of course the vision quickly expanded from there, where it's not only that that will change. It will be the entire interaction with technology around us that will change.
AI is not going to replace the entirety of the process on creating a movie, but it will replace parts of it, where you will have started with an idea and then it will help you draft the first set of scripts, and then you'll refine that, and then you will repeat this process a few times, and then you have an end result, probably a combination of human and AI, where this is indistinguishable.
You are going through a set of iterations on selecting the voices, redoing the lines, sound effects. Maybe you created the voices that you care about and you want to bring those voices inside of the production flow. You need a pretty solid platform to be able to do that. And we kind of combine all of that in one place for you.
Welcome to Grit. I'm Jubin, partner at Kleiner Perkins, a show where we go beyond the highlight reel and explore the personal and professional challenges of building history-making companies. Today on the show we have Matty, co-founder and CEO of ElevenLabs, the audio company building infrastructure for how we speak and listen in the world of AI. They are in the eye of the hurricane of all AI hurricanes right now. And it's an absolutely incredible story of starting in Europe and building what looks to be one of the preeminent businesses in LLM today. Enjoy the episode.
So your early employees, how much of it was like, I don't know, researchers, scientists versus... these rules start to get very blurry and fuzzy, but researchers in our mind, these are like people that can create net new work, new paper, get it out onto the world, and this is like a true innovation that it could make any paper at any of the conferences and would be one of the best papers. So that's the researcher.
Then you're right, there's standard full stack engineers. But then we also have like a category of research engineers. So people that can take an existing paper and improve it or do some of the modifications on the inference side to make the model serving and more optimal and quicker or better. So kind of in those three categories.
Initially we would optimize for researchers and research engineers. So that's why we started very much remote at the very beginning, where we wanted to hire anyone who was the best in the world wherever they were. And we knew that the only way for us to get to the best talent would be to hire the researchers from... we had a person from Korea, from Europe. And that was like kind of the big part of like, let's hire them wherever they are. Let's assemble the best researchers in the world in one place.
And then of course full stack engineering came kind of in parallel at the same time to start building the product.
Okay. And so your first, let's say your first 10... how many were each of those responsibilities?
So my co-founder, the smartest person I know, also an incredible researcher. So that's what he used to do at Google and previously at university. So he was one of the researchers, but on top of... we hired two people in the first 10, one, two people, then additional one research engineer.
But the research engineer, like in the early days, you kind of do everything, whether it's data, whether it's improving the model. So it like kind of infuses. But I would say from the first stand, including me and my co-founder, four of the people would be doing research or research engineering, two, three people would do engineering, and three people would do go-to-market growth.
So pretty quickly we started looking into how we can get the technology out there too, to both the self-serve distribution and then sales-led distribution through working with some of the companies.
Yeah. And before we get too far ahead, like maybe for those that don't know, what do you all do?
So we are ElevenLabs. We do a combination of research and product work, product deployment work focused on AI voice space. We are on a mission to be the voice of technology, and we do it for two key ways. One is in agents platform offering. We help people build conversational agents across customer support, for personal AI, across media entertainment to make interactions interactive.
And then second is our creative platform offering, where we help people create narration, voice over, dubbing, add music to their work in one holistic suite. So really stretching the use cases where voice can empower, and now even more on the omni-channel experience and agent and omni-modality experience on the creative side.
What's crazy is the company started pre-GPT-3, right? And you were, if my math is correct, like 25 years old at that point, living in Europe and recruiting researchers... like that doesn't... I'm trying to like... that doesn't really add up to me. Like how does that... how does that... like what are they even... what were they working on? Like what research were they even doing? How did that... how did that work?
No, it is, you know, it's like in a way it was also the best. In many ways it was the best. We started a company in 2022. It was kind of still the year where crypto, metaverse were kind of on their final downhill moment. So AI, like nobody was working on AI, which gave us an advantage of two things.
One of us, we had a good amount of time to start work before the GPT moment, where like a true viral moment, where that gave us like kind of advantage of working before others. And two, the people we were able to hire in those days were true missionaries and excited people about the space.
You mean you're not competing with Zuck back then?
You're not competing with other big players, although happy to compete with them. I think we have a great spot for a lot of the researchers to do the best work of their lives in audio and beyond. But the main thing is that all the people that join you at the time are like true missionaries. They really believe in the mission, really believe in what the voice can do for the world.
That was still like... currently it became normalized that voice is going to be the future interface. It's an important part. Back then nobody really cared about AI. Nobody really cared about voice. And me and my co-founder, so we started a company, we knew each other for 15 years. So we are best friends from high school, took all the same classes. We both love mathematics and everything around it. And then through the years did everything together. Worked together, traveled together, lived together and are still best friends. So the time is on our side.
And the idea came from Poland. We are both from Poland. Then there's a very peculiar thing. If you watch a movie in Polish language, like a foreign movie in Polish language, all the characters get narrated with one single voice. So whether it's a male speaker or female speaker in the original movie, in Polish you get just one voice narrating every...
Is that unique to Poland?
It's unique to the post-communism countries. There's like five countries in the world that do it this way. In Poland it's like one of the few, which is... it's like almost hard to imagine, but it's a terrible experience and it still happens for most of the work today, most of the content today.
So that was the initial trigger point of like, hey, we know this will change. We know that with the invention of transformer a few years back, with the diffusion models, audio will be able to get created in a way that's actually heavily emotional and has a good intonation. You can recreate any voice you want. And with knowing what's possible across audio space, decided to start ElevenLabs.
And of course the vision quickly expanded from there, where it's not only that that will change, it will be the entire interaction with technology around us that will change. If you can create synthetic speech that's high quality, the voice will be such a powerful medium for interaction. And start ElevenLabs in 2022.
We also, you know, we through the years prior, so he was at Google, I was at Palantir, we would meet and do hack weekend projects together across a variety of different domains. I'm curious how you stumbled on your idea as well. But in our case, we were exploring new technologies and seeing what they can do and how we can implement some of the mostly ideas around them, less so commercially focused ideas, and one of those was audio.
But that gave us a good kind of glimpse into how we can work together. We were so excited about going full-time and dropping our relevant companies and starting ElevenLabs.
Crazy. What an insane story. And then when the... when I guess what was it? GPT-3 came out was a year later.
So ChatGPT came December 2022, January 2023.
Not long into the journey. Did you basically scrap everything?
No, it was... so in a way a lot of our ideas were based on the foundational stack anyways. So they were like just validated how relevant they are. So like it was almost a parallel invention path, just a different technology. We of course have a lot of the transformative diffusion ideas coming into our work too.
So when we started in 2022, we would use some of the most relevant work recreating what should be done in text-to-speech. And then when GPT moment came and a lot of the technology around this was validating that we are doing the right thing because the ideas were giving good results.
And two, it kind of gave you this amazing overlap of what's possible, where you could have ChatGPT now and you could voice on top of it. Suddenly you can do content in such a much easier way where ElevenLabs has even more advantage.
I guess, you know, like at the time, so in 2022, most of the synthetic speech just was so poor. It was still very robotic. It's still... I don't want to name some of the big other companies, but you can imagine the hardware devices around how like that's like... it's still not good. And back in 2022 it was even worse. So it was truly robotic speech and nobody could figure out how to make the speech sound human, so like actually represent the emotions.
And that's where 2022 for us was all focused, like how can we get the text to understand the context in a much better way. Understand the emotions around the text. Is it happy text? Is it dialogue? And bring that in the same way that a voice actor would or human would. When you read, you understand that context and then you bring those emotions across.
So that was the big innovation we brought in 2022, where text-to-speech sounded human. And the second big was being able to recreate voices. So how can you effectively encode a voice and the characteristics of the voice, and then when you produce synthetic speech, it still carries the style of the voice, the accent, the dialect, whether it's narrative voice or conversational voice. How do you bring that all across? And we figured this one out.
But a lot of those ideas were based on similar ideas that the LLMs would carry as well. And then of course a lot of the nuance that were required for the audio space.
One of the things that I've been thinking about that I wanted your opinion on, and maybe this is just like a very stupid thought, but like we have this concept here in Silicon Valley about like crossing what's called the uncanny valley. And the uncanny valley is this concept where basically machines become indistinguishable from humans. You cannot tell if it's an LLM talking to you or writing to you versus a human.
And so far, certainly in text, I can within 10 seconds tell if an LLM has written it or not. It's very, very obvious.
Based on the em dash or...
Based on the em dash, based on just the way that it writes. It's not even just the em dash, based on...
Like not even random. Like, forget about the bolding and the em dashes, right? Because nobody actually writes like that. Just the way that it actually writes does not sound human to me. And it's very obvious the minute that somebody sends me something, immediately I'm like, you didn't write it. Just like something inside my brain is like, that's not a person that wrote this.
Okay. And so, you know, maybe audio, if it's coming off of text, is a similar paradigm today. And everybody continues to say, oh no, no, it's happening. Like it's going to happen for sure. And one of the questions that I've been asking myself, like, you know, these models have gotten a lot better. And they're still getting better. And everyone's like, well, as long as it keeps getting better, it's definitely going to sound indistinguishable from us.
And, you know, I've listened to the like NotebookLM from Google. Like I've listened to all of this stuff. I've done the use ElevenLabs to clone my voice. And I'm not even saying it's not a knock, like I'm not even saying it should or shouldn't sound like me. And like maybe it shouldn't. Like maybe if you're from India, your dialect should be Indian. Like maybe if you're from Iran, your dialect should be something else. Maybe if you're from machines, your dialect should be machine. You know, like I don't know.
And on the LLM, quick question back: is it across all use cases you can tell, or you can like, you know, like if it writes a copy for your email or writes a copy for your post, or is it... what... across all settings or just specific settings?
I think unless somebody has to really... I think the only way that I generally can't tell is if somebody writes down what they first thought, then puts it into the LLM to tweak what they thought, and then goes back and then like unbundles the things that sound like an LLM. But I think that's across every use case. I have seen that.
Yeah. Do you agree, by the way?
No, I don't think... I think... yes, in many settings it's obvious where somebody is an LLM and some of the things you mentioned make it obvious, but of course like there's something about the style, like if somebody goes the lazy effort of hey, write this for me, immediate tell of like it's easy.
But I see more common the kind of the last thing of what you mentioned. So somebody will write the thoughts, they'll rewrite the thoughts so they are like more human. And I don't know if we are classifying that in the same category, but in those cases I think it's very not... or it can be very good.
I wouldn't classify it in that same category. I'm just saying if you tell an LLM to write something for you...
Just like if you tell an LLM to speak something for you, is it always going to sound like an LLM?
Yeah. So I think if you just use like a default, likely yes. I do think now you're going to kind of like pre-prompt it where it becomes a little bit less of a tell.
And in some settings it's as good or better than the human would. And let's take a customer support setting where we work a lot. If you're working in customer support, the baseline you're comparing it to is usually an IVR flow or like a decision tree of if this then that, or a potential human that has maybe not had as much of the training and will try to help you but might not be able to help you in that setting.
I find the LLMs can provide as good, if not just on the text level, if better than human experience where you get the support and you know you are comparing it to something that's...
I totally agree with you. You're using Sierra and you're using ElevenLabs on the back end, and Sierra is partnered with United, and you're comparing the experience that United would give you pre and post. It's night and day.
Yeah. Yeah, but I'm thinking even on the human side, if you are comparing, you know, just... so even let's ignore voice, and I have my take on voice as well, where just on the text level, if you are comparing customer support done by AI agents versus humans, I think many people wouldn't be able to tell, and it's probably a better result altogether.
If we are thinking about like more of a creative case, like a narration for a movie or a conversational dialogue, I think there, yes, it's a little bit harder to get on a high-end part to get something comparable on the voice side on the uncanny valley piece.
I think in the narration space, I think you can get the results from voices that people won't be able to tell are not narrated by human. And we've seen this across so many places where an audiobook created is just so good that people are like both immersed, and I don't think if you kind of ask them point blank is it better or worse, they would be able to tell.
In the conversational setting, I don't think we passed uncanny valley for all types of use cases. So I think we passed it in call center application. I think we passed it in some of the customer support experience use cases, but let's say in immersive media or gaming or personal AI type spaces, I don't think it passed that kind of Turing test in all settings.
Like we are speaking now, we can interrupt each other at any point. We can move pretty quickly, pause, we can... there's like a reasoning and voice that's combined to get us one. That moment has not yet, I think, happened for conversational setup.
But you're convinced it will.
I think it will. Yeah, I think it will. Hopefully we make it happen. So yes. I'm biased as well, I guess.
Yeah, you're biased. That's fine though. Like... but even without voice, so you think in that conversational setting, you would be able to tell today, I presume, if you're messaging with an AI straight away that this is AI, or like after a few exchanges?
Yeah. So I think it will happen.
So I have two points that I'm making. One is certainly in most cases I can tell. Second, I don't even think it's a bad thing that I can tell. I guess what I'm really questioning is why do we want them to sound and feel like a human?
Meaning if I am chatting with United, okay, or if I'm on a phone call with United, I don't actually care if it's a human or an LLM on the other side. In fact, I would rather know. And what I care about is how quickly can you get to resolution. That's all I care about. And in fact, it would maybe give me a little bit of peace of mind if I know what's on the other side of the person that I'm talking to.
Oh, no. I 100% agree on that part. Like ideally you do know it's pre-prompted and it's a good experience, right? So I think like yeah, I would decouple those two points too. It's like you want the resolution as quick as possible. And then two, does being human voice or human experience help you to get to that resolution? And in some cases it does, in some cases it doesn't.
It's like in customer support, no, you want the content, you want the actions, you want the intelligence to be well connected, you want to know the information about you. You want to know you're speaking to an AI agent to help you resolve that case.
And then let's say you are in a gaming context. You are in an immersive game in Fortnite. That's a kind of example of a game we worked with Epic Games to bring Darth Vader experience into game. You want to make it feel as emotional as possible, as real as possible. Just like Darth Vader.
Just like Darth Vader. Exactly.
And by emotional here, you want a pretty raspy, pretty serious tone of the experience, and customer support you don't. But then there is a very different experience depending on who you are.
So like you know, let's say you or I want to ask quick to the resolution, quick responses across the conversation. We recently worked with a customer support company in healthcare in Japan, and they had this case where a lot of their population calling in is older, and then they have a small proportion of people that are younger.
It was effectively a widget on top of the website, and they had the information about who is reaching out for help. And both the voice part, but also just the style of how the information gets delivered, was drastically different between those groups and led to much different results between those groups.
So if you had a younger person reaching out for help, you wanted relatively quick, you wanted relatively short in style, straight to the information. If you get an older person, you wanted to make it slightly more calmer, less emotions, a lot longer responses, longer slower responses. So in this case, it really mattered depending on the context of who is calling in.
And in the same way as you think about a human, we don't... you know, like if you have a human responding on customer support in that same client, what you really wanted to do is to be able to alter depending on who you are speaking with and what type of experience you deliver and then what leads to the resolution.
And that's where I think it matters that you make agents versatile to help across all those different use cases. I don't think we should force them to be emotional or human in any of those cases. It's more of like how is it able to adapt to the use case at hand and kind of bring it to the resolution in a way that's understandable.
Yeah, it makes total sense. All I'm... maybe the other point that I'm making on the... because I think we're in agreement on the should it sound like... should I know who I'm talking to on the other side? Okay. But I also think that there's a thing happening now where everybody in my world just assumes that, including you, that we are definitely going to pass the uncanny valley.
It is going to happen. There will be a world where we cannot... if we choose not to, it will be indistinguishable whether we are talking to a human or a machine, both in text and audio format and potentially even in video format. Right? If you're going through YouTube Shorts or whatever and you can't tell the difference if this is a real or not.
And all I'm saying is that hasn't happened yet. Like the YouTube Short videos have infiltrated when I scroll through. I can always tell the audio. I can pretty much always tell the text. I can almost always tell for purely LLM created and generated, right? The models are getting way better, right? They've already gotten way better. We haven't crossed it yet.
Everybody is telling me we're obviously going to do it, assuming that scaling laws continue to persist. And all I'm asking is like, is there a possible world where that's not true? That's like kind of my only... like is that... like I feel almost dumb asking the question because it's like sacrilegious around here to question the scaling laws. But I'm just curious like...
Yeah, so two answers there. I think one is it's like depending on what you are solving for, this answering this question is important or not. Like in a similar way, like as we think about delivering the best experience, in many cases whether you pass uncanny valley and all those doesn't matter.
Doesn't matter and you should agree because the bar is talking to some United agent that's like already terrible. You wait in queue for 10 minutes. So I'm not disputing it.
But so I guess on kind of like, you know, as I think about a lot of Silicon Valley companies, I don't think it... like, you know, the goal in itself should matter. And I think there's a second piece to it which I think is important, which is in the same vein, I think it's like a lot of obsession about the agent or the AI kind of the buzzwords kind of proliferated the mission of the company itself in many of those places.
Where at the end of the day it doesn't really matter across all of that. It really matters about are you solving the problem for the customer in a quick and good way. So that's kind of the part one where I think I agree with the general trope that many seem to be like focusing on solving without understanding what you're solving for.
But to the second thing, I think it's going to happen. I do think uncanny valley is in many cases already crossed. I think you might be more so also more attuned to it given you are living and breathing every day than most of the general population.
I would say it's almost more like how do we make sure the whole world is prepared where you aren't able to distinguish and you want to get information. I think this will be super important that you know that there is an AI agent on the other side or that content might be created with AI. You want to know that information or at least know that this is generated by human and kind of assume that everything else is AI.
So I am like, you know, 99.9% that in most of the cases the uncanny valley will be crossed and we need to like kind of work on preparing on how you make sure that society functions as a whole.
And by the way I believe you. Not... I mean I do think you're talking your own book, but I also believe you and that's why I'm asking you. Like you've been engrossed in this, you know better than anybody.
You know, like I honestly think if even if it doesn't cross for kind of mission of our company, we still will be... will be... that's why I think it's like not as kind of tied in that side.
So you think there's more than 50% chance we'll not cross uncanny valley? So I do want to make a point about ElevenLabs because I do believe that you do not need to cross the uncanny valley in order to be a huge company.
No, but do you think that uncanny valley will not be crossed in a LLM setting and voice setting? And it's more like, okay, if it is crossed, then every movie moving forward will always be generated by an LLM. Why in the world would you create a movie that it's just so much more work to do all the things if you can just tell it what you want it to do?
But it's like, you know, like I don't think it will happen the way that... and so maybe we are like I'm answering a different thing. I don't think it will be end-to-end in any short perspective of time. I think it's going to be more like middle-to-middle, where AI is not going to replace the entirety of the process on creating a movie, but it will replace parts of it.
So that's why the middle-to-middle, where you will have started with an idea and then it will help you draft the first set of scripts, and it will help you vocalize the first set of scripts, find the voices that are right for that script, and then you'll refine that, and then you will repeat this process a few times, and then you have an end result that's in the end probably a combination of human and AI.
And I think this will end in a result in the category two of what you started with, where you kind of have AI or like human AI, AI say human AI human, where this is indistinguishable. I think in that case this will happen. I think this is where the 99.9%, I think it's in that category.
Will you be able to generate a movie from scratch at level of quality and creativity, ingenuity as the human would? For like, you know, like in many cases, I don't think you will, but there will be definitely use cases where yes, where that still will be possible.
Yeah, I think it's going to be like a... you know, like same same like in the current... I think let's take movie as an example. Like even currently you have rare this kind of sparks of incredible creativity where you're watching something that's completely new or different.
And I think this will still remain. Like you will need a lot of idea, the process generation from the human side, but then you will be able to generate a longer tail of content that's okay, it's still enjoyable to watch, but might not have required that kind of entirely new idea.
The interesting question that I keep circling back to is like if you've crossed the uncanny valley for a three-minute clip, why can't you cross it for two hours? And if you do cross it for two hours where I cannot tell, then why wouldn't a director just give their entire script that they wrote to an LLM and then it can be like put it in the style of Goodfellas with the punchiness of whatever, right? And it just does it.
And then maybe there's some tuning or whatever. I'm just saying like, and if that's true, right? If that is possible, what that would be insane like...
No, but and I agree and I think this will be possible, but coming up with Goodfellas...
Or writing the script for Goodfellas or like the book itself...
That's going to be extremely hard.
Agreed. That's going to be extremely hard. And then there's another piece which is like it's... you will be easier to create software, to create content, to create movies.
So now you are kind of even more attached to the storyline, to the history of something than ever before. And that history will frequently go back to has it happened in real life? Has it happened? Is it the story I used to follow when I was a kid? Is it a universe of characters I know and love and follow and other people do as well? So we can kind of speak.
So I think it's going to be possible to do it, but it will be harder to create the attachment to the human across all of those use cases.
Totally. Where... what percentage of the revenue is B2C for you versus B2B enterprise?
It's closely approaching 50/50.
Yeah. So it's a lot on the B2C is the creative use cases, narration, voiceovers, dubs, music. And then on the B2B it's huge conversational AI push with our agents platform. So everybody's building for conversational use cases, making... and it's in voice, but increasingly also sometimes not only voice, where like an omni-channel experience where you do voice and text interactions. So that's kind of skyrocketing across.
Yeah, it makes sense. I mean on the B2B side, like one of the great use cases that we have seen is support.
It's great because it's generally a cost center and it's great because LLMs are very good at taking that unstructured nature and putting it into a structured format that it can automate away. And in so many cases, like it makes total sense to me that you're powering that workflow.
Yeah. And 100%, it's kind of the place where we see just such a quick adoption. But an interesting thing that started happening is the shift from this being a customer support place where you reach out as a user when you have an issue into increasingly being more of a kind of a front-facing experience, customer experience across the entirety of your product or user journey.
So from, you know, like in the traditional way, if you got an issue, you would reach out through an email. Now we see companies working on bringing that as part of their website experience itself. So recently we worked with a company in Italy, the biggest real estate company, where they effectively placed an agent that as a buyer and seller of a company you can reach out to to get information about that real estate.
And they had like historically a huge supply and demand mismatch. So the seller of a property wasn't available to pick up the phone to give the information about the property, and on the other side you had a lot of the buyers that were just not the right buyers for the property. So now you have an AI agent that you can call in and reach out and get information about the property.
And then that information gets aggregated, given to the seller of the property, and then they can reach out to the right ones, helped improve the qualification rates like 20 to 60% of people that are progressing to the next step. And before you would only reach out if you had a problem.
Going through that experience. So it's kind of interesting where increasingly it's kind of the part of the experience itself.
Even in our own work, we love trying to do it — like, how can we elevate our experience? We of course had, because of the B2C and B2B angle, so many people reaching out for help, credits, how to use the product. So we shifted it where now you have our agents both on the website itself. So you can speak with an agent about ElevenLabs the moment you open the website. Then when you log into the product you have an agent that can help you navigate through the product experience, will open items for you, speak with you about how it works. And then of course on top of that you can reach out to support, which is also powered by our agents. So now you have like a full fleet from understanding to using to reaching out for help, to even if you reach out for the sales part, we work with our own agents to help with that process, and it's been amazing.
It's been both amazing for the experience itself, elevating and of course solving the issue. But second — and you kind of spoke about this briefly earlier — we notice a behavior change where if you know you can speak with an agent, your behavior changes. And let's say you're talking about your use case, you are much more keen to tell everything about the use case because you know you are not judged. You can be a little bit more expansive. You have all the time in the world to tell about the use case, and this has in turn meant that we can provide you a better experience. Even in the classic kind of BDR, SDR categories where you can reach out to ElevenLabs — all usual flow, you leave your details, you can request a use case — but then if you want you can accelerate and give more information by talking to an AI BDR, and then we loop you in to the right person a little bit quicker, and that has seen a great adoption as well.
How big is the company now?
So we have 330 people now.
Everywhere. We have the biggest bases. So we still run it remote, so everybody can work remote, but we started investing into hubs where you can work in person, and our biggest ones are in London, New York, San Francisco, Warsaw, and now we are starting some in Tokyo, Bengaluru, and São Paulo in Brazil as well.
And you just raised at what was it, like 6.6 billion or something?
And you let people take secondaries.
Actually there was only a tender. Yeah. Only secondary.
