Switching to English now because we're going to be talking about all of this with ElevenLabs CEO Mati Staniszewski. Welcome to our show.
Thank you so much for having me on.
Thank you. We know ElevenLabs mainly for text-to-speech and transcription, but you've grown far beyond that. What's your core business, if you should describe that to our listeners?
So as a company, we uniquely combine both research and products. So we build foundational models for everything across the AI voice space and then build products around that to help solve how we, people, humans, interact with technology around us. And that's the core mission of what we want to solve. We believe that it's finally the time with AI development where you can adapt technology to the way we want to communicate, and that's what we want to solve. So, the biggest part of our business today is building, effectively, models for interaction. So, combining the transcription, the intelligence, the generation of the speech and how you orchestrate that all together.
And then how you bring that into the enterprise context as voice agents. So, how we can help across customer experience, how we can help across communication. And that's the biggest part. So, today we've already seen so much of that potential happening here in the Netherlands with some of our customers and partners like KPN, where we are solving that across. So, that would be the biggest part. I think, as you mentioned in your opening introduction, it's also a great timing for us as we've seen a huge adoption across enterprise with the recent round announced just yesterday, where so many companies are building voice agents, bringing voice agents into production across the world.
Yeah. Now, I already mentioned we could describe you as the highest-valuated European AI company. To what extent are you European? You're from Poland, if I'm correct. You have your headquarters in the UK. You also have a US presence. You now have an Amsterdam presence. What's the situation, Mati?
It's, you know, it's... I like this question because it's... We as a company, when we started, our aspiration, and even today, we would describe ourselves as a global company. We always wanted to build for global markets wherever you are. And we never kind of tried to think about the category of like, you know, where do we feel like we need to say we are a specific location. But the truth is we started in Europe, continental Europe. We have the biggest team today in Europe. We have our headquarters in London. And still today, out of 800 people, close to 500 are across Europe. We are planning to open locations hopefully in most European countries and show what we can do and impact globally. Of course, the US is a big market for us. We have our office and entities there too, and that's continually growing. But if we can contribute back not only to talent but also to the ecosystem, for grants, for impact programs, for investments, we would love to do it. I would love for Europe to show what they can do. Of course, you know, I'm extending away from your question, so interrupt me.
So, but you know, I think the big question is like, what can we do as Europe to really innovate and bring it to another stage? And I think there are good ways of us actually showing that. And I'll make a specific case.
I think infrastructure, energy, GPUs are a hard problem. Developing models outside of voice models is also a tricky, tricky problem for that space. But where I think we have an incredible opportunity is the applied AI part of the equation. Europe, across most of the countries, has incredible social systems, whether it's healthcare, whether it's education, whether it's benefits
that are distributed to every person. Now, the benefit of those is that you can actually distribute the AI in an incredibly easier way and diffuse that across the economy in a way you just cannot do in most other economies.
I think we should be leaning into that. And maybe as an extension to your question, to what extent do we feel European? We want to innovate in Europe with that technology too. That's why our work stretches not only to businesses but also with governments across the European continent.
Right. Let's dive into the tech, Daniel. That's something you're very much... Yeah, very interested in. You see the frontier AI labs right now pushing out models. I think the interval is 10 days now between frontier models. Does ElevenLabs have a similar increase in speed, pushing out new models? Because V4 is already a new step up. How do you see that?
I think the iterations in voice models to truly make a step change take a bit longer. You've probably seen since the beginning of the year, and we spoke a little bit, of course, before the show about the technology, and you probably have some feedback about what you've seen in the space too. But there's a lot of voice models that are being developed. But a lot doesn't mean a lot of good ones. And that's, I think, the one piece that we are the most proud of is I have a brilliant co-founder, Piotr, who is my best friend of 15 years, who's been able to both build the models and assemble, I think, the leading research team in the world for everything around voice. And that started in 2022 with the first model, V1, that was at the time the first model to cross the uncanny valley of speech. Turn text into speech, get the right context, intonation, emotion, and finally get the content and story in a way that you would actually like to enjoy listening to it. Fast forward to today, to this week literally, we are releasing our latest version, V4. And once again, it's a model leading on benchmarks. It's leading against Google, OpenAI, any of the startups that are building, and not leading by small percentage points. Leading by a significant leap, almost 10% over the previous part. So, less frequent step changes, I would say. But when they happen, they still matter. And as we saw this week, you can still build something magnificent.
Yeah, but so I'm very curious, what is your edge then? Because the companies you just named, Google, OpenAI, are way bigger than you with probably more researchers on this topic as well. What's your edge?
Well, one is definitely Piotr and the research team behind it. So that's an amazing part. You know, I think the first piece on the research side in voice today, maybe first across any AI model, you need compute, you need data, and you need the architecture.
A lot of the big companies will benefit from the first two, the compute and the data scale. In voice and audio, the architecture problems are still some of the biggest problems you need to solve. So you don't need as much, you still need some good amount of resources, which we have, but you also need extremely good ideas of how to reconfigure the model, and that's what the team has been able to do, and that's the edge. So just the excellence of the researchers on the team and having the best ones is what matters.
Now, second thing on research, and I'll add a little bit more real quick on research, is now how can you feed back the cycle from what you see in the field to improve the models too? I have actually a very specific example here in the Netherlands. We worked with KPN.
Pronouncing Dutch or transcribing Dutch postcodes, tricky challenge. Most of the models fail on this. So we worked with them to create a specific transcription model that improved it from roughly 60% accuracy to now over 82% accuracy that the transcription is happening, and that's much better than any model. So that ability to bring the feedback from the local field into the models is extremely important. And then, as you ask about the longer-term advantage, research is a component, but as we think about our value, we combine research with the product, how you bring not only AI, but how you bring it into production, how you bring that applied AI part of the picture, how you combine it with your systems, how you configure, how you connect to CRM or Ciscus or the telephony work, and how you deploy it. And then over time, and the last one is of course how you can build an ecosystem around this of people contributing their assets, contributing the simulation tests, contributing the guardrails. So people building for it can not build from scratch and benefit from that ecosystem.
Can you shine a little light on the balance between all these? Because you mentioned a large part is research, there's compute, there's data which needs to go in. Where do you lean in most to get that competitive edge, as Daniel just tried to sketch?
Yeah, that model is the key piece. The model is the key piece, and then second, in audio, which is slightly different to other spaces, you don't need as much the data part, you need the high-quality annotated data. So we invest, we have over 1,000 people contracting with us which have been trained by voice coaches to annotate that in a specific way. Because the usual labeling efforts don't work. You need people that not only understand what is being said, but how it's being said. What emotions are used? What accent is used?
You're cultivating and making this bespoke data. You're not just grabbing large sets of spoken data, you're really ripping off books.
Yeah, no, exactly. You need very nuanced... in an amazing way, the voice carries so much nuance. It carries different, I mean, emotions and social norms.
Exactly, it's the social norms.
The emotions that you pass through, you cannot pass them through text. It's a much more rich way of communicating. Imperfections, the intonation, all of it matters. And to be able to capture it, you need people who know how to capture it, and you need to train it. And that's where... So, we have the model, then the specific automated data and specific categories of work too. And of course, that's just the research, and the real advantage is how you bring the research into the product and how you deploy the product internationally.
But I think you do 49 languages now, right?
90? 90? Okay, sorry. I'm sorry. 90 languages, but for all these 90 languages you all have to get into the nitty-gritty details, get that annotated data for every minor detail that's unique to a certain language.
100%, you need to. And that's, you know, what we would love to do is solve interaction between people and technology in every part of the world. And the only way this is possible is if you really spend the time on the data part. But, you know, maybe an interesting piece, so you might notice we started the company, I'm from Poland originally. In Poland there's a very peculiar thing that if you watch a movie, a foreign movie in Polish language,
effectively the way it's dubbed is you have one voice narrating all the voices.
All right, we don't have that here.
So, and it's crazy because whether it's male or female or age or person, you have one flat voice.
And when we started, we wanted to break down the language barrier, that then evolved to breaking down the full communication barrier. We created the model in a way that's architected to work across languages.
Which is a different way of doing it. Before, people used to do just English and then figure out how to extend it to the other ones.
And in our case, we built it from scratch so it can account for that nuance that you mentioned. And today, thankfully, we can do multiple languages, the nuances, whether it's Catalan in India or Odia in India, it's all of the different parts that are needed.
Yeah. Now, doing a voice in any given language is one thing, but having a person's voice specifically is the big one. Actually, the most famous example, slightly famous in that, is our colleague Arjen Robben, who's also a talk show host at BNR, approached you when he had vocal cord issues, maybe you know the case. There we go, and he made a clone of his voice which he was able to use to communicate while he wasn't able to speak. So we were already aware of how this can benefit people in such a position. At the same time, Daniel cloned my voice.
It was remarkably realistic for only 10 seconds of audio. How are you able to approach a person's voice so specifically?
Yeah, so there are those risks and that balance of how you can use it.
be misused by certain people.
Exactly. So you need to be able to build a technology, and we take the responsibility of how you use that technology too, and how you can build the right safeguards so it's deployed in the right way. The first piece across all the audio that you generate is watermarked. So, what this effectively means is that if the audio is generated and brought into the public, there's a hidden watermark that allows us to understand and trace back who generated it and how they generated it.
This is in the file properties then, or also in the audio?
In the audio itself. So you cannot just...
Can be analyzed by police, for instance?
We can assist the police, yes.
