Mati Staniszewski co-founded ElevenLabs in 2022, and has since scaled it to the $11 billion leader in AI audio. He's credited it with capturing the humanness of speech through realistic emotional inflection, and they're now expanding into everything from agentic workflows to music. Cheers.
Maybe a good place to start is, describe to me how… I know how an LLM works at a high level. Describe to me how an audio model works. If we were Karpathy-style looking to build a toy one from scratch, how does it work?
In the early days, you try to replicate it exactly like you would replicate it with the human body. You would completely try to reproduce a machine, analogue machine, that would create a vocal tract effectively. Then that progressed into trying to create effectively a digital signal for speech. Bell Labs was one of the first to try to create a structured set of signals that would represent the speech. That is a first precursor to what we would do today. Then you would try to stitch in phonemes, effectively different sounds of how we would speak as humans, and then try to concatenate them together. It's another important part in that equation where you would... Based on the most probabilistic approach of the next word, you would effectively try to bring the phonemes from your library of phonemes and bring them together. Then down to the modern history where now we effectively do similar neural nets in other domains. You predict the next sound based on, of course, the context of the previous sounds if it's a streaming speech. If it's, let's say, a context of audio, you will use combination of predicting of the phonemes, but you also use the contextual text element of that work. Here, credit to my co-founder, Piotr, who effectively came with that new idea of how you can now create voice models which are both reliable, high quality, quick, where you would bring a lot of the ideas from transformer models, from diffusion models, into the speech space. That prediction of the next token in the phoneme space wasn't something that was possible. We spoke briefly about this, of how you can operate on the text, on the waveform space. There's also Mel spectrogram space. Usually, you do text, Mel spectrogram, waveform.
Sorry, what's the spectrogram space?
It's like a visual representation of how the speech sounds across pitch, across energy, and then you transform that into a waveform. When WaveNet came along and Tacotron models, they would effectively use text to Mel spectrogram, so that visual representation, and then how you decode and encode that into the waveform to bring it across. Piotr figured out how to abstract some of those steps and decode and encode them a lot better. That predicting of the next phoneme was one of the big piece. Second big piece was, how do you bring that context into the equation? What I mean by context is, if a voice actor was reading a textual copy, you would know that, okay, this is a dialect sequence. I need to produce a dialect. If it's a happy sentence, I might need to pronounce as a happy sentence. But what happens before and after comes into the equation, and you need to bring that across. Then there's a last big piece. Voice model has the sound of how you intonate the given fragment. But the second big part is the voice itself, the characteristics of accent, of style, of prosody across that voice. When you actually try to vocalize something, when you create that voice model, you turn text into audio, you need the text. You also the voice reference of how you want it to be spoken. Here was the second big innovation. Apart from context, it's how you decode and encode those features. When Bell Labs came with their initial representation of speech, the big piece there was you would have effectively hard-coded parameters for that speech. With ElevenLabs models—
Hard-coded parameters for enthusiastic speaker, British accent, that kind of stuff?
Exactly, that kind of stuff. The set of pitch elements that you can select, set of energies, spectrograms you can select from. In our approach, effectively, you would give the model open-ended ability to select what those parameters should be. It's not going to be British, Polish, Spanish, English speaker, but the model will deduce them themselves. The same for other sets of parameters that are not hard-coded, whether it's the enthusiasm, whether it's the sadness, et cetera.
You're saying Britishness is an emergent property in your voice models?
Exactly. Those two big parts, so it's encoding and decoding of how you create the voice. Super hard problem before, and figured out, too. How you then construct that in a sentence, how you get the context across so you can predict the next volumes, so how you bring them together in a reliable and stable way while doing it quick. These were the two first big innovations in the voice models that continue to today.
But okay, so if LLMs reason about text and word subparts tokens as the way they think about the world, what is the equivalent of a token in a voice model? You mentioned phonemes a bunch. What is that representation?
We start the voice embedding effectively for the speaker. You need that reference when you produce and create the speech. Of course, in the input to the voice model, you still get the text, and you bring the speaker and coding. Then, when you produce speech, you do operate on the waveform or effectively on the phoneme level of that speech. Then when we go the opposite, so of course when we—
Sorry, what is a phoneme? Fill in my understanding.
It's like a syllable deconstructed even to smaller elements. These are effectively the human sounds you can produce. These would be the most close to that representation. But of course, in our models, now it's going to be a combination of not only operating on phoneme level, you also operate on the text level. You operate both in sync because when you are predicting the context, you need to understand how that sentence will get constructed, and especially if it's more of a streaming real-time use case in a voice agent setting, you need both parts to work across. It's similar to how you operate on the token level on the tech side, we operate on the token level on the audio side.
It feels like a big part of the magic of Eleven was your voices were much more human-sounding. How did you accomplish that?
I'll give you a quick synopsis of how we think about the models on the text-to-speech side today. In any model, you need the architecture, you need to compute, you need data. Architecture innovations were one thing. The data part was the second big thing. With audio, you will have a lot of audio data available, but frequently you will not have it annotated in the right way. You won't have which speaker is speaking when. Some of the "what" is annotated, but the "how" isn't. As we are speaking now, what's the emotions that we use? What are the accents that we use? We would invest a lot internally on effectively creating our own data labellers, our own team, to be able to create those data sets that will be better. That was a combination of, of course, semi-automatic techniques and then manual techniques. Actually, a lot of the models that we did afterwards actually span out from a lot of that research, too. Speech-to-text model initially was a model we did for ourselves because the models on the market just weren't good to annotate that data. Then another brilliant research area on our team was being able to construct it so we could span it out as a model that we brought to the customer.
You've just been doing useful stuff in voice, and that has emerged with a whole bunch of products that you might not have expected because you find you're building useful stuff.
Exactly. That's a combination of data, of being able to do it automatically, create a team that's coached on voice, on how to describe it, because most of the labellers out there just aren't as well-versed on understanding the audio and voice, helped us a lot to bring that back. Then, of course, deploying those models in production, seeing how customers interact with them, having them annotate all of the data helped us refine those models over time. A very interesting thing on the side. We spoke about the speech representation. The first guy who created the speech representation is a guy called Kempelen, Wolfgang von Kempelen. He created those analogue machine that would represent effectively a human vocal tract and try to produce that sound. He had to spend decades on that, and that started producing valves. But that was the same person that created a chess machine, the first viral, let's say, chess machine that would simulate playing chess.
Is this the Mechanical Turk?
It was called Turk. But the crazy thing behind it, it was operated by a human, and it was all a fluke. That's where the Mechanical Turk came from, which actually we use in our data labelling production to make that work there.
Sorry, we jumped right in. But if you describe the Eleven business today, people think of you as the speech company. How should they actually think of your business to the extent you can describe the big areas? Text-to-speech, speech-to-text, voice agents. Just break down the business for us.
In a nutshell, I'll describe ElevenLabs as a research and product deployment company. We build foundational audio and voice models, and then build a platform for businesses to transform how they communicate with their customers, with their employees. That will apply through AI agents from customer support, sales, hiring, training, all the way through to marketing and storytelling for our creative tools. In that set, we've created all types of foundational audio models. Text-to-speech models for producing speech, speech-to-text models that work over 100 languages and happily beat others on benchmarks, all the way through to conversational models of how you loop them together to music, to other domains of audio. Then, of course, beyond the models, when you actually bring them to production, that's where the second level of the platform comes in, where that meets the businesses on the specific use case. On the agent-specific example, it would be how you now connect those models to the knowledge base, to telephony, to the integrations that you need to perform the actions, how you evaluate and monitor the agent that it behaves in the right way, how you build the right safeguards. On the creative side, on the marketing side, it's how do you create a good ad so you can create a good video voiceover for one of the campaigns. How you create an article that's narrated with a specific voice that represents the brand in a good way. That's where we combine the models and understanding of the customers we work with into one policy platform.
Every platform company has this question about how far they go into applications. How do you think about where you go horizontal and power the whole ecosystem versus where you develop application? Because you can imagine there being a whole ecosystem of closed captioning tools that grow up that, again, are built on the ElevenLabs tech. It's not necessarily a space that you would have to go after yourself.
I think the big difference between... In your question, today, we see ourselves as a platform where if you're building a horizontal use case in your business, a great place to come. If you have a lot of domain specificity, that's where I see a lot of the application companies forming over time, where that's specifically not the spaces we will go into.
I think it also is interesting when the tech is moving as quickly as it is here. It's one thing with SaaS where you get these vertical-specific providers, but I would imagine one of the biggest risks for you guys in being intermediated is if there is, like in this example, a closed captioning service that is on a two-versions-old version of ElevenLabs and hasn't upgraded. That's a problem because you want people to be using the latest and greatest model that you've developed, and you'll be deploying new capabilities every week. I presume that's part of your thinking, is that just when it's moving that quickly, you need to go direct in a lot of cases.
That's right. In the closed captioning, here already now, we know that our services is going to be able to tackle 99.9% of the cases that customers have. Then there's added benefit of we work with healthcare customers where we will create a custom model for those customers where we'll get that transcription perfectly. The context is the tricky thing in closed captions, where we talk a lot about a lot of technical stuff on this.
That's where you need effectively a dictionary of where you do the tech beforehand, which, as we work with the businesses, we know we need to embed in that creation process.
We're talking a bit about product here, and one thing I note is that LLMs are amazing, and you have the usage stats of ChatGPT and Gemini and all the popular LLMs where they're working and people use them a ton. It feels like there's a big product overhang when it comes to voice, where the leading-edge voice models are incredibly capable. Yet, I was driving home the other day, and I needed to read a PDF, but I was driving. I said, "Okay, I'll just have my phone read the PDF to me." You can try and hack it with iOS screen reader, but it doesn't really work with the scrolling. Then in theory, you can upload it to Gemini, but you're trying to get it to not summarize it, and it actually just hung when I tried to press the Read This to Me button. There was no way I could get my phone to read me something, which seemed like a fairly basic feature. All cars advertise voice control, and yet, it sucks. Separately, if you want to input something to the navigation, just no car has a good version of that yet. Maybe Tesla does, and, and, and... Why does it seem like with LLMs and Claude Code and everything, we are using all the capabilities of the intelligence, whereas with voice, we're living 10 years ago somehow?
Well, I'm thinking would I agree with the premise that we are 10 years behind. In the lived experience of people day to day, they're using Siri's transcription, which has gotten better, and it's still way behind the leading edge. There's definitely a piece of… I think the technology in many of those cases already, there's a deployment gap to what you are saying. It's like an automative for some of the big companies are not adopting that quickly enough or bringing that into the production. But plenty of different problems that you need to fix along the way. The quality of voice models for them to actually sound good, this is only a last three years thing.
Yeah, but it's three years. It's a three years thing. Cars have over-the-air software updates now.
That's three years for the first voice model that can narrate text async. Two years ago, we can start seeing the real-time version of that. I think the real break was a year ago, where you could start seeing that in production. Then I think over 2025, the big piece that hasn't been possible is how you connect now the real-time voice interaction with something which I think you're referring to. It has context of what you want to do, what is the material that you want to read, how does it connect to set off your preferences from the past and get that across. I think that only recently became possible, and where we've seen the big adoption across the enterprises leading on the technical side. I think this year, it should be in the automotive side too, or some of the applications.
You think we'll start seeing great voice models in cars this year?
This year for their own cloud use cases. On car, in car, so without connectivity, not yet. There's deployment, of course, gap, of how you bring that into the gaps. But I think the next two years, three years.
How about the PDF reading use case?
But how should I have done it?
Back in the day... I'll pre-empt this with a story to cue ElevenReader, but we had this problem. We had so many audiobook authors come into ElevenLabs. 2023, we released the first software. We had a lot of creators and then a lot of audiobook authors or book authors that couldn't afford professional narration and wanted to create an audiobook. However, none of the companies accepted AI audiobooks.
You can't sell an AI audiobook on Audible or something?
Exactly. Audible would block AI content. We had no choice. We need to create an avenue for them to—
Ah, because there was no distribution for AI audiobooks.
Exactly. We created ElevenReader, and that came with a functionality where you can upload your PDF, you can upload your text, and have it read out loud with a number of incredible voices, whether it's Sir Michael Caine all the way through to estate and working together with Sir Richard Feynman, where you can have that—
You are working with the Sir Michael Caines of the world?
Exactly. Then you can actually read it out loud.
That works extremely well. That works. Now, how can you do it? Gosh, I do want everything read to me by Michael Caine. It's a great voice. Shouldn't you guys have a consumer app where I can just do the common voice things? I want to be able to have an Eleven app on my phone, and then if I upload a PDF to it, it can do the common things that I would like, such as have it read it to me.
Yeah, that's exactly ElevenReader. That works.
The phone makers allow third-party keyboards. Do they allow third-party transcription engines? Will they, do you think?
The phone makers you said.
The OS makers. Yeah, not all of them. With Android, you can work through it. There's variations of that, Nothing. tech, and others.
I feel like if you had a popular Eleven app that allowed for transcription, people would use it a bunch, and maybe eventually, Apple would say, "Oh, we should allow third-party transcription engines if that's what people want." It seems like they might be going in that direction.
Recently, they announced that it will open up the LLM ecosystem. Hopefully, they will do the same with voice ecosystem, which is similar. I think it's rational to do when it's moving so quickly.
The voice assistant paradigm is one of the oldest UI paradigms in computing, like the, "Open the pod bay doors, HAL," from 1969. I will claim it's not working yet. Siri doesn't have the intelligence. Then on Gemini and ChatGPT and those apps, I want to use the voice mode, but I don't know about you, it just doesn't work. Sometimes I'll be using my phone, and I'll use the iOS keyboard transcription to type in the field and then say a bunch of stuff, and then send it off. But this suggests to me that consumers really want voice mode that works, and yet it's just not working yet for the major LLM apps or for anyone. Why isn't it working yet?
It is pretty hard to do because you want two things. You want to be able to say things that you want, but you want sometimes for it to execute it, sometimes to wait for you to finish and add something in the sentence. Sometimes you want it to be interactive, so it asks you questions back to clarify and get some of the additional detail. All of that is actually pretty hard. That's where the magical ideal version of a voice agent for us comes through, where you need the speech-to-text element, you need the transcription side, you need then the turn-taking mechanism. When do you finish sentences? When is it likely based on silence, based likely on the context? Then sometimes you want it to speak back and clarify, or at least give you the text back to clarify, and then maybe execute a set of instructions. That problem is still very hard. I agree with the claim that this orchestration side has not passed a true conversational agent Turing test, where it behaves as you would expect from another person, where you can say—
That's the simpler way of saying what I'm saying is that we have passed the Turing test with text LLMs a long time ago. We're actually nowhere near that on voice LLMs. It's interesting how that's a final frontier. I feel like it's going to work in specific domains. In customer support calls, passes the voice Turing test. It works well.
Let's take another spectrum of that. An interactive gaming experience, a truly interactive as you would have with another human in that game, it's so hard and further out there, we haven't passed it yet there.
But I think that's a combination of even a simpler version of within that. Sometimes you might give a response immediately back. Sometimes you need a tool call to get additional information from the database, how you orchestrate that. That's probably the most common thing we see as we work with some of the companies out there is you want those systems to orchestrate extremely well where if it's a conversational use case, pretty simple, you can root the agent to speak with it. But if you need to authenticate, if you need to pull additional information from the database, what do you do? How do you handle that graciously? That's where it gets tricky. To that extent, I would say that that's just getting there. We'll hopefully see that. Our goal is to pass the voice Turing test in all those cases or the Turing test for all conversational agents outside of voice, too. I hope we will all be there in the next year or so.
For subscription businesses, a lot of revenue is lost in that last few seconds before the checkout. Someone has to get up and find their wallet, or they mistyped their card number, or they hit an error, and they just give up, and you lose the sale. For a company like ElevenLabs, adding hundreds of thousands of subscribers, even a tiny bit of friction like that, it would really add up. But that's why ElevenLabs uses Link from Stripe. Customers save their details once, and then they can check out in seconds across more than a million businesses with saved credentials. If you want a faster checkout for your customers, you should turn on Link from Stripe.
Are you guys working on personalized voice transcription where it feels like part of the way we're making it hard for ourselves is when I speak to Siri, I have a bit of an accent, and so it sometimes has a hard time understanding me. But my accent doesn't change. It could just get good at listening to John. But my understanding is it's not, it's just running the global voice recognition model. I'm guessing it's the same for ElevenLabs where you're running the global voice recognition model. But again, you have an accent. If you walked up to someone in a coffee shop and said two words, they might have a hard time understanding it because they're not putting it through their Mati Polish accent filter. Where's this going with actually interpreting the person that you know to exist on the other side?
I have a very tricky one to detect. My voice is frequently used in the tests.
You're part of the test suite.
For text-to-speech, for speech-to-text, for everything. It's pretty—
But again, trying to parse your voice in a global model is just making life hard. It's like, have a Mati-specific model.
On the speech-to-text, by transcription, exactly. The big part now that we are bringing in is you have two parts. One, effectively a person or a voice-specific detection, which is true for the accent side, but it's also true for a crowded room. We have an incredible research team that's able to continually do both the accuracy high, but also add things like speaker detection, of course, noise reduction. But then the second part is also keyword detection. There are specific words that you would want to say in those settings that you want to effectively monitor for. We spoke about, let's say I'm going to the coffee shop and order things. The set of actions the coffee shop would expect me to do.
This is information theory. It's like they can just listen out for the coffee words.
Exactly. Then try to match it to the closest proximity. Both things will help. In a setup where you have my voice, perfect. You can decode it and code it on that. If you don't have my voice or even if you want to double amplify it, we already support effectively keyword detection, which is useful for real-time setting and async setting. Back to Cheeky Pint transcription, you could effectively pre-generate that from the previous podcast and look for a set of words that you would use traditionally in that.
You do the keyword detection already, but how hard are the… I want to get superhuman transcription performance by feeding it an hour of Mati audio before it listens to Mati, and then it should be able to do a much better job transcribing. Is that just a really hard research problem?
No, solvable. We think we can roll it out in one of the next versions, which is hopefully in the next month.
You think this year you're doing person-specific transcription?
Person-specific transcription. We can already dial our speakers extremely well. If we are speaking, we can, of course, dissimulate who is speaking when, which is in transcription side, apart from accuracy, diarization is one of the harder problems. We do that extremely well. Now it's going to be effectively what you're saying, fine-tuning based on the speaker that I want to listen to, which we know will be important. In a healthcare setup, it's such an important part. You're in an operating room, you're a doctor, you want to say a command, then you want to really be able to listen to that one person's specific piece. You have a hardware device at home, let's say it's a pilot that helps you control the TV. Here, too, you will want that to listen to you versus, let's say, the family running around.
Or maybe you want it to everyone.
You could decide that, but in many cases, you want to be able to specify that.
It's great because there's still so many unsolved research problems.
Yeah, there's just breakthrough after breakthrough coming in the domain of voice models. How about on the flip side, when it comes to speech generation? In the Zoom "touch up my appearance" feature, I've always thought about that in the context of voice, where... Should you offer a de-accenting filter for voices? Or even there's one podcast that I like to listen to, but the voice is a little mumbly, and I always thought they should put it through a de-mumbling filter just to make the—Or slow it down. Yeah, make the enunciation a little better. But all these things, again, like Photoshopping an image, there's no reason that the… Have you thought about voice-to-voice, basically, rather than voice-to-text or text-to-voice?
Yeah. There are two big parts. One on the speed generation side, similar. So many innovations still there. There's a wider piece, and we released the V3 model that we're solving that for the first time. It's like, can you control speech? You can have the text-to-speech. You generate something that sounds emotionally great. Previously, until the end of last year, effectively, you would rely on the model to decide what's the best performance. You could regenerate it, but ultimately, model decides the best performance. That's where the controllability came in, where we can finally give it cues of, say it in a slower way or change how you deliver the dramatic pause or any cues that you give. To be able to do that, you need the architectural changes and the data that we created over time where you annotated what was said and how it was said, so you can actually train the model to do that. Today, finally, you can have both speed generation or entire voice agent experience with what we call expressive mode, where the agent knows the emotions on the other side. If the person is stressed, it can react and be reassuring, and that's generating an LLM response on the reassuring side and response in that set of emotions, too. That breakthrough was super hard to do. That, of course, stretches to a lot of what you said. It could be some version of speech enhancement, either real-time or in a pulsed setup to change how that's delivered. That's relatively recent innovation, and we know it can still be so much better. The edge cases of how you want to describe it is pretty large. That's one. Then the second part of the question, which is a huge question, that's speech-to-speech models. As you said, our approach, as you think about voice agent, conversational agent, is effectively a cascaded approach. You use transcription or speech-to-text, LLM, text-to-speech, and orchestrates all of that together. Then you have a speech-to-speech, which goes directly from speech, and there's a speech response on the other side.
When we say speech-to-speech, is that the idea that it doesn't go through text as an encoding in the intermediate?
Oh, interesting. For performance reasons? For accuracy reasons?
You usually do it for latency. It is faster to run a model that does not have to transcribe and then generate.
Exactly. It's quicker, but on the flip side, you lose reliability. You lose like all visibility into the parts of the pipeline.
In emotionality, we think you can deliver both on both sides extremely well, and maybe you can make it more controllable, too. Today, we are optimizing heavily on a cascaded approach.
I'm sorry, a cascaded approach is? Is the speech-to-text... Going through the text layer.
As we work with a lot of the businesses and enterprises, they will need that visibility into what happens. They will want to execute certain tasks on top of that. They want a good visibility into each of the steps and great accuracy of all the models. But beyond that, they can abstract away what's the LLM layer, what's the intelligence layer, the integrations are easier in that system. That's where we are betting a lot of the research work of how you can make that great, and we think we can make that great. In speech-to-speech, as you think about maybe more of a companion version of the applications, that's where that will flourish because maybe the hallucinations aren't as important, but the latency is a little bit more, and maybe hallucinations are even a feature. Maybe in the future-future, just to finish that part, you will have some version of combination of the models. That for low complexity, easy models, you will have speech-to-speech, and for higher complexity we will have to cascade it.
I was going to ask about this. You know the way there is research on how the invention of writing changed human brains and just changed the neural pathways in ways beyond
Do you observe that speech-to-speech models think differently than cascaded models? It sounds like they're dumber.
They are definitely dumber. You need a smaller model.
But that's interesting, that forcing models to reason about text. I know they just have much more in there as well, but they're smarter.
Yeah, but it's like, if you are getting speech-to-speech, usually you will use smaller models, so it's still quick.
I see. It's also just a model size thing. But are there interesting differences beyond correlates like size?
What I can say is slightly different to your question. The people interacting for voice and the performance we see for how they interact with the business changes just by nature of interacting with us. A good example, you can contact ElevenLabs and register for your interest, you go through the form, and at the end of that, we have supplemented that, that instead of going through the form process, you can speak with our agent and leave more details. What happened there are two things. One, people were actually much more keen to leave the forms through speaking with the agent, so we would go through the form a lot easier. But second, they would be a lot more open-ended in terms of what the use case are. They would start giving us information about the wider set of use cases, the complexity of the use case. The writing out was tedious and tricky—
This is like an open-ended adventure game. Open-ended. You could ask follow-up questions, you could clarify.
But people were just more at ease and could trust the system while doing that, that it's working. That helped us a lot. Then three, which maybe it's more of a technological barrier, it also works across all languages. Now we have leads from all parts of the world coming in and leaving their details. We did that use case, and now we have a few different companies building their SDR versions of that, too, to help them capture the leads coming in from banks all the way to actually one of the automotive companies that leaves that where people are just more keen to speak through voice.
