This Startup Built the Infrastructure Powering Voice AI

Yeah, I'm excited to be here.

So, we help other companies build voice AI features and applications. Everything from AI notetakers to AI capabilities in contact centers to real-time voice agents. Healthcare companies are building ambient medical scribes on top of our voice AI infrastructure platform. So, we have about a million developers that have signed up to the platform. We have about 10,000 customers. Last year, around 250 million voice hours were sent through our platform, and we're now doing...

Yeah, we're now doing almost 2 million hours per day. So the run rate on that would be, you know, like 700 million voice hours for this year, and it's continuing to grow week over week. So we really are focused on helping other companies have the voice AI infrastructure and primitives they need to just innovate around voice and to build innovative products and features, either like standalone or in their products.

Yeah, for sure. So notetaking is a really popular use case that companies are shipping with our platform. So if you've used Granola or Fireflies notetakers, those use Assembly AI's platform.

Yeah, that's right. Also in the hiring segment. So if you've used Metaview or Ashby's notetaker. Okay. Ashby. Yeah. So notetakers are a really big segment. We're also pretty widely deployed in contact center. So there's a big contact center company, they're called Calabrio. They provide contact center software to big brands like Delta Airlines.

Most likely. Yeah. So right now, product teams are shipping products and capabilities around voice with our platform. But then we're also seeing enterprises. So right now we have a lot of Fortune 500 enterprises that are building AI capabilities themselves within their contact center operations or within their trust and safety operations around voice to automate workflows for their teams. So it's a mix of product teams, but then also enterprise companies that are looking to build with voice AI. So Zoom, for example, has used our infrastructure for a number of different capabilities. Yeah.

Yeah, there were like 10 other companies in the AI cohort.

Yeah, so I started a... I mean, the really early story is I started a company in college, and it was not an AI company, but through that experience I taught myself how to code. So, you know, you wouldn't need to do this now with AI, but back then I bought a bunch of programming books on PHP and Python and Django, and I just read them all and taught myself how to code and started building like SaaS software products and launching them as little startups.

Like, what did you... one was like a fundraising tool for college organizations. One was a tool where small businesses could like put up a QR code and then customers could text them, get a number that they could text feedback to. This was like anonymous feedback capture through text message, bunch of random stuff. And I learned from that that I really loved programming and that I really loved the creative journey of building something and like thinking of this, you had this like fuzzy picture of a product or something in your mind and then you could just, you know, through programming build it and see it come to life and get feedback. It was this very like addictive kind of feedback loop that I really enjoyed.

So after college, I just kept programming and kept, you know, learning about everything from like security to machine learning, and I really got interested in machine learning. So this is like support vector machine time. This is not like deep neural net time.

Eventually I joined a machine learning team out in San Francisco. So I moved to San Francisco, joined a team at Cisco, the company Cisco that's based here. It's actually like a few blocks from where we're filming right now is where my job was in Dogpatch. Yeah. And that was where I got, I started getting learning about neural networks and getting into neural networks. So to give you a sense of time, this is like 2015. So like very early into, you know, the neural network. This was like right when...

I think it was like 2013 or 14. It was around that time. Okay.

But I do remember going to the very first TensorFlow meetup down at Google's headquarters in like 2014 or 15. So it was, you know, that was the state of things at that time. So a couple things happened around that time. One is I saw that AI and neural networks and deep learning were going to take off over the next decade, that it was such early days and data, compute, algorithms were just going to continue to get stronger and stronger, and they were going to bring, you know, really disruptive new capabilities to everything from self-driving cars to natural language processing and, you know, everything that we've seen over the last 10 years. And so I just really felt like, wow, that's an area I was drawn to and wanted to keep going deeper in. And then I bought an Amazon Echo when that came out in 2015.

It was crazy because my experience with voice recognition prior to the Amazon Echo was that it sucked, right? Siri didn't work. Everything sucked. You didn't use it.

But the Amazon Echo, you could use it across the room. So far field, it worked like, you know, great distance from the microphone. Your TV could be on. You could be, you know, shouting to it over a bunch of background noise and it would still work. And what was so incredible was that it worked so well that I found I was building new habits around the product because it was reliable.

So setting timers, asking for the weather, playing music, it was just like... I was blown away by how well those things worked over voice and that it was reliable. And so I started looking into voice recognition technology to just build stuff on my own and to innovate and experiment on my own. And there was nothing in the market at the time. It was either like bad or... so the company Nuance at the time, which was like a big...

Yeah. Exactly. I got in touch with them and I, you know, was trying to find their developer SDKs and you had to pay, I think it was like, you know, thousands of dollars up front.

And then they would mail you a CD-ROM with a developer SDK on it. And I didn't even have a CD-ROM drive in my laptop. So, you know, for me, I was used to the Twilio or Stripe style developer experience, and this was like the opposite of that. So all these like ideas kind of came together for me and I got really excited about the idea of what if you could, you know, use like these new deep learning algorithms, all this innovation that was happening around AI and deep learning at the time, and to build way better voice AI capabilities and technology and then make it super easy for any developer to build with those things. Because for me, I remember getting access to Twilio, you know, back in the day. So like 2012 or 2013 getting access to Twilio...

and I was just overwhelmed with creativity because I was like, wow, now I have like, I have access to this cool technology, me as a college student developer, and like I can experiment with all these really cool ideas. And that has been the whole spirit about Assembly. You know, we're not an application company. We're really focused on being a developer infrastructure tool that just makes it really easy for a developer, whether you're, you know, a college student or a high school student or a development team at a Fortune 500, to have access to this amazingly powerful technology effortlessly and with a really good experience around it. So that was the kind of the mix of ideas that came together and then, you know, is still what we're focused on, you know, delivering many years later.

Some of them.

You couldn't use AI. If you said AI in your... it was deep learning was the thing to say. If you used AI, it was like a scam.

That was the state of things.

Yeah. Exactly. Exactly. My experience as a founder has been one that I think you have to be obsessed with the problem that you're solving because what keeps me going is like I really want this product to exist that we have not, you know, finished creating yet, you know, and I'm obsessed with like that problem and wanting that to be realized. Whereas I think if you're just focused on like a business opportunity, you know, then and you're not obsessed with the product or the subject matter, you can find yourself losing interest. But I think if you're, you know, you should pick... my lesson has been like, and what I share with other founders, like you should pick a problem that you're just obsessed about. Because when I think about like what would I be doing if I wasn't building Assembly, you know, I don't know, like I'm having a lot of fun building Assembly. And I use our product all the time. Like I'm constantly dogfooding it and playing with it and showing people demos and it's fun. So I think that, you know, picking something, working on something that like you want to exist because you want to be a user of it has been really fun to...

Yeah.

Yeah. And the first years were just this like long slog where it wasn't clear that it was ever going to work.

Yeah. Yeah. So I got into YC summer 2017, solo founder. I had pretty much just started working on the company and I got in late. I applied late, got in a week before the batch started and I was just so stressed out because you had all these other companies that, you know, had like real products and they were further along and they could iterate so much faster. But like I picked a very hard product to work on back then and I just remember being, you know, so stressed out and so overwhelmed back in 2017.

You get feedback, you can iterate on it that night, right? Whereas like our early users would give us feedback and it's like, all right, we'll get back to you in a month when we have a better model. And YC is three months. So we had three, you know, iteration cycles. It's not a lot of iteration. And no one was building with voice because like what I have now realized is that this whole ecosystem of technology needed to be created like so that you can build voice AI applications. What do I mean by that? I mean, yeah, you need really good voice AI models, but you also need LLMs, you also need vector databases, you also need WebRTC, you need mobile 5G, like you need this whole ecosystem to build what we're seeing developers build. You know, what I remember like 2021, we started to get transcription models to work like decently well, but the interesting applications were not just like transcribing something, it was sentiment analysis of a phone call, it was summarizing a meeting. But if you wanted to do those types of tasks back then, you had to train sentiment analysis models or train summarization models, right? So whereas now you can use Assembly and, you know, an LLM and like what you can do is it's insane and it's super easy to build and innovate around voice data.

So it took a while for that ecosystem of technology to come together. And I think it goes back to probably what I was saying earlier where if in 2017, if I had put together like a business plan, I would probably been like, I shouldn't work on this because, you know, the market is so small.

Yeah, exactly. The market's so small and all the technology sucks and like why am I working on this? But I, you know, I thought it was a fun problem to work on. And I was really fortunate that back then, so Daniel Gross was our YC partner. He had worked at Apple. He saw the state of voice recognition technology and he was a really early believer. And so he invested personally in the company and really supported the company early on post YC. And we just had these like early believers in the company that knew it was very early. And I think for me, I always had conviction that it was kind of like self-driving cars where, okay, once you... it wasn't a question of like product market fit, like will people want voice AI technology, it was a question of when will it have product market fit, when will it get good enough, like self-driving cars. And I think that in hindsight we probably could have gone a bit faster, but you know, I was learning. And yeah, it took a while, but to give you, you know, a sense, it was like the first couple years, not much progress. We're like three, four people. And then I think we landed our first, you know, real customer in 2021.

Like real legit customer, still a customer of ours. A contact center company. Okay.

And raised our Series A in January of 2022. And then to date we've raised about...

Yeah. So COVID was a big part of it. So during COVID, all of a sudden you had way more voice data created and captured over the internet than ever before. With remote work, podcasts started to get popular around that time. Our models started to get better. We started to use, you know, modern transformer architectures trained on more data. So like transcription was getting better, was getting cheaper. Then you had other NLP models like BERT that made it easier to do summarization or sentiment analysis on top of transcription. So you had like the ecosystem of technology starting to come together where it was easier to build a voice AI application. And so there was acceleration in the TAM around that time. And Accel led our Series A and they saw that too. And then between then and now, obviously, it's just to continue to accelerate, you know, pretty dramatically. We raised about $160 million in the course of like, I don't know, three years or something. Less so or yeah, like three years. So it took a while to get to that point, but then once it did, it really started to accelerate.

So like analyze all the calls that went through this call center for the past week and like...

Bad, bad like NPS scores or things like that.

Pretty much all, you know, use cases on top of pre-recorded audio. And then now, there's still a ton of growth in those use cases. Like usage to our non-real-time APIs is still growing, you know, 200% year-over-year. But real-time use cases are exploding because the real-time capabilities are now way better, way lower cost. You have, you know, ecosystem of real-time technology to build real-time notetakers, real-time voice agents. That market is really taking off, that use case. But back then it was all non-real-time.

Yeah. The very first API I built was a real-time API.

But the real-time models back then, I mean, in general, all the models were bad. Real-time, which is a lot harder, and it's only been, I would say, really within the last probably 18 months that real-time models have started to get like, cross a new threshold. And that's why you're seeing real-time voice AI use cases take off because that threshold has been crossed in the last 18 months across accuracy, latency. And it's still not perfect, right? Like there's real-time speaker identification that, you know, doesn't work that well today. So there's still a ton of room to go. But what I have found is that all of a sudden these technologies cross some threshold where, and it's so hard to pinpoint what that threshold is, but all of a sudden they cross some threshold and there's product market fit and it just takes off. And so, you know, for the first time we experienced that was around 2021. And then now we're experiencing that again. And I would say that's across both real-time and non-real-time again. So some of the newer non-real-time models that we have, they're more like general-purpose like audio understanding models. So they can, you know, identify speaker genders and they can capture like background sound effects and they can just understand and capture audio with new capabilities and that's unlocking new use cases. So what innovative teams can do is it just keeps kind of... what they can do just keeps kind of increasing.

Yeah, definitely like learned a lot of growing pains. And I think I learned a lot through that period where in the early days, in the early days it was kind of just like, you know, no one's paying attention. Stressful in a different way. And then now it's like, okay, there's a huge, you know, opportunity. We really want to be a leader in the space and really like executing against that opportunity. So I would say we, you know, when you raise a lot of capital quickly, I think that puts a lot of pressure on a company and on a team. And I think that like the biggest lesson I've learned is like you just have to kind of trust your instincts as a founder. Because you really, you know, understand like the market you're building in and the right like company-market fit you need and stuff. And so I think like it was hard to know, I think it was hard to know like what were the right instincts to follow when we started to see a lot of acceleration in the company. And that just like took a bit of time, but now I think our instincts have become like more refined as a company and then me too as a founder.

Yeah, so there's a couple of very exciting things we're seeing. So, real-time voice agents are like, they work really well now. It's kind of wild. Yeah, real-time voice agents, they work really well now. So I think those are being rapidly deployed because the success rate of a real-time voice agent is high enough now where they can deliver a good customer experience over customer support, you know, frontline like reception tasks. And I think as a consumer, I don't know, have you interacted with one yet?