Zapier CEO: Why Companies Can't Ignore AI Anymore

You bet. So GPT-4 launches and Zapier calls a code red.

The quality of difference between 3.5 and 4 was vast. If this is the pace at which this is going to move, this rewrites everything. We went from roughly just under 10% of folks using AI daily to now over 50% of the company using it.

It launched the first version of our AI fluency rubric. We're now on V2 of this.

The most important shift that is happening right now is software will predominantly be built and used by an agent and not a human.

That is a good question.

Yeah, I'm stoked to be here.

Sure. So GPT-4 launches in the spring of 2023. Zapier calls a code red. Now what led up to that? We had started working with AI a little bit before ChatGPT had gotten released. Now, ChatGPT was released in the fall of 2022, so about 6 months prior to GPT-4. And my co-founders, Brian and Mike, they had started to play around with LLMs, Generative AI, etc. And, you know, we're kind of having that spidey sense, that tingling sensation that something important was happening here. Didn't know exactly what. And so, Mike was running, he was the president of the company and was running all of product and engineering and he said, 'Hey, I'm gonna step out of that job and I'm gonna go work as an IC because I feel like it's so important to figure this out and I can't figure it out if I don't put my hands on the keyboard and play with this stuff.' And so him and Brian kind of off to the side of the company just messing around with this stuff. And I would wager at the time most people looked at that and thought that's just like weird co-founder pet project stuff.

Sure. I mean in hindsight probably so but I think obviously there was folks that were even more appreciative, you know, like that were working on this far ahead of us. So in some ways we felt like even behind but they were messing around with it. I remember one of the first things they built was this texting bot and so it was a Twilio phone number and you could text it and you could text it questions and you would just talk to it. So I remember being like, hey what should I get my wife for her birthday or I've got this weird top and just stuff like that and it would reply back and it was this fun conversational experience. I think it was like a month or two later ChatGPT launches and you're like, ah of course, this is obvious, that there should be an amazing product around that. But even with the ChatGPT launch, it was not necessarily a code red moment for us yet. It was just like, wow, this sort of felt obvious. Like it felt like the obvious extension of what these LLMs were able to provide. And so, what we did for the company is I remember posting about it in our general channel in Slack and saying, 'Hey, here's this cool new product. I like new products. You like new products. Like, go play around with it. See if there's something interesting we could do with it.' And that was kind of that. I remember some teams started to incorporate some of the concepts in the roadmap and some people started dabbling with it here and there but it was still a minority of the company that was doing much with it. And then of course over the course of the next six months we started to have a handful of moments where we more seriously played around with AI. So I remember we had a customer that built a private app that put AI steps into his apps and we started to see how powerful that was and we're like, wow that's pretty dang cool. We had a customer that built a basically he called it voice to invoice. And so it was a zap where you could talk to your phone and he built it for his wife who ran a gardening shop. And so she would be at clients and say, hey, they want to put this many violets, this many roses, this many tulips and we agreed that the budget was a hundred bucks or whatever. And that would go from voice to a transcript to Zapier to AI to a spreadsheet and then it would eventually spit out a QuickBooks invoice. And I remember seeing that and going like, wow, you could have raised a million bucks for that idea like 10 years ago and he just did it. So that was pretty impressive. And then I remember we had our company summit where we get everybody together in person once a year and we had customers come who were using a bunch of AI talking about this stuff there. And we had our internal teams doing workshops showing off what they were learning. And I remember all those workshops were like kind of standing room only. So, we started to have these moments where you could feel the excitement starting to grow internally, even if we weren't doing much with it. And then the GPT-4 moment hits. I remember it was a Thursday night being on a call with Brian and Mike late at night and we were chatting for I don't know, must have been 90 minutes, maybe even a couple hours. I'm not sure. But the thing that really stood out to us was a few things. One, it was only six months between 3.5 and 4. The quality of difference between 3.5 and 4 was vast. It had gotten so much better and the cost improvements were also notable. Those three things made us think, okay, if this is the pace at which this is going to move, this rewrites our product roadmap, what we're investing in, how we operate everything. Our level of ambition for what we are trying to do at Zapier is not meeting the moment. And so that's when we made the decision to call a code red. And I remember the next morning 7:00 a.m. Pacific, we had the whole exec team on a Zoom saying like, 'Hey, we need to rethink it all.' By Monday of next week, we had started enacting a plan. And that plan had a whole bunch of stuff as part of it. But the singular most important impactful thing that we did, a lot of stuff was just kind of noise and swirl. But there was one thing that really worked. And the one thing that really worked was our hackathon. So we paused the company for a week. Everyone in the company went hands down on keyboards and said, 'We're gonna start building with AI.' Engineers were using the OpenAI APIs. Non-devs were using ChatGPT just to get a feel for it. And pre and post that hackathon, we went from roughly just under 10% of folks using AI daily to now over 50% of the company using it daily as part of their workflow. And so that was kind of our first big moment with AI.

So I do think there's a couple ingredients that make hackathons really good if you do them correct and they're not rocket science. They're things anybody can do. First, you do need to do just a tiny bit of prep. It doesn't have to be a huge amount, but in our case, what that prep looked like was procurement. You can't do a hackathon if people don't have access to these tools. So, you got to make sure that people have access to the tools. Second thing, need to make sure you have some governance figured out. Are you going to let people put PII and data stuff into these new tools or not? So, you need to have that figured out. And three, some very basic enablement. Like how do you get started? What's the first thing you should try? You know, if you do those three things, you'll be fine. Most people are going to figure this stuff out. Like you're going to have smart folks in your organization. Then you got your hackathon. Just let people go. Like just say, 'Hey, your job is to build, go explore.' That's part of the fun of a hackathon is you let people's creativity take hold. But then at the end of the hackathon, you want to do one key thing, which is you need to have a demo day, show and tell, whatever you want to call it, needs to be a part of it. The reason that's so important is you get two things. One, you get accountability. If people know they got to show their work, they're actually going to do it. They're not going to just sort of phone it in. So that's kind of the stick side of it. But the carrot is you're showing your work. And so now everybody gets to learn from each other. So they can see, oh, you did. How'd you do that? Like I didn't know you could do that. That's super cool. And so you get this proliferation of ideas that then people start to take back into their day-to-day work. And that's what you're really after. You know, the reality is even at Zapier, most of the things that get worked on in a hackathon don't directly become like product features or operational workflows or anything like that. Most of it gets thrown away. But the ideas, the concepts, the ways of working, that stuff does stick and lives on.

I think there's a couple ingredients. Some we controlled, some we didn't. Things that we controlled were basically continuing to beat the drum. So we continue to have show and tell as part of our weekly all hands. So people continuing to show off here's my workflows, here's what I'm working on. We have internal Slack channels where people are again sharing new techniques, new ways of working, new demos that are constantly being out there. And so just the more you do that, the more people just continue to see cool stuff, cool stuff, cool stuff. And it just becomes pretty infectious. So I think that's pretty important. It was important to us that we were becoming AI fluent inside the company. And so when we were running up against people who were being successful, we asked why. What are you being successful? What are you doing? Like we were trying to learn from the best. And when we were coming up against folks who weren't being successful, we were like why? Why are you not being successful? And so we would start to port the ideas from the best to the folks who were struggling. The folks who were struggling started to either, for some of them it was just like, hey, they just didn't know how or they didn't know how to get started. They needed encouragement or there was just some other benign blocker that just needed to get fixed up. In some small cases, we also had just AI resistance, people who were just dogmatically opposed to that. Those folks, we said, 'Hey, if that's how you feel, that's okay, but this is who we are, and so for you to work here, it is important that you embrace this way of working.' And so that's a choice you need to make. Do this or you choose a new employer. And so we presented that to them. And like I said, some small number of folks left. And there was many career, we've seen a number of career changes. I would wager this is true across tech. But over the last two years, we've seen more folks than I'd ever seen doing career pivots where they've shifted totally out of tech. And I suspect AI plays a part of that. Where they've gone to pursue some other passion independent of tech, which okay, great. I think that's good when people go do those things. I'd say the last thing that has been impactful that we don't control but is just as important and deserves just as much credit is the technology has gotten better. You know the language models continue to improve and improve and improve. And the more they improve both the language models themselves but also the harnesses, the technology around it, things like that, it also just opens up more use cases. And so things that were hard before become possible now. Things that were impossible before you can actually do and it's hard now. So that has also been just as important at getting more and more people to use them is they just get better and better.

I think I would just come back to just making it a consistent and important part of what we do through hackathons, through show and tell, through learnings, both providing the space and time to go work with the tools but also putting constraints into saying like, hey, it is important that you do this, this is who we are and so there is going to be standards and accountability and things like that that are associated with it. And so you kind of had that carrot and that stick and then you let it bake. Like it does take time, people don't just wake up and be like, aha, I've gone from zero to perfectionist at AI in one day. The more you work with it the better you get like any skill, like any technology.

Yeah. We've measured a lot of stuff. Nothing that we're sort of consistently holding ourselves to a particular goal around. We've measured token spend. We've measured lines of code. We've measured PR releases. We've measured all the stuff that kind of people talk about. You know, I think of the token maxing phenomena that's going on right now. And with token maxing, my perspective is it's almost the correct idea. What I mean by that is it captures a lot of positive things, which is an encouragement to use the tools and to use them aggressively and prolifically. That is probably a good thing. But within token maxing, you have two types of usage. You have your kind of slop cannon style usage and then you have people that are using it to generate this phenomenal learning loop where they're shipping a feature, learning faster, etc. One of that style of usage is way better than the other. And so token maxing mixes those two things together. And so you need another mechanism to discern which is which. And this is where having a leadership team that is steeped in AI is so important to help drive that accountability through the organization because once you get to a certain size it becomes very difficult for one person to go inspect all of that. We have an 800 person org, I can't go inspect every single one of these. I can inspect some, but I need help to really go like, ah, that's really good what that person is doing, more of that. Or hey, you're kind of cheating the system over here, maybe intentionally, maybe unintentionally.

Yeah, I think this was a decision I've seen this decision get repeated and I've been asked about it a number of times now and the reason why we did it is not well understood and so I feel like folks are copying our tactics and not necessarily copying the strategy here. And so I think it's really important to understand what is holding your company back from taking the next step in your own AI transformation. Now, in the case of Zapier, we held off on putting an individual in that role for a very long time. In fact, we were two years into our transformation journey before we said, 'Hey, we want to have someone in that role.' And the reason I held off on doing that is I felt like it was really important that everyone in the company treated their own AI upskilling as a first order priority. I did not want it to be, oh, that's that individual's job or that's that team's job and they're the AI team and we're just doing everything. To me that would have been really disastrous to have just a small group of people who knew how to do this stuff and the rest of us stayed in the dark ages. And so, yeah, I said, 'Hey, no, we're intentionally not going to go do that.' And for two years, that's how we operated. Now over the course of that two years we went from sub 10% people using AI every single day to then 97% or whatever was the stat, nearly everyone. And once we got to that there was new bottlenecks that emerged. And so one of the interesting bottlenecks that I think a lot of folks are bubbling up against today is that you can look across their organizations and you can find many 10x individuals. You can find many folks who are being accelerated with AI. But if you look around and say where are the 10x teams or where even better where are the 10x companies, that's a lot harder to answer the question and say, oh, I see who the 10x companies are, the ones that are generating that much more revenue with that much less cost. And often when I ask this question the answers folks point to is they point to places like the labs or these AI-based startups. And while that's true, when you actually go look under the hood at how these companies operate they aren't actually operating all that different than how old school companies operate. So, I would wager they're growing off the back of extreme product market fit and less so because they've 10x'd their operational throughput through AI. Now, it's starting to change, you do hear stories, the labs are two great places where I do think they're starting to get that payoff loop happening, but it is still even spotty inside of those organizations. So, we started to notice that and we said interesting. I think our challenge is not necessarily a technical one. I think our challenge is an organizational one. Redesign the factory, so to speak. And so much of this was about roles, responsibilities, and where do we insert AI into the loop? Where do we take a human out of the loop or restructure this team or restructure these operating procedures? And a lot of those are people problems. And it just so happened that Brandon and his team inside of Zapier were also doing a really good job at their own team transformation. The people team was pretty advanced both compared to people teams elsewhere, but also compared to just any other function inside of Zapier. And so it made a lot of sense to me to say, 'Hey Brandon, you're doing a good job with this for your organization. Why don't you help the whole company with this problem?' And so that's why we chose to do that. But I could very much see that in another company they might say actually our bottleneck is a technical one. And so we need a CTO or an architect or somebody who takes that position. To me you have to answer this question based on what is the problem you're trying to solve and not necessarily just go, oh, Zapier's doing good AI stuff. They have a people leader in charge. We'll make our people leader be in charge.

The main thing that I saw was if you just looked across a bunch of the ICs in the organization they were all AI-pilled. Like they were all just building tools, rethinking their workflows. They were rethinking how the org should work. All of these things that you as a leader would want from a forward thinking function, they were pretty leaned in. Now, it wasn't universally true across the whole people team, but I just seemed to have a remarkably high number of case studies coming out of that organization that just made me go, huh, something is in the water over there.