Zapier Made AI Fluency a Hiring Requirement. The Rubric They Use | CEO of Zapier

We looked at that and said, "Holy cow, we need to rethink our road map." And candidly, this could be a threat as well, too, if we don't react to it.

Yeah, thanks for having me, Sophie.

We started the company in 2011 focused on integrations specifically and we quickly realized that folks wanted more than integrations. They did want workflows but when we launched workflows in 2016, you know, it wasn't exactly obvious all the different ways that people would use it. And I remember one of our very early power users of that feature, this guy in Australia, he messaged me and was like, "Zapier is incredible. I had built an entire business and like application on top of it." And I was like, "Oh, that's interesting. Tell me more." And it turns out he had built basically like gag apps. So one was called Seinfeld quote and the other was called Kanye text. And effectively what you could do was you would put in like a friend's phone number and then he had hooked it up to Stripe and Twilio and basically you could buy like a smaller, medium or large package and depending on what you bought it would basically spam your friends with either Seinfeld quotes or Kanye texts. And it was like all built on top of Zapier. I was just like holy cow. Like I had not anticipated people building like full-on applications or like businesses on top of the platform in that way. And so that was a very early eye opener. It basically was like oh we're just thinking way too little about what this thing could become.

So that would have been around when we launched workflows. So that would have been around 2016 or so, which was like 5 years in. You know, it took us a while. Like integrations was like such the focus for the first, you know, couple years, you know, probably about 2 or 3 years in, we're like, we got to go figure this workflow thing out. And, you know, it took a year for us to get that, you know, product ready and shipped.

Yeah, certainly like everyone uses agents to call it the superset for everything. I tend to think of agents in the difference between agents and workflows on a spectrum. A workflow is deterministic. It behaves the same way every time. It's like a computer program. You say, "Hey, I want it to do this, and then I want it to do this, and then I want it to do this, and I want it to do this." And then you're like great. Then the computer will go execute it the same way every single time. And so you get like reliability, you get cost, you get all these sorts of things. And so that's what a workflow actually looks like. An agent to me is different. An agent is something you give a goal and you give it instructions on how to complete that goal and then maybe you give it access to data or tools or something like that. And then every time you know the agent has the opportunity to go complete that goal, it goes and figures out how to do it. But it has its own logic, its own reasoning, its own way about going about it. It might be different than the way you would go about it. And as a result, you can give it a wider variety of tasks because you don't need to know all the possible ways that it needs to behave to go solve the problem. But you're also trading off against reliability, cost, things like that, too. And so to me, agents, workflows, they sit on a spectrum. And you might want one for a particular job, and you might want a different one for a different type of job.

Well, I think obviously like agents is blowing up in the age of AI like there is so much more use cases that have been unlocked because of the ability to work with AI and so that part of the puzzle is where all the momentum is today. That said, what is interesting is that there is also been a renaissance of workflows as well too. And I think the reason why is because a lot of folks maybe have like a mental block or had just sort of said, "Oh, I can't solve like this is not for me. Like I'm not an engineer. I'm not tech off. I don't know how to go solve this problem." And so when you go sit down and you know talk to people now, they have this appreciation that AI exists. And so their mind's like, "Oh, what problem can I solve?" And they start saying, "Can you automate this? Can you automate that? Can you automate this?" And honestly, like nine times out of 10, no AI required. It's something that they could have been doing all along, but they just didn't realize it that that was something that was available to them. So, in a lot of ways, I think AI has been this like catalyst for all of us to be more generative with our ideas and say, "Oh, there are like and get more excited about the automation potential." And the irony is, of course, much of that stuff we could have been doing all along.

The key insight that we had was we had seen the ChatGPT launch in the fall of 2022. You know, I think a lot of people that was like whoa moment for us it was like this is cool but it didn't yet like cause us to like monumentally change our behavior. Instead we were you know we shared the product launch in our Slack channel in general. It was like hey check out ChatGPT. This is a cool product. We like cool products. You probably like cool products. Check it out. Some of the ideas in this might be useful in our own product. So if you know you're a product manager, an engineer and like you're trying to think of like cool interesting like problems you can solve in automation on behalf of our customers, this is another tool in your toolbox like you know go think about using it. And that was kind of it. Then over the course of you know the next like four or five months there was more areas where we just started to bump into users partners doing interesting things with it where that sort of like organic you know more casual adoption we started to just get like a little more forceful a little more forceful a little more forceful here or there and that all culminated though with the GPT-4 launch which happened in the spring of 2023. The reason that set off the code red for us was that one, it was only six months between 3.5 and 4 and 4 was a vastly superior model to 3.5 and the costs kept going down. And so we looked at that and said, "Holy cow, if this represents any sort of trend for the future, we need to rethink our roadmap. We need to rethink our internal operations. There is a monumental opportunity for the companies that jump on this. And candidly, this could be a threat as well too if we don't like react to it." And so I remember a Thursday night call with my co-founders where we were just like trying to figure out like what are we going to do? What are we going to do? What are we going to do? And ultimately we called the exec team in at like 7 a.m. the next morning and we're like, hey, here's what's on our mind. Like this is a code red moment. We need to go figure out what that means. Now ultimately we had never called a code red internally. So we didn't actually have a concept of like well behaviorally what does that mean? And so we had to go figure out what exactly do we mean by go red? Like it sounds serious but like you know beyond just sort of everyone like throwing their hands up in the air and being like ah let's figure it out. Like what is this practically?

And we did a bunch of things in the like days and weeks following that. But there was one thing in particular that like really stood out as a very effective technique and that was the following week the following Monday we kicked off a companywide hackathon. So not just engineering but you know marketing and sales and HR and finance and all these functions and we said hey we want you to go build with AI and you know for engineers that probably meant building features with you know the OpenAI APIs for you know our sales team or our marketing team that might be just using ChatGPT and like trying to have it do research for you or do tasks for you or things like this. What we saw was over the course of that week, it was like a lot of people just like got a lot of experience using these tools and we did a show and tell at the end of it, a demo day where people showed off. There was a bunch of learnings and all that sort of stuff. And the culmination of that was that pre that hackathon week there was probably just under 10% of folks using AI like daily for their work and after it was just north of 50%. So almost, you know, 40% of the company, you know, went from very little to no usage of the technology to, you know, now using it pretty frequently. And so that was probably the single most effective thing that we did to jumpstart acceleration of how we think about AI inside of Zapier.

I so since that time I have talked to probably hundreds of companies founders CEOs executives and the singular most effective tactic that I've heard is some concept of a hackathon builder sessions lunch and builds like workshops like you hear that repeated over and over and over again amongst the companies that are seeing the most success. It makes sense to me like you see a very different mentality between the people who are using the technology versus the people who are pontificating on the technology. The folks pontificating about it, there's a lot more fear-mongering. There's a lot more sensationalism. The people who are actually putting their hands on it like they have just a like a richer understanding of where this is a powerful technology and how it can help them. They understand the limitations and what it is not capable of, where their role is. And like the thing I end up just coaching people on is like AI is a it's a tactile thing where it's like you benefit so much from using this stuff. And so I recommend it wholeheartedly and I think that is not just like I wouldn't say Zapier is an n of one here. This is like an n of you know 100 plus where I've seen now that this is a very effective technique.

You know, I think we had just seen a couple things. We'd invested so much in just providing opportunities for folks to learn and have space to kind of figure out how to use the technology. And as we'd seen that, we definitely were starting to see that the folks adopting it the most were almost like these super contributors inside the company. Like their productivity was sky-high. They were able to do things that they couldn't do before and they were providing leverage for themselves, for their team, for the company in all sorts of interesting ways. Also we had seen you know adoption of AI go from 50% post that first hackathon to now I mean I think by the time we announced that we were like at 97% or something like that. So basically everyone in the company was using it regularly and so the thinking there was like hey if you're not using this stuff coming into the company like you're just going to be behind inside of the company like Zapier. So at this point in time this is kind of just a job requirement to be successful inside of a company like Zapier. The second thing was it was an important signal for our employees but also to the market where it was you know we want to be known as the type of company that is going to invest in this technology and invest in like the future of work and so we wanted our employees to know that like hey we are going to be on the cutting edge and so to the extent you're fearful about your skills continuing to be modern continuing to be relevant like you don't have to worry about Zapier being a place that's going to support that we are 100% going to stand behind you support you in like making this transition to this sort of new AI world. And just as importantly, we wanted people out there in the world who were maybe in companies where that wasn't true, but they were excited about working this way, maybe doing this like in their spare time to say, "Hey, come work at us. Like we want you here like this is going to be awesome for you." So we said about that as well. I think the biggest challenge was we had to just define what the heck does AI fluency mean. You know, I think a lot of companies at this time were struggling with this where they wanted more AI adoption in fact I'd say companies even today still are struggling with this like they want more AI adoption inside the company but they don't have a good way to talk about it like what does it practically mean what's the rubric you know at the end of the day to say is this person good or bad and so we just set about like a very simple exercise which was we went to leaders in every function we went to the power user in every function and just said hey what are the things that you feel like are baseline level skills for people working with AI in your function. So if you're an engineer, you need to be able to do this. If you're a designer, you need to be able to do that. If you're a marketer, you need to be able to do this. So on and so forth. And you know, we said, okay, what's sort of like the basic things like what would be the type of stuff you kind of just expect people are doing these days? What would be the type of thing that we go like, oh, you're kind of ahead of the curve? And then what would be something that you'd be like, holy cow, you figured that out? Like we haven't figured that out yet. Something that would just like really surprise you. And so we just made like a simple rubric and said, "Hey, this is kind of the, you know, not acceptable, acceptable, adaptive and transformative." And we put that out there and said, "Hey, this is what it looks like." You know, I think alongside that we said, "Hey, we fully expect this rubric to change." It certainly has a year in because the technology moves very, very quickly. So, you know, it is an important exercise to do this, but it's also just as important to recognize that the slope here is like more important than any, you know, dot along the way because of how fast the technology is moving.

So, in effect, we've created these based on it's like any other rubric. Like if you were trying to hire for, you know, like a content marketer, it's like, well, you probably want to see that they're good at writing. You probably want to see that they have good ideas. You probably want to see that they, you know, have some understanding of like search or like headline writing. Like so you kind of know like, okay, these are sort of like the skills that are required to be good at this job. You've probably worked with people before where you're like, oh, this is the best of the best. This is like pretty good, this is like not very good, and you just create a rubric out of it. It's not unique for AI. Like at the end of the day, you do the same thing. I think where I see folks really struggle with is you mentioned like leaders struggling with this. I actually think leaders are one of the biggest reasons like companies are held back because a lot of these leaders are not putting their hands on the keyboard and not using the technology. And so they're asking, "How do I create a rubric for this?" And the reason they can't do it is because they haven't used it. So they don't know what does good look like, what does bad look like? And you know that is where I find like so many leaders are in a bad position themselves because they're running a dated playbook. They sort of grew up in this era where they had once upon a time their skills were fresh modern best of the best but now they've sort of like you know grown into this management job and they've kind of lost touch with their craft and the craft is changing like how they do the work is actually changing and those folks have not gone back and stayed relevant on this. I grew up with a bunch of folks in the medical field in my family. And if you're a doctor or a pharmacist or a nurse or any of these people, you're always doing continuing education. Like you always have to go back and do these things because there's, you know, maybe there's a new like technique for surgery or there's like a new medicine that comes out that sort of impacts these things. To me, these are like, you know, folks in the medical field who never did their continuing education. And it's like, well, of course you don't know how to like stay fresh on AI because you're not putting in the work yourself. And I would just caution if you're a leader in one of these positions like you got to go figure out how to rectify that because you're not going to be able to lead your team effectively. If you are not yourself being a user of this technology and it doesn't mean that you have to be like the world's foremost expert at this stuff. You just got to like start and play around with it and you'll get better just like anybody else.

I think there's a couple areas that we see the most like one we've pushed it into our product in so many different areas. And so it's not just like that our operations are different. It's that we are building AI capabilities for our customers. And so when you look at the growth curve of our users who are using our AI capabilities versus the ones who are not, it's like a night and day difference. You know those like those cohorts are growing just vastly vastly faster than any other cohorts. And I think it just speaks to the power of the technology. It's like this is a really powerful way to work. Internally there's sort of two ways that I think about this. One is easier to measure the ROI. One is much harder. So the whole like 100% of people are using AI daily. I think of this as like a floor raiser. This is you know let a thousand flowers bloom. Everyone is using it as like a personal tool to help them with their day-to-day. If you were to ask me if you ask our CFO if you ask any of these people like what ROI is there on that? It would be I would be hard-pressed to tell you. I know that there is because I see it in my own work. I see it in other people's work. But if you ask me to measure it in a very concrete way, I just don't I'm not even sure how I would like go about that problem. And so there is just like a leap of faith there where I'm like I know this is working, but it's hard for me to sort of point to like the ROI impact of that. Then you have like your ceiling raiser projects. These are places where you can look at a very concrete business result and say like this is working. So for example in our customer support team some 50% of our customers requests are now being fielded by AI and those folks get one much faster response times but they also for those set of questions cite higher customer satisfaction as well too and so that's one of those things where you can very like concretely measure hey this is working it's better than the way that it was working before and I think that's the thing when people ask like how do you go about measuring AI productivity The answer is like we you don't really measure AI productivity. You measure your business outcomes. And so generally that's like are you improving like sales throughput or conversion rates or are you you know decreasing the cost to serve in some way somehow like all the sort of KPIs that you probably measure today. Those are the things that you should be seeing improvement on if you really are deploying the technology effectively. And so if you're trying to look for how do I actually go impact the business, you probably have to start with those KPIs and work backwards and say, you know, what are the bottlenecks that are causing us to not be like more ambitious with these goals and then figure out, okay, can AI like meaningfully change the fundamental unit economics of how that works in a particular way.

I would say the point is that there is like a little bit of play that has to happen first. Like it's really hard with something new to just wake up and go, okay, if you cannot show like monumental business results, I will not let you touch this, you know.