We looked at those three things and said, "Holy cow, if this continues with any sort of pace at all, this changes the entire industry." And so, we felt like we needed to take a quick breather and say, "Okay, we got to reset roadmaps, we got to reset how we think about our vision, we got to rethink how we think about automation." Hence the code red.
Welcome to the Talking AI podcast, where we talk AI with both experts in the field and early adopters. I'm your host Matt Page, and we're here to demystify AI for you, so you can get some value from it. Let's talk some AI.
Zapier has more AI agents than it has employees, and across its platform, nearly 600 million tasks have already been automated, with the number climbing every second. Literally, there's a running counter on their website. And Wade Foster co-founded Zapier in 2011, and he has led it ever since, turning a scrappy Y Combinator startup into the $5 billion plumbing of the SaaS era on barely a million dollars raised. Now comes what may be his most exciting chapter yet, deliberately disrupting his own company before AI does it for him. Welcome to the show, Wade.
Yeah, thanks for having me.
And I'm excited for this discussion. Zapier is one of those iconic names in the space, and you've run it for 15 years now. But I want to go back to March of 2023, when GPT-4 first dropped, and for the first time in company history, you issued a company-wide code red, and effectively shut down the company for a week. What was going through your head at that moment in time? I think it was a seminal moment for everybody, but I think few leaders took it to that extreme, and really saw what was coming.
Yeah. Well, the GPT-4 launch for us was pretty eye-opening. We had seen, obviously, ChatGPT launch, we'd played around with it. The product is fantastic. Really enjoyed using it. But it didn't really create this crazy sense of urgency inside the company, at least not yet. It was, "Hey, this would be cool. How can we operate better with this? How can we make better products with this?" It caught our curiosity.
The GPT-4 launch, which was about 6 months later, had a couple characteristics. One, 6 months later, it was pretty quick. Two, the difference between 3.5 and 4 was pretty big. The capabilities had just advanced a lot. And three, the cost curve was meaningfully coming down. And so we looked at those three things and said, "Holy cow, if this continues with any sort of pace at all, this changes the entire industry."
And so we felt like we needed to take a quick breather and say, "Okay, we got to reset roadmaps, we got to reset how we think about our vision, we got to rethink how we think about automation." Hence the code red.
And a big part of what we did around the code red was, you know, we did a lot of stuff, but probably the most impactful was the hackathon, where we paused the company for a week and we said, "Hey, everybody, just go build. If you're an engineer, play with the APIs. If you're not an engineer, then go mess around with ChatGPT." That was really the main tool at the time. And just get a sense of, "What is possible? What is coming with these things?"
And we saw our daily usage of AI from our employee base go from about 11% of folks using it to over 50% in 1 week. And so that really was kind of like that jump-start that we needed to say, "Okay, something important is happening here."
How was that received? You literally, essentially, stalled the company for a week. Like, I got to imagine some people were like, "Yeah, okay," and weren't taking it as seriously. How did people react to that?
Yeah, this is 2023, right? The AI frenzy is just starting. It's not in peak fervor as it is now. It was a mixed reaction, honestly. I think there were some folks who had already been using the technology who felt like I was behind. They were like, "Come on, Wade, we should go faster. We should go faster. We should go faster."
There was definitely folks in the company that felt like this was unnecessary, it was creating chaos where there didn't need to be any. I was called sensational and things like that. And the way I think about all of that was just it was a moment where I had to really help educate the company, really had to be clear about what I believed and what I felt was coming, to help people understand why this was such a critical moment and it wasn't just sounding the alarm bells unnecessarily. And so it definitely was like a moment of just, "Whoa, what's going on?"
So you got people excited. You got up to 50% usage. But I feel like after a lot of those you have the excitement and then human nature kicks in. You go back to how you did things. How did you actually get that to stick in terms of—
habit formation, which I think is one of the biggest undervalued things of this entire transformation? We're creatures of habit at the end of the day.
We like the way things are.
Yeah, I think there's a couple moves that help it stick. One, you got to just keep doing show-and-tell. So in our all-hands we would have show-and-tell where we'd show off, "What are you building with AI?" And it's not just engineers. We just have everybody from across the company, myself included, that would just show off things that we're doing.
And people learn by seeing. You'd see someone do a cool thing and you go, "Oh, I should try that out," or, "Ooh, wow, I have that exact same problem." So just the act of seeing people and getting exposure to it, you start to feel the art of the possible.
You periodically step back and run more of those hackathons, so you give people more space to come back in and pick up where they left off, see the new capabilities. That's one of the fun and exciting things is that these models are constantly getting released—new models that have new capabilities, new powers. The application layer is figuring out how to do more things with them. The harnesses are getting better. So, there's this pace of improvement that is pretty invigorating because the things you can't do today, you are very likely to be able to do not that far in the future. And so, you just got to get people in the habit of keep trying. The answer isn't, "Hey, no, it's not possible." It's, "Not yet. It's not possible yet."
Quick break in the pod. I keep hearing the same pattern with companies that talk to. Claude's helping employees move faster, but in many companies the business itself hasn't changed. The value's still trapped in isolated chats and experiments. And that execution gap is why forward-deployed engineers have become one of AI's most talked about deployment models. They embed with your team instead of advising from the outside. It's also why the FDE model is now central to every client engagement we lead at Hatchet AI. As an official Anthropic partner, we embed Anthropic-certified FDEs to identify high-value business problems, build and deploy the solution, and put governance and security around it, then transfer the capability back to your team. If your Claude rollout's still mostly individual usage, check out how Hatchet AI FDEs work at hatchet.com/claude-fde. You can also find it in the show notes. Now, back to the show.
And in most of these areas, you just keep the exposure up, keep giving people places to do it, keep sharing, and along the way, the momentum starts to build. It's like a snowball.
Yeah, and you mentioned the application layer. There's different layers of the stack. You got the frontier models, you have the application layer, obviously chips and things like that. How do you think of the level of disruption, moats, differentiation, all of these things? Cuz literally Zapier is a workflow automation tool, right? And that's one of the things that AI is best at, and people can go into Claude or Codex and have it do things. How do you think about differentiation in this new era? Cuz you're obviously not going out and building a model, but you're leveraging models in a major way throughout your entire platform.
Yeah, you know, I think a big part of it is you have to think about what are the things that the frontier of the models, what are the things that they can't do or they want to? That's a really important question. Sam and Dario and the leadership teams at those companies have been pretty forthcoming about the places that they intend to invest and the places that they sort of want the ecosystem to invest in. And so you like to pay attention to that.
And then you really just try and understand what are the things that are really valuable to customers that those companies can't fulfill. So for example, most of these folks are pretty nervous about vendor lock-in and they want to have the best capabilities no matter what model company it is. And so, a lot of these application layers have model routing baked into them or have exposure to different tools. Zapier does as well, where you can pick and choose from the best. You can allow Zapier to sort of figure out what the right tool for the job is.
You're trying to figure out what are the capabilities that is not best served by AI. So right now there's a lot of token maxing going on and it's in the best interests of these labs to have people continue to do token maxing, but it turns out there's a lot of workflows that you benefit from having deterministic workflows that run on code that are backed in a reliable way. Those are things that are not possible, or you have this hybrid agent workflow setup and you want something like Zapier to sort of manage those processes end-to-end.
And so, a lot of this is just trying to think through where are the places that customers really value and you're just not going to get it from one of those labs, and how do you build a great product for those types of use cases that are going to be really important to a segment of the market.
Yeah, there's several rabbit holes I want to go down here, but you triggered one thing: the deterministic versus probabilistic nature. And Zapier's put out this benchmark called Automation Bench, which is effectively this leaderboard. I'd actually like you to talk through what it is and what it's scoring, but I think it hits at that point at which when a model is executing on a workflow, it's not perfect all the time. Even when you're using some of the best frontier models out there—even I see a GPT-5.6, which the day of this recording just launched about an hour ago or so, it's listed on there. So, what's the purpose of Automation Bench? How do you think about that? And what's leading this benchmark?
You bet. So, Automation Bench is a benchmark that is used to test these various models on end-to-end workflow execution. And we have a whole set of tests that we test against, and it's across six business functions: sales, marketing, operations, support, finance, and HR. These are the types of workflows that you or I, any white-collar worker will do every day working inside these tools. And it's meant to test how effective these models are at completing those tasks and at what price points.
And as you can see, if you look at the leaderboard, the top of the leaderboard right now is GPT-5.6 Soul Max, and it's scoring 18.1%. So, the best model right now is performing these tasks only 18.1% of the time.
And I think what that screams to me is that, "Hey, for a lot of things, these workflow-related tasks, you probably don't yet want to delegate these to a model to go do them." And furthermore, they're all expensive as well. So, if there are parts of your workflow that you actually can run deterministically, you're going to have higher reliability and lower costs. But, there are definitely workflows where you want an agent to do it: working with unstructured tasks, writing code, generating emails—there's a whole bunch of things that only a model can do.
And so, that's where these hybrid agent workflow setups I think are so powerful and underrated right now, where you can get the best of both worlds. And that's where Zapier just invests a ton of time, where you can in natural language describe what is you want to go build, and we're going to help optimize what gets built on the other side. So, it's not just purely a model running amok, token maxing, and you spending a lot of money on these things. You're getting something really built for the task you want done, and we're trying to be smart about how that gets built so that it isn't just willy-nilly spending tokens for you left and right.
Yeah, and I think that it's an interesting pattern cuz you're allowing AI agents, the models to leverage technology in a deterministic way. It's very similar to how a human would go and leverage technology in a sense. Are there other bottlenecks or constraints as to why the top-tier model is only scoring 18%? Are there other culprits at play other than the fact that it's probabilistic versus deterministic? Are there avenues to actually increase that capability?
Yeah, and look, I can sort of only speculate because I'm not building these models. But from what I understand is a lot of how these frontier labs are optimizing is they're optimizing for coding models. And one of the great things about building these models to focus on coding is that coding is easily verifiable. You can have tests that run against code, and you can see: did the code work or did it not? Did it perform the task or did it not?
And so, it's a really great place for AI to work because you can know that there is a factually correct way to do this or not.
Now, there is some knowledge work that has that same characteristic. Maybe there's accounting workflows. You either balanced the budget or you did not, right? Those types of things I suspect the models will continue to get a lot better at. But, there's also a lot of white-collar work where it's a lot more subjective. Is this a great email or not? Maybe we might have different styles of communication, and we might say, "Well, I really like this type of email. It was short and tight." You might say, "Well, I like this other email that had a lot of detail in it." So, there's these other types of tasks that happen inside of knowledge work that have a little more variability to them.
And so, I think this is where when we talk about white-collar work and AI being able to automate a lot of this stuff, I do think it will be able to automate it, but the quality aspect, as long as there's sort of a human evaluating and judging it, those tasks I think are going to be trickier for these AIs to objectively say, "Hey, it is absolutely better." I think you could have some humans that say, "Yes, it is better," and I think you could have some that say, "I don't think so. I think I'd rather have it this way."
Which is also why I think you also want to have multi-models. And you're seeing this now bear out where these models have different capabilities. And so you might say, "Well, I really like having the Opus model be the one that talks to me because I really like how it talks, but I really want to have this GPT-5.6 model write all the code because it's like a really good engineer on this stuff." So you're starting to see—it's weird to call it a personality, but it is almost like a personality that sort of emerges from these different models.
That gets into what I think Anthropic just put out something around consciousness in the frontier space, which is a whole interesting separate topic. And then you have people where they're identifying patterns of AI in terms of writing and whatnot, and they're purposely trying to do things to look not like it's AI, which is counterproductive in some ways.
But on the token maxing front, I think you're in an interesting position in Zapier. You're both using AI internally, obviously, so there's a cost component there. But Zapier, literally the product itself, people are consuming tokens using the product. So you're getting hit from both sides. How do you think about model efficiency? We had Coinbase's CEO recently come out on optimizing which models they were using, saving 50% roughly on tokens, and token usage is still growing. But how do you think about that in the nature of your business where it's on both sides of the equation?
Yeah, well, I think this is a big opportunity for us to help our customers navigate a lot of this stuff. When you go into, I think, many organizations, they don't really know what is the best model to choose for a job. Most people don't sit down and think about that kind of stuff, and so they just say, "Ah, I just want to use the best model for the job." And they might be doing a pretty mundane task where it's like, "Ah, build my daily brief," but you're sticking GPT-5 on it, and it's like, "Whoa, that's just way overkill. You don't need that type of model for the job."
And furthermore, I think these models are starting to get where they are very smart, and the incremental advancements that we're seeing in them are now past what is required for many types of tasks inside of the workplace. And so that's why you can see somebody like Brian Armstrong at Coinbase say, "Hey, we're going to do a lot of optimization around our spend here because some of these open-source models are plenty capable for a wide variety of tasks inside of Coinbase, and so we no longer need to be on the frontier for certain types of tasks."
And that is where I think we're going to start to see a lot more optimization into the future, and so it's going to be really interesting to see how this plays out. My guess is that we're going to live in a world where most of the tokens are consumed by open-source models, but most of the spend still goes to the frontier.
Yeah, and that's a great point. And it is funny, there is this element of model FOMO where it's, yeah, maybe the lower-tier model would be okay, but maybe I get that extra bit of something from using the best—I'm guilty of that all the time. But it comes down to, especially when you're doing things in a more structured way, the evals and evaluations. So how do you think about that, and is there a method for actually creating evals to determine, okay, this model is perfectly fine for XYZ task?
Yeah, what I've learned about what makes a good eval is a couple things. I think one, you really do want to have your eval be tough. It should be really hard for these models to achieve these tasks. And the best evals are typically evals that are actually kind of easy for humans to do, but really hard for these models to do, because now you're testing something really interesting where you're like, "Huh, there is a true capability gap there." So, I think that's one that's really important.
The second thing I think that's really important is it's really valuable to have the test data entirely private. This prevents basically the models ending up benchmark maxing on these by accident. I think one of the things we've often seen is that as these benchmarks get leaked, all of that stuff gets leaked into the training data, and then all of a sudden the model is like hyper-focused on fixing it. And so, the model gets good at the benchmark, but it doesn't necessarily get better at just the things you might actually be exposed to in real life. Because ultimately, a benchmark is still just a proxy for the type of work you do in real life. It's not actually real life.
And I think this is what can happen a lot of the times when folks get frustrated with AI in the real world. You'll see these benchmarks where it's 99% or whatever it says—"Isn't it a crazy model?"—and then you go use it yourself on a task, and you're like, "Dang, it kind of sucked at that. How is it so good at this, but it kind of stinks at this other thing?" And it's because the benchmark is still just a small representation of all the knowledge that humans might ask the model to go achieve.
