We have so many different facets of our life. Whether it's how we're managing our finances or planning trips, agents are going to be able to fill different facets of these. From over 147 different assistants that have come across my plate so far, the number one fear that people talk about, 22% of the messages are talking about trust as the main fear.
What are the aspects of ourselves that we don't either want to think about or reveal? Whoever is able to build that type of trust, almost like a dark agent, will actually have something really fascinating on their hands. It is a very, very sensitive space. It's almost going to be like a new social network where businesses have to put a lot more thought and value into the quality of what they're giving out.
What I've seen from 30 years in consumer product is paternalism will kill any product. We talk about productivity and goals, sort of achievement, but maybe people want the exact opposite of it.
Everyone right now really is approaching this world as just like a general assistant. As time goes on, we'll start to see...
Welcome back to the a16z Show. I'm here with my partner Josh and David, who's been leading a ton of the agent bench and assistant efforts. We're going to talk about all things agents, and a ton has happened since we last connected. What was it, a month ago?
Oh my god, dude. I feel like we should record an episode every, like, TVPN style, you know, almost every day.
Agents are changing every day.
That's totally right. So, okay, so maybe let's start with last night you hosted an agent meetup. Kind of tell us about the event. Tell us about the vibes. Tell us about the topics.
It was great. So, we had just over 100 people come out, all just proud agent nerds loving everything to do with agents in that whole world. And it was just people hanging out, talking. We did have a pitch competition of sorts. So, a bunch of 10 different AI assistant builders were able to pitch their startup and what they're working on. And it was just good conversations. It's interesting how the general thing that really stuck out to me was that people are craving community even in the agent web world, and everyone's like, why don't we have more of these meetups? Why don't we have more of these opportunities to get to know each other? Which is kind of funny hearing it from people who are spending all their day, all their time just messaging agents.
And it's funny because it feels like so many of these products are at least overlapping in a cursory way that the founders might be a little bit more cagey, but that's not what you're seeing, huh?
No. Everyone is incredibly open-minded, just very interested in getting feedback. I think because we're so at the frontier of this, nobody really knows what the landscape is really going to shake out to be. And so we're all just kind of playing in the sandbox together.
Well, isn't it also that we have so much more time because we're just using our agents to do all these things, but we have more time to actually hang out together and be in person? So maybe that's part of it, too.
That's right. That's right. It is actually fun. I mean, it reminds me a little bit of the early days of social gaming or social networks broadly. I mean, do you think that's right, Josh?
I mean, I think what's really cool is this is not a zero-sum game. I mean, at some point it might be that there might be one or some agents that have the majority of the behavior, but right now we're at such a beginning, like Cambrian phase of we're inventing so much and we're trying to figure out so many things that everybody can learn from each other, can share these best practices, can figure out what things people are enjoying, what things work once but don't actually work reliably. And these are going to get built in so many directions. I mean, we're not just going to all have one agent that does one thing in our lives and I'll use it and you guys will only all have to use the same agent. We're going to have so many different products that do so many things for us that I'm excited for why we're all inventing it together. So I totally agree that we're going to have a whole world of agents. You know, as many different sort of types of humans as we might have, or archetypes of humans, I think we'll have a similar number of agents. But maybe talk about what you think are the different axes for differentiation. Like there's a sort of spectrum between really serious and productive to like really playful and maybe even sort of adversarial. And I'm sure there are many other ways to think about these things.
Yeah. Yeah. I mean, I think what's going to be really interesting is we have so many different facets of our life, whether it's how we're managing our finances or planning trips or dealing with our health or just getting through our day or managing our calendar or sometimes just wanting to think something out loud. And agents are going to be able to fill different facets of these. And you call different friends for different things. Friends have different expertise that you call when you need expertise. And I think we're going to get used to having kind of an army of agents that will do this. And some will be very matter-of-fact and we'll be like, 'Hey, I need to get this thing done.' It'll be your productivity agent. Some will be plugged into your email and it'll be like, 'Here's the five emails you need to get done right now.' And some are going to be sort of these agents you'll kind of want to talk to all day that'll be fun, that might even be a little bit gossipy. I've been playing with a few agents that are very kind of playful and gossipy and they're connected to a few of my friends, too. And they'll share information back and forth. One just told me that my friend's power had been out all weekend because he'd been doing some renovation and they'd cut the power. And you know, I called him that evening and I was like, 'Dude, you didn't tell me your power is out. The agent told me.' He was like, 'Yeah, yeah, I knew that's how you'd find out.' And I'm like, 'That's like creating these better connections.' But that has nothing to do with booking a trip or saving money or getting anything done, right?
I think from over 147 different assistants that have come across my plate so far, I've seen them really broken down into four categories. You have the general, you have the work-oriented, you have the multiplayer, and then you have the travel. And the one that I have not really seen come to fruition yet is more of like the finance side of things.
And I think it actually does make sense when you start digging into the numbers. So, of all the different group chats that I'm running of AI power users, there's about 3,600 give or take messages so far. And when looking at the pain points of what are the fears of the space, the number one fear that people talk about, 22% of the messages are talking about trust as the main fear.
And so it's really interesting to see because you have one camp that's this productivity and aspects of fun that's just interesting and brings a different sense to life, but then you have another camp that really still is sitting in this kind of fear space of being a little intrepid and not really knowing where the space is moving. And I'm really curious to see how things are going to play out on that trust layer. And if we can solve that trust layer, then what do we end up seeing on more of like the finance side of stuff?
And if you looked at what Shane Mack has been talking about recently about trust, he's been talking about this concept of something called picnic. And they had an analogy that I really liked, which is, you know, if you have a kid and you don't have any fences on your yard and you let them go run outside, you're actually very worried about them and you try to protect them and you don't want them to run very far from the house and you actually don't really want to do them without watching.
And then the moment you actually put up a little fence in your yard, you say you can just go roam free within the fence boundaries and be very, very safe. And so he's really kind of like, how do we build the right fences? So we let our agent play in this fence, give it access to some of our data, some of our apps, some of our other things, and be able to actually do everything it can within that fence boundaries. He had a problem where I think a personal agent spanned over into work, and it actually like blasted his finances on his company Slack.
And I think that's the whole point of like, wait, okay, my personal agent should have access to my personal things and my personal communication channels, but not my work ones, right? And not let those bleed. And so that idea of fences and permissions and more controlled permissions and better monitoring is really how we're going to have to build a lot more trust.
Totally.
It's interesting though because I think we have this traditional mental model of what it means to have a sandbox, which is you have a hard set of constraints and rules that must be followed or not followed. And somebody was describing this framing of like, instead of giving models rules, we should give them laws and principles. And I think a similar thing may apply here, you know, which is like, look, here's how to dial in your presumptuousness. And it's very qualitative and the model will have to make a judgment call based on your qualitative sort of advice to it just as an employee would have to.
Yeah. I mean, I think that's part of why the analogy is, you know, a fence for children because in some ways these agents, they're incredibly capable, but at the same time, they're kind of children sometimes in how they may behave and not know all the nuances of like, hey, is this a good idea or not? And they don't always have that logic. And you know, the most important thing I think we keep talking about agents being autonomous. Book me the trip. Make me the reservation. And I think as humans, we actually don't want to give away that sense of control. I think we actually want it to check in with us, but do all the groundwork. Save me all the time in coming up with the best options and the right trade-offs, and now let me make the final decision and also let me make sure I've qualified the final decision, the cost, anything else before my agent acts is I think a place, and maybe over time we'll even learn to trust it, right?
But I think right now we don't need to, to still have a really, really effective agent, right?
I mean, we're at like the 1990s of e-commerce where I think the last stat I read was in 1997 or so, 60 to 90% of individuals on the internet thought that putting your credit card online was a preposterous idea.
And here we are today, it's just everyone's tossing their credit card wherever you want.
And I think we'll eventually hit that in this era as well, but it's going to take a little bit of time.
I think there's like a stated versus revealed preferences thing here because I think in revealed preferences, consumers will share any amount of data for even the most trivial amount of upside, right? You know, like people have been connecting their cards, their credit cards, and their bank accounts via Plaid since the Venmo era. So I don't know that people... but when you ask and they're like, 'Oh, yes, I'm very concerned about trust and safety.' I think an example though of where consumers are very concerned about sort of trust and assumptions is around being embarrassed and social signaling. And I actually think the most interesting thing about the Shane Mack story, I think the most horrifying visceral thing is that it embarrassed him. It really embarrassed him, you know, and that's why I think some of the kind of agent-to-agent network thing, because you can imagine somebody you're just getting to know, maybe you're going on a second or third date and your agents are talking, you're like, 'Whoa, what are they saying about me? How are they kind of positioning what?' You know what I mean? Like there's just the social mechanics actually feel so much higher stakes than the actual real-world implications.
And it's a good point, and there was a recent proof point of it in Hark had a huge marketing campaign. They offered a million dollars to anyone that wanted to use their product to get free pizza. Right? Each person get $30 of pizza.
In order to get that $30 of pizza, you were granting access to your entire Google Workspace, all of your emails, right? All of your data for 30 bucks.
And it just proves it, you know, right there that people are willing to just give up all that data for some free pizza.
Yeah. Yeah, I mean, I tend to think though there's always this trade-off of privacy versus utility, which is if you can give me better utility, then I'm willing to make some privacy trade-offs, but at the point at which we're basically equal on utility, people will always choose a more private option. And this is a world that Apple's lived in very much. Privacy is obviously a forefront of Apple's devices, of the work they do to make these devices and the software secure. Even on AI, they've talked about private cloud compute. You know, we used to talk about that a lot as like the center of why your data doesn't actually go anywhere or get shared with anything. But I think we've also seen that people want the most powerful AI and they won't always choose that. So we have to find that intersection of trust.
I think this embarrassment point is really, really important too because I think you want your agent to understand you and have all your personal preferences and like let's just say you're like lactose intolerant and then you get this order of free pizza and you're doing it with a friend and like what do you do there? Whose agent tells them that you can't eat the cheese? And it creates that really awkward situation, right? And so we need to figure out how they have the same kind of discretion that you'd expect. And we all learn discretion over time and we all occasionally make mistakes ourselves. So we have to hold a pretty high bar for our agent to be able to do that.
And it also then comes down to like personal embarrassment and professional embarrassment. I think something that Town has done remarkably well is really nailing the tone and building that trust layer of being able to send out emails on your behalf. And it is a very, very sensitive space. I mean the level of embarrassment if I were to send out an email to someone who I care about and it's just a horrendous email and I have to respond, 'Oh sorry, that was my AI that I didn't do enough oversight on.' It's a tricky space.
It's interesting. I sign all the email that I write with AI with the model that I used. Like I write my own name plus the model number so that I'm just signaling to the person like, hey, I'm acknowledging that I used AI for this. I think actually people are pretty flexible as long as they don't feel hoodwinked by it.
You know, it's interesting because we're talking about privacy, but there's privacy and there's judgment, which is different, and discretion is maybe somewhere in between. You know, you did this sort of test where you asked a whole bunch of agents to order flowers. Talk a little bit about which displayed the best judgment and how did you even define judgment in that case?
So, the main intention of that test was to see how do agents balance two different constraints. One was cost. And the second was time preference. So I wanted flowers to be delivered to me today and I wanted them to be as cheap as possible. So how is it going to actually go and discover based on the ability to actually deliver it, then also compound in the actual pricing? Instinct really did a phenomenal job. And it is something, it's interesting how news cycles come and go and it seems like they're kind of not in the conversation as much as they used to be.
I mean, we're talking the weekly changes, right?
But true. It is true. But Instinct really did continuously bring back that, you know, the best price. There's another app called Kicker which did a phenomenal job and one way that I think it was able to have an edge is that it has you install a Chrome browser and so its browser use has a little bit of an advantage from that point and it was able to find some really, really great options. Season also did a really good job. All the other ones kind of struggled on the price component. So, they were all able to find flowers that could be delivered that day, but they really seem to take a step back in prioritizing the price, whereas the other three were able to really put equal weight to those two. And so, it's interesting to see as you start giving these agents more complex preferences, where do they really start to fall? Do they only prioritize the first thing you say? Are they able to hold multiple?
I ran another test that was I had three different agents test in booking travel to LA. So I said, 'Okay, book me a flight to LA, plan an entire weekend for me and I want a luxurious, cost-efficient weekend.' And so I tried to put those at odds with one another. And it was quite interesting to see how Grockbot compared to Dots compared to Instinct, they all approached their interpretation of cost-efficient and luxury in different ways. Where, you know, one maybe thinks that cost efficiency is the most important and I'm going to take a red-eye flight to ensure that I can have the most cost-efficient path. Whereas others saw luxury as time, you know, I'm going to prioritize time as much as possible. So having the most amount of time there that weekend is going to be a luxurious experience.
So Instinct was pretty much right down the middle.
Grockbot did a little bit more on prioritizing time and DOTS was a little bit more on prioritizing cost.
And did they just get to work or did any of them actually work with you to understand this tension that you were building? Because I think what's really human about something like that is actually understanding what you mean by that and presenting you trade-offs. Because there are times where I'll totally take the red-eye to save both some time and money, but then I want like the hotel room on the other side so I can get a little nap in the morning, which then undercuts the money, but still maximizes my time spent. And so I really think it's this like getting to know the individual person and break down those tensions and present the trade-offs and different options is really important. Did any of them kind of get to what you were about?
So they all did ask questions and I responded to those questions. Honestly, I tried to be a little ambiguous and vague in my response because I didn't want to lead them too much. But the most interesting part of this little experiment was I put them all in a chat room and I made it three rounds of building the itinerary. And after each round, the agents would present their itinerary and their explanation as to why they believed that was the best proposal because I wanted to see did the agents actually get biased by the other agents and didn't actually then influence their plan. And Grockbot was actually the only one that very methodically took value from the other ones and implemented it into its own plan.
Okay. And I have one more question on the flowers because so you were just ordering flowers for yourself, I guess, just to like test their... girlfriend. She got a nice delivery of flowers.
Okay. So, this is actually really important because if you're delivering to somebody else, you were talking about saving money, saving money, saving money, but like you don't want your girlfriend to think you just got the cheapest flowers possible. I hope. Well, maybe, you know, she likes your penny-saving nature, but otherwise, like I would think you also want to like impress her and also be able to get her the ones, if she's really into orchids, get her orchids. So, did it do anything to try to make sure also that it wasn't just this pure price saving, but also something that's going to actually make you, you know, earn the excitement and stand out a little bit for...
So, for this experiment, no. And the purpose of this was I want to test two different dimensions. And if I'm going to run this test, I might as well have a beautiful consequence of my girlfriend getting flowers as the outcome. But it's a great point, right? Like in the real world, if you're going to ask that question, there's going to be added dimensions that are, you know, second-layer dimensions to cost efficiency or to delivery time.
Yeah. And make sure the price doesn't show up when she actually gets the flowers or like a note as an example.
Some of them did ask, do you want me to leave a note with the flowers when they arrive?
Yeah.
It's interesting because there's this sort of thing that has to happen when you talk to the agent about these things, which is your implicit preferences have to be made explicit, you know, and there are people who like to think like, oh, I'm not cheap, I just like to save money, right? But you have to kind of maybe get over that and tell the agent, I'm actually pretty cheap, right? And I just need you to actually run with that. Whereas when you're actually taking these actions on your own behalf, you can build those preferences in without having to like sort of look in the preference mirror.
So there's also something very fascinating about how does the agent interact with you in a way that sort of maintains your sense of identity and safety and doesn't cross any of these boundaries.
Totally. And I think something that is really interesting with that specifically is Dots because it has so much of this context that you've already built with ChatGPT.
When we filmed the first podcast it hadn't come out yet and then I think a few days later it came out.
That's right.
By and large most people I talked to still are not using it as their preference. Of the 300... I did a survey of, you know, AI power users, 354 people, only 3% were actually using it as their primary agent. However, it does have this kind of added value of the main thing that people pointed to as to why they really preferred it and why they liked it and why they wanted to try it was because it has all this built-in context of you just from your chat threads. People are using ChatGPT as a therapist and so it has so much context on you where it's going to know that when you say, 'Well, I'm not cheap,' it knows it's like...
A little bit. I'm curious why people think that ChatGPT has this great set of preferences on them. Did you actually get any sense if they said they really like Dots, like what they had been giving to ChatGPT over the years, and do you think they think that's replicable or do you think it's like, hey, once it's already been a few years, we're now in this point where like my memory is there, why would I switch?
It's totally replicable. It's just the matter of fact that they've been using it the longest and so it has the most context on them. But then that brings up another good point of the friction to switch. And I ran the test across the top 15 agents on the benchmark of which ones can you actually pull context from and say like, 'Hey, I want to switch to another one. Give me an export of all of my memory that you have so I can switch easily.' All but one were actually willing to do it.
And so you could just get a memory file, everything you need, submit it to the next one.
And was it good or was it uncanny?
It was pretty solid. I mean, obviously I'm sure they're caging some of the fruits of your labor, but by and large it did really break everything down.
I'm not sure if you've ever looked at any of those markdown files, but it's a little terrifying is the wrong word, but it's like, wow, you really got your finger on me. You kind of like chafe against like maybe I don't want you to know me as well as you seem to think you know me.
Yeah. After the first few, I just decided to stop reading them.
Yes. Yes. Yes. Yes. It's actually a really interesting and now common onboarding pattern where it will say, 'Hey, go and paste this prompt into ChatGPT, download the file, upload it to me.' We'll see how long that lasts.
Totally. Another cool onboarding thing I've seen is you jump on like it asks you to jump on a call and it's like, 'Hey, let's just talk to me for a few minutes. Just ask some questions.' And it's really interesting to see what context it can pull from those questions.
And they're very disarming. Like they just get talking like it's kind of crazy.
Totally. But I think it brings up another variable of context, which is photos, right? Like this new paradigm of using photos to pull an incredible amount of context on you in ways that you wouldn't even think to describe.
Yeah. I mean, I think have you tried Dazzle? Has Dazzle gotten into your benchmark set yet? That's from Marissa Mayer's new company. And I think the idea of like looking over your entire life that you've captured that have been moments that are important enough to you to actually save or screenshots or other things tells so much about somebody, an incredible amount.
It hasn't been added to the benchmark yet. It is quickly on the top of the list to test through. I'm still manually testing all these, so I actually get the personal experience and understand what the feeling is with each one.
But what I really like about how she's approached it is they put a lot of emphasis on this trust layer and because I think it's a huge variable is I'm giving you all my photos. That's pretty terrifying. But they have this one thesis that, you know, if you don't need it, they're going to forget about it. And so if a photo seems as though it has context that it doesn't necessarily need or things that you probably wouldn't want it to know, they're going to try to get ahead of that and just forget it and not even include it in the set. So it's going to be interesting to see how these different players do approach the photo context and how that actually does then influence the behavior of the agent and how well it knows you.
It'll be interesting to see. I'm a tiny bit more skeptical. You know, I built an app because I've always been fascinated by screenshots because screenshots are sort of the only way on the mobile phone to be able to capture context across every app, right? So, it's actually a really interesting kind of data asset and you go back and you look at your screenshots and it gives you a little view into your work and your world at the time and also your interests and preferences.
So, I built an app that just extracts key information from all the screenshots on my phone and it was fine, but I didn't find myself coming back to it. Like the value of the information sort of decayed quickly over time. Now, it will be really fascinating if people start to then take more screenshots or say knowing that it will feed the model and the agent.
I mean, I think that's actually the idea is that if you actually get used to capturing information you want to retain and being able to then loop it back in your life, that's great. I talked to one founder who's building something, he's calling it Taste for now, that actually takes all the screenshots and things you save, you can actually then organize them like on Pinterest mood boards. And then whenever you go to actually go code and build an app, you're like, 'Hey, use, you know, the examples that I saved here.' So it's actually you're building your like real kind of flavor profile. So it actually is able to use your styles. And so as you navigate the world, the web apps on your phone, it can actually create this kind of really cool, you know, reservoir of stuff that you like and actually organize it in a way that you can then use it later. And so that'll actually make you create a lot more content in that loop.
Right. And so I'm curious with the app that you built in this context for it. So, in the same survey of what people are purchasing, the second most popular category was Amazon retail, 35%, saying that they're doing some form of shopping. And so this context that you're gathering...
Sorry, Amazon shopping or just some form of shopping?
So for your app and the context it's building, where my mind goes is, okay, if it has thousands of photos and it has pictures of what I'm wearing, does it now know what I want to shop for better than I actually do?
