How Enterprise Leaders Use Agentic AI to Transform Teams and Drive Scalable Impact with Andy White

There's been a lot of discomfort around how do we build automation into the sales agent so that it can automatically approve deals of a certain size. But there's fear over what happens if it goes off the rails and approves more things than we want. So understanding what does good look like for the humans so that we can measure agents in the same way. And if that doesn't exist, we have to go and build it.

One of the challenges with Agentic working and all this new stuff that we're trying to figure out is people don't know what that journey looks like. It's also new. Several of our assumptions throughout the way have changed 3 months later. And adaptive problems are ones where when you go to try and solve them, you frequently have to go back and solve it again because you didn't solve the right thing. We learned very early on that we had to be working very closely with the direct end users of the technology we were building the solutions for and have them be part of really the development team. A good way to think about a lot of the agentics is if you brought in a team of interns. So if you brought in 20 interns to help your sales team, how would you support them? And how would you make them effective? Because one of the things that's so important with an agentic solution is testing and tuning based off of results. And you have to do that with the business team of experts that know what good looks like in that space.

That's a great question and I would start with...

Good morning, Vivian. Thank you so much for having me here. Excited to talk with you today as always.

I've been at Salesforce for 16 years. I did join through acquisition and I think that was 13 years ago, maybe 12. And so Exact Target became the marketing cloud. We learned a lot of what to do and what not to do through that acquisition. That's informed how we've gone on and done more and bigger acquisitions. But it's been a whole lot of fun. I feel like in that time period, I've had multiple careers and the company has evolved substantially. I'm sure that's been your journey too being here for more than 10 years and yeah it's been a whole lot of fun and I think what's the most exciting is right now I think is the most fun that it's ever been and we have a front row seat to the future of work and what does that look like with humans and agents working together and Slack and voice and so many cool things that we get to figure out internally and help our customers with externally.

One thing that really stands to mind or comes to my mind is that I learned some words that helped me and it's adaptive versus technical problems. I'd never heard of adaptive problems. I've experienced adaptive problems but having language to say this is an adaptive problem because what we do at Salesforce and what I think a lot of technologists do is try to solve problems. We sit down and we say you know what Vivian we'll get the right people in the room together and we can figure this out. But one of the challenges with Agentic working and all this new stuff that we're trying to figure out is people don't know what that journey looks like. It's also new and several of our assumptions throughout the way have changed three months later where something that we thought was true ended up not being true. And adaptive problems are ones where when you go to try and solve them, you frequently have to go back and solve it again because you didn't solve the right thing. You weren't asking the right question. They also typically involve like changing the way people work or what they do. And that means it's like this zigzag of progress that is not linear and straightforward. And being able to put your finger on it and say this is an adaptive problem was a huge unlock because then everybody's thinking about it the same way. We're not expecting linear progress. We're expecting experimentation. We're expecting trial and error. We're expecting failures. We're expecting that six months from now half of what we thought was right on what we were trying to do is not. But that was a huge unlock because I think I and others started out some of this journey of like okay it's going to be ABCD. This is what we're going to do. And it's like nope it's an adaptive problem. So that's been a huge thing of changing my mindset and my language and how I bring people on the journey with that on what to expect and it builds resiliency because it part of it is proper expectation setting. You know what I mean? It's almost like if you and I are on a boat ride together and I'm like it's going to be bumpy up here so let's be ready for it.

It was early on in some of our first agents and so the first couple that we built was one for help.salesforce.com at salesforce.com. Our help portal which is providing customer support externally to our customers and then our sales agent which is an internal agent which is helping our sales team the sales process helping them identify problems, opportunities, helping them collate and find information. And we learned very early on that we had to be working very closely with the direct end users of the technology we were building the solution for and have them be part of really the development team. And that's different because lots of times when you're doing a technology implementation, you'll sit down and gather requirements and the technology team will go away and build a thing and then you'll come back and you'll maybe do a prototype or you'll do UAT and that does not work. And one of the reasons it's very different is in each of those scenarios we're augmenting teams where we're bringing capacity and capabilities to these teams. We're changing the way they work and it's not just giving them a tool but it frequently it's extending their teammates and how they work together. So on the help portal, we're adding more capacity to our customer support engineers. They're now working with this whole new team of tier one agents that are helping to support customers and that they're having to give feedback to and improve. A good way to think about a lot of the agentics is if you brought in a team of interns. So if you brought in 20 interns to help your sales team, how would you support them? How would you make them effective? But because this capability is joining the team of the group that you're trying to build the solution for, it really causes you to think differently about how are they involved in the very detailed implementation, but as equally important with the go live because one of the things that's so important with an agentic solution is testing and tuning based off of results and you have to do that with the business team of experts that know what good looks like in that space. We can give you tons of examples of that, but it's really caused us to work much more closely together. And frequently when you go live with a technology solution, once you get out of hypercare, it's kind of set it and forget it. You might add some new capabilities over time, but it's not this kind of daily what happened. Let's go look and let's then tune either adding more capabilities, more data. Because of two things. One, agentics are nondeterministic and so they behave in interesting ways sometimes that we don't expect. And number two, people are responding to agents differently than they do humans. They ask different questions. So one of the things we ran into with the help portal right away was customers were asking the help agent different questions than they ask our support engineers. And so it was things we weren't ready for. It was things we weren't expecting. And it was interesting because it's caused us to change the way we think about how we're providing support. I'll stop because I just said a whole lot.

And you asked if they were resistant. I mean, everything you said is true. And I think some groups were a little resistant to it. Others dived right in. But the big thing was making sure that any team that we're going to be working with knows that they're signing up for managing this new teammate as part of that go live work that there's a huge burden on the business team to make sure that the agentic solution is performing the way they want to and that they're helping it know what good looks like. It's like that with the example I gave. You wouldn't expect to bring on 20 new interns and say, 'Okay, good luck. Go do this task that we asked you to do, right?' You would expect to onboard them and to give them feedback and training along the way.

Totally. And it's actually more of a line of business problem. And they do own the accountability because that's exactly like an interesting other interesting early learning was a RACI exactly that who's accountable for the agent and you would think from a technology perspective oh well like it's a technology IT and it's like no it's the team that they're augmenting which is interesting and then you start getting into all things of like what level of access should they have? Do they go in and adjust prompts and some of these other things. But that accountability, the RACI is 100% something that we've had to build a muscle on of making sure that the business stakeholders know what they're signing up for and are ready for it.

Yeah. The good news is on so much of agentics it's taking something that we know two things that we know and combining them in new ways like we know technology and we know people and that's really kind of what you're bringing together but you have to view it as that fusion. There's been a lot of discomfort around how do we build automation into the sales agent so that it can automatically approve deals of a certain size. Like why are we even asking for human intervention if 95% of the time we say yes? But there's fear over what happens if it goes off the rails and approves more things than we want and it's like well we have to look and we have to give it feedback and we have a capability for this. It's exactly what we do with humans. You have the same risk with a human, right? A human could make a bad call, bad decision. You give them feedback. If they do it a couple of times, you remove their ability to do it. But it's that merging of those things coming together. And as far as the actual capabilities of the business teams, some teams are more technical than others, but really what's required is to understand what good looks like of the team they're a part of. Which is helpful. Now what we found when we started doing some of this work is that some of the human work is not well instrumented. So understanding well what does good look like for the humans so that we can measure agents in the same way and if that doesn't exist we have to go and build it. And that's where the teams work together but you basically have to have a baseline of what are we trying to measure. And the big thing that the skills that the business teams need are number one, understanding their business very well, understanding the outcomes that they're trying to drive, and then having ways to go and look and measure that both for agents and humans. And then we get into prompt tuning where we're using prompts to change the way the agent performs. That can be done together with the business team. What we found has worked well for more technical teams is giving them access to copies of production environments like basically a second UAT where they can go and do experimentation with the agent and see if the responses improve or change. The other thing that we're working on with our technology teams at Salesforce, so the teams that build the product we sell is experimentation for just that reason. Because what we'd really like to do is be able to build experimentation like to have two identical topics and say fire this one 80% of the time, fire this one 20% of the time, and then let's see at scale how they work. But really the skills that they need are curiosity, performance measurement of their existing team and then an understanding of prompts which are natural language. And so these are skills everybody can have. You're just bringing them to bear maybe in new ways.

Yeah and what we've done typically is give them access to a copy of production. There are some exceptions to that where we have given actual production access but that's normally due to the technical capability of the team for the most part what we're doing is we found that we had to create a completely new environment for them to be able to do tests with the prompts and an environment that had their data sets, their workflows, and their actions. And that's the way we've kind of gotten around that because we don't really want to be making changes in production to do these tests. But that's been a learning lesson. The environment has been a learning lesson.

Well for sure they should use Trailhead and you know there's multiple courses on agentics and large language models and prompts those are all very helpful. But also just experimentation one of the things that large language models are so good at is telling you what are great ideas for prompts and so using you know Claude from Anthropic and ChatGPT from OpenAI and Gemini from Google using those and saying like, 'Hey, this is what I'm trying to do.' I think that's a great thing and a skill and muscle for people to build and using multiple large language models in parallel with each other and seeing which gives different results. Having them tune each other like, 'Okay, I want to do this. Here's my existing prompt. What are some of the things I should be thinking about?' But ask like using large language models to help them. How would you go about it? What am I missing? What am I not thinking about? How would you write this? And then again, I think it's super powerful using multiple large language models to do that. Now people need to be careful with company data and making sure that they're not putting in anything into large language models, anything that's not approved for their work. That's why Agentforce is great, right? Because it's approved and trusted. But I think you can put in generic information and still learn from that. But a lot of experimentation using multiple tools is my recommendation.

We've got a couple. Maybe I'll give you an example of a big learning, but it wasn't a flop. It was just like a what were we thinking? And it was on our help portal. So when we first built out the help agent, we grounded it in all of our support data. So if you think about everything that a customer support engineer and Salesforce has access to, that's what we loaded into the support agent. And then we noticed after it was solving people's problems, it was pretty cold. It wasn't really empathetic and it didn't act the way one of our human customer success agents do and there's a reason for it. It's because we didn't do any training on soft skills. None. And so when we onboard a customer support engineer at Salesforce, we train them on the heart and the head. And there's an expectation of how they provide customer support. And it's like going back to my example before, if we think about these things as bringing together technology and humans, you'll do better because you would never do that to a new support engineer. You would never say here's all of our documentation. Go start talking to customers. And so that was a big learning and it's something we've taken into account with all of our other agents and really baked it into our best practices that you start with what are we trying to do and what is the personality persona that we want to drive as a part of that. So you could think about our engagement agent that is reaching out to customers to try and book opportunities is very different than our customer support agent. So that was a big learning. I'll pause in case anything stood out to you about that before I go on.

So the other failure was we started out kind of having a race to build as many agents as we could. It was really like we actually had an agent that was for getting ideas about what agents we should build. It was an ideas agent and so like I would say another big problem that we ran into is agent proliferation. And so having a bunch of agents but having them be minimally effective in their capabilities is not great. And so that was a lesson and a failure of we learned what was much more important was focusing on really great agents and having fewer of them. And kind of back to the other points I made about the importance of the stakeholders and business teams knowing what they're signing up for about the work they need to do to make it an effective part of their team and really change the way they work. That's super important and each one of those teams that built an agent needs to be signing up for that. And so if it was a technology-driven transformation of like, hey, I built this agent, but the business teams aren't incorporating it to how they work and paying attention to what it's doing and continuing to give it more capabilities and access to more data, it's going to fail. So quantity versus quality was another big lesson learned. Any response to that or any question on that?

It's really based around a couple things and I will say I do expect in the future there will be thousands of agents at companies but the tooling will mature and evolve to allow for much more robust orchestration and easier scale. But where we're at today that's very much the right approach. So what we try to think about is what is something that has broad impact and aligns to the capabilities that we have today from an agentic action perspective. And so you don't want to bite off more than you chew. Like there could be a great opportunity, but if it doesn't align with your business capabilities or technology capabilities, it's not going to be successful. So we've really been trying to think about these highly impactful areas. And we call them hero agents that like our customer support portal that has access to all of our customer base and it's got a huge opportunity for deflection automation. We're able to provide support in more languages than we could before. It's really amazing. We have another area which is sales and really thinking about what's our website presence on salesforce.com. How are we reaching out to customers that say we can contact them through lead management like if you come to one of our events how are we following up with you and engaging with you that's called our engagement agent and then our sales agent which is where the internal focuses version of that helping our sellers and then the last one is our employee agent. So what's kind of a generic multi-purpose support agent for our entire internal company and those are the big buckets that we look at but it's really about what's the blast radius and then what are the capabilities of the agent today. There is another criteria that I would call like a micro agent where it's for a very small team but it could be highly effective for that very small team. So, our group that focuses on customer renewals uses an agent and it's one of our most active agents from a number of times it's used per day, how the percentage adoption, but it's a very very small team. And so, like it won't show up on the radar at all out of our 75,000 people because it's like 50 people, but for them it's highly impactful. And so, those are the kind of things we're thinking about like at the top line, what's the blast radius and then what are the capabilities? And sometimes that means some of these smaller purpose-built agents make a whole lot of sense for that individual team.

Yeah. That's a great question and I would start with an area of your business that you think is ripe for change. Either an area where you would like to see things done at a much bigger scale or you think there's a great opportunity for improvement or you've just got a real pain point and I can give you an example of that for us. I'll give you two with the help.salesforce example that I've given multiple times. We didn't provide any bot assistance on our website. Nothing. We had no chat bots. It was either a live human that you were calling, a live human that you were doing async ticket engagement with or a live chat with a human. So when we sat down and said we have the opportunity to provide agentic support in a bunch of languages and provide real-time translation and all of these things but it was lower risk for us because if things didn't go well or we had a problem we could turn it off and it wasn't a capability we had provided before. So lots of upside, very low downside. And that allowed us to also iterate more quickly. Because speed is really important. Being able to see and get signals about how the agent is performing is really important. Like getting it into production because you can spend so much time in theory and analysis of what is this thing going to do? And I told you even our best guesses of what we thought it was going to do was based off of how in this case it engaged with our human support engineers which was different than what people did with it when it went live. And so getting it live looking at what's going on and doing that in a low-risk scenario with lots of upside is absolutely what I'd recommend. The second scenario for us was our engagement agent. And so this used to be called SDR agent sales development representative but for 25 years Salesforce has had more leads than it can follow up on. This has bothered our CEO Mark Benioff from day one. But it boiled down to we didn't have enough capacity to follow up with every lead. So when you attend one of our events and you say yes Salesforce, I'd like to hear back from you. We score you. We put you that lead through an AI ML model and we say Vivian is a one because she was just here for the free lunch. She doesn't care about the product. Let's send her a marketing drip campaign to keep her interested. Or Vivian is a five. She is ready to buy. This is super important to her and her company. Call her right now. We would only follow up with the threes, fours, and fives. Everything else that was a one and two would be maybe an email drip campaign from the marketing platform. But now with agentics, we're following up with every single lead every time. And the quality of those follow-ups is better than what our human sellers would do, the email quality and the reachout quality, which is quite remarkable. And so that's a good example of again in this space where we're doing nothing, those lead scores one and two or maybe we're doing a marketing campaign, there's very low risk to say let's have a two-way conversation with those customers. Now, as we've proven that that works, we're now going into lead scores three, four, and five, and having the agent take a stab at that. Which is allowing us to scale our human SDR team in different ways. But those are two examples that I would recommend for CIOs that haven't gotten started. Look for an area where you have a business problem that's clear, an outcome that's clear, and maybe a little bit of a green field solution where you can start from scratch, and if things go south, you can turn it off and you can go quickly.

100%. And that is such a good story and it's actually my favorite of the change that's happened at Salesforce because that is so successful and now that team is really scaling that agent much further. My boss Joanne Zerello says that humans are for impact and agents are for scale. And that's a perfect example of we have this team of the SDR team that is now really being thoughtful about what's the roadmap and how are they expanding the capabilities of this engagement agent and then the engagement agent scales it in ways that they never could. It's quite impressive.

I'll give you an example. I'm sorry, Vivian. Go ahead.

An example of that that's so cool is our very first deal that closed with the engagement agent. So, we did two experiments when we launched that thing. We had it go after the lead score one and two because they were lower risk for us. And then we had it do agentic follow-up on lead score three that had initially been reached out to already by one of our human SDR agents, but it had not moved into an opportunity, meaning the deal hadn't proceeded to the next level. And the first one that closed was the second scenario where an agentic email did a follow-up on a customer that had already been engaged with by a human but hadn't moved on. And they followed up with the agentic email and said, 'Yeah, I would like to hear more. Book me an appointment.' That's the first deal we closed. So, it's a perfect example of humans and agents working together where that deal wouldn't have closed before because the human SDR moved on to something else. They either forgot about it or didn't follow up or it was a bad time for the customer. It's pretty cool.

Yeah. So, I think number one, curiosity. You know, we've talked about that multiple times, but being curious. And when you see something in the news or an article, read up on it. Like a new AI model that you've never heard of before. What is that? Why is it popular in China or Japan or wherever the news story was about? So, doing that. Number two, experiment. So, using multiple large language models. And number three is being really important in that experimentation. Changing your mindset. So, as you're doing your day-to-day work and your personal activities, say, 'How could I use a large language model to help me?' And try to retrain because you honestly won't know what works in your flow or what works with these large language models until you try it. And continuing to try things because everything evolves so quickly. This whole idea of whatever you did six months ago does not hold relevant anymore. Like you can't say, 'Oh, I tried that on ChatGPT six months ago. Well, try it again.' And I think that idea of making that into a virtuous cycle of experimentation and learning and application that would be my recommendation.

I mean, there's so many things. What I'm looking forward to from a work perspective is I think this year is going to be a huge unlock for us on making really meaningful change in how our employees are using our agentic solutions internally. We've got a couple foundational things that we've been doing with some of our business processes and our data that are really required to unlock these bigger outcomes. But I think it's going to be the year of real capabilities where today we have a lot of solutions that help with knowledge and action but being able to really assist with some key pain points for our sellers and our employees. I'm really excited about like it's going to be a big unlock for them with what these agents can do. So I'm excited about that and I'm also just excited about our ability to use large language models to augment ourselves. You know, one of the things that kind of blew my mind with for myself is I did a project in Python that should have taken me days and it took me hours and I was just so impressed with that and I'm not the strongest, right? And so if it can make me that much better, how can it augment the rest of my team who are engineers that are much better than I? And so I think just that idea of continual experimentation, learning, being hands-on, like you said, really excited for continued improvements.

Thanks Vivian.