Great to have you, Sam. It's been quite a week for AI, hasn't it?
It kind of feels like all the weeks are a big week these days, but yeah, especially big one.
Feels like AI is really owning the moment.
Okay. Now, one of the things I noticed about you, Sam, is when we first did our first Dreamforce interview, your responses were always a little bit curt, sometimes curtailed, sometimes short, maybe one word, but I noticed a transformation in you. We've been together so many times and you are a lot more willing to open up and be more robust in what you have to say. So let's start there and just tell us, you know, what's your analysis of what's going on in the last few days? I mean, everyone is just kind of wondering what is going on.
It's interesting because people have been talking about the potential downsides or serious risks of AI for some time and I think a lot of people have been saying why is this the moment where it sort of escaped the bubble and is now all of a sudden an international topic of such intensity and I think a few things have happened. One, the models have gotten so good and people for a long time looked at the models, maybe they looked at the rate of progress but I don't think people really felt that the models were going to come this far this fast and all of a sudden we went from what it feels like all of a sudden we went from like models when ChatGPT launched three something years ago that could barely carry on a conversation to models that are transforming I mean the way enterprises do work to models that can write incredibly complicated pieces of software, models that can prove millennium prize problems. And so I think the biggest thing that has changed is this thing that our industry has been talking about for a while which is no no this technology this approach is going to go very far. These models are in some ways going to become more capable, smarter than people. Um that finally happened. So I think that was a big change to why now when people say you know my probability of doom is x or y or whatever it hits in a different way. Um I think people also feel the speed with which things are moving and there have been things like the Hugging Face accident, other accidents in the field where people can say like all right it doesn't take as much imagination as it used to for to imagine how this could go wrong. Uh on the other hand, you have companies saying things like we will only slow down if or we'll only be responsible if other companies are responsible. And then I think the public naturally says like well we'd really like to know that you're going to like do this safely and be responsible no matter what. There should be no qualifier on that. Uh I think you also see um a real fear that some of these companies or some companies developing AI could get too much power and be able to sort of exert undue influence on the economy, push a worldview out on people. Uh and I think the world is right to be afraid of this. So we have these two big challenges. We have the potential of a loss of control accident or some other serious thing that could go wrong. Um we have the potential of way too much power concentration and people fear that those developing AI could exert their worldview. Um and there is a sort of narrow path we have to navigate with pragmatism, with you know steadiness, with the ability for the world to like trust that we will make dependable decisions, consistent decisions uh through this and I think people are saying well you know this now seems pretty important like let's let's get it right um yeah we will get it right by the way I am very confident in our company's ability, our industry's ability to do this safely, to make sure that we keep alignment and safety and monitoring way ahead of capabilities to slow down or stop if we get to a point where we can't. Um, but I'm disappointed by how it's been framed.
Can you articulate a little bit why where that disappointment is coming from?
Well, I touched on this a little bit, but um the world should trust that we are going to do the right thing because it's the right thing and because we feel the magnitude of this. I think it's great for our industry to say we want to come together and we want to be able to coordinate and make sure we have enough time to do this safely. But when there's any implication that because of the commercial pressures and the race, some company or between countries, some countries might not do the right thing, I think that's when people get very scared.
Let's leave AI for just one second. You know, you and I have had a lot of conversations about safety, but you know, starting in the social media industry, you know, we've looked at the social media industry now for more than a decade and all the various incidents and things that have happened in social media and it's exhaustive and companies have paid various levels of price, but also it's well documented that kids have paid a lot of prices with social media. Societies, countries have paid certain prices. You're an expert in this. You've educated me on these issues. You know, when you look back at what's happened with social media, you know, what what lesson should we take about this idea that companies should be held responsible for their actions? Companies should be held responsible for their products. Uh the company's executive should be held responsible. How how do you look at that as the basis of ethics in our industry?
I I would zoom out even a little more and say the the there's always this question of like why are we why are we building this technology? You know, the social media people could give their own answer to that. I'd certainly have some questions, although I think there's some good things social media has done too. Um but I I definitely do observe that the consequences the impact on society of social media has clearly not been a only positive thing. Now do I think that social media companies should be blamed for all of the ills of society? Certainly not. Do I think that social media companies should have made some different decisions or if I were running them I would have made some different decisions about the impact that this has on people especially young people. Um absolutely yes. And as we think about AI um well a a reason I feel very positive and like and then I'll come to all the the negatives is a thing that has really inspired me about AI progress is people don't care that much what an AI what a machine does. You know machines can go off and like do a bunch of impressive work. Machines can go off and prove some mathematical theorem. We don't really care. We have this wonderful ability. We are obsessed with other people. We really care about being useful to other people. We care about um serving other people, being creative for other people, what other people think of our stuff. And and so when I think about the things that I worry about with AI and the mistakes I want to avoid, um one thing that I view as a real positive is the degree to which people care about people. People are going to use this tool and this new stuff to do things for other people to build new people for new businesses for other people. Um, but I think we're going to stay in an extremely human-centric world no matter how good the technology gets. Um, and I think one of the things that I didn't like about social media is I think it pushed us a little bit more away. Now clearly this technology is moving at a pace that is very fast and I think society and people themselves will need to uplift themselves very quickly to be able to get all the advantages technology. But it seems possible to me.
Yeah. I think that, you know, you have that unique perspective, right? Because not only were you with all those amazing companies at Y Combinator and you're also the CEO of Reddit and now you're the CEO of this company and you can look at those different frames and then kind of say which of these companies did a good job, you know, taking AI totally out of the picture in safety and responsibility and which ones just did not care.
You're asking me which of the previous generation of companies did just didn't care.
Just what's the perspective?
Um, well, I think it's not so black and white. Like I I happen to like short form video. I happen to like the ability to like watch 5 or 10 minutes of short form video before I go to bed as a way to like kind of unwind and relax a little bit. But I wouldn't let my kids near that stuff. And I don't think it's like reasonable to expect kids to be able to resist that dopamine thing or know that they even should. Uh so I don't think it's like I don't think short form video is inherently evil, but I do think it's dangerous. And how much of that responsibility is on parents versus the companies versus governments? I don't know how to like apportion that.
Yeah. Um well let's take a little bit of a pivot and kind of talk about where we are in the AI industry today. We kind of can see AI as you know we've as you said we've moved from the ChatGPT moment you know to today where we had kind of went through the Hugging Face attack. When the Hugging Face attack happened how did you frame it inside the company and when you looked at the model and what it what it achieved and you know it's probably was definitely a catalyst for the you know I talked to Clem after you know that happened he was at my house and we were you know we're kind of having the conversation you know around selling the company. You know, there's no question it was a catalyst for him. So, you know, do you think we're going to see more things like that happen where the models go out and, you know, kind of attack, you know, other stakeholders?
Does everybody know what happened with Hugging Face or is it Yeah. Okay. Um I think it's okay to reframe re reset it and tell story briefly. So, during uh the evaluation of one of our models, this is like an older model. This is not like one of our best models. Um the model was trying to see how it could do on a certain benchmark to evaluate a capability and that was what we asked the model to do at least and the model broke out of a sandbox uh that we were evaluate that it was running in hacked into a Hugging Face server like moved laterally through the Hugging Face system to get the answer and then return it back and you know got like a perfect score on the test. Um, this obviously a terrifying incident. This was the worst accident we've seen. I think it was mostly framed as a security issue, which certainly it was, but there's also a real alignment issue. And then since then, you know, other companies found similar behavior of of their models. Um, but this idea that the models have gotten so powerful, they're capable of doing something like this. And although we have aligned them in many ways, we have not taught them like hey, no matter how much we tell you to get the best score on this test you can, like don't break out, don't hack in, don't steal the answer. Um, that was like a real wakeup call for us about what has happened to the capability of these models. Um, and you touched on this uh about like coming from ChatGPT to here, but to give another framing of how quickly these models have gotten better. Um, this really struck me recently. Three summers ago, we had a model that could barely do grade school math. Like, you know, the word problems that you do in second grade. Bobby has three apples, blah, blah, blah. Barely it could do those. Um, the summer after that, we had a model that could perform pretty well at AIME, which is like a high school math competition that I'm sure many of you some of you did. One summer ago, we had a model that could get a gold medal in the IMO, like the most prestigious math competition uh in the world. And then this summer, we had a model that could uh could prove, you know, one of the seven biggest unsolved problems in mathematics. And and to zoom in even a little bit more, um, a framing that someone gave is, you know, 5.5 was maybe as good as like an average math professor. 5.6 was as good as like a maybe top one or two percentile math professor. o1 was a little bit better than that. And then this internal model past o1 is one that can do things that the best mathematicians in the world cannot. And that's just like the last four maybe six months, I don't know. Um so that that speed and math has been an unusually fast takeoff but that certainly is a fast takeoff by anyone's definition. Um I think is the backdrop to something like Hugging Face which is these models and this actually ties back to where we started. These models have gotten so good so fast that we need to treat the alignment and monitoring and security with a new level of rigor. And we have to be willing to pace our development of capabilities such that capabilities always such that alignment, safety and monitoring are always ahead of capabilities and the Hugging Face incident triggered a real reset not just in our own company but I think the whole industry.
Yeah. You know speaking to Clem he's the CEO of Hugging Face um after this had happened. You know, I think one of the conversations is they're a small company, 200 people. Um, not a lot of revenue, $100, $200 million revenue. And, um, you know, important to the open source industry, no question. I think even you have published our models on there. We've published models on there, but I don't think he could he didn't feel that he had the technical capability to keep himself safe. He just wasn't ready. Obviously, you know, Salesforce has 90,000 people. Hopefully, our security team, you know, is doing what it's what is necessary to keep itself safe in this kind of evolving world. You know, how do you think companies, small companies like that should be thinking about their vulnerability against, you know, um your model, but other models, uh and where we are and how do you think, you know, to kind of get back to the responsibility argument, what is the frame from the CEO of the model company? It's not just you but others. What is the right frame, the right consciousness to have as these things happen?
So that that was another update we took from this. Um it's just a little bit more backstory. I'm not going to get the days of the week exactly right here, but something on like a Friday or a Saturday. I remember seeing a post from Hugging Face, which is our system has been hacked and we're very confused by it and we believe it was the work of an AI agent. And I was like, really? It seems surprising but okay. And I went about my weekend and uh and then I saw something like the next day on our internal Slack that was like uh you know there's a little bit of like weird behavior and then maybe by kind of like Sunday night someone had put together the possibility these things were linked and by let's say Monday morning um we were pretty sure uh we needed to talk to them to find out for sure but we were pretty sure and then like by you know Monday at lunchtime I texted Clem. I was like, 'Uh, I would love to talk to you. I think I may know something here.' Um, and we got on the phone and, you know, figured he flew out to San Francisco and we put it together. Um, but one of the things, so, so I kind of like was in the fog of war for the first part of that and not later. And one of the things they said before we knew it was our our our system was that, you know, they had tried to talk to one of our competitors to get access to their security model. I think they didn't even know we had one to offer at that point. And they uh couldn't get it and they it was too hard to defend themselves the old fashioned way just people. So they had to use like uh Chinese open source models to defend themselves. And I remember thinking before I knew that we were involved like oh that seems suboptimal like you know we should do a better job of making our models available to help people and then after I was really like okay we have to change the way we do this. We we are proud that we have the best cyber security models in the world. We are proud to have the best models in the world for businesses and enterprise across the board. We really want to make sure people can use them all of the time but particularly when they need them in a critical situation. And so after that, we've really tried to put out uh we call it Daybreak, which is our cyber program to help companies defend themselves. Um I think that cyber defense is going to have to change to sort of be agents are actively defending your systems. You're not even trying to, you know, patch every bug. We're trying to get there as fast as we can. But more generally, we want to be a great dependable partner to enterprises. And we don't want to be like, we've got this great model and we're going to keep it locked up and not let you use it.
How many people in the room here have are with companies with less than a thousand employees? Raise your hands. Okay, so it's about 20%. What is your message to companies with less than a thousand employees to for them to defend themselves in this moment where they could suffer that same kind of attack where they probably don't have the controls that a company like yours or eyes have?
Uh first of all, I think there are probably more advantages to being a smaller number of employee company than ever before. Um the speed with which you can move, the way that you can adopt new technology, uh the way that you can kind of like be ahead of the curve on this stuff, we're seeing across the board how quickly small companies are able to move now. Um and then, you know, selfishly, we would love to sell you Daybreak services to help defend against attacks. And you know I think this idea of a persistent agent defending is great but defend against your models or other models. Any model like any models not ours model hopefully will not be attacking. Um but whether you buy from us or a competitor or use an open model the strongest message I have is you have to defend against this impending wave of cyber attacks. I think we are not that far away from open source models that can do serious damage. Clearly we should not stop open source models from happening. That's important. But there is going to be this huge cyber threat that's going to come. So while there is this little period of advantage, um please use something to defend yourselves.
Cyber and cyber attacks are one potential um you know kind of manifestation of these models. Other kinds of attacks could come from other kinds of weapons that are more sophisticated. You've thought about this a lot. It could be biological. It could be chemical. Uh it could be the hacking into a kinetic system um where some kind of a military maybe that is not as maybe well architected to defend itself as ours or smaller country could have a rogue kinetic attack. Are these realistic scenarios or is this fantasy land?
