IBM's $10 billion bet on what comes after AI (with CEO Arvind Krishna) | Masters of Scale

I'll point out to you underlying GPU pricing which is what all of these things run on has doubled in the last 6 months.

Yeah, look, the advantage is client intimacy. Knowing your clients. The advantages are around trust. I don't think we have knowingly ever done anything wrong with a client's IP or data or people. Those are advantages. I think our people are incredibly technically adept. I would say they're experts in the areas where they have spent time and energy. So those are all the advantages. Now technology keeps changing and I'll be the first to acknowledge sometimes we are really good at predicting where it's going to go and we kind of get ahead of the wave and do that. You mentioned a few of those. The mainframe certainly but this is now 60 years ago was one of them. I would say the IBM PC may have been another one of them. I think embracing Java in the internet era was another great one. And then you sometimes miss them.

That we missed. I would actually turn around and say despite inventing the IBM PC, client server was another one that we missed. So I think it comes into you could probably miss some and you kind of know you're going to miss some sometimes and you know you're going to I think because if you don't take risk you're never going to succeed. So part of it is hey I'm getting a lot from that one. I kind of want to keep my focus there and kind of if you're not two three years ahead of the wave you're going to miss the wave. Mhm. But the whole point then is can you do enough of those where you can be with the wave as opposed to way behind.

There's always all those things. I think when we won Jeopardy with Watson, I think it woke the world up because I think for the first time in a long time, AI started doing something people thought it could not do. Unfortunately, it woke the world up completely in the sense that a number of other companies started investing very heavily. Now, we had an advantage. We might have been able to succeed, but I'm not putting on 2020 vision looking backwards. The mistakes we made were actually much more of a strategic nature. As opposed to creating building blocks, we wanted to create solutions in verticals that I think is a mistake. As technology shows in the beginning, it's too quickly to the application. We went too quickly and we wanted to make a monolithic application. Mistake number one. Mistake number two, we picked a domain which is perhaps the hardest of them all which is health. Mistake number three, how many IBM sell to doctors and how many IBM deal with the FDA? Like none. So you picked the wrong solution set in an industry you know nothing about and with a customer you know nothing about. Other than that it was pretty good.

So we are not going to be a hyperscaler which was two of the four you mentioned and we're not a foundation model provider. I'll say something provocative. I think foundation models are going to become commodities by the way not that far out. Is it a year, is it 2 years, is it 3 years? Commodities doesn't mean that they don't have value. Gold is a commodity. So is iron. So commodities means that there is very little switching cost to go from one to the other. Second, I think that right now the token price on all of these is going to go way up. It just has to to justify the capital investments. You put those two together and there's going to be a huge motivation then from everybody who's using them to say I need to optimize. I need to use each one but in the most economic way possible. Our role is to a let our enterprise clients do that. Two do it in a way that is safe and right now there is very little demand for on premise or smaller models which are effectively then 100th of the cost to run. So I'll use the analogy in a very crude way. In the end, an automobile, which I'll include trucks into it, is an automobile at some gross level. If you step back, I guess you could, for those in the suburbs, take your kids to school in an 18-wheeler every morning. You could go milk shopping in an 18-wheeler. Then you'd ask yourself, is it really the most effective vehicle for that? But if you're moving homes, which you do every seven years on average in the country, it is the most effective vehicle for that. I think right now we're using the 18-wheeler for everything.

And this is the transition that I'll predict will happen within 24 months. I'm not sure it'll happen within 12 months. I'll point out to you underlying GPU pricing, which is what all of these things run on, has doubled in the last 6 months on a per hour basis. So it's getting more expensive to use these tools and right now when you're pre-public it's okay to lose money because you're gearing towards number of customers. I think you've seen this before, right? It used to be called eyeballs and then it suddenly became the economics are important. So I think we're right maybe a year from that point.

So, first let me be clear because I can sound a little bit cynical about the economics and I actually am. That said, I think AI is an incredible productivity tool. I think those who don't take advantage of it will be perpetually disadvantaged compared to those who do. So, let me begin by saying that it's going to optimize how you market. It's going to optimize how you write code. It's going to optimize enterprise operations. It's going to optimize how you sell. It's going to optimize how you get your daily work done. So there is no question about it. It's going to make a profound and deep impact on all of those things. By day zero, I mean it's time to sit down, take it seriously. You're not in the experimentation phase. So this is not like you're in high school. Day zero, the race is about to start. Put yourself in the blocks and start sprinting. But by that I mean take three, four, five things, not a hundred, and learn how to do them at scale because that'll teach you how do you get all your change management done. How do you get your data organized? How do you really get people motivated to change a process? So do a few things at scale, learn how to do that really well. Now do 10 and then give yourself the confidence to do the next 20.

Yeah. So unfortunately I think corporate performance is even more of a differentiated K. If you look at corporate performance actually it tends to be a 2080 rule more of a power law. Then the K is really more of a 50/50 I think if I follow the economist correctly here it's more of a 2080. And so 20% get it. They go forward. They're jumping into it. They kind of are going to get their returns and 80% are either not getting a return or don't quite know what to do. If you're in that 80, figure out what is it that you should do and it probably doesn't matter where you start as long as you're starting to try to do it at scale to make a real difference to your bottom line.

Find that 20 or 30% of them who are motivated to say I want to learn a new way to do things. So I think curiosity, a willingness to adapt is more important. I think we're getting hung up on I need to know AI like a PhD in computer science. I think that's the wrong thing because that's for the inventors of AI. That's not needed for the deployers of AI. There's also this idea that AI is going to save me a lot of money. It's going to be very efficient for me and I think for a lot of businesses when they start implementing those results don't necessarily come. Now you you guys have talked about that you've unlocked four and a half billion dollars of efficiency from AI. You're doing something. But there are also these hidden costs as you mentioned tokens like how do you balance what are the costs versus the efficiency and what you should be expecting.

Yeah. So this was my point of scaling. I would probably turn around and say that for our first six months to a year, we were probably spending more than we were saving because if you think about you're putting a couple of hundred engineers to work at it, that's an incremental cost. The underlying infrastructure aka the tokens if you're doing it on public is an added cost. There's opportunity cost also of not doing other things with these people that could have resulted in revenue. That's a cost. Now once we learn a rinse and repeat method that you're not doing it across two or three but across 10 or 20 when you're saving a billion dollars a year well that's a lot more than the cost of a couple of hundred people after year two we were definitely getting a return that was 10x compared to what we were spending and now at year four we'll be I think over 5 billion from a baseline of 22 spent so that's not an incremental five over last year that's an incremental saving compared to our year-end 22 spending. So that is tremendous. That is more than enough to offset any extra expense. And there was a CEO I was talking to about some of these issues, the ineffective the inexact nature of some of the outputs you get from AI, right? I don't want to call them hallucinations, but the things that don't go the way you want.

Well, what he was saying was that the money that he was saving by having his engineers use AI that on the few cases where it was wrong, he had to spend so much trying to find what was wrong and fix it that he wasn't actually coming out ahead. Yeah. So, I actually think that that is an edge case of how you use AI that I think is actually wrong. I think that you should try to use AI in a case where you're not going to have to go undo 6 months of work or undo having spent hundreds of millions. Take customer service. If it gives a wrong answer, you got to undo one customer service answer. Then you can put all kinds of evaluations and checks. So AI can check itself to make sure that you're not like way off in the wild. Okay, it may be slightly off, but you're not way off. So you can put checks and balances and this is the sophistication of how you use it. So when we use it for example for our software developers to help them code I don't think they realize it. We actually have checks built in to make sure that what it is suggesting is not absolutely crazy. Right. But do you don't you I mean as with a human worker you have to expect that sometimes it will go wrong. It will go wrong. I kind of turn around. If I look at customer service, I think the stat which would be a good one is 85% of the time the human people get it right and 15% of the time and I say of course humans get angry, humans get pissed off, humans may not like the tone of the person on the other end if it's a call. Humans are sometimes overconfident. I'm sure we all remember things perfectly, right? I mean perfect. You and I do. I'm not sure. Have never told us that we are completely in the wrong and remember it completely. So humans have all those too. I think AI is at least if you keep it constrained to some extent it's probably 95% correct. So the eval I'm talking about is more like saying when you think you're way off punt it to a human. Don't try to venture into the underconfident range.

No it's not actually. You'd be surprised if you tell it don't pretend to be confident. You will get from it that hey I'm not quite sure that this is as this thing and you can put a something to check it who's rewarded on actually pointing out the other's mistakes. So now you have multiple models working at the same time. I mean I guess where this leads me is there's a lot of it's just like humans you put four eyes we call it four eyes in software development you put two people one is coding one is checking their work just like the models one is doing something the other is checking its work and when you but when you have two models instead of two people does that mean that you need fewer people I mean that is one of the you know you got some grief a few years back for saying your back office was going to get smaller which seems like it's small change compared to the things that some other CEOs are saying right now. Do you think there's going to be a lot of displacement?

So I'll address both parts of the question because it's a and it's not a. So our software developers are probably 40% more productive today than they were two years ago. So I'm not saying it's a one month over two years they're 40% more productive. So you could turn around and say that means you need 40% fewer developers. We actually tripled our college level entry hiring this year. Tripled compared to last year. So you say wait that seems off. No because this is what people are missing. If my cost of software development is going down that means we can make products that were not economically affordable 3 years ago. If we can do those we can get more revenue. Add an appropriate margin. So why wouldn't I get more people? These are value creating sales, marketing, consulting, development are value creating. Then there is the 20% I'll call it is what you need to run the operation. So is it compliance, is it accounts payable, is it procurement, is it all those things. It's not going to go down to zero. But I would not be surprised if about 30% of the total headcount in those areas is not needed within a few years. That's the statement I had made and I'm actually still consistent. Note I just said we tripled our entry-level hiring. So while there is some decrease on this side in about 20% of the enterprise, there is a big increase in the remaining 80% of the enterprise. So I actually think net we'll have increased demand for jobs but there is some displacement which is always a little bit painful and always has happened with new technology.

Well, actually I think that if we can get five to 10 points of productivity in every enterprise around the planet, everybody is a beneficiary. Tech may be the early beneficiary, but I will note everybody in tech right now is losing money in it. So we can claim is it going to be a long-term beneficiary or not? I think there's open questions in that. I think everybody's going to be a beneficiary. I think that in our societies, at least in the West, the responsibility of business leaders is to provide an opportunity. So we want to help our people get upskilled. We want to help our people get reskilled. We want to open up that there are other opportunities or jobs. We can't force them to do any of that. So then it's on them. Do they want to take advantage of those opportunities and step up to do it? And I would say that answer has always been about 50/50. Some do, but a lot of people say, 'I don't want to get retrained. I want my old job.' Okay, I'm sorry. That's not going to happen. And those folks that's ends up being the responsibility of government and society not necessarily of the business. Correct? We try to be compassionate. We don't force people out like in a day but if over 6 months or 9 months they are not willing to learn the appropriate skills where they needed that's actually bad for the other 90% who are around them.

So first for the few geeks in the room a mainframe is actually one box but a mainframe is designed differently. When we say data centers nowadays, what people are intuitively implying is there is a collection of similar boxes, hundreds, thousands, tens of thousands, maybe hundreds of thousands of them in a single data center and the work is such that you can divide it up amongst all of these and they can talk to each other if they need to collaborate. That's the network or the optics that does all that. That's a data center. A mainframe, while it could be used in that context, rarely is. A mainframe is really useful when you have one piece of work that has an incredible volume. Example, airline reservations. If we sell you the seat, you probably don't want to sell the same exact seat on the same flight to somebody else. That would be inconvenient. So that is a different kind of workload than you asking an AI model a question and Joe asking it a question and me asking it a question. That can be divided up because it doesn't need to know all the three answers. It knows what it's going to do. So the work is inherently can be divided or parallelized. Part of the reason I asked is because you know IBM was is sort of the mainframe shop, the OG mainframe shop, right? And there was a time where mainframes were sort of seemed like they were I don't know pay everything was going to the cloud and that has shifted like suddenly they're back. I'm sure the mainframe people don't like the idea of saying they're back, but I remember in 1993 there was I think it was Time magazine where they were showing a mainframe dressed up as a dinosaur and it was called the death of the mainframe. It was only 34 years ago and then every 10 years people talk about the death of it. I think you got to be a bit more astute. What is the workload that is great for a mainframe? What is the workload that's not good for a mainframe? And as long as you stick to that, I really am a believer in fit for purpose. The same way as a GPU is probably not ideal for running your smartphone because you kind of want your battery to last all day, not be over in 3 minutes. So there is a fit for purpose that is underneath these things. Where do you do AI training? That's one kind. Where do you do inferencing? That's a second kind. Where do you do web serving or streaming? And well, and if you want to keep things secure, you want to have them on your own premises. Sovereignty also comes into play because especially if you're outside the US, people care deeply about which government has control over the tech stack. So all of those things come into play for where you want to run things.

So first for the good news, at least in our case from the things that we've been running for the last few months, Mythos didn't find anything that other models didn't and couldn't find. I'll call that the good news. Here's the bad news. We have a lot of people, tens of thousands who are experts in using these models to try to find vulnerabilities and then go fix them. So for the expert, they could already do all this using other models. We will completely acknowledge that Mythos is way easier to use than the past. So what it did do was it opens up the attack surface to where I don't need one of those 100 experts to go do it. I can now do it with somebody with average skills. So that is definitely something to be worried about. So when Mythos came along, it's not just Mythos, the ability of these foundation models to actually help you write code, to understand code is also there at the same time. So the same thing which could be used to exploit, we could turn around and say, can I use it to fix at least all open source? And that answer became a very quick we can. So we said as opposed to only worrying about oh my god I got all these things and like okay I have my list of 10,000 of them. We said can we do something but it's not altruistic. It is good for society but we do intend to charge people a fair price not a usurious price for it to say can we instead turn this into a utility where people can come to us we can be a clearing house so that they can get their fix after giving us the vulnerability but we can share to others who are inside the closed set also that hey your friend here found a vulnerability we're not going to tell you which friend and we're not going to tell you where they're using it so that that information is anonymized and protected. But you can actually get the same fix if you want. And yes, we are throwing a lot of people at it. But despite throwing that many people, without using the current AI tools, it would have been impossible for us to go about saying that if you give us a piece of open source, we can actually give you what is in our belief a very well constructed patch of fix against that vulnerability.

It's the old I think it was Billy Sen, right? Why do you rob banks? Well, that's where the money is. Why are you attacking cyber infrastructure? Well, today that's where the data is, which is where the money is. That's why I began by saying the good news is that it's not really a brand new. The bad news is simply it'll be done faster. So nation states have been doing this for decades. But you would say three or four nation states were capable of doing it. Maybe that opens up to a couple of dozen. That means more. And I guess it means organizations that might otherwise not have been targets. Smaller midsize it's the bar potential function has come down. So it's easier to target more. Right? So we all have to be a little bit more prepared even before we reach the scale of. I would turn around and say if you don't think you're protecting yourself it is only a matter of time it will come.

So, let's go back to your very first question. If you can get ahead of the curve and if what you're doing is hard enough that you actually have a couple of years advantage, our industry, the tech industry has shown that you can create outsized returns for yourself and outsized returns for your clients. In doing that we felt that quantum is going to be one of those. We actually came to that recognition many years ago. Then the question became can we do the hard science it takes to be able to make progress. I would say earlier this year we convinced ourselves of that. The evidence of that is both in our $10 billion investment because that means we expect to see a real return on it as well as in the government agreeing to invest because that is a sign that they did their homework and agreed it is now time to scale this as an industry. So I think you should think of quantum as doing the following. CPUs, we've had them for 60 or 70 years, do a lot of great problems, right? GPUs came around and did a different kind of problem. They did matrix math that allowed AI and other things to happen. But in some sense, it's not that CPUs couldn't do it. They were 10,000 times slower to do it. So last summer, summer of 25, they could simulate a five atom molecule. I'll be honest, a five atom molecule, a really good computational chemist, if it's a simple molecule, could probably solve by hand. Maybe an expert, but they could do it by hand. So you'd say, okay, your quantum can do it, but who cares? Last winter, November, December, they could do a 300 atom molecule. That's good progress. Now you're getting beyond what you can do by hand, but you could do that on a normal supercomputer pretty easily. So you say, okay, you still haven't told me that this is interesting. In April, they did 12,000 atoms. They're now getting into the protein realm. When you're in the 10 to 30 to 40,000 atom range, you're in the protein realm. So this was a piece of a protein called trison. We're pretty sure that in another month or two we'll be at double that range which means you can solve trison. If you can understand the properties of a protein using a few minutes of computation, you can now understand which molecule aka a drug may bind to it to stop its bad behavior. We've now opened up a new pathway possibly for health which didn't even exist.

Correct. So I think we're solving problems now whether it's in I talked about biology and molecules because I think most people intuitively get that that's a hard problem but there are problems deep inside things like fluid dynamics in computations to do with aerodynamics to do with how liquids flow inside pipes all of these are problems that are now coming right about now within the range of quantum computers. Years to solve. Understand where quantum may be in two or three years. So understand what kind of algorithms you may need to develop. So when it is there, you don't then spend two years doing all that. That's what I would recommend to people to do today. But that makes it an easy and that's not a choice then. It's not a dilemma to say which of the two do you do?

We want to be known for not just being technologically innovative. I think that we had that reputation. We weren't always great at making that easily accessible to clients. So I think we want to be in the position where we are bringing all of our innovation to clients in a way that they can easily consume. That's one big piece. Two, we were about 20% software in 2019. We're now about 45%. I think that number will keep going up at a few percent a year. So we'll be much more in that space than anything else. I think we'll become known as one of the exemplars of how we deploy AI and agents to not just improve our own business but to help improve our clients' business.

So I would tell everybody the most risky route is taking zero risk. What happens in any business that takes no risk? That means that you're kind of trying to extract profit or what an economist would call rent from what you already have. But that means you're giving everybody else the opportunity to clone you or copy you, be innovative from the bottom. So they will pick off the most profitable parts of your business. So now you have a declining profit pool. Now you begin to have declining profit pools and if you're conservative by nature, you're going to invest even less because you're getting smaller. So you're going to accelerate your decline and you'll be approaching a cliff without realizing it. Go read history and see how many companies behave like that when you sit and watch them. Right? It takes about five years and you begin to hit a decline and then you're going along and then five or 10 years later somebody comes along and either bites you up and chews you up and spits you out for parts or you actually just go and fall over into oblivion. So I call that the most risky. So how do you maintain enough innovation that you're actually growing? All innovation does not pay off. Innovation is risky by nature. So, you got to say, how do I manage it where I'm generating enough profit to pay for innovation, recognizing that not all of it will have a long-term return, but enough of it should. That's the nature of it.