Business AI in 2026: What’s Working, What’s Not, and What’s Coming (w/ Databricks CTO Matei Zaharia)

As I said, at minimum, LLMs are really great at basically search and approximate matching. So, at minimum, it'll help you surface that information and then you do something with it. Maybe in the extreme, it can actually, with high probability, just give you the right answer or give you a comprehensive report that tells you this is a little bit unclear or whatever it is. But people, you know, there's so much like people just see the sort of sci-fi version of AI, which, you know, they ask it to do things that it can't do.

Mhm. Yeah.

No, that's it. Yeah.

Yeah, it's a great question. Yeah, I think there's a lot of hype around it for sure. So clearly, you know, you'll get all kinds of answers. I do think it is useful for certain things and we're figuring out over time how to make it better for more of them. But not for everything. And I think one of the challenging things about it is that it's so easy to get a demo that works okay, you know, 50% of the time or works in a few examples, and then you think you solved the problem but actually getting it to be like highly reliable and diligent is not for sure. Like some things that it's good at and they're being used are any kind of search basically, trying to, you know, surface information, put it together, it helps you find things, but usually you have a human in the loop who looks at all the results. Coding agents, again, depends how you use them. It's not going to code a complete application for you that's like flawless and works, but it can help you navigate a lot of code, prototype things, do a small amount of code, things like that. And then there are others, you know, there sort of others that are exciting but maybe harder to get going.

Yeah, you know, I do think we can figure out how to use current models better. Actually, in my perspective, models haven't gotten that much better say in the past year compared to like the year before or the year before that. So they are in some sense slowing down and you could do similar things with today's models that you could like a year ago. Maybe it's cheaper. They've gotten faster and cheaper for the same quality, but that's sort of what it's like. So these models are not perfect. They make mistakes and they, you know, they can fall over, especially if you give them a lot of context for example, they'll just like stop working well. So we have to figure out, you know, what to build with them that's useful. So I would say a lot of the advances in actually applying these models in a way that's useful have been partly like user interface, like HCI advances. Like just as an example, everyone knows when ChatGPT came out, people were excited. But before that, OpenAI had something called InstructGPT which would follow instructions. The difference is you send that one message and it sent you one response, you couldn't send a follow-up message. So in some sense like ChatGPT was a user interface innovation where like when the thing misunderstands your question or makes a mistake, you can tell it like are you sure that's right or, you know, think about this and then it like follows up. So and that chat form factor worked really well and same thing with the coding agents, like even the initial like tab completion ones, right? You press tab, it shows a few options and then, you know, some of them are bad but like you pick the one that you like and continue. That's, you know, that's a UI that made it good for that particular thing. So I think we'll figure out, I mean both the techniques for like building agents but also how to, you know, get them to interact with people. And some, yeah, now what kind of things will it like, can it replace like all employees in like some company or some department? Probably not, at least not if the models stay at the current stage. But there are a bunch of things that it can accelerate and make better. Actually, the other thing I'll mention is there are things now that like people don't look at at all. For example, you just can't have humans like read vast numbers of documents that are posted every month in the financial markets or things like that and maybe you can have an LLM look at it. Even if it's not perfect, it will give you some summaries or some insights that are useful. That's actually one of the biggest use cases we're seeing is this sort of batch data analysis where like no one would look at this unstructured data. Now you do something with it and, you know, that it's not perfect but like you take that into account in what you do downstream.

Yeah. Yeah. And then this is also where I think the user interface things. Like for example, what if it gives you those and then you could in a chat you could send a follow-up like you could say, 'Oh, give me more that have this use case which I'm seeing popping up this week or whatever,' right? Like that would be an HCI sort of thing where suddenly like this is useful. You never as a user you don't have to look at all 10,000 leads, but you can ask some questions about them and get the ones you want.

One of them is like not being clear about like what outcome they want or maybe how to evaluate that outcome. That's one of them or like, you know, you build something but you can't tell how good it is. So, I'd say it's not necessarily a technical reason because if you knew how to evaluate, you might be able to build a better agent or whatever for the thing you want, but you just don't know like for example, you don't know that it's going wrong in a certain set of questions. And you just like deploy it and then it doesn't work and people don't like it. Yeah, I mean maybe even higher level is like the wrong expectation about what it's going to do. Although I think people are learning more now about what's possible and what isn't. But I would say okay if you do kind of know like what AI could do, what's practical to build today and you try to do something, you invest, it still goes wrong. Maybe it's because you didn't have a good plan for evaluating like either during development or even once it's deployed. Can you tell how it's doing?

Yeah. You know, the way I think about that is you have to ask to close that loop of feedback from users or from whatever the use case is as quickly as possible. And I think it's easier than ever now to build prototypes of things. And with especially with generative AI like your prototype of an agent could just be a prompt or it could be you just tell it like go do this and maybe it's like really slow, maybe it takes a lot of steps but you can see, you know, does it have legs like is it likely to do this task or not. And then you can get people to try it and I would go even further and say like as an exec like you should try it, right? You should like sort of be critical, like, you know, try using it for a bit, see what happens. Obviously may not be possible for everything. But that's what I would do, having the tight feedback loop. If you're starting with saying well we need to first like staff up the team with like 20 people then we need to like pay to build a data set then we need to like spend three months investigating the framework we'll use to build it and like all that, you already invested a huge cost when you know maybe there's a very quick way to prototype to see is it possible, like what is it good at?

Yeah. And another thing you can do here, it's like prototyping any software. Let's say there's a piece, you know, you think could be useful, but you don't have it yet. Like let's say if we collected or computed like a new data set in some way that would be useful. But you know, doesn't mean you need to build like a huge pipeline and a big like serving system and stuff for that. You can try running an agent where like you just handcode that information. You tell it, you know, here's this stuff and like see what it's going to do. And see if it would help to build that. We do that a lot here when we think about like what we want the assistant in the product to see. For example, like we can prototype stuff without fully building it out and then see, oh yeah, if we provided this type of, you know, tool to it or whatever, it would do better or it wouldn't do better. Yeah. So anything like that that can help you close the loop to get feedback, quick iteration, quick feedback loop. Yeah. Like sort of look for like what's a way you can tell like could it work, you know, at all and then if it can you invest in making that like more scalable, more solid, faster, whatever it is.

Yeah, yeah, I mean the other way you could do it, I think that the traditional way, like we kind of know if you know that okay if we invest this much time in a thing we can build it, then it makes sense to like perhaps go all the way ahead. But I think here with a lot of what people are trying to do with AI it's still pretty experimental, you don't know what's going to work. There might be a few things that are getting into that realm where like for example let's say you build some question-answering agent or something, you know, and you say well if I gave it this one, this other API to like, you know, look at customer whatever like transactions in real time or something, like could it answer questions better? Like, you know, there are small changes like that where you're like pretty sure that yeah if I build this it will do it and you can just add them in. But if it's a very unclear, like you haven't seen many people build this type of application, then I think you need to be in more of that experimentation mode.

So obviously it's going to depend a lot on the company. I would say for sure anything where people need to like find information or even synthesize and summarize information, but there's a human in the loop working on stuff, I think it's very likely that you can help them do that and accelerate it. And that's I'd say at least like half the use cases we see. And even when I use stuff internally here, it tends to be that it's like, you know, I have some question about what's happening in the company and there's information across documents, tables, maybe many different data stores, maybe there's a little bit of analysis I want to run on it, and, you know, but it's all out there and like if I had to do it, I would just need to spend a lot of time finding each thing and putting it together. So that type of thing I think for sure you can do. And, you know, so like for me for example as CTO here, like I don't know what are examples of questions I asked. I mean one, you know, one example would be like for our Agent Bricks product, like what are the top, you know, concerns in like recent customer interviews? So it's about looking at all those customer interview notes or which customers, you know, talked about this, like use the latest version of like vector search or something, right? That's like joining a table with which has the actual events and like is the person using this or not with this unstructured data essentially in the document, like let's say they complain about latency but they didn't try the low latency version of like vector search or something, like that's kind of what I want to find. So these type of things are good but it can apply to anyone. Like for example we also have, you know, when there's any issue with a service here, like the engineering team is paged and normally as an engineer you get a big, you know, you get a whole bunch of dashboards you can look at and logs and things to figure out what is going on. So the team that runs those also built some little agents that, you know, while when the incident is happening the agent will go and run and sort of try to send you anything it found that looks unusual or you can ask it questions. So it's something like this that's good. That's the easiest one. There are cases where maybe you can automate more of a process if there are some standard decisions. I think as I said before one of the biggest ones is if you have like, you know, unstructured data like text or images, documents that like a person wouldn't even look at, for sure you can use AI and get a lot of value from that. And we've actually seen this with some companies that are, you know, who a big part of their business is essentially analyzing some data sets and they had very custom pipelines for it, you know, NLP models, regular expressions, whatever it is, entity matching, you name it. And some of them are switching to LLMs and finding that it's, you know, more flexible and you can get it to give good results. So, yeah, these are some of the ones. The things that I'd say the AI can do well is on its own, like diligently doing a task and getting lots of things right like for, you know, over a long duration and just giving you the right answer at the end. That's pretty tough. So it can do that very well. If you have a human in the loop who can verify stuff, then that's good or and who can even correct them by sending a follow-up message, then it works. There are some rare cases where you can have an automated verifier and that's where I think actually AI is most powerful today's AI. So for example like optimize this assembly code for like matrix multiplication and, you know, try to minimize the time it takes to run this benchmark while passing all the unit tests, like that's people have done that and they found like new, you know, new micro sort of algorithms and stuff for that using AI because it doesn't have to be right every time it just has to make like thousands of suggestions and then one of them might be better. Yeah. By the way, to capture some of these things if it's interesting, one of the things we're putting out is we have this sort of frontier AI benchmark which is not getting it to do like math olympiads or like, you know, programming contests or like weird obscure humanities last exam questions with obscure facts. We call it OfficeQA. All it is is you have a bunch of documents like PDFs from the US Treasury and you try to answer some questions that involve looking at a bunch of data like, you know, compare the interest rates in different years or whatever and tell me like what the, you know, peak years were in this time period or whatever. And so we specifically designed it so it's things that don't require like advanced math or world knowledge, finance, anything like that. They just require reading documents like a high school student could do them. But they do require diligence and, you know, like focusing on it and getting all the steps right. And this is one where like current agents and models get maybe like 30, 40% at best on this.

I think that there's a couple of times when. One is if you want it to be very sort of deterministic and predictable. So you always, you know, get the same answer, that's pretty hard to do with LLMs and with AI today. And it's going to change anyway when you change your agent even if you get it to run that way today. And then I think the other one is for cost really, like you have to see if there is, you know, an easier way to do it, like both the monitoring cost of running it, latency, you know, yeah, maybe the other one would be if it's an adversarial setting where like the stakes are quite high of getting it wrong, you know, if someone can sort of trick the agent or whatever. So, because that's another like the weaknesses of LLMs today, you know, they sometimes get things wrong. They're not deterministic and they're all attackable through adversarial inputs. If someone really wants to get it to like, you know, say a bad word or whatever like or you can or worse. Yeah. Or worse. Yeah. So, yeah. And it is happening sometime that people prototype a thing with agents and then they realize actually I could have done something else. Yeah.