An honest look at AI agents | Pioneers of AI

You should never take a broken process and use AI to fix it. It is clear that there's an immense appetite for these agents, and in a sense, maybe this is the first of these agent platforms to sort of go viral on the consumer side. And so that starts being super interesting to just explore the diversity of things that people are interested in doing with our agents. You don't know the creative ways in which people may be using these agents for nefarious purposes till you really deploy it at scale. You can do all the red teaming you want in house before you go, but actually when you go at scale is when you discover all of that.

Hello. Great to be here.

I decided to make the jump in 2017 because already we were seeing a divergence in terms of the type of research questions that you could take on. I think academia is wonderful in that you have this influx of new students year after year, and that is just such a source of genuine curiosity. So I find usually, you know, they ask wonderfully different questions, which really stimulates the exploration and the curiosity. On the other hand, when you're doing research in academia, you know, you do have to build in a lot of time for the learning process and the development of these young talents. The teams tend to be smaller, more junior, where on the industry side, you can put together larger teams with a mix of engineers, scientists, designers, user research folks. You get these bigger and multifunctional teams that just allow you to tackle bigger problems or do bigger projects. And so fast forward to today, I would say there's some of these same differences but magnified significantly by the amount of investment that is going into AI in industry. Whether it's compute, whether it's data, whether it's talent, the size of the team, the ambition of the projects is just on a very different level. That being said, you know, like research doesn't always happen at scale. Sometimes some of the insights come from much more, you know, small teams working really diligently on a specific problem. So I think there's still a role for both, but very different mode of operating.

Interesting. I think I'm still digesting some of these lessons. You know, sometimes you don't know that you've learned something till suddenly you're in a context where you can apply those lessons. It was an immense learning opportunity for me to be at Meta at the time that I was. The FAIR, the fundamental AI research team I was leading, grew, you know, from a few, you know, maybe a hundred people to 600 people or so. And so, you know, one of the things that I developed over the years is the ability to push teams to have ambitious projects, ambitious road maps, and a clear line of sight to a concrete goal. And so coming into Cohere, you know, bringing some of that culture of how do we work as teams? Maybe the teams are smaller, smaller company, smaller research team, but nonetheless, how do you set up the team in a way to tackle a goal together? That's one of the things that has been quite helpful.

Oh, I think it's about clarity of the objective. Like, do you have a clear sense of whether you succeed or not? Having clarity of that north star in a way that's measurable helps really align people towards a goal. Now you have to make sure it's a goal that's worth doing and that people are excited, that they have a path to be successful, they have the right resources, all these other things, but it's a lot easier to get buy-in of people, of the research team, leadership, and so on, once you have that clarity of goal.

There's two core pillars. One of them is we do build our own models, and the other one is this agentic AI platform that we build, which is really the core of the product. We have North, an agentic AI platform. It's designed to ingest the models that our modeling team is doing and deploy that so that we can, you know, bring that into different companies and governments who have the ability to benefit from AI agents, coordinate these agents to build automations, and in the end of the day bring a lot more productivity to their work. You can think of the models as the engine, and you can think of North as the vehicle to essentially deliver AI. And there's some places that are ready to ingest like a raw model. Our models are very good, for example, in terms of their multilingual capabilities, in terms of their reasoning capabilities. Our models are also really interesting when you want a very good trade-off between performance and efficiency. But actually more and more we're seeing a lot of demand for having that engine be built into a vehicle that can deploy AI. So that means, you know, there's a UI. Your end users can actually, you know, chat with the model, but they can also build their own agents, they can exchange agents across a team, we can build automations that orchestrate multiple of these agents, and building up a much more cohesive experience for how you bring AI into your workflow. And I would say the other really important distinguishing aspect of what we do is to deploy on premise with very high security and privacy guarantees. So this isn't just like a, you know, API or a cloud thing. This is something that you can run locally with your own data. So you can have a conversation, or your agents can see actually all the internal data from the company without that data coming all the way to Cohere. But locally deployed, it means that you have the ability to leverage all the business intelligence in your data and in your systems internally and do that in a way that's secure and private. So, especially in regulated markets, financial markets, healthcare, government services, you do have obligations to give strong privacy and security guarantees. And so there's a lot of potential for that and a lot of interest.

Same. Exactly.

So like, take anything I say with a great, great grain of salt, and hope by the time this lands in your viewers' inbox, you know, we may be way off in terms of our understanding of that. But there's a couple things. One, it is clear that there's an immense appetite for these agents, and in a sense, maybe this is the first of these agent platforms to sort of go viral on the consumer side. And so it's super exciting to see, you know, I don't know, I think the last number I saw was 150,000 of these agents getting created. Each agent with their own data, with their own incentives, and with their own set of skills. And so that starts being super interesting to just explore the diversity of things that people are interested in doing with our agents, just from like almost a sociological point of view of what do people want out of agents. That's a fascinating, fascinating development over the last few days. The second consideration is all of the safety questions. You know, is this vulnerable to prompt injection?

Yeah, like the ability through a prompt to inject information that would lead the agent to take on malicious behaviors. Out of that, are we going to discover many different ways in which, you know, we didn't think of vulnerabilities in our systems? Quite likely. So it opens up the door to discovering, and it opens up the door to new vectors of harm. But I also think, you know, the faster we learn about this, the better we're able to build in the right mitigations for it. And in many cases, you don't know the creative ways in which, you know, people may be using these agents for nefarious purposes till you really deploy it at scale. You can do all the red teaming you want in house before you go, but actually when you go at scale is when you discover all of that. The third piece I want to highlight about this OpenClaw platform, which I think is possibly even more fascinating, is the interaction between all of these agents.

So so much of AI, you know, we build agents thinking they're going to be unique. They're going to be deployed and they're going to go out interact with a stationary version of the world. Now suddenly we see what happens when you deploy against all these other agents. They start interacting. They start negotiating. They start trying to offload some of their work. All these other interesting behaviors. I've been looking for a platform to really study these multi-agent systems at scale, and we haven't had one in the space of agentic AI. We've had it in a few other contexts. For example, you know, with autonomous vehicles on the roads, we're starting to see some of these effects. And as much as on the vehicle side, you know, for many years we worried about the trolley problem. And in the end, like the problem that really turns out to be hard is like all the Waymos stuck in a parking lot, like spinning their circles, not able to get out because they're in some kind of like pathological loop with each other. So what is going to happen in this case with all these LLM agents talking to each other and exchanging information, deploying their skills, using tools? I think that sandbox and the ability to look at the network effects across agents is going to be super interesting, and so I'm quite curious to see what comes out of that and from this what we can learn in terms of some of the essentially like social norms we need to build into these networks so that we don't have too many pathological behaviors.

I think one thing to keep in mind is we're still very, very early days in terms of deployment in enterprise. And so whereas on the consumer side we're already seeing the rise of the adversarial behaviors very, very early, I think on the enterprise side we're not at that stage yet, frankly. Like we're seeing all the ways in which the agents are not meeting expectation of the users, but it's not adversarial. I think there's a bit of a gap in terms of those two environments. So as a consumer, like you're kind of responsible for your own privacy information, and I think we've long stopped expecting the platforms to handle that for us. On the enterprise side, there's still an expectation that the platform will provide high levels of privacy. And so we can't afford to take some shortcuts on that. And so I think like that's one of the major differences I'm seeing in these early days. But again, so much more that we need to learn on this.

I spent the early part of my research career in academia where it's very natural to share all the artifacts of your research, maybe not in all fields, and certainly, you know, in some of the fields whether it's medicine and healthcare and even some of the biology and so on, there's probably different standards in terms of openness. But in computer science it's pretty standard to share the code, the data, the paper, all of the artifacts that capture the results of the research process. And so open source is about essentially like sharing all the information that was used in the course of the research in a way that facilitates the reproducibility of that research result. When it comes to LLMs, in recent years, we've seen much more movement towards open-weight models, meaning that the weights of the network are shared such that someone can download that and essentially like rerun the network locally, fine-tune from that, then adapt it, fine-tune it, and so on and so forth. But often some of the other artifacts are not shared. So you might not have the training code, you might have only the inference code, you might not have the training data. With respect to Cohere, I would say, you know, their position on openness is one of the reasons that drew me to them. I think they have a great track record of sharing the results, whether it's Cohere Labs, the results of their research, or whether it's the Command line of models that are coming out of our modeling team. I think there's incredible value in terms of making our own work better, holding a high bar, and also really empowering the community to keep building with us.

There's a couple things. I think you talk about, you know, both like the formal evaluations and then the more informal experience of the model, which takes a different shape depending on what's the model. I think more and more we also have the challenge that as we're building general intelligence, it's not, you know, whether it's the automated metrics or whether it's the human experience of the model, both of that becomes much more multi-dimensional. And so, you know, let's say you have just a model for translation from one language to another, like there's some pretty standard metrics, BLEU, ROUGE, and so on to do machine translation, and then there's the experience of it. But in the experience of it, you're just going to expect to look at translation. You're not going to start expecting like a chatbot that can, you know, solve all sorts of your personal problems, gives you some advice, handle your email, all these other things. Like it's a narrow set of qualitative experiences as well. As the models get more and more general, models that suddenly are multilingual, multimodal, including chat and reasoning, agentic behavior, tool use, you start having a huge surface of automated tests. But you also have like a huge surface for humans to form an intuition, and it becomes quite difficult to know, you know, what to look at, quite frankly. And we have that problem in practice. Like we train up a model, we think it's a good model, but then it's like out of this whole set of evaluations, like which one do we look at?