Good morning. One of the foundations of scientific discovery and technological progress is really building on each other's ideas, sharing ideas. Each of us, of course, builds on the discoveries of others who came before. And so I'm here to advocate for a vision of AI that is open, that is responsible, that is democratized. And it's really a pleasure to address this topic this week with all of you at an event where there's so much open exchange across many different actors of this ecosystem.
Now, each of you have seen the progress through your own eyes just in the last year. I was here a year ago, and it's a pleasure to be back. So much has changed since then. We don't need to go back a whole decade to compare. Just in a few months, we have seen how much progress has happened, in particular in some cases going beyond even what science fiction might have imagined for us.
We've seen increasingly large models become the norm. This is a very telling plot from the 2023 AI Index. It shows a clear exponential acceleration in terms of the model size and the intensity of the effort to train these models. When the models are trained well, the size of the model is one of the best predictors in terms of performance, in terms of how general the capabilities will be.
Now, as much as we've made progress building larger models and better models, more general, more powerful, a lot of our most important work remains, and that is building models that have common sense, building models that actually bring lasting values, that foster genuine, authentic interactions, and that really enhance our understanding of the world as well as our understanding of other people. And so this is the journey that is just beginning.
I'm going to use our time together today to share some insights, in particular perspectives on the AI transformation, the trajectory, some of the gaps that we're seeing as well that we're going to be working on going forward. Beyond all of this, I will share how we think of making progress using a culture of open science, using a culture of collaboration, and how that's going to shape the trajectory of AI going forward.
To really ground us in that trajectory, I do think it's helpful to look at a concrete example. I know we've talked a ton about language models this year. Let me start with some of the work on a more modest task of AI, really this task of detecting objects in images and segmenting the image with information about the object. This is a task we've worked on for decades, but if we look back, 2012 was really a pivotal time in terms of progress in this area.
For the first time, we have models built on neural networks, deep learning technology, much of it invented here in Montreal, which for the first time could give us better prediction accuracy over hundreds of types of objects. This was 2012. By 2015, we had models that gave us some localization of the information, so bounding boxes around the objects of interest, telling us not only that there's a bus and a clock, but where in the image are the bus and the clock.
By 2017, we integrated technology that allowed us to have not just bounding boxes but actual delineation at the level of pixels. By 2019, we had semantic segmentation of the whole image: in the front, in the back, all the aspects of the image were associated with concept with high-level accuracy.
This is huge progress within the span of seven years. I don't know if there's any correlation with the fact that academics go on sabbatical every seven years, really a cycle of innovation that completely transforms what we can do in terms of applications, in terms of new products, and in terms of new experiences.
Where are we today? Well, in April 2023, we released Segment Anything, a new model, a universal model for image segmentation. One of the keys to this one was to realize that there's not one unique segmentation for all needs, and in fact, as our models become more and more performant and more and more general, they also need to become more controllable, so we get the information that we need and the information that we want, not the information that the machine has determined it should provide us.
And so the innovation of Segment Anything is to allow interactivity, whether through language, whether through mouse clicks, whether through select boxes. You can indicate which objects you want and which objects you don't want. It was released openly, including the code, the trained model, a demo. In the first month since the release, 1.7 million people tried this model with a wide variety of different images. We also leveraged the model internally at Meta.
Beyond this model, we publicly released a large amount of data. We released millions of images as well as billions of mask tokens of the objects in the images, which then were used by developers around the world to build new applications.
Once you have a strong model such as Segment Anything, you can build lots of different things. On the input side, one of the things that we have now is modularity of visual contents, and so with that we can use that to essentially leverage information towards reasoning tasks. So, for example, counting the number of cats in an image. On the other hand, on the output side, you can also use the output of the segmentation to build up new information, so going from a 2D object, inferring what may be the 3D shape, animating that through space and time as well. These are just a few examples of what can be done.
Now, as we were building this model, there were really some key innovations that allowed us to do them. One of them is the annotation strategy, and the other one is the model architecture itself. And to achieve the strong results we had, we adopted an approach that we call the data engine: co-evolution of the data annotation and the model happening in parallel. The better a model gets, the faster we annotate the data; the better the data annotation that we get, the better the model improves, and so on and so forth.
This is data and model co-evolution, which is going to be critical to progress in AI and lots of other fields going forward. Now, the fact that we released the model openly, as I mentioned, doesn't in any way limit our ability to innovate at Meta. We've integrated this model into Backdrop, a feature on Instagram. Some of you may be using it. It allows us to cut out an object of interest and replace the information in the background of that object. It can be used to create stickers and so on and so forth.
And so I really think the important thing to note is when you do commit to an open approach, it doesn't slow down your product innovation. In fact, in many ways, it accelerates the innovation because you are leveraging the power of a much larger community to drive that innovation. In this case, outside of Meta, Segment Anything was adopted by several other companies, in particular companies focused on data annotation, companies that provide the data to train many of the large language models that we are seeing today.
One of these companies, Roboflow, integrated it very quickly. In the span of six or seven months, they were able to launch this towards 50,000 users to accelerate their workflows. They labeled over 13 million images. We're talking about 21 years of labeling time that was accomplished in about six or seven months using this technology. This is just one company. Beyond those, there are many different applications that used the model towards a completely different domain, particularly in the space of healthcare, in terms of medical imaging, in the space of conservation of endangered species, and so on and so forth. We're still seeing the impact of this today.
Now you may be wondering why Meta as a company would do such a thing. And so let me spend a little bit more time explaining really the philosophy behind this and the commitment. In many ways, it's part of our DNA to be open. It started from the basic stack, the software stack on which Facebook was built, and it continues to this day. Specifically for our AI research, the organization which I lead, FAIR, has been in place for 10 years. We have a 10-year commitment, and we are going forward with this.
Over those 10 years, our researchers have released over a thousand different repositories covering all areas of AI. Our real goal down the line is to achieve advanced machine intelligence, is to solve the mysteries of AI, but we know very well what a big challenge that is, and the more people we have participating in this, the more likely we are that we will be successful. As I mentioned, we've released over a thousand different projects. Some of the ones that are most notable are No Language Left Behind, a machine translation model; more recently, our Seamless model that does live image translation; DINO, our image feature model, which allows you to build self-supervised features for many models on which to build higher-level stacks and fine-tuning.
And so this is just a quick summary of the things we've released in 2023 alone. 2024 is just starting, so I expect we have many more exciting things in store for you. Overall, really, the commitment to open science stems from a belief that economic growth, of course, needs to be paired with responsible development.
And so when we open source a model, we enable a larger number of people to take part in the conversation, different stakeholders to get their hands dirty and really see what is going on behind the curtains, look at the model weights, look at those recipes we've implemented in terms of safety, look at the data, and enable a number of new innovations and research, in particular with the academic community that doesn't always have the resources to train some of the larger models. It really gets that technology out into the hands of many, many people, and this is particularly important also for populations that are underserved, in particular in the global South.
Now, taking such an approach isn't something we do lightly, and in fact a lot of work goes into figuring out how to do this. You can't decide in the very last days of a project to open source the project. It starts from day one when we decide from the beginning that we will build this project with an eye to open science. As we build the project, it sets a very high bar towards quality and towards responsibility, towards how we will make sure that the model actually goes towards leaving the world a little bit better.
Really, in the end, this is our commitment to the world, and this is why also each of the releases is assessed on a one-by-one case basis. It's not an absolute strategy. There are some cases where we've held back models because we felt, from a level of safety or from a level of quality, that the model wasn't ready to be released, something we assess chronically.
Now, probably our highest visibility release is our Llama series of models. I think when I was here last year, we had just released Llama 1. Really, that was released for the research community. The interest was enormous. We've seen in the first few months of this over 10,000 different requests from researchers interested in the model, that certainly signaled that there was a wider appetite for this. And so we came back in July with Llama 2, a model that was available both for research as well as commercial purposes.
Over the first few months, we had 160 million downloads of the Llama 2 model, and we've seen upwards of 19,000 model derivatives built on top of Llama. There is a whole zoo of animal models out there populating the community. Overnight, we saw startups using this, we saw established companies, of course, but also a lot of private sector institutions building on top of Llama. The fact that the model is open helps build the confidence towards what they are building on, and we have a lot more to see.
Really, the timing is excellent to be here. Just last week, we released our Llama 3 model. We are still seeing what will be the impact of that, but right now I would say that we've had incredibly positive reception. We released an 8 billion version of the model, a 70 billion version of the model, which I do think are the best open source models that are out there right now at any size. We also have a larger model that is still training, and we are looking closely at the performance of that. It'll be exciting to see what it is.
The model, compared to previous versions, I think has a lot more nuance in terms of the language. It has also a much more sort of direct tone and ability to have more concise conversation. One of the challenges of building these models isn't just to digest all the data — this is the part that we call pre-training, ingesting all of the data — but it's also, once you have ingested all the data in this very raw model, how do you shape the behavior of the model towards something that is useful, towards something that actually provides value to people.
In parallel with the Llama model release, we have launched Meta AI as the interface for most people to access this model. It is obvious that not everyone's going to want to download the model weights and start running them on their own instance. For many people, it's a lot easier to access this through some of the conversational interface that they are used to. Meta AI is available, including in Canada, which I'm thrilled about, through Instagram, WhatsApp, Messenger, Facebook. You can have access to it, you can launch into a conversation, you can get an idea for yourself of the performance of that model as well as send feedback.
It's also integrated in some of our AR and VR devices, which has been a whole other challenge, in particular building a model that can interact through conversation. For some, such as the Ray-Ban Meta glasses, brings many other challenges. The types of questions that people ask, the type of information that they want is completely different. I'm really thrilled again that the smart glasses have been launched in Canada just yesterday, with a version of the model that we built in our research teams not so long ago, just a few months, and already it's coming out.
Really, some of the visual, audio, and spatial components are quite different in the experience through the glasses compared to language models. We'll continue to learn from that experience and innovate and build better models over time. Whenever we launch these models, whether it's for research purposes, whether it's through products, we look carefully at a number of prerequisites, and in particular all of the data that we use goes through legal and privacy review in a very formal way.
We look at a lot of safety benchmarks as we're building this, some external safety benchmarks that the community is working on. We partner with many organizations externally, Partnership on AI, MLCommons, AI Alliance, and so on, to define those standards and benchmarks. We use a lot of human evaluations, and we do some internal as well as external red teaming to ensure that the model meets a high standard of safety. We've published quite a bit about how we do that, as well as open sourcing some of these tools, including Purple Llama, as a good reference for how to do that.
But really, our goal is that every time we move towards another launch, we raise the bar in terms of the technology, but we also raise the bar in terms of the safety, availability, and our understanding of these problems.
Now, when you build an entirely new AI system such as the glasses, you're going to need to leverage the full virtuous circle of AI: the data, the compute, and the model as well. And so in this case, we've also built a dataset specifically for the glasses. This is our Ego4D HowTo dataset. It allows us to pair the full experience. If you take this task of like building an omelet, a very common task, but wherever it happens around the world, we have different words for this, we have different utensils, and in the end, our omelets look a little bit different for each of us.
So if we're going to build AI systems that are going to be deployed at the scale of the planet, we also need to be thoughtful about collecting data and leveraging information across this planet. And so our Ego4D effort is one such effort towards building data through a worldwide consortium. The data has been open sourced as well, including to the participants, to allow them to build their own model.
Now, where are we going from this? Look, there's a ton of new things to do. Let me briefly give you a few thoughts about where we are going, more from a research perspective on these questions. One of the things that we are paying close attention to is really the ability for AI to have cognitive abilities, the ability to not just go in terms of generating language, but also manipulate symbols and so on and so forth.
There's a lot of work to do in terms of reasoning, in terms of common sense, in terms of performing tasks in many different domains and taking actions and so on and so forth in a way that is robust as well as safe. I have talked about this question of building models that are increasingly general and increasingly large. Let me reemphasize the point that the focus isn't just on scale. The focus is on having models that are more general, but also having models that are more controllable.
This is my Y-axis here. In many ways, if we just build on the generality axis, we will completely have missed the boat on what we need to do. Generality without controllability is really like intelligence without socialization. It's without purpose and without common sense, without values and guardrails. So this is going to be incredibly important to get right over time, to do this and shape the models in a way that brings positive value.
Beyond this, in many ways, as we're working towards achieving generality as well as controllability, there's a lot of hard problems that we need to solve. Our chief scientist Yann LeCun has published essentially a white paper on the roadmap, the research roadmap. I invite you to read it if you're interested. It calls out many different components that we need to focus on, and this is some of the work that's happening in our research lab right now.
The ability to build perception systems where the AI agent learns through observation as well as through interaction, the ability to build memory systems — what to remember, how, when, how to use that information — the ability to have an actor that can execute plans, whether it's through digital agents, virtual agents, physical agents, robotics, and so on, and a cost model. I talked quite a bit about the controllability aspect. This is really the nexus of that.
At the core of all this, a key component is the world model. This is essentially a digital twin for the real world built from data that can be used for reasoning and planning. That's going to be a major focus in coming years: a world model that has the ability to make predictions that we can use across multimodal input as well as actions. That's going to be essentially a universal prediction machine. This is a huge goal for the next few years, not for us alone, but for us to do along with many collaborators on our path towards AI systems that are more transparent, more inclusive, and higher quality.
And so with that, let me conclude. It's been a pleasure to be here. I wish you a wonderful two days. Thank you very much.
