There was a number of releases of Llama — Llama 1, 2, 3, 4 — and I would say each of the releases they got flagged for, but for different reasons every time.
Hey, welcome to Approximately Correct. I'm Alana Fish.
And this month on the pod, we have Joelle Pineau.
She was formerly at Facebook leading their AI team. And when she visited us here in Edmonton, she was actually here for the reinforcement learning conference that was being held here.
We had RLC at the University of Alberta bringing in some of the top thinkers in reinforcement learning here. Joelle was one of them and we were very, very excited to talk to her. She's done some fascinating stuff — too much stuff for one episode. So we wanted to focus on her work in open-weight and open-source models and a lot of the amazing things that she did while at Facebook around that area.
Yeah, and it's a fun thing to talk about because I think it had a lot of impact on what happened next.
Open models have always been one of the cornerstones of language modeling, AI research. It's one of the reasons that things have progressed so quickly. People shared what they had done in that way, giving access to the actual model itself. And that was something that Joelle really stands behind still, and did a really good job of defending against some of the flak that got thrown for that choice.
Yeah, and definitely talking to her, it was obvious she was prepared for that backlash. She had talked about it before.
Well, I mean, that kind of shows just how important she knows that this open source approach is, and I didn't realize how important it was when it came to further scientific advancement in this area.
So it was really kind of inspiring talking to her about that.
Yeah, she's a great personality in AI and it was fun to have her in.
So, here's Joelle Pineau.
Joelle, thank you so much for joining us today. We're so excited to have you on Approximately Correct.
Great to be here. You've done a lot of things with your career. You've worked as a professor and then you also worked at Meta or Facebook on the research team. What made you think about taking that leap from one to the other?
Yeah, a lot of it is about sort of the context in time. And so I started my career as a professor in 2004 and spent a number of years at McGill building a lab, teaching, doing all these awesome things and some of the less awesome things that come with the territory. And there came a time around 2016, 2017 where I feel I had gotten a good handle on how to do that job. I was still enjoying it, but I think I was up for a new challenge.
And it was the point in time where more and more of the research on AI was happening in industry labs, and in particular industry labs afforded some resources, compute, you know, the concentration of talent, these multidisciplinary teams to take on bigger challenges. And so I was frankly just really curious to see what that would be like and what kind of research I could do in that setting.
That sounds really exciting. Did you have a sense of what about being an academic would help you in that setting?
I don't think I thought about it that much in those terms.
But like while you were in the job, did you see like...
Forward looking, and so particularly I think, you know, I started talking to the folks who were at FAIR, the Facebook AI Research Lab, which then became the Fundamental AI Research Lab when the company changed its name, but anyways, FAIR stayed. I think they didn't know the change acronym, and really the mission, you know.
I started talking to that group for a couple of reasons. One, it was the only research lab at the time in industry that had a very strong commitment to open science and sharing the work openly. And that sort of for me was quite important — this notion that we should be doing science in a way that enables verifying the results, but also enables the collaboration and all the things that come with this openness property. And there were really good researchers at the time who were working specifically on dialogue systems, people like Jason Weston, Antoine Bordes, Dan Burel, and you know, I kept on reading their papers and I frankly just thought it'd be a lot more fun to collaborate with them than to compete with them, and so might as well join the party.
Yeah, good. So Facebook is known for that open research, and that was one of the things that drew you to it.
Yeah, they also caught a bit of flack, I think, for releasing Llama.
There was a number of releases of Llama — 1, 2, 3, 4 — and I would say each of the releases they got flagged for, but for different reasons every time.
And was it worth the flag?
Absolutely.
Yeah, I was most involved in Llama 1, which was released in early 2023, which is a project that came out of FAIR, one of the teams that I was sort of supporting broadly as I took on more of a leadership role. And I think the model was super interesting. We released it with a research license, intending research groups to be building on that and grow from that. And so that's where I was most involved in terms of what we call the go-to-market process, meaning, you know, like what's the assurance process we put in place to share that.
I would say in Llama 1, a lot of the discussion across the community — on the one hand, we had a number of people who were incredibly enthusiastic to just get their hands on that model, both the ability to understand how the models were built, the ability to run it locally for many different domains. That property to run a model locally where you can run it on your own data, which for confidentiality reasons or other is valuable.
It was incredibly useful. There's a lot of academics that were incredibly excited to have a model that they could use also. Where there was more criticism was on the question of how to safely release these models and whether by opening up essentially the books to these models you were enabling some bad actors to get their hands on some pretty powerful technology.
So we had some good discussions, and you know, some very cordial, some a little bit more aggressive on this topic. And I think to me it came down to a couple of things. One, you know, there's a lot of lessons we have from the computer security community, right, that do depend on a lot of open protocols. And really the lessons from there seem to be in large part having open protocols means that we can learn very quickly about what are the flaws, what is vulnerable to attack, how do we build up mitigations to these attacks. And so for me the thought of releasing a model such as Llama 1, which is frankly still really early days in terms of the curve of progression of AI, means that as a community we had the ability to work with a much broader set of people to make the models safer.
And the other thing is the process — I alluded to it, this go-to-market process. Internally we did every time go through quite a bit of testing internally, and to some degree with each release we got better at doing that. And so our own protocol for testing, for validation, got better every time. And in some cases we worked with external partners. There's a team at Stanford, Percy Liang and colleagues who do the HELM evaluation and so on. So it was like an opportunity as a community to just say like we're just going to keep on getting better at doing this if we let more people into the work rather than do this behind closed doors only within one or just a big companies.
So the rewards outweighed the risk for that model at that point in time.
I was convinced that the rewards were far bigger than the risks.
Okay. And when we say, you know, we're talking a bit about the Llama project specifically, when you say it was open, like is it like the model is open or is it the training data and everything as well, or what is actually released?
Probably the best way to describe it is as an open-weight model. I quite like that definition. And I think there's been some, you know, semantic arguments about whether it really is fully open source. It's not. We didn't fully release the data sets for it, though we released a large set of different artifacts. It wasn't just the model. There was some inference code and so on. And so there was a set of artifacts, but really, yeah, I think open-weight model is probably the best term for it. We did release a paper that does list the data sets that were used to train the model, something that subsequently very few people have done.
Okay. And why do you think there's a reluctance to release that as well?
I think quite frankly as a society — I'm not going to say even the community, as a society — we haven't quite sorted out how we think and how we feel about the use of internet data at scale, including data that is produced by a number of authors and creators in the training of these models. And so while that conversation hasn't happened to a sufficient degree of resolution, I think a lot of big companies feel like it's quite, you know, it's putting yourself out there to give too much information, and may have bad legal outcomes in terms of the work.
Now, you know, I think the culture I insisted on internally for me was always if we're willing to do something, we should be willing to talk about it. And so if you're willing to use the data for training purposes to release the model with a research license, then we should be willing to say what data sets we have used to do that. So that's how we, you know, that's how we got to this outcome with this project.
It's obvious that, you know, that openness is something you're pretty passionate about just talking with you right now. But you know, what is it about it that you think is so important for research going on? Like why insist on this openness?
Openness is a means to something. You know, I believe pretty profoundly that as scientists, we are all building on each other's contributions. None of us is an island. None of us sort of creates despite this myth of this, you know, creative genius. You know, we all build on the results of each other. And so the more we can share in terms of not just the outcome but the process of our work, the more we can empower others to contribute.
So I think like research moves faster, more people can contribute to research, more people can benefit from the results of research, and I think it sets a very high bar in terms of the quality of the model. And by quality I mean both like the performance quality and more the responsibility aspect. When you've set out as a goal to share the result of your work, whether it's a paper or a model or something else, it sets a pretty high bar. You're like opening up the books on the work that you've done.
And I think it just pushes each of us to do better work, and I have seen this many times with teams. You know, you're going to go out with this work, you know, it just really motivates the team to do their best work, which I think is good.
So it really comes from that, you know, openness is like a means to get to those aspects.
Since you've left one company and maybe you can think and speak more broadly about the community of companies that are working together, what would you say to encourage other companies to consider open source?
Open weight.
I've tried many times to share.
I'm not the first to ask you. I'm surprised.
When I showed up in the offices of colleagues and friends at DeepMind in London, and we've had, you know, we've had great discussions over the years with many researchers across industry. I think at the end of the day, you know, there's different incentives that go into decisions. And for many of these large companies, quite frankly, the decision of whether or not to share a model openly has to make business sense. You know, like I love being idealistic, but we also need to be pragmatic about this. If a large company is funding research, these large companies obey essentially like a business logic to how they run their business. That's why they're successful. And so there has to be a good business sense for doing that. And I'm not privy to the strategy of all these other companies, so it's hard for me to know what makes sense or not for them.
I will say from Meta's perspective, there was a strong culture in terms of open software ecosystem already. A lot of the original software stack of the Facebook platform had been built on top of open protocols. It is a company that, you know, also sponsored, created, and then shared PyTorch, which is now like such an amazing part of the ecosystem. So that understanding of the value of open software and openness in general, and how it can be a powerful accelerator for a company, was pretty deeply in the culture of the company, right?
And so it was maybe an easier sell than in other places. You know, I often shared my experience with others, but whether or not it made business sense for them, I think they have to figure it out. As we are recording this, we're sort of 24 hours past the point where, you know, OpenAI has released its first couple of models in many years, which is super exciting. I'm really glad to see them commit to that. And I think they found a point in time where it made sense for them to do it. I'm glad because I do think the whole ecosystem will benefit from having more options.
Yeah, apparently there was like one of the — it was like the biggest downloads Hugging Face had seen for a long time was when that model loaded.
The servers were busy yesterday and probably till today and a few more days to come.
Because there was a sense that like that OpenAI was trying to — I think somebody said — pull the ladder up behind them, that they were saying it is, you know, we — it's not safe to release these models, but it really was sort of a market capture situation. I don't know if you want to comment on that.
I've certainly heard people make this argument, and that the reason that in many places they argued quite strongly against open models, including, you know, in Washington, was the strategic question. I didn't hear that directly from them, and so I will, you know, leave it to them to actually say what was the strategy behind that, but that's certainly one of the hypotheses, right? And I guess their more recent actions say maybe they are interested in sharing, which I think is nice to see.
Yeah. Is there a sense that things are more trending? Like you talked about a lot of different cultures, a lot of different incentives. Is there a sense that things are trending towards that more openness, or is that something you've noticed?
We're going to have to see. When we shared the open weights for Llama, it was a pivotal moment because at that point there were essentially no large companies who were willing to do that, and it showed, you know, it can be done.
Yeah, it kind of takes like somebody to take that leap first.
Exactly, to show that it can be done. Then I think, you know, another important pivot point happened when some of the Chinese models started becoming available — DeepSeek, Qwen models and so on — showing that you could share really high quality models in some cases trained with a lot less compute. And so I think that was another sort of piece of the puzzle put together. It's always really hard to predict the trajectory. I do think the recent release from OpenAI is an interesting one. So we'll have to see how it plays out.
So I guess we had just talked a little bit about like some of the concerns when it came to open models were about misuse. You open something up, it makes it easier for people to use it for harmful purposes. Now I know you've argued in the past you need to look at it through the lens of marginal risk. So can you run us through like what marginal risk is? What's that idea about?
Yeah, this is a framework we developed with several other collaborators, and we shared a paper at ICML, I think 2024, that described essentially a framework to think about that. And it closely mirrored a lot of the work that we'd done internally at FAIR when thinking of our own releases. The core idea is that we should be looking at what are the risks with respect to specific risk vectors.
You know, to just be afraid of models in general, I think, doesn't allow us to make good rational decisions. So being specific about what are the risk vectors — something like is a model likely to create misinformation, is a model likely to be amenable to being repurposed, for example, creation of biological weapons, is the model likely to generate images, for example, non-consensual sexual images — and so identify specific risk vectors.
Is that what you mean by vector? Like a specific...
