Thomas Wolf | Open Source AI, AI Robots, & Learning AI

Thanks for inviting me.

It's long. It's always hard to really understand how you would have been if you took another path. I know that for myself. You have to picture that in the 90s, AI did not really exist. I had no artificial intelligence class. So maybe if I grew up nowadays, I would directly go to AI. For me, I knew this thing existed. I was like, I like math. Computer science was not yet a job really. They had this ARM PC, and so I was looking for something in math, and math itself looked a bit too difficult for me, like research in math. So physics is maybe good. So that's why my first track was physics. I really enjoyed that, but I also discovered that physics experiments and real life experiments are just really tough. They take a long time, most of the time they just don't work. They're annoying. Real life is annoying. So during my PhD, I was like, it's really tough. You also need to have the... Usually a physics experiment goes like that: you will have an idea, and then you will find brands basically to pay for, for instance, the cryostat that we used for cold temperature superconducting materials. That's what I was working on, which was basically what today's quantum computers are. I was doing this kind of Josephson junction, which is one way to do qubits, and this thing cost one million. So you need to find money, you need to buy it, then you need to build it. It takes back one year to build basically. So the whole experiment was basically three years, and then you discover at the end if your idea was right or wrong. A lot of my ideas were just not good ideas. Oh, back on the desk and having a new idea. I don't know if I can do three experiments in 10 years. Maybe also I was young, I wanted to move fast. So that's why I was like, okay, maybe physics is not for me. I need something that moves faster. I really liked a lot of things. I really liked writing as well. So that's why I was discussing with a good friend who was a lawyer. I was like, we write a lot as lawyers, so why don't you try? Okay, and so I took an evening class in law and I became a lawyer. I liked that a lot as well. It was really moving much faster. You had a lot of cases and people coming, and I felt very helpful, helping them. You're an expert in the field. At the end, I really started to have this pool of startups that were helping quite a lot, and some of them were starting to do deep learning. That's basically where I discovered what deep learning was and what machine learning was. I was writing the patents for them. I was like, okay, this is really the same equation in a way, right? Instead of minimizing free energy, you're minimizing loss, you're maximizing probabilities, however you see it, but that's really the same math. I was like, that's cool, and the experiments look a lot faster. It's not three years, like two weeks or a month. So I got really excited about that. I would say on the side, I've always been enjoying programming, always as a side hobby, but the perspective of making it my main occupation was very attractive as well. So that brought me there. Quite a complex path to end up doing what I do.

Yeah, it's possible. I've never really... It's hard to know because I would need to compare a version of me who didn't do law. But that's for sure. Law is all about studying the impact on society of something we build, which is law, and that's all the things. It's case studies, and you see, oh, this actually brought this person to this, and then you should change law because that's not good, or we should add something, or we should actually take into account these real life cases. So it's all about real life stories. I think that's probably something I kept in my computer science work right now, which is there are humans behind all these algorithms that will use them, and that will actually change, good or bad, their lives. I think the first thing for me is, and in retrospect, I think that's where I maybe... I don't think I'm a very good programmer. There are a lot of much better programmers at Hugging Face now, but I think I'm pretty good at defining user interface, how people will use my software or the software we build. This is all about thinking about the people that are using your software. Maybe this is kind of a very law or social science aspect or something like that.

Most of the thing, even the start of the project, starts usually with me seeing the type of user experience I want to see, the type of user interaction I want to see. Even very... All of the libraries I've helped to get switched out, Transformers mostly built myself, Datasets, those are the two I really myself. Accelerate was one I strongly pushed for the idea, and Diffusers there's also another one. All of them start with this idea: this is the lines of code I want to write, and this is the effect they should have, and now let's build the back end that's running that makes all of this possible. So you start there, and then you build a lot of things. The last part is also very long. Before you release, you play with the thing a lot, you try all these cases, you try all these things, and maybe you adjust. Often it's only when you start really high thing built that you can really play with it and you really fully understand how people will use it. But this last part is also quite long, it's a few days, and sometimes you change, and sometimes you decide I want to change completely the API. Something that's not natural, it has to be extremely natural. The perfect library for me is one you don't even need to take a look at the docs, it's so obvious how you use it. Nobody really reads docs, nobody really wants to read it. So anytime you're like, okay, you should have a very small number of abstractions. If you need to list 10 abstractions you need to learn before you use a library, it's annoying. You should have, I think, two or three abstractions, that's the right level. They should be very intuitive, almost non-programmer, non-software developer minds. You should almost be able to explain them to your parents. If your parents are software developers, I don't think they are. So I think Transformers, I could even explain the abstraction to my dad, who was a doctor. Maybe with a little bit of that, but yeah, it should be to this level.

Yeah, yeah, I think it's very true. Usually the selling is basically just putting yourself in the shoes of someone who wants to use your thing and explaining why they should use it. It should be very obvious, it should be directly, oh yeah, it's exactly what I used to do, that's exactly how.

I wrote recently a short on that, the Einstein AI models, where I explain that they have some limitations that I found very frustrating, and which are also similar limitations that I hit in my research, which was it's making the experiments. It's pretty long, so it's still very nice if we speed up a lot of the work you need to do, but it's only a very small part of being a researcher. It's maybe the one that takes the longest time, but it's a lot of engineering and just being very rigorous. But the tough part is which question, which question are you exploring and why? I think a lot of people who haven't done research really don't understand how this is crucial. They just think, oh, you just do experiments, but we give the experiments. When you're a student, when you're a PhD student, sometimes you're lucky, you have a great PhD advisor, they say, oh, this is the question, and it happens to many of my friends. They had amazing PhDs, great publications, and after that they were on their own when they became researchers, it was very difficult. That's the moment they understood that actually 90% of their success in PhD was due to their advisor having very good ideas around which direction to explore. This transition for many researchers, also here when I come and I see friends, professors at Stanford, you see this really difficult transition between being a student researcher where someone helped you a lot in this general thing, and you still feel like you're doing a lot because you're building the experiments, you feel like you're doing it all, but the moment you need to find yourself the right question to ask, that's a very tough moment. Maybe like a writer in front of a blank page, you could write every book possible, which one do you want to write? I think models are very good at all the work once the question is asked. They're very good at maybe giving you a plan for the experiments you should do, and very soon they will be able to be plugged also into the machines to do the experiments themselves. But the real thing that will give us the future super relativity or way to go, I don't know, faster than light something, will be someone just asking the right question. I think they're pretty bad at this.

Yeah.

Part of it is... I don't know which part of this is transferable to AI models because they are really different. Sometimes we tend to think they're really like us, but no, they learn totally different than we do. But for humans, I think there is this strong will, and you see that in every researcher, a good researcher, and it's a very common point. We also see some very successful entrepreneurs. It's like they really want to do things differently. They really want to explore the thing that's outside of the most probable or most likely thing. They want to play against the rules, they want to challenge some of the rules. That's something I was really liking when I was young. I was really following the rules all the time, very perfect student, professors were always very happy with me. Problem is, that's not a good thing for wanting to change the world. That's a hard skill to learn, but you can learn that thing. Some people have that naturally since they're young, they're very going against the grain.

I can only speak for French education. It does not prepare you very well to that. It does in a way. I think there's two ways you can do that. It's maybe like a U-shape, that's just something I make up now, but when you were speaking I was thinking that's pretty true. Sometimes when you start tackling a new field, you have this first thing where you have a lot of interesting questions because you don't know anything, so you're very free. A little bit like children, you're very curious, you're ready to ask basic questions. Then you have a deep, and then you're in a valley where you feel like now you've covered it all and basically you're uninventive. I think if you pass this, then you reach maybe the Einstein level, which is you know the field because you also see some old researchers that actually still have very interesting ideas. I think it's because they passed this valley and they're again in this thing where you actually digested all the knowledge of the field, but now you're ready to ask good questions because you build on top of the knowledge. But it's tough to reach this. A lot of people stay in the middle. Like gradient ascent, you want to maximize, you can't be at the valley, you should not stay in the maximum probability valley.

It's now nine years ago because I did that during my paternity leave where my second child was just born nine years ago. That was a gift for my first child, like a fun gift. In a way, it became a little bit of a blueprint for any project at Hugging Face, which is funny because it's not at all a Hugging Face thing. But what happened is I built this stuff also because I had this idea of how I wanted humans to interact with this. I wanted you to have this... Just to explain what it is, it's a big box of sand and there's a projector on top that's quite high, and projector color that depends on the height. So you have this kind of cartographic, dynamically cartographic colors. For instance, you can make mountains white and then water if you want, and when you move the sand you see this changing, and it's very fun. It's like a computer interface but it's all made of sand. It's a non-flat screen way to interact with a computer. We made also some fun games where basically you have some animals that are walking, and when you move it they're like they got in the water and they have problems. I had a lot of ideas. When I built it, I was like, that's really cool, it's very fun. I actually totally destroyed it again and I rebuilt it slowly by taking pictures step by step, and I made a full tutorial to explain how you could build that at home and what did you need. It was pretty cheap back then, you just need a piece of wood and a bit of sand and a beamer basically. This went viral on Reddit, that was the first time. I understood that was the first time I understood actually people like when you do that, when you explain how you can build something cool themselves, they really get excited. A lot of projects we did at Hugging Face and a lot of the open source philosophy is a little bit like that. There is this cool stuff, and because it's open source, you can actually build it yourself for free. You give it and do the nice experience yourself. It's like Lego blocks. I love Lego and this type of direction is also super interesting for physical things like robots. I would love to build robots also myself and feel like they are actually something that I own, that I know how it is, I handle the pieces, and I even actually trained the model myself, so I fully understand this thing. That was also the first time I discovered you could actually have a bit of impact on the world with something you program. Most of my previous software were just internal things I coded on my computer, but I never shared them with anyone. The first time I was like, oh, actually if you share it, people can really react, and then you get a lot of people reaching out. Still up to date, I got one last month, someone said, I cannot install the magic sand, sorry I just don't maintain it anymore. It's nine years ago.

I don't think so. I even feel safer with an open source robot next to me because I feel like I can open it and understand what's inside. Otherwise, you really have to trust the robot vendor, or even anyone who touches the robot actually, because you can change hardware, it's not so hard to hack into stuff. I feel like if I build it myself and it's open source, there are also some nice advantages. For instance, for LLMs, you can run locally, you can maybe change it, adapt it. There's a lot of reason I would feel much safer with an open source robot at home than a black box closed source robot. But maybe it's also kind of just philosophical.

Yeah, it's very interesting. It's very slow as a book. There's definitely a lot of very interesting points. I feel like I started reading the book really thinking, well, we do it because we want to make an art piece on AI. It's a side project, it's not Hugging Face, but I have some artists I know well and we try to use art to talk about AI and to have people. Last year we did a performance where people would interact with an AI and it was fun. This AI was local running, I programmed it, it was running so fast, answering so fast that people thought that was a human. They were like, wow, when I ask GPT I need to wait for the latency, because it was local. Now we want to make something around Frankenstein, and so I was reading thinking it could be a good metaphor, but more and more I think it's very different. AI are actually very different from creating life. That's a part of this piece I wrote also about AI scientists. I think they're really heading this direction of being very useful assistants, but I don't feel like they have any... For instance, you have this discussion of consciousness in AI. I'm very down to earth. I feel like they are like a bike. If you turn the pedal, the thing works. If you stop turning, there's no token coming out, you just stop. So it's not human. You just tell it, but if you stop turning, it stops working. So it's hard to imagine that there's anything close to consciousness in them.