Runway’s Cris Valenzuela: Building The Hollywood-Friendly Video AI That OpenAI Couldn’t

It's a dream. You have this thing that basically conjures whatever you want. And the first thing people struggle to do is not to use the tool. It's like, what do you want to make? What is in your head that you want to make?

Yeah, first thank you for having me.

Yeah, we're neighbors.

We started, the three founders met in NYU and we started the kind of the initial beginnings of the company while we were at school in New York. And so it was very natural for us. It wasn't typical back then, to be honest. It was like eight years, nine years ago. But I think we just, I love New York. I love the city itself, the people, the culture. It just feels very special and it's becoming very special for the company as well to be able to be based here in New York.

Runway is an applied research company that builds video and world models. And then so what a world model is is kind of an extra tier of AI research and AI capabilities. These are basically models that can understand and take actions within the world maybe in a similar way that we do. And then the way you get there is by training large video models. And so we've been working on this for quite some time and now our models are getting deployed in entertainment, in movie making, in robotics, in many other industries, in gaming. And so Runway builds these world models, pre-trains them, develops them internally, and then we build products on top of them. And so those are kind of the best way to think about it is like an applied AI research company that builds the products, builds the infrastructure, and of course builds the models to make these world models work out in the world.

A lot of the paradigm of AI that we've gotten into until now is based on this idea of training large language models. So you've probably heard about LLMs, right? LLMs are just large AI models that are trained on language alone. And so they're trying to get a sense of reality by describing reality. The thing about language is that language is not reality. Language is an abstraction of reality that we humans have created to try to approximate what things are happening, right? So a glass of water is not really a glass of water. It's like we made up the symbols and the representation of that. And we train models on that language alone. What we do with world models is slightly different. We train them on video data. And if you show video data and models enough kind of information in pixels, there's enough information about what water is and how it should behave in containers, in other spaces, that the model starts to understand slowly, not by describing it, but by simulating it. So that's a good way of thinking about it. Language models describe reality, world models simulate reality.

It depends. We need to partner with partners, with industry folks. We have people. We buy data. We collect data. Depending on the task at hand, you might need to source from different points. It's a very technical challenge to do.

We started the company in 2018 and I think back then there wasn't a lot of intuition into how to build the models and the kind of models that we're building these days. And I think over time you build some sort of experience and intuition into what are the best ideas and approaches you can take to scale deep learning models for them to work at this kind of pace and level. And I would say that most of the best research labs these days are not, I've been working on this for at least a couple years. And I think that separates the kind of, I would say the best from like the new kind of like the ones that are trying to build the best models. And I think for us that's been the lesson. We've been working on this for quite some time and we have a natural sort of research ideas and approaches and experiences that have allowed us to differentiate more than others.

It's kind of actually a myth that OpenAI sunset Sora. When we saw it first coming up, it was like, it was the first ever like real competition we felt we had. And now like two years after they announced that they shut it down. A few things are true. One is like it's very hard to build the kind of products and models that video requires and the users kind of demand. And specifically consumers are just very, yeah, very hard as well. For us it's always been around, well, we're building towards a world where we think these might become the foundational like base on which you make every piece of content out there. From ads to media to films to things you share with your family, with friends, or just things you make for you. And so the level of craft and the level of kind of control that you need to have in the tools needs to be critical.

When people think of Runway, when people use Runway, one of the things that keeps coming up as a core differentiation is that it's not just like one prompt and you kind of make a thing. It's you're in control. This is a tool and you kind of like will it in the way that you want and the outputs are a reflection of your process. And I think the way you think we think about it might be different from how others might have thought about it because you can see how different like building of products and models is. And I think from at least from us, from our professional enterprise consumer side, which is a lot of what we're focusing on, that really reflects really well and people come back to us because of those kind of things.

Yeah, that's a good question. So yeah, we've, the three founders met at ITP. So ITP, the best way to think about it is it's art school for engineers or engineering school for artists. Which is another way of saying that it's a way to explore like the recently possible, the things that are adjacently like or new in terms of technology and like possibilities from a technical standpoint. At the time when we started kind of working on first ideas of Runway, the deep learning revolution that we're in was kind of beginning to take shape and form. It was 2014, 2016, 2017. And a lot of what we were trying to do at the time was really understand what was possible because like we need to remember that in 2018, the state of the art and I think we suffer from collective amnesia because we forget this. The state of the art AI models, the thing that what people were like freaking out about was a model that could distinguish like a cat from a dog. That was like revolutionary at the time. And to be honest, it was because you never reliably could do it with any other process or technique. But suddenly now you can go from identifying objects to, you know, identifying different kinds of breeds of dogs, which also at the time was like fascinating. And so the applications of AI and kind of deep learning back then were very boring, if that's a good way of saying it. You were, I don't know, people were thinking about classification or like financial systems or security or organizing data. And for a minute I thought there was like maybe something else that was more interesting coming from a mix of both engineering and kind of art background, where you can see this technology basically as a tool to express or to create or to imagine things. And so really what we tried to do really early on once we start training early models both on kind of computer vision and then on generative side was to just build things, prototypes. A couple of things we built back then was we built this film kind of recommendation system or think about it more as a search engine that you can search for visually for shots. So I remember I went to like New York to a street in New York. I recorded a bunch of people walking. Then I put it to our system and then the system returned me like seven films in which like the same type of shot was like, you know, in a scene. And so I remember Children of Men. It was like the same shot but like way way nicer done. And so you got all those like kind of opportunities of oh, you couldn't do that before. Now you have the system that can do it. What does it mean? I don't know. We're just building, experimenting. And one experimentation led to the other, led to the other, led to the other. And then eventually we're like, oh, we should take this more seriously and start building a product around it and a company around it.

No, I think to be honest, we when we started it was too early. I think we're building something that there wasn't a comparison. There wasn't a reference point. So it was hard to wrap your head around as to why this would ever exist, you know? And again, this is why I mean we sometimes have collective short term amnesia because now there are a lot of things that are obvious, but like a couple years ago it was a lot of things weren't all these obvious. And so the use cases of what the models mostly because in a way the models weren't working as well. So you were like, why would I ever use this? And I think a lot of what I had to do was like prove that you can get better models by improving both data and algorithms and training and everything else. And once you cross that threshold, you start kind of realizing more of the use cases more early on. But I would say that early on if you're doing kind of research and discovering new things, it's really important to have an open mind and be kind of less obsessed with what you know and more of what you can discover. And that has allowed us to understand how the models can be used for things that I could never imagine for, which is exciting. And from a business perspective, what I like thinking about the models is instead of like a company winning a market or like being good in solving specific need, it's more about creating a net new market. You created something that didn't have a name and you helped people do things they never thought they could. And that becomes the new standard. That's how I like to think about what we're doing.

So you know, back then I don't think there was a sentiment or a point of view in terms of how you should feel about it. I think it was at the most it was just very exciting. Like you were able to do things you couldn't do before. I remember we trained this model that you could write, this is like before transformers anything else. But it was a website, it was a web editor and I trained it on like a couple of authors that I like. So it was Bolaño and like Jane Austen and a few other folks and you could write something and you could get a completion of a sentence kind of as if that person was being your editor or was like helping you. And it was kind of cool. It was using like LSTMs and it worked in the browser in real time. So it's like a cursor for coding from like 10 years ago. And when I shared it with people and with writers, I think people just got excited. It's like oh this is very cool and they could think about all the things they could use it for. And of course it wasn't as good as the stuff we have right now. But I think the excitement when you show people projects like that and demos like that is it begins to open their minds into what else can they do. And I think that was at least my perspective on the whole field was like this is a whole new medium. It's a whole new way of thinking about our relationship with technology. And if we really need to think about them from a creative standpoint, then what we need to really think about this is how do we get creative minds to be part of the process. Because one thing that becomes clear maybe kind of touching back on what we just spoke is that if you design the systems in isolated ways with one particular point of view of the world, then you're not going to get like the kind of things that you can do with Runway kind of models and stuff because you need to have more of a creative artistic point of view when you actually create tools for them. And so I think we've always thought about it that way and the excitement that we had back then I think we still have it today.

Well there's always this utilitarian kind of point of view about models where people are like well why would I ever use this to make a movie if I'm not a filmmaker or like I'm not trying to sell something I don't need ads, right? And I think an underappreciated aspect of all of this technology is the reward it gives you personally to be able to make stuff like this. And this is something I would say technology has allowed us to do over time. We just sometimes forget about it. I compare it to like going to the gym, right? So if you go to the gym and you exercise, a lot of times you're going there not to become an athlete or like to monetize your performance in some way and get sponsors because you're going to be the best athlete at particular things, right? You just go there because you feel good. It's good for your body. You feel, we're all excited to do it hopefully more and more, right? I like to think of sometimes these models and the things that we're building as kind of a gym for the mind where you are exercising your creative muscles. And once you exercise that, once you practice that enough times, you like it and you're willing to spend money in it because it's a reward mechanism that you're activating your brain that you never had a chance of activating before. You're able to make things. You're able to creatively concept about things you never thought you could. We have users who are engineers. We have users who work in finance who you've never might have thought of like a Runway user, but they are creative, they still have creative ideas, have creative minds. They're thinking about things they just don't have the technical experience or the kind of resources to execute some stuff. And when you provide them with the things and it helps them be creative and they're going to be able to and willing to pay for it. So there's definitely that component of it that it's like an answer to well how do you monetize this is I will build things that people like to use a lot. And people make hundreds of videos per day. We have people making thousands even like a week. And of course on a more utilitarian aspect, well of course this is incredibly useful for studios and agencies and content and gaming studios and everyone who's in the business of pixel making. If you're making pixels for a living, this is a completely new way of making pixels with way cheaper and way more controllable methods.

There are a few aha moments of different products we have over time where you start seeing how valuable things are. And so again like on the more individual consumer side, you can just see the stats of people coming back and generating over and over and over again. You're like why are you generating so much stuff? And like you start understanding you interview people and you realize they just like they like it. They're obsessed with it.

And then for companies we have times where I don't know remember like the late night show with Stephen Colbert started to use Runway. And having seen them it was an interesting like kind of concept and then like realization because the person who found out about it just saw like a video about Runway and then started to use it. He didn't tell his colleagues he was using it but he was like 10x-ing his work. And so everyone thought he was just like doing some insane amount of work overnight. He was just using Runway as his weapon and then at some point he's like okay I'm going to tell you what I'm using. I'm using this thing called Runway and then they became one of our first kind of enterprise.

So we had an early version of Runway that allowed you to kind of automate a lot of the editing process of making things. And that's kind of the mantra of Runway is like if you want to make something it should be fast whatever you want to make. An early model was a way of just editing things and removing things from scenes that would have otherwise taken you hours or maybe days. They were able to do in a couple minutes. And they were using then those kind of films or scenes or videos into like the show. Into like the Colbert show is like runs live but you have like segments that are recorded. So those things were using Runway. That was like I don't know like four years ago and have many times in like our lifetime where that happens over and over again where like there's a film that was like recently screening on theaters that had a couple scenes that were Runway. And like when you know about what they managed to do and how they did it, it's kind of insane. It's great. You're starting to see the beginnings of like that creative revolution.