Netflix Top Tech Exec Stone on Integrating AI

Yes.

Okay, so where to begin? So product in tech at Netflix covers actually more disciplines than even are in that name. So think about a group of engineers, data scientists, designers, product managers, consumer researchers. That covers many, many different areas of the business. So from supporting content decision making to studio production to promotion to the experience, hopefully all of you use on Netflix, to how we deliver content across the world through our content delivery network and newer business areas like live, games, and ads. So think of it as multiple disciplines covering lots of different domains for Netflix. So quite a span of different problem spaces. When we brought product and tech together, the opportunity from that was to maintain the functional excellence or the craft excellence that comes with those disciplines, while orienting the team towards the problems that are most important to solve. So the most important member problems, ad problems, games problems, content problems, and then the team can function in a really cross-functional way. So it's less about this is a product organization, this is a data science team. It's teams that are oriented around the problems that they're solving.

I'm not a fan of the blame game. Most people in Netflix are not. It's one of the benefits of our culture. I do think it allows us to ship things faster. It's related to making sure that we're solving the right problems and being focused on that and having impact from it, rather than having additional cycles of alignment across many teams, and to create a culture where people feel like they're on the same team. The other thing that supercharges it, which I'm sure you're all going to be talking a lot about, is AI. So that picks up the pace a lot on velocity. We're able to do ideation and prototyping and testing and get teams focused on what's the best way to use this technology for the problem. So I think the way we've structured things allows us to see the business end to end, to connect dots across many domains for the benefit of members, and then leverage technology more quickly to make sure we get a faster innovation cycle.

I don't feel like there's a right answer to that question. How would I reflect on that time? So, we are constantly looking at ways to advance the business. I think it was a great opportunity to look closely at Warner and to think about what that opportunity was for Netflix, for members, for our content business. I think the best thing that came out of that time was the discipline to make sure that we continued to drive the Netflix core business forward, and that we were focused on the things that actually mattered for the business. So when I reflect on how the leadership team and many folks across Netflix went through thinking about what that integration would look like, thinking about the Netflix business with and without Warner, there was a level of discipline and focus that I was really proud of. So that I think is a benefit no matter how it turned out.

I think Inner Positive is a great example of how we continued to move the core business forward, even at a busy time, with a lot of other things happening. When we first met Ben and his team, I guess it would have been late last year, we got to look under the hood at a script by filmmakers, for filmmakers, and the way they thought about the problem solving. A lot of the solutions that are on the market are things that do not deeply understand the creative process or what it means to be a creator trying to bring a vision to life. And so the models and tools aren't really well suited for the problems that come up on a production set every day. We have been working on many of those problems internally. We definitely have a build leaning for much of that. So we've been working with productions and building our own tools. And Inner Positive was so nicely complimentary with this, framed in terms of how do we enable creators to bring those visions to life instead of thinking about how could we do something without human creativity remaining at the core. So I think part of that was the benefit of Ben, obviously as a director, actor, producer, so deep in that space, really speaks the language of content, vision, and creation so well. The human element of that, and he's had a really long history of being curious and advancing technology as part of it. So to see those things come together, it felt like Inner Positive had built something great, and it was going to be a great addition to things that we were already building. And it also helped us frame in terms of the philosophy we have, which is we're trying to enable creators. We're not trying to do something separate that is divorced from the actual process of content creation.

Yeah. So I think that there's a huge advantage that comes from Netflix has been so innovation-oriented since the beginning days, far predating my time at Netflix, of course. And even things like if we take specifically machine learning and AI, are not new to Netflix either. So that's been core to how we thought about personalization. It's been core to building studio production tools for a very long time. This is a supercharged version. It's a huge step function in what's possible, but thinking about how technology is going to allow us to innovate is a very familiar muscle at Netflix. So we were very quick to say, what are the areas of the business where we think this is going to have impact, focusing on how do we deliver something better for members. So that I think everybody in this room probably feels it. It's tough to keep up with the pace of it. It's evolving more rapidly than we can even wrap our heads around, and that's been quite an undertaking. But there's an excitement and a natural inclination at Netflix to be curious about those things, but also be pragmatic about it. I think this is another thing about the Netflix culture, which is it's not technology for technology's sake, it's technology for the application of how do we make great content and deliver it to members around the world. AI supercharges that. And so that's been an advancement in a way of thinking we already had. But it's change. It's change management. It's figuring out which tools we want to use, where the impact actually comes from, how to get people comfortable and educated on it. So that's been a big role for portions of my team to make sure we're driving that across Netflix, which I'm sure lots of different companies are going through. Are we thinking about this in the right way and applying it in the right areas? So we spend a lot of time on that education and guidance.

I don't do that alone. You know, there are, of course, when some of the first gen AI models hit the scene a few years ago, I think there was a tendency to panic. And so part of my role was to say let's not panic. Let's think through all the applications this might have. Let's start to get really familiar with the technology, what it can do, what it can't do, how it's complementary to our current approach or might disrupt it. So there's something about a leadership role in a time of change and guiding a technology evolution that is very principled and focused and helps make sense of something that can be very overwhelming. So that was a big part of my role, especially in the early days. And that comes with spikes, you know, that comes when certain models are launched or there's new people on the scene. And what does this mean for us? And it's come with a big part of my role was shaping some of the principles or how are we going to think about the applications across the company. The reason I say I don't do it myself is I could set the focus areas. I could build understanding about what is the tech, how are we going to use it? But then a lot of other people are behind building it into our infrastructure, the products that we deliver to members or other parts of the business. And that's where I say I don't do it alone. So based on that initial like, let's take stock of the landscape, let's be pragmatic about where that impact is going to come from, let's make sure we prioritize it. The Netflix team is incredible at taking that type of opportunity and direction and making it happen. That's been true for AI. It's been true for new business bets that we've made. So that's where I have a lot of trust in my colleagues.

The core of personalization for members has always been trying to deeply understand titles that you in particular will love. That's a very data-fueled exercise, and most of it is based on what you've watched and what you've liked in the past, and what similar people have liked in the past, so that we can do the right title at the right moment for you. So that is, at its core, both a data and engineering exercise that's been behind personalization and the product teams at Netflix before we brought together product and tech. And the engineering and data science teams are extremely data-driven thinkers in solving problems. So I don't think that bringing data and engineering closer to product changed the degree to which data is leveraged there. I think the thing it did was, as I mentioned earlier, making a faster cycle on what are the problems we need to tackle. Where is data helpful? Where is judgment helpful? Where do we need to hear directly from consumers about how we're solving problems? And then personalization becomes a mix of consumer understanding, really good product experience, judgment and design, and data, machine learning, AI that helps pull together that context.

So this is where I think one of the most important things is that we can learn through multiple sources. So we can learn by designing an experiment that we test in the product, which we do all the time. And it's a great way to learn how members are actually going to experience or engage with something. But we also learn from directly asking consumers, what are the current pain points with the Netflix product? How would you think about a new experience? And we marry those two things, the quantitative and the qualitative. And there are times when they disagree where a member will say, this actually is making discovery more difficult, even if our intuition and then some of the experimental data would have said otherwise. And that becomes a really tough decision-making exercise that the product leaders have to weigh. There's data, there's judgment, there's what we're hearing from consumers. There's also another angle to it, which is each experiment on its own gives you a set of data, understanding how that change impacted members. But our job as strong product leaders and stewards of this experience is to see how that's going to add up into a cohesive experience. So I would say the times that data can lead astray are when you look at something in isolation and you think about if we make this change, what's going to happen to the member experience. But part of the job is saying we're introducing lots of new content types, we're introducing new features and UI and backends across TV and mobile. We want to have a very personalized, interactive, immersive experience that feels cohesive and delightful. Is it adding up in that way? And that's where you have to be careful that you're not having a very insular, narrow view based on an individual set of data points versus looking at the full picture.

So the richer the information that we can pull together about, as I mentioned, like your viewing experience, what your preferences are, as well as a way to represent the content that we have. So think about that as if you watched a certain film. You might say this is a horror film that's maybe kind of kitschy from the 1990s. Some of the new technologies allow us to do that type of description of the content in a much richer way. So we can marry this rich understanding about the type of content that you love and enjoy with a rich set of understanding about what is this film or TV series or podcast about, with a rich understanding about the world, like what's in the cultural conversation right now, what's trending in terms of focus areas, and then bring that together into an experience that feels extremely personalized and immersive. And more conversational means that you can use things like natural language to say, like knowing what you know about the type of content I like and knowing the content catalog that Netflix has to offer and the context that I'm sitting in and I'm watching with my family. Did I have a tough day? Am I looking for a laugh? That conversational experience is powered by that rich context across members, content, and the world around us. So that comes to life in things like the voice UI experience that we're testing now, and we've done different tests or beta experiences using your phone and natural language to help search the catalog. That's enabled by a different approach under the hood that you don't always see, but helps to enable the great member experiences.

That's not reality. And I don't think that's not reality for only Netflix. So many companies we've gone through bringing live over the internet versus broadcast have gone through a similar experience. It was really tough. I am glad that we took very big swings early on though. So that includes things like the Paul Tyson fight, standing in a launch room with the team as we approach 65 million concurrent viewers, dramatically record-breaking. I think it added up to more than 108 million people watching at the same time when just six months earlier, approximately, you know, we had maybe a hundredth of that size watching smaller events. So it really went from 0 to 100 fast for us. And taking that big swing meant we learned very quickly about where our systems needed to evolve. So a good example of that will be if you're talking about video on demand, we're able to place those content files very close to where you live. And when you press play, it is sitting there cached and ready to go. And pretty seamless. For live, it's going from the camera to the cloud to the content delivery network to your device in almost real time. And there is no room for error along that path. And the room for error gets very large with a high blast radius when a lot of people press play at the same time. So it's definitely a challenge. We had to rethink a lot of things that we had built over time. Open Connect, which is our content delivery network, has been a huge benefit to have as part of live. So while it was very highly tuned and optimized for video on demand, we've extended that to support live. So one part is the tech, one part is the skill set to run live operations, to have launch rooms and real-time triaging as inevitably something goes wrong. And how do we make sure the member experience stays very stable? And to go from Paul Tyson as an enormous swing, taking a lot of risk to NFL games, and now we're well over hundreds of live events. We do multiple every week. We just did World Baseball Classic in Japan, which was nonstop for several weeks with the teams working around the clock. It feels like a capability that we've got, but it doesn't make it easy to do.

The superpower of Netflix is that we're the combination of programming, content, and product. So the special sauce comes from that. And the challenge has to be solved by both parts of the company. So right now, I mean, we're not new to competition. I think that that's been true for a long time. The nature of that competition is changing. The nature of consumer preferences is changing. Time is finite versus the number of options that are out there. But in some ways that's a familiar challenge to us. Like we were always competing with other things people could be doing with their time. So we need to have the best content. We need to have that available with amazing experiences across both TV, when you're prime time, sitting with your family in the living room, and on mobile, when you want something snackable, something flexible, something that like meets you while you're in your commute or entertaining your kids at a restaurant. We need to get across more of those moments, which I think is a combination of best content, best product experience. So things like the expansion to new content types. So live is a good example of it. Games is a good example. Podcasts. More recently, we revamped our TV experience last year to be more flexible across those range of content types so we can expand into them, and more recently launched some changes on mobile that help with navigation and introduce a vertical video feed. So, together with the new content types and different discovery experiences or engagement experiences across TV and mobile, I think it positions us really well for the ways that entertainment time and competition are evolving, but that's a relentless focus, you know, like, it's always been. It always will be. And the more we stay focused on what we control, which is deliver a great member experience across those things, the better positioned we'll be.

It's one of the toughest challenges, especially for the product and tech team. I would say it's one of the things we're most focused on. The last thing I want to do is throw a set of things into the product and hope for the best. Part of our job is to make that feel cohesive and like somehow we understand you perfectly. When you sat down, we knew which type of content to really feature on the homepage. We knew how to merchandise that content so that you know what it's about and what that experience will be. And that motivated some of the changes we made to both TV and mobile so that it was more flexible, not just in what you see, but almost more importantly in what you don't see. So we built that in a more modular way that has more flexibility so that as we continue to add content types, we don't end up reflecting that complexity in the homepage or in your discovery experience. So that's new things that you may notice of how we've changed the navigation on the home page to be more prominent. We recently have rolled out just below the billboard or the most prominent title on the home page, a set of content bars that gives entry points in different aspects, so into podcasts or into certain genres. Those are just a couple examples of ways where we're having to evolve a reliable pathway for our member to understand. If I'm in a mode of wanting to watch podcasts, hopefully we show you podcasts on the homepage, but we also want to make it easy to find. Those things are easy to find, to continue watching. That problem has gotten more complex as we have more to offer. It's a good problem to have that we have more to offer, and we have to make sense of it. And that's where a lot of the testing and consumer insights help us to shape how we solve the challenge.

Well, they're definitely increasing in popularity. So I don't think that's unique to Netflix. There is something about podcasts as a creator type that the visual around it is especially compelling, you know, and it adds focus. What we hear from consumers is it makes it even more immersive when you're able to have the video plus the audio. But honestly, there's a whole range of how members engage with podcasts. Some is on the TV in the living room, some is on the mobile phone while you're on the move and listening to something. And so we've tried to support all those different experiences, because I think there's different preferences that even vary across types of podcasts or types of consumer behavior. And same for games. So we have mobile games. We have the playground app, which launched recently for kids games, which is a great, safe, special place of kids games, separate from advertising or other monetization as part of the app. And then we have cloud games on TV, so party games. So that's going to be a very different mode. You know, it's family game night, which is different than you're sitting a kid down with a playground game on a mobile phone. So the consumer behaviors are not uniform, which means we need to have a product experience that flexes across all of it. And we see that especially for some of the newer content types compared to the more core film and TV series.

Technical talent. Especially right now. Yeah. So the type of talent that I see most attracted to Netflix are people who are curious about how innovations and AI are going to change applications that clearly will be impacted by it. So there's some talent that is drawn towards, of course, frontier model companies and a focus on building the foundations of the new tech. And then there's a lot of talent, myself included, that is very excited about the applications of it. So the entertainment industry is going to be a great example where there's applications of AI from content creation, promotion, personalization. And so that of course there's always great competition for great talent. But Netflix has continued to be a draw for a lot of that great technical talent, which I'm not surprised by. But it's a difference in approach across all the different types of applications of AI.

Yeah, I honestly think one of the most impactful will be the member experience that we build around entertainment. So that's the more personalized, the more interactive, the more immersive that I think AI is going to enable. That helps to solve a consumer frustration that's brewing, which is there's so much content, how do I make sense of it and what's right for me and what's right for me in this moment? So that's going to be a big one. That's been a strength for Netflix for a long time, and I think it needs to continue to be a place where we're state of the art and continuing to innovate. Of course, there's also going to be applications for content creation. So those are spaces where the example like Inner Positive that we talked about earlier, I know that impact is going to be coming from humans staying at the center of content, creativity, and storytelling. And a company like Netflix giving the tools that enable that. So bring your vision to life. Have state-of-the-art tools that are available to do so, which opens up all kinds of new possibilities for some titles. So we've seen examples of this now. So Berlin season two in Spain is a good example of it, of being able to use AI tools to actually be able to recreate digital doubles of the set, to be able to come up with new ideas and reshoot and reframe and relight things and bring visions to life that they can see. I think that will change how creators actually bring their stories to life. But the role of Netflix there is going to be to make sure we're offering that full set of tools and the best in class. And then AI and ML have been a big part in tech generally, of how we make sure that content finds its audiences around the globe. So that's where subtitles, dubs, how we localize the product in the UI, how we even localize the title treatments, which is basically how the title is written on the box art that helps to promote or merchandise it. Those are all things that I think AI will help to supercharge. But each of those examples I gave, whether it's member personalization and interactivity and immersion, it's content creation, or it's content promotion and kind of traveling around the world, are places where tech has already been very powerful. So now it's sort of the next era of what's AI going to do in those spaces? So I see opportunity across all of it.