【播客】Netflix没有绩效 | CTO Elizabeth Stone | 技术版图 | Pitch to Play | Open Connect | 独特的企业文化 | 留任测试

Hi everyone, this is Best Partners, I'm Dafei. When we talk about Netflix, what's the first thing that comes to mind? Is it the Upside Down from Stranger Things? Is it the political maneuvering depicted in House of Cards? Is it still that famous red N logo? To most people, Netflix is a Hollywood-level entertainment production company. But in our tech world, Netflix has another, more hardcore identity: it is a top Silicon Valley tech giant, a pioneer of microservice architecture, and the originator of chaos engineering. Many people may wonder, how deep is the technological moat of a streaming media company? What is it like to work as a software engineer at Netflix? To unravel these mysteries, today we'll have an in-depth conversation with Elizabeth Stone, Netflix's current Chief Technology Officer, through the perspective of renowned tech blogger Gergely Orosz. This conversation not only reveals how Netflix handled the recent record-breaking boxing match's surge in livestream traffic but also delves deeper into Netflix's unique, even counterintuitive, engineering culture in Silicon Valley. Today, I'll share some insights.

First, we need to correct a common misconception: many people think that Netflix engineers simply maintain a website and a few apps, ensuring videos play. Elizabeth's data in the interview might surprise you. If we add up the consumer interactions, product-service calls, and decision-making data that Netflix captures daily, the number is staggering: over one trillion events daily. Note that this one trillion isn't just a simple log line count, but business-significant events. The technological complexity behind this far exceeds our imagination. Netflix's technology team doesn't just serve end-users; their reach extends to every aspect of the entertainment industry. Elizabeth mentioned that Netflix's technology landscape can be roughly divided into several huge segments. First, there's the streaming experience everyone is familiar with. How to make videos play smoothly under such massive concurrency, even in extremely poor network conditions, so that users feel a seamless experience? This is itself an extremely high-barrier engineering challenge. Second, there's studio technology. This is the biggest difference between Netflix and traditional streaming platforms. Netflix is one of the world's largest film and television studios. They possess a complete self-developed toolchain from scriptwriting, filming, post-production to distribution. We'll discuss this unique concept, known as Pitch to Play, in detail later. Third, there's Netflix's heavily invested gaming and advertising businesses in recent years. Especially the construction of the advertising technology stack—for a company like Netflix that has long adhered to an ad-free subscription model, building a complex monetization system from scratch in an extremely short time—was a significant undertaking. Finally, there's the business and payment system supporting all of this. With over 190 countries worldwide, different currencies, different payment channels, and complex partnerships, all of this requires an extremely robust underlying architecture. So when you click that red play button on Netflix, you're not just triggering a video stream, but a vast digital empire built by thousands of top engineers, processing trillions of data points daily.

In the interview, Elizabeth introduced a core concept called Pitch to Play. This is not only Netflix's business process but also the soul of their technical architecture. So what exactly is Pitch to Play? In the traditional film and television industry, technology is fragmented. A production crew uses one system for filming, another for editing, and distribution relies on a third-party CDN. However, Netflix's technology team has built an end-to-end closed loop. Imagine this scenario: a Netflix crew is filming in a remote corner of Europe, while a producer in Los Angeles needs to see the day's footage immediately. In the past, this might have required physically transporting hard drives or using extremely expensive and slow dedicated lines. But Netflix developed its own media production suite. Elizabeth explained that this system completely modernizes the way media files are transmitted. Footage from the shooting location can be quickly and securely uploaded, instantly reaching the review room in Los Angeles via Netflix's self-built high-speed channels. The producer's feedback can then be fed back to the European set in real time through this system. This seamless technological connection greatly shortens the production cycle and reduces communication costs. And the end of this closed loop is the renowned Open Connect. If you're a tech enthusiast, you've definitely heard of this name—Netflix's self-built content delivery network. Elizabeth revealed that Netflix has deployed over 6,000 Open Connect server nodes globally. Covering over 175 countries, this is more than just placing a few servers in various locations. Open Connect physically connects directly to local Internet Service Providers (ISPs). This means that when you watch Netflix at home, the video traffic often doesn't travel across the ocean from a distant US data center, but is read directly from your doorstep, or even from a server in your local ISP's server room. This physical-level optimization, which doesn't even need to go through the public internet backbone, is the fundamental reason why Netflix can provide extremely low latency and extremely high picture quality. Elizabeth mentioned that this vertical integration capability, which intervenes from the source of content production and controls the last-mile transmission, is a super move that other streaming media platforms that simply rely on third-party cloud services or CDNs cannot replicate.

If video-on-demand (VOD) is Netflix's comfort zone, then live streaming is the uncharted territory they have actively stepped into in recent years. In the interview, Elizabeth spent a lot of time reviewing a recent epic battle: the boxing match between Jack Paul and Mike Tyson. This match will be held in November 2024. It is not only a grand event in the boxing world, but also a milestone in the history of internet live streaming technology. What does 65 million concurrent streams mean? This is the largest online live streaming event in human history, surpassing not only the online viewership of any Super Bowl but also far exceeding the scale of any previous esports event or concert live stream. Elizabeth admitted that although they had made ample preparations before the game started, the air in the entire command center froze when the numbers began to spike. This wasn't Netflix's first time doing live streaming. Back in March 2023, they tested live streaming technology with a Chris Rock talk show special, followed by a Love is Blind reunion special. That one wasn't perfect; in fact, it could be said that there were obvious technical glitches. But it was that failure that became a catalyst for Netflix's technological evolution. Elizabeth recalled that after the show malfunctioned, the team didn't fall into mutual blame but quickly entered a rapid learning mode. This learning was put to the ultimate test in the Paul vs. Tyson match. On the night of the match, about 100 people gathered at Netflix headquarters in Los Gatos, with many more waiting remotely. Elizabeth described the scene: everyone had a laptop in a makeshift room. Some people even used wired connections, fearing Wi-Fi issues. They also prepared VPN backups and had dedicated broadcast commanders wearing headsets to communicate directly with the broadcast trucks ahead. As the preliminary matches began, the number of concurrent users exceeded their expected peak, and the system began to show signs of stress. The buffering rate started to climb, which is the most critical metric for live streaming. Once users start seeing the spinning wheel frequently, the experience will collapse. At this critical moment, Netflix engineers demonstrated extremely high professionalism. They did not sit idly by but began to adjust traffic strategies in real time. Elizabeth mentioned a detail: in the live streaming field, once congestion occurs, you can't simply ask users to try again later like with on-demand streaming, because live streaming only happens once. The team activated the 40-50 page If-Then decision tree from their contingency plan. For example, when bandwidth in a certain area is tight, how will the algorithm degrade the image quality to ensure smoothness? How can Open Connect dynamically redirect traffic to adjacent nodes when an ISP's link is congested? While the final experience still had some flaws for some users, the system didn't crash. It withstood the instantaneous surge of 65 million users. Elizabeth joked that she felt ten years older that night, but when she woke up the next morning, she saw a post-mortem document prepared by the team overnight. Instead of shifting blame, it detailed what they observed, where improvements could be made, and what the next steps should be. This is precisely the growing pain and glory of Netflix's transformation from a video-on-demand giant to a live streaming giant.

Having discussed the core technology, we must talk about the soft power supporting all of this—Netflix's renowned engineering culture. This is perhaps one of the most controversial yet most enviable cultures in Silicon Valley. In the interview, Elizabeth confirmed a fact that many employees of large companies envy yet are confused by: Netflix indeed doesn't have formal performance appraisals, no scoring, no KPIs, and no grueling, semi-annual PPT presentations. This sounds wonderful, but Elizabeth points out that this actually places higher demands on managers and is more demanding on employees' self-motivation. Netflix replaces this with two mechanisms: first, continuous and honest feedback. You don't need to wait until the end of the year to find out if you've done a good job. If your code has problems or your collaboration style makes people uncomfortable, you'll receive feedback immediately. This feedback is two-way, even to your superiors. Secondly, there's 360-degree feedback. This is an annual process where everyone can send feedback requests to their collaborators. This feedback isn't to determine your bonus, but to help you grow. So, since there are no performance reviews, how does Netflix decide who should stay and who should leave? This touches upon Netflix's most core and most brutal cultural cornerstone: the Keeper Test. Elizabeth explains that the core question of the Keeper Test is that managers need to ask themselves, 'If this person were to resign tomorrow to go to a competitor, would I do everything in my power to retain them?' If the answer is no, then perhaps this is not the most suitable employment relationship. This may sound cold, but Elizabeth emphasizes that it is actually a two-way street. Employees should also ask themselves: Is my manager empowering me? Am I doing the best job of my career here? If the answer is no, then employees should also consider leaving. This culture has created Netflix's extremely high talent density. According to data from SignalFire, Netflix ranks among the top tech companies in terms of talent retention and talent standards. Elizabeth believes that high talent density is itself a huge attraction. The best engineers want to work with the best people. When you know that your colleagues are capable, responsible adults, you feel a great sense of professional satisfaction. This culture has also given rise to another principle: Context, not Control. In the preparation for the live broadcast of Paul vs. Tyson, although there was a commander, the decision-making power was delegated to front-line engineers most of the time. This is because Netflix believes that those closest to the code and systems have the most accurate information. The manager's task is to provide sufficient background information, such as the strategic significance of this live broadcast for the company and our acceptable risk tolerance, and then let the engineers decide what to do.

When it comes to talent, Netflix has long had an unwritten rule in the recruitment market: only hiring senior engineers. For a long time, Netflix even had only one engineer level: senior software engineer. However, in this interview, Elizabeth elaborated on the major shift in this strategy. About a few years ago, Netflix has begun introducing a new job title system and has started recruiting junior engineers and recent graduates. Why is there such a change? Elizabeth explained that this is an inevitable consequence of business development. As Netflix's technology stack grows larger, not all jobs require architects with 15 years of experience. Some tasks, such as exploring new technologies and developing specific modules, are actually more suitable for young people who have just graduated, have active minds, and a natural affinity for new technologies. More importantly, Netflix has realized the importance of internal talent development. Rather than forever competing for expensive senior talent in the market, it's better to establish a mechanism to cultivate top engineers who fit Netflix's culture. Elizabeth proudly mentioned that Netflix now has a well-established internship program and a process for recruiting recent graduates. The addition of these young people has brought new vitality and perspectives to the team, but this does not mean lowering standards. On the contrary, Netflix's selection criteria for junior engineers remain extremely high. They value potential and cultural fit, especially a strong sense of curiosity and self-motivation.

When mentioning Netflix's open source, people may immediately think of Chaos Monkey, the crazy tool that randomly kills service processes in production environments to test system resilience. However, Elizabeth told us that Netflix's open source contributions go far beyond that, especially in the field of video encoding. Netflix is one of the founding members of the Consortium for Open Media. Committed to promoting the openness and standardization of video coding technologies, such as AV1 encoding. Why would a commercial company do this? Elizabeth's logic is that by driving industry-wide advancements in encoding technology, videos can maintain higher quality with lower bandwidth. Data shows that Netflix now requires 60% less bandwidth to transmit videos of the same quality than it did in the past. This not only saved Netflix a huge amount of bandwidth costs, but more importantly, it allowed users in areas with poor network conditions to enjoy high-quality streaming services. This is a typical example of Netflix improving the user experience through technology. For this, Netflix's technology team even won nine Emmy Awards. Yes, you heard right, a technology team winning Emmy Awards is unique in Hollywood.

In a time when all tech companies are talking about AI, Elizabeth's attitude towards AI is remarkably calm and pragmatic. She didn't talk about how AI will disrupt everything, but focused on how AI can be used as a tool to improve efficiency. Netflix is experimenting with various AI coding assistants to help engineers handle tedious tasks such as document writing, code migration, and anomaly detection. Elizabeth clearly stated that AI is not a silver bullet. She believes that while AI is very useful in certain areas, such as prototyping and data analysis, human judgment remains irreplaceable in core, high-risk system decisions. Netflix's strategy is to provide various AI tools to engineers, allowing them to explore and experiment to find the most suitable usage for their workflows. This bottom-up AI application strategy also reaffirms Netflix's core culture of 'Context not Control'.

Throughout the interview, Elizabeth showed us a Netflix that's less of an entertainment company and more of a top-tier technology lab. From its backend architecture handling trillions of data points to its physical hardware directly connected to ISPs, from its ability to withstand the largest live stream traffic in the world to its management philosophy of daring to eliminate performance evaluations, every technical decision and management move at Netflix seems to challenge industry norms. Just as Elizabeth advised all engineers, stay curious. Whether you're a beginner or a seasoned architect, in an environment like Netflix, the most important ability isn't how many languages you can write, but whether you dare to ask, 'Why do we need to solve this problem?' Is this the best solution? This is Netflix, a place that uses cutting-edge technology to tell the most tender stories. Thank you for watching this video, see you next time.