Cofounder, Openai
Search every verified Vicki Cheung interview, podcast appearance, and on-the-record quote — each transcript cross-checked by AI and human review to confirm speaker identity. Vicki Cheung, a cofounder of OpenAI, has discussed the challenges and lessons learned from managing cloud-native infrastructure for machine learning research. In a 2020 talk, she recounted an incident where an intern accidentally launched a job that consumed all available compute resources, which she estimated would have cost roughly a million dollars to complete. She emphasized the importance of setting reasonable limits and implementing cost monitoring to prevent such overuse, noting that the infrastructure had been made "too easy to scale." In a 2017 keynote, Cheung described OpenAI's use of Kubernetes to manage a cluster spanning multiple cloud providers and on-premises hardware, scaling up to 4,000 nodes. She explained that the infrastructure was designed to support large batch jobs, some running for weeks, and to allow researchers to scale experiments from one core to 10,000 cores as needed. Cheung highlighted the value of open-source software and Kubernetes' flexibility, which enabled the team to build custom tools while shielding researchers from underlying complexity.
“One job was taking over our entire infrastructure, and we discovered that our intern was so excited about how easy it is to scale all this infrastructure that he had launched a job that was just going to consume all the compute that he could.”
“Had he been able to run this experiment, it would probably cost on the order of a million dollars to finish running the experiment and also consume all of the available infrastructure.”
“The moral of the story is that it's great that cloud native infrastructure is super easy to scale and also quite usable, but always put in reasonable limits.”
“Cost monitoring is super important, especially when you have a system that can scale so easily.”
“We're a non-profit AI research company and our mission is to democratize the AI technology.”
“Because of that we care a lot about open source software — we rely heavily on them and we also care deeply about benefiting from and contributing back to the community.”
“We're mostly running large batch jobs — jobs can span hours to days or even weeks and sometimes we run tens of thousands of copies, which puts a very different strain on the cluster.”
“Our infrastructure needs to be flexible enough to be reconfigured super easily — the goal is to empower users to run experiments by themselves without needing engineering support while shielding them from underlying complexity.”
“Up until a few weeks ago we ran a cluster across three different cloud providers and our own physical hardware — we scaled up to 4,000 nodes and it went pretty well; we had no major outages.”
“Coop allows us to ship infrastructure as internal products — it needs product-like qualities: a consistent UI that's understandable, hides complexity, but is composable so you can do inherently complex things without ripping your hair out.”
“We had to change so that we can train in parallel: you have a master parameter server and you split your training into parallel workers that interact with their own copy of the game, which lets you scale training.”
“Researchers can scale from one core to 10,000 cores very suddenly (paper deadlines happen) so we dynamically resize our cluster and wrote an autoscaler that provisions new nodes exactly to fit missing resource requirements using AWS APIs.”
“By default we put everything on spot instances because it's much cheaper — of course people can opt out if they didn't implement checkpointing, but these kinds of policies are very easy to implement and cleanly separated from the scaling logic.”
“At the most basic level Kubernetes exposes very good primitives and interfaces we can build on — it strips away a lot of complexities of the underlying infrastructure and exposes a clean API for both researchers and maintainers.”
Keynote: Building the Infrastructure that Powers the Future of AI - Vicki Cheung, Member of Technical Staff & Jonas Schneider, Member of Technical Staff, OpenAI OpenAI is a non-profit research company that does cutting-edge AI research. Our mission is to build safe AI, and ensure AI's benefits are as widely and evenly distributed as possible. This means democratizing the technology and releasing our research publicly. As a result, we rely heavily on open-source software. The majority of our experiments run on our Kubernetes cluster that spans Azure, AWS, and our own data center. Kubernetes an…
Sign in to search the full transcript archive, filter by topic, and access every quote from Vicki Cheung.
The summary and quote tags on this profile are produced with AI assistance from verified, first-person interview transcripts, then checked by our team to confirm the speaker's identity and the accuracy of every quote. See how we verify →