Natalie Glance on content moderation
From KDD 2025 - Applied Data Science Invited Speakers: Natalie Glance · · Association for Computing Machinery (ACM)
“We have a simulated bot approach that pressure tests conversations to check if Lily says anything inappropriate. We also use OpenAI's filtering model to check all outputs for objectionable content. This combination has been working well for us.”
On , Natalie Glance, Chief Engineering Officer at DUOLINGO INC, spoke about content moderation during KDD 2025 - Applied Data Science Invited Speakers: Natalie Glance on Association for Computing Machinery (ACM).
About Natalie Glance
Natalie Glance, Chief Engineering Officer at Duolingo, has spoken about the company's use of generative AI to improve learning efficacy. At KDD 2025, she described a fine-tuning approach for language models that involved sampling overly difficult conversations, simplifying the responses, and tuning the model, which she said made about 70% of sessions easier for learners. She also discussed the "video call with Lily" feature, which uses AI to conduct natural conversations with learners, and noted that the team focuses on "total words spoken" as a proxy for engagement. Glance stated that the company gave the video call team a budget of millions of dollars for AI spending, allowing them to decide how to allocate funds between cost optimization and model quality. At Config 2025, Glance discussed Duolingo's engineering and design culture, noting that the company outlawed the term "MVP" in favor of "V1" to emphasize quality while pragmatically cutting corners. She said the company runs hundreds of A/B tests in parallel and described a recent change replacing the "hearts" mechanic with an "energy" system, which she said was initially met with little leadership support but proved positive for both monetization and daily active user growth. Glance also mentioned that designers are starting to use AI for "vibe coding" complex animations to better communicate their vision to engineers.