Senior Staff Machine Learning Engineer, Data & Eval

Airbnb was born in 2007 when two hosts welcomed three guests to their San Francisco home, and has since grown to over 5 million hosts who have welcomed over 2 billion guest arrivals in almost every country across the globe. Every day, hosts offer unique stays and experiences that make it possible for guests to connect with communities in a more authentic way. The Community You Will Join: AI and ML are at the heart of the Airbnb product. From Trust to Payments, and from Customer Service to Marketing, we rely on ML to ensure that guests and hosts have the best possible experience with Airbnb. The Core ML team is responsible for driving Airbnb Assistance Engineering initiatives by adopting Generative AI technologies to enable an intelligent, scalable, and exceptional service experience. The team develops and enhances AI models, ML services, and tools including LLM fine-tuning and optimization, RAG/Search, LLM evaluation and testing automation, feedback-based learning, and guardrails for a wide range of applications at Airbnb. The richness of Airbnb's data, the complexity of its marketplace, and the variety innate in our product mean that we need to operate at the state of the art of AI practice. We are committed to long-term innovation to solve complex problems, and to do that we need experienced ML The Difference You Will Make: In this Senior Staff Machine Learning Engineer [... source excerpt omitted ...] and tooling). Your work will define how we measure quality, how we turn feedback into learning signals, and how we continuously improve models and products safely and efficiently. You will partner closely with product, engineering, design, operations to build evaluation systems that are trusted, scalable, and actionable - connecting offline metrics to online outcomes. A Typical Day: Define evaluation strategy and success metrics for GenAI systems, aligning offline evaluation with online business and customer experience outcomes. Build and scale evaluation frameworks (golden sets, synthetic data, automated regressions, rubric-based grading, LLM-as-judge where appropriate) with [... source excerpt omitted ...] t. Lead cross-functional quality initiatives across product, ops, and engineering, driving clarity on what “good” looks like and how teams act on evaluation results. Develop and productionize pipelines for dataset creation, model monitoring, evaluation-at-scale, and continuous testing (pre-deploy and post-deploy). Drive technical decisions and architecture for evaluation and data infrastructure, balancing speed, rigor, cost, and safety. Minimum Qualifications: Educational Background: PhD in Computer Science, Mathematics, Statistics, or related technical field (or equivalent practical experience). Industry Experience: 10+ years building, testing, and shipping ML/AI systems end