The Benchmarks Game: Why It's Rigged and How You Can (Really) Win - Darius Emrani

Video Description

AI benchmarks control billions in investment and shape entire markets - but the game is rigged. In this talk, I'll expose the three "cheat codes" companies use to game benchmarks: * Cherry-picking comparisons (xAI's selective Grok-3 graphs) * Buying privileged access (OpenAI's FrontierMath funding) * Optimizing for style over substance (Meta's 27 Llama-4 variants on LM Arena) When Andrej Karpathy says "I don't really know what metrics to look at right now," we have a crisis. I'll show you why Goodhart's Law guarantees benchmarks fail when billions are at stake, and more importantly, what to do about it. You'll learn: How to spot benchmark manipulation (with real examples) Why 39% of score variance is just writing style A 5-step framework to build evaluations that actually matter for YOUR use case How pre-deployment evaluation loops separate reliable AI from constant firefighting Drawing from my experience building evaluation systems at Waymo, Uber ATG, and SpaceX (where bad evals literally crash), I'll show you how to stop playing the rigged benchmarks game and start measuring what actually matters.