How Companies Hack Benchmarks

Video Description

In this video, I dive into the controversy surrounding the Leaderboard Illusion paper and what it reveals about systematic flaws in LLM benchmarks—especially Chatbot Arena. As someone who’s followed the evolution of these leaderboards closely, I was shocked by the extent of data access disparities and selective reporting. This is a wake-up call for the entire AI community. LINKS: https://arxiv.org/pdf/2504.20879 https://lmarena.ai/?leaderboard https://blog.lmarena.ai/blog/2025/two-year-celebration/ https://techcrunch.com/2024/09/05/the-ai-industry-is-obsessed-with-chatbot-arena-but-it-might-not-be-the-best-benchmark/?guccounter=1#:~:text=Bias%2C%20and%20lack%20of%20transparency https://www.vktr.com/ai-market/the-benchmark-trap-why-ais-favorite-metrics-might-be-misleading-us/ https://arcprize.org/arc-agi#arc-agi-1 https://www.lesswrong.com/posts/8ZgLYwBmB3vLavjKE/some-lessons-from-the-openai-frontiermath-debacle https://epoch.ai/frontiermath https://x.com/random_walker/status/1917516403977994378 https://x.com/karpathy/status/1917546757929722115 https://x.com/alexalbert__/status/1916878483390869612 https://x.com/lmarena_ai/status/1917492084359192890 https://blog.lmarena.ai/blog/2025/two-year-celebration/ RAG Beyond Basics Course: https://prompt-s-site.thinkific.com/courses/rag Let's Connect: 🦾 Discord: https://discord.com/invite/t4eYQRUcXB ☕ Buy me a Coffee: https://ko-fi.com/promptengineering |🔴 Patreon: https://www.patreon.com/PromptEngineering 💼Consulting: https://calendly.com/engineerprompt/consulting-call 📧 Business Contact: [email protected] Become Member: http://tinyurl.com/y5h28s6h 💻 Pre-configured localGPT VM: https://bit.ly/localGPT (use Code: PromptEngineering for 50% off). Signup for Newsletter, localgpt: https://tally.so/r/3y9bb0 00:00 Introduction to The Leaderboard Illusion 01:02 Understanding LM Arena 02:05 The LAMA Four Controversy 03:17 Systematic Issues in Benchmarks 09:05 Community Reactions and Criticisms 15:39 LM Arena Team's Response 20:33 Conclusion and Final Thoughts