RE-Bench: measuring AI agents at AI R&D vs human experts

Samuel Albanie • December 20, 2024

Samuel Albanie

About

AI research. Note that opinions expressed are my own. However, for conflict of interest purposes, please note that I'm employed by Google DeepMind (GDM). This almost certainly biases my judgment to some degree. I think GDM is pretty great. Other related content: - misc/outdoor stuff channel: https://www.youtube.com/@samuelalbaniemisc - https://x.com/SamuelAlbanie - https://bsky.app/profile/samuelalbanie.bsky.social FAQ: Software used to make videos - keynote on mac (to make slides) - Adobe Premiere Pro for editing

RE-Bench: measuring AI agents at AI R&amp;D vs human experts

Samuel Albanie

About

Latest Posts

The Agent Company: Benchmarking LLM Agents on Consequential Real World Tasks

Prover-Verifier Games improve legibility of LLM outputs

Deliberative Alignment: Reasoning Enables Safer Language Models

Alignment Faking in Large Language Models

You May Also Like

Elevate Your AI Testing

GMKtec Mini PC NAS, G9 N150 Dual Boot Win Linux Desktop Computer 12GB DDR5 64GB EMMC + 1TB M.2 SSD, 4*M.2 NVMe Slots, Dual NIC 2.5GbE, USB-C/USB 3.2 10Gbps, Network Attached Storage

MeLE Quieter DL Mini PC Windows 11 Home, N100 4GB 128GB, 2.5G Dual LAN,IoT Industrial Desktop Computer Support Windows 10 11 Linux Ubuntu Debian 4K Triple Display, Dual HDMI, All-in-One USB-C

10Gtek USB WiFi Adapter, for PC, AC600M USB WiFi Dongle 802.11ac Wireless Network Adapter with Dual Band 2.4GHz/5Ghz for Desktop Laptop Support Windows 10/8/7/XP, MAC OS, Linux etc

TP-Link USB WiFi Adapter for PC(TL-WN725N), N150 Wireless Network Adapter for Desktop - Nano Size WiFi Dongle for Windows 11/10/7/8/8.1/XP/ Mac OS 10.9-10.15 Linux Kernel 2.6.18-4.4.3, 2.4GHz Only

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge R660XS R760XS

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge XE9680

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge R6625 R760 R7615 R7625

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge HS5610 HS5620

YMZ Smart Watch for Men Women, IP68 Waterproof, Bluetooth Calling, Fitness Tracker, Sleep Monitoring, 1.85&#x27;&#x27; DIY Dial, AI Built-in, 100+ Sports Modes, 7 Days Battery, Compatible with Android &amp; iPhones

Intel Core i9-12900KF Desktop Processor 16 (8P+8E) Cores up to 5.2 GHz Unlocked LGA1700 600 Series Chipset 125W

Asus ROG Strix 1000W Gold Aura Edition (Fully Modular Power Supply, 80+ Gold Certified, ATX 3.0, Cybenetics Lambda A+ Certification, PCIe Gen 5.0 Ready, Axial-tech Fan, Aura Sync, 10-Year Warranty)

TERRAMASTER F8 SSD Plus NAS - 8Bay All SSD NAS Storage Core i3 8-Core 8-Thread CPU, 16GB DDR5 RAM, 10GbE Port, 8 Heat Sinks Included, Palm-Sized Network Attached Storage Peak Performance (Diskless)

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge R660XS R760XS

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge XE9680

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge R6625 R760 R7615 R7625

OWC 64GB DDR5 4800 PC5-38400 CL40 2Rx4 288-pin 1.1V ECC Registered RDIMM Memory RAM Module Upgrade Compatible with Dell PowerEdge HS5610 HS5620

be quiet! Dark Power Pro 13 1300W Power Supply | ATX 3.1 Compliant | 80 Plus Titanium | Digital Regulation for PCIe 5.0 and PCIe 6+2 Graphics Cards, | 2X 12VHPWR Cable Included | BN500

MINISFORUM MS-01 Mini Workstation Core i9-12900H 32GB RAM 1TB SSD Mini PC, HDMI+2X USB4 8K Display, 2x10G SFP+ Port, 2x2.5G LAN Port, Support M.2 2280/22110/U.2 SSD/RTX 3050 Graphics Cards

Loading...

RE-Bench: measuring AI agents at AI R&D vs human experts

YMZ Smart Watch for Men Women, IP68 Waterproof, Bluetooth Calling, Fitness Tracker, Sleep Monitoring, 1.85'' DIY Dial, AI Built-in, 100+ Sports Modes, 7 Days Battery, Compatible with Android & iPhones