Llama 4 Explained: Architecture, Long Context, and Native Multimodality

Julia Turc • April 9, 2025

Julia Turc

About

AI explainer videos from a former Google Research engineer, now startup founder. Expect American optimism with a solid dose of Eastern European cynicism. Find (free) supplemental materials (slide decks, paper reading lists, Colab notebooks) on Patreon 👉 https://www.patreon.com/c/JuliaTurc

Llama 4 Explained: Architecture, Long Context, and Native Multimodality

Julia Turc

About

Latest Posts

Transformers & Diffusion LLMs: What's the connection?

Text diffusion: A new paradigm for LLMs

The physics behind diffusion models

How LLMs survive in low precision | Quantization Fundamentals

You May Also Like

Essential AI Training Tools

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 &amp; PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super &amp; AMD GPU Compatible

Skytech Legacy Gaming PC Desktop, Ryzen 7 9800X3D 4.7 GHz (5.2 GHz Turbo Boost), NVIDIA RTX 5080 16GB GDDR7, 1TB Gen4 SSD, 32GB DDR5 RAM 6000 RGB, 1000W Gold ATX 3.0 PSU, 360 ARGB AIO, Wi-Fi, Win 11

HP Victus 15.6&quot; 144Hz FHD Gaming Laptop, Intel i5-12450H, 32GB RAM, 1TB PCIe SSD, NVIDIA GeForce RTX 3050, Backlit Keyboard, HD Webcam, Win 11, Blue, 256GB Docking Station Set

AtomMan G7 Pt Mini PC AMD Ryzen 9 7945HX(16C/32T, up to 5.4GHz) 32GB DDR5 1TB PCIe4.0 SSD Micro Computer, HDMI+DP+USB-C Output, 2.5G LAN, WiFi7, BT5.4, 4xUSB AMD Radeon RX 7600M XT Graphics Gaming PC

STGAubron Gaming Desktop PC, AMD Athlon 3000G 3.5G, Radeon RX 580 16G GDDR5, 16G RAM, 512G SSD, 600M WiFi, BT 5.0, RGB Fan x4, Windows 11 Home

Samsung 990 EVO Plus - 4TB PCIe Gen4. X4, Gen5. X2 NVMe 2.0 - M.2 Internal SSD, Speed Up to 7,250 MBs, Upgrade Storage for PC-Laptops, HMB Technology and Intelligent Turbowrite (MZ-V9S4T0B/AM)

SAMSUNG 870 EVO SATA SSD 500GB 2.5” Internal Solid State Drive, Upgrade PC or Laptop Memory and Storage for IT Pros, Creators, Everyday Users, MZ-77E500B/AM [Canada Version]

SAMSUNG 870 EVO SATA III SSD 4TB 2.5” Internal Solid State Drive, Upgrade PC or Laptop Memory and Storage for IT Pros, Creators, Everyday Users, MZ-77E4T0B/AM [Canada Version]

SAMSUNG EVO Select Micro SD-Memory-Card + Adapter, 128GB microSDXC 160MB/s Full HD &amp; 4K UHD, UHS-I, U3, A2, V30, for Android Smartphones, Tablets, Nintendo-Switch (MB-ME128SA/AM) [Canada Version]

Crucial 64GB DDR5 RAM, 5600MHz (or 5200MHz or 4800MHz) Desktop Memory Kit, UDIMM 288-Pin, Compatible with 13th Gen Intel Core and AMD Ryzen 7000 - CT2K32G56C46U5

Intel Core i9-12900KF Desktop Processor 16 (8P+8E) Cores up to 5.2 GHz Unlocked LGA1700 600 Series Chipset 125W

Asus ROG Strix 1000W Gold Aura Edition (Fully Modular Power Supply, 80+ Gold Certified, ATX 3.0, Cybenetics Lambda A+ Certification, PCIe Gen 5.0 Ready, Axial-tech Fan, Aura Sync, 10-Year Warranty)

Logitech G305 LIGHTSPEED Wireless Gaming Mouse, Hero 12K Sensor, 12,000 DPI, Lightweight, 6 Programmable Buttons, 250h Battery Life, On-Board Memory, PC/Mac - Black

Logitech M185 Wireless Mouse, 2.4GHz with USB Mini Receiver, 12-Month Battery Life, 1000 DPI Optical Tracking, Ambidextrous, Compatible with PC, Mac, Laptop - Black

Logitech G203 Wired Gaming Mouse, 8,000 DPI, Rainbow Optical Effect LIGHTSYNC RGB, 6 Programmable Buttons, On-Board Memory, Screen Mapping, PC/Mac Computer and Laptop Compatible - Black

Logitech G435 Lightspeed and Bluetooth Wireless Gaming Headset - Lightweight Over-Ear Headphones, Built-in mics, 18h Battery, Compatible with Dolby Atmos, PC, Mobile - Black

NEW SteelSeries Arctis Nova 3 Multi-Platform Gaming Headset - Signature Arctis Sound - ClearCast Gen 2 Mic - PC, PS5/PS4, Xbox Series X|S, Switch, Mobile

New SteelSeries Arctis Nova Pro for Xbox Multi-System Gaming Headset - Premium Hi-Fi Drivers - Hi-Res Audio - 360° Spatial - GameDAC Gen 2 - Quad-DAC - ClearCast Gen 2 Mic - Xbox, PC, PS5/PS4, Switch

Loading...

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 & PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super & AMD GPU Compatible

HP Victus 15.6" 144Hz FHD Gaming Laptop, Intel i5-12450H, 32GB RAM, 1TB PCIe SSD, NVIDIA GeForce RTX 3050, Backlit Keyboard, HD Webcam, Win 11, Blue, 256GB Docking Station Set

SAMSUNG EVO Select Micro SD-Memory-Card + Adapter, 128GB microSDXC 160MB/s Full HD & 4K UHD, UHS-I, U3, A2, V30, for Android Smartphones, Tablets, Nintendo-Switch (MB-ME128SA/AM) [Canada Version]