Retentive Network: A Successor to Transformer for Large Language Models (Paper Explained)

Yannic Kilcher • September 13, 2023

Yannic Kilcher

About

I make videos about machine learning research papers, programming, and issues of the AI community, and the broader impact of AI in society. Twitter: https://twitter.com/ykilcher Discord: https://ykilcher.com/discord BitChute: https://www.bitchute.com/channel/yannic-kilcher LinkedIn: https://www.linkedin.com/in/ykilcher BiliBili: https://space.bilibili.com/2017636191 If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribestar.com/yannickilcher Patreon: https://www.patreon.com/yannickilcher Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

Retentive Network: A Successor to Transformer for Large Language Models (Paper Explained)

Yannic Kilcher

About

Latest Posts

[Video Response] What Cloudflare's code mode misses about MCP and tool calling

[Paper Analysis] On the Theoretical Limitations of Embedding-Based Retrieval (Warning: Rant)

On the Biology of a Large Language Model (Part 1)

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

You May Also Like

Upgrade Your iPhone Experience

SURPHY Silicone Case Compatible with iPhone 13 Case 6.1 inch (2021), with Camera Protection Liquid Silicone Soft Gel Rubber Phone Case Cover with Microfiber Lining (Pink) Global Recycled Standard

JETech Silicone Case for iPhone 13 6.1-Inch, Silky-Soft Touch Full-Body Protective Phone Case, Shockproof Cover with Microfiber Lining (Black)

GuaGua Case for iPhone 13 6.1 Inch Liquid Silicone Soft Gel Rubber Slim Thin Microfiber Lining Cushion Texture Cover Shockproof Protective Anti-Scratch Phone Case for iPhone 13 2021 Yellow

JETech Case for iPhone 13 Pro 6.1-Inch, Non-Yellowing Shockproof Phone Bumper Cover, Anti-Scratch Clear Back (Clear)

Wireless Charging Case for AirPods Pro 1st / 2nd Gen with Type-C Charging Port, Compatible with AirPod Pro 2nd Generation Replacement Case, with Blue-Tooth Sync Button, No Earbuds

Cleaner Kit for Airpods Pro 1 2 3 Multi-Function Cleaning Pen with Soft Brush Flocking Sponge for Bluetooth Earphones Case Cleaning Tools White

Apple AirPods Pro 2 Wireless Earbuds, Bluetooth Headphones, Active Noise Cancellation, Transparency, Personalized Spatial Audio, High-Fidelity Sound, H2 Chip, USB-C Charging

Wireless Charger, 3 in 1 Charging Station, Fast Charging Station Foldable Charging Stand, Wireless Charging Station for Multiple Devices, Compatible with lPhone 17/16/15/14/13 Pro Max Airpods Watch

Anker USB C Charger Cable (6ft 100W, 2Pack), USB 2.0 Type C Fast Charging Cable for iPhone 15 / 15Pro / 15Plus / 15ProMax MacBook Pro 2020, iPad Pro 2020, iPad Air 4, Samsung Galaxy S23+/S23 Ultra ClimatePartner certified

Anker USB C Cable, [2-Pack, 6 ft] Type C Charger Premium Nylon USB Cable, USB A to Type C Charging Cable Fast Charge for Samsung Galaxy S10 S10+ / Note 8, LG V20 and Other USB C Charger (Black)

[ 3+3 Pack] LK 3Pack iPhone 13 Screen Protector 6.1-inch with 3Pack Camera Lens Protector, Anti-Scratch Easy-Installation,9H Hardness, HD Tempered Glass Film Global Recycled Standard

Spigen Case for Galaxy S25 Case, Rugged Armor MagFit [Fast Wireless Charging] Compatible with MagSafe - Black

Spigen Case for Galaxy S25 Case, Tough Armor (Ai) MagFit [Wireless Charging] Compatible with MagSafe - Black

Spigen Case for iPhone 16 Case, Tough Armor Designed for Apple iPhone 16 - Black

Spigen Case for iPhone 16 Pro Case, Tough Armor [MagFit] Compatible with MagSafe - Black

[ 3+3 Pack] LK 3Pack iPhone 13 Screen Protector 6.1-inch with 3Pack Camera Lens Protector, Anti-Scratch Easy-Installation,9H Hardness, HD Tempered Glass Film Global Recycled Standard

3 in 1 Charging Station for iPhone, Wireless Charger for iPhone 16 15 14 13 12 11 X Pro Max &amp; Apple Watch - Charging Stand Dock for AirPods Global Recycled Standard

3 in 1 Charging Station for iPhone, Wireless Charger for iPhone 16 15 14 13 12 11 X Pro Max &amp; Apple Watch - Charging Stand Dock for AirPods (Blue) Global Recycled Standard

JARGOU 3 in 1 Charging Station for Apple Wireless Charger for iPhone 16 15 14 13 12 11 X 8 &amp; for Apple Watch Charger Wireless Charging Station for Multiple Devices for AirPods 4 3 Pro

3 in 1 Foldable Charging Station for Apple Wireless Charger for iPhone 16 15 14 13 12 11 Series, Wireless Charging Station with Night Light for Apple Watch Charger for AirPods 4/3/2/Pro/Pro 2 (Black)

Magnetic Wireless Charger for iPhone - 3 in 1 Mag-Safe Charging Station for Multiple Devices Apple - Charger Stand Dock for Apple Watch iPhone 16 15 14 13 12 Pro Max Plus Airpods

4-in-1 Wireless Magnetic Power Bank 10800mAh Portable Charger with iWatch Charger, QC4.0+20W PD Fast Charging USB C Battery Pack for Magsafe, iPhone 16/15/14/13/12 Series, Apple Watch etc. (White)

eazpower for Magsafe Power Bank 10000mAh, Magnetic Battery Pack Wireless, 20W PD Fast Charging, Compact Portable Charger with Built-in Cables and Stand for iPhone 16/15/14/13/12/ Apple Watch/Airpods

for Magsafe Car Mount【20 Strong Magnets 】Magnetic Phone Holder for Car Phone Holder Mount Dash【360°】Cell Phone Mount for Car Accessories,Car Mount for iPhone 16 Pro Max 15 14 13 12 Plus MagSafe Cases

VANMASS [85+LBS Strongest Suction &amp; Clip] 2025 Upgrade Car Phone Holder [Patent &amp; Safety Certs] Cell Phone Mount Truck Stand for Dashboard Windshield Vent for iPhone 16 Pro Max 15 14 13 12 Samsung

[ 3+3 Pack] LK 3Pack iPhone 13 Screen Protector 6.1-inch with 3Pack Camera Lens Protector, Anti-Scratch Easy-Installation,9H Hardness, HD Tempered Glass Film Global Recycled Standard

Loading...

3 in 1 Charging Station for iPhone, Wireless Charger for iPhone 16 15 14 13 12 11 X Pro Max & Apple Watch - Charging Stand Dock for AirPods Global Recycled Standard

3 in 1 Charging Station for iPhone, Wireless Charger for iPhone 16 15 14 13 12 11 X Pro Max & Apple Watch - Charging Stand Dock for AirPods (Blue) Global Recycled Standard

JARGOU 3 in 1 Charging Station for Apple Wireless Charger for iPhone 16 15 14 13 12 11 X 8 & for Apple Watch Charger Wireless Charging Station for Multiple Devices for AirPods 4 3 Pro

VANMASS [85+LBS Strongest Suction & Clip] 2025 Upgrade Car Phone Holder [Patent & Safety Certs] Cell Phone Mount Truck Stand for Dashboard Windshield Vent for iPhone 16 Pro Max 15 14 13 12 Samsung