RWKV: Reinventing RNNs for the Transformer Era (Paper Explained)

Yannic Kilcher • September 22, 2023

Yannic Kilcher

About

I make videos about machine learning research papers, programming, and issues of the AI community, and the broader impact of AI in society. Twitter: https://twitter.com/ykilcher Discord: https://ykilcher.com/discord BitChute: https://www.bitchute.com/channel/yannic-kilcher LinkedIn: https://www.linkedin.com/in/ykilcher BiliBili: https://space.bilibili.com/2017636191 If you want to support me, the best thing to do is to share out the content :) If you want to support me financially (completely optional and voluntary, but a lot of people have asked for this): SubscribeStar: https://www.subscribestar.com/yannickilcher Patreon: https://www.patreon.com/yannickilcher Bitcoin (BTC): bc1q49lsw3q325tr58ygf8sudx2dqfguclvngvy2cq Ethereum (ETH): 0x7ad3513E3B8f66799f507Aa7874b1B0eBC7F85e2 Litecoin (LTC): LQW2TRyKYetVC8WjFkhpPhtpbDM4Vw7r9m Monero (XMR): 4ACL8AGrEo5hAir8A9CeVrW8pEauWvnp1WnSDZxW7tziCDLhZAGsgzhRQABDnFy8yuM9fWJDviJPHKRjV4FWt19CJZN9D4n

Latest Posts

[Video Response] What Cloudflare's code mode misses about MCP and tool calling

Yannic Kilcher

[Paper Analysis] On the Theoretical Limitations of Embedding-Based Retrieval (Warning: Rant)

Yannic Kilcher

On the Biology of a Large Language Model (Part 1)

Yannic Kilcher

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Yannic Kilcher

RWKV: Reinventing RNNs for the Transformer Era (Paper Explained)

Yannic Kilcher

About

Latest Posts

[Video Response] What Cloudflare's code mode misses about MCP and tool calling

[Paper Analysis] On the Theoretical Limitations of Embedding-Based Retrieval (Warning: Rant)

On the Biology of a Large Language Model (Part 1)

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

You May Also Like

Titans: Learning to Memorize at Test Time (Paper Analysis)

Efficient Streaming Language Models with Attention Sinks (Paper Explained)

Something Weird Happens When E=−mc²

Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained)

Mustafa Suleyman on Superintelligence, Microsoft’s OpenAI Deal, Broligarchy | The Mishal Husain Show

Tech author Karen Hao: 'The Sam Altmans of the world are going to be fine" | Science Quickly Podcast

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Retentive Network: A Successor to Transformer for Large Language Models (Paper Explained)

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Paper)

Visualizing transformers and attention | Talk for TNG Big Tech Day '24

LSTM: The Comeback Story? [Prof. Sepp Hochreiter]

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

MIT 6.S191 (2024): Recurrent Neural Networks, Transformers, and Attention

Flow Matching for Generative Modeling (Paper Explained)

What are Transformer Models and how do they work?

Scalable MatMul-free Language Modeling (Paper Explained)

FlashAttention - Tri Dao | Stanford MLSys #67

MCP vs API: Simplifying AI Agent Integration with External Data

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (Paper Explained)

Transformers, the tech behind LLMs | Deep Learning Chapter 5

RWKV: Reinventing RNNs for the Transformer Era (Paper Explained)

Yannic Kilcher

About

Latest Posts

[Video Response] What Cloudflare's code mode misses about MCP and tool calling

[Paper Analysis] On the Theoretical Limitations of Embedding-Based Retrieval (Warning: Rant)

On the Biology of a Large Language Model (Part 1)

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

You May Also Like

Titans: Learning to Memorize at Test Time (Paper Analysis)

Efficient Streaming Language Models with Attention Sinks (Paper Explained)

Something Weird Happens When E=−mc²

Mamba: Linear-Time Sequence Modeling with Selective State Spaces (Paper Explained)

Mustafa Suleyman on Superintelligence, Microsoft’s OpenAI Deal, Broligarchy | The Mishal Husain Show

Tech author Karen Hao: 'The Sam Altmans of the world are going to be fine" | Science Quickly Podcast

[GRPO Explained] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models

Retentive Network: A Successor to Transformer for Large Language Models (Paper Explained)

Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (Paper)

Visualizing transformers and attention | Talk for TNG Big Tech Day '24

LSTM: The Comeback Story? [Prof. Sepp Hochreiter]

Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention

MIT 6.S191 (2024): Recurrent Neural Networks, Transformers, and Attention

Flow Matching for Generative Modeling (Paper Explained)

What are Transformer Models and how do they work?

Scalable MatMul-free Language Modeling (Paper Explained)

FlashAttention - Tri Dao | Stanford MLSys #67

MCP vs API: Simplifying AI Agent Integration with External Data

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters (Paper Explained)

Transformers, the tech behind LLMs | Deep Learning Chapter 5

Master AI Research Today

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 &amp; PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super &amp; AMD GPU Compatible

soundcore by Anker P20i True Wireless Earbuds, 10mm Drivers with Big Bass, Bluetooth 5.3, 30H Long Playtime, IPX5 Water-Resistant, 2 Mics for AI Clear Calls, 22 Preset EQs, Customization via App

The Pandy - Emotional Healing on Demand, Realistic Panda Plush for Emotional Support, Mimics Natural Movements for Realistic Comfort, Pandy Ai Plush with Ultra-Soft Fur, The Comfort Hug You Have

BASESAILOR iPhone 16 15 Pro Max Charger Cord,USB to USB C Cable 6.6FT/2Pack,Type C Charging Power Cord for Apple iPad 10,Samsung Galaxy AI S25 S24 S23 S22 S20 Plus Ultra Flip,A52 A53 A54,Google Pixel

Spigen Case for Galaxy S25 Case, Tough Armor (Ai) MagFit [Wireless Charging] Compatible with MagSafe - Black

Freenove Ultimate Starter Kit for BBC micro bit (V2 Included), 316-Page Detailed Tutorial, 225 Items, 44 Projects, Blocks and Python Code

Mother of Learning Arc 2: Mother of Learning, Book 2

soundcore by Anker P20i True Wireless Earbuds, 10mm Drivers with Big Bass, Bluetooth 5.3, 30H Long Playtime, IPX5 Water-Resistant, 2 Mics for AI Clear Calls, 22 Preset EQs, Customization via App

Portable Carpet &amp; Upholstery Spot Cleaner,15Kpa Powerful Suction and Spot Remover, Deep Stain Cleaning for Carpet, Pets, Stairs, Car Seat, Couch, with Dual Size BrushHead and Cleaning Solutions

Ailltopd Portable Carpet &amp; Upholstery Spot Cleaner, Powerful Suction and Spot Remover, Deep Stain Cleaning for Carpet, Pets, Stairs, Car Seat, Couch, with Dual Size BrushHead and Cleaning Solutions

Loading...

Seasonic Focus V4 GX-1000 (ATX3) - 1000W - 80+ Gold - ATX 3.0 & PCIe 5.1 Ready -Full-Modular -ATX Form Factor -Premium Japanese Capacitor -10 Year Warranty -Nvidia RTX 30/40 Super & AMD GPU Compatible

Portable Carpet & Upholstery Spot Cleaner,15Kpa Powerful Suction and Spot Remover, Deep Stain Cleaning for Carpet, Pets, Stairs, Car Seat, Couch, with Dual Size BrushHead and Cleaning Solutions

Ailltopd Portable Carpet & Upholstery Spot Cleaner, Powerful Suction and Spot Remover, Deep Stain Cleaning for Carpet, Pets, Stairs, Car Seat, Couch, with Dual Size BrushHead and Cleaning Solutions