Account

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

Umar Jamil • August 24, 2023

Video Thumbnail

Umar Jamil

About

I'm a Machine Learning Engineer from Milan, Italy, teaching complex deep learning and machine learning concepts to my cat, 奥利奥. 我也会中文.

Latest Posts

Video Thumbnail

Titans: Learning to Memorize at Test Time

Umar Jamil

Video Thumbnail

Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Umar Jamil

Video Thumbnail

Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer

Umar Jamil

Video Thumbnail

Reinforcement Learning from Human Feedback explained with math derivations and the PyTorch code.

Umar Jamil

You May Also Like

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

Umar Jamil2 years ago

BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token

BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token

Umar Jamil2 years ago

How Attention Got So Efficient [GQA/MLA/DSA]

How Attention Got So Efficient [GQA/MLA/DSA]

Jia-Bin Huang2 weeks ago

Llama 4 Explained: Architecture, Long Context, and Native Multimodality

Llama 4 Explained: Architecture, Long Context, and Native Multimodality

Julia Turc8 months ago

AI Optimization Lecture 01 - Prefill vs Decode - Mastering LLM Techniques from NVIDIA

AI Optimization Lecture 01 - Prefill vs Decode - Mastering LLM Techniques from NVIDIA

Faradawn Yang6 months ago

Attention is all you need (Transformer) - Model explanation (including math), Inference and Training

Attention is all you need (Transformer) - Model explanation (including math), Inference and Training

Umar Jamil2 years ago

How Rotary Position Embedding Supercharges Modern LLMs [RoPE]

How Rotary Position Embedding Supercharges Modern LLMs [RoPE]

Jia-Bin Huang1 year ago

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware Training

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware Training

Umar Jamil2 years ago

How DeepSeek Rewrote the Transformer [MLA]

How DeepSeek Rewrote the Transformer [MLA]

Welch Labs9 months ago

Rotary Positional Embeddings Explained | Transformer

Rotary Positional Embeddings Explained | Transformer

Outlier3 months ago

How LLMs survive in low precision | Quantization Fundamentals

How LLMs survive in low precision | Quantization Fundamentals

Julia Turc6 months ago

How diffusion models work - explanation and code!

How diffusion models work - explanation and code!

Umar Jamil2 years ago

Rotary Positional Embeddings: Combining Absolute and Relative

Rotary Positional Embeddings: Combining Absolute and Relative

Efficient NLP2 years ago

Tensor Processing Units (TPUs) Explained

Tensor Processing Units (TPUs) Explained

The Coding Gopher2 months ago

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

Umar Jamil2 years ago

What Are Word Embeddings?

What Are Word Embeddings?

Under The Hood9 months ago

Understanding LLM Inference | NVIDIA Experts Deconstruct How AI Works

Understanding LLM Inference | NVIDIA Experts Deconstruct How AI Works

DataCamp1 year ago

MAMBA from Scratch: Neural Nets Better and Faster than Transformers

MAMBA from Scratch: Neural Nets Better and Faster than Transformers

Algorithmic Simplicity1 year ago

Visualizing transformers and attention | Talk for TNG Big Tech Day '24

Visualizing transformers and attention | Talk for TNG Big Tech Day '24

Grant Sanderson1 year ago

Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)

Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)

Umar Jamil2 years ago

Home AiChat Shop Saved Account