Signin

Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer

Umar Jamil • April 23, 2024
Video Thumbnail

You May Also Like

Video Thumbnail

Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion, Recurrent, Convolution, Math

Umar Jamil

Video Thumbnail

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

Umar Jamil

Video Thumbnail

Understanding Mixture of Experts

Trelis Research

Video Thumbnail

Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)

Umar Jamil

Video Thumbnail

Simply explained! KAN: Kolmogorov–Arnold Networks is interpretable! Mathematics and Physics

Bin Yiu

Video Thumbnail

Coding a Transformer from scratch on PyTorch, with full explanation, training and inference.

Umar Jamil

Video Thumbnail

The KV Cache: Memory Usage in Transformers

Efficient NLP

Video Thumbnail

Mistral 7B’s secrets: a talk by Guillaume Lample

Photoroom

Video Thumbnail

Rotary Positional Embeddings: Combining Absolute and Relative

Efficient NLP

Video Thumbnail

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

Umar Jamil

Video Thumbnail

FlashAttention - Tri Dao | Stanford MLSys #67

Stanford MLSys Seminars

Video Thumbnail

Mistral 8x7B Part 1- So What is a Mixture of Experts Model?

Sam Witteveen

Video Thumbnail

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware Training

Umar Jamil

Video Thumbnail

Geoffrey Hinton | On working with Ilya, choosing problems, and the power of intuition

Sana

Video Thumbnail

Fine-tune Mixtral 8x7B (MoE) on Custom Data - Step by Step Guide

Prompt Engineering

Video Thumbnail

BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token

Umar Jamil

Video Thumbnail

Attention is all you need (Transformer) - Model explanation (including math), Inference and Training

Umar Jamil

Video Thumbnail

MAMBA from Scratch: Neural Nets Better and Faster than Transformers

Algorithmic Simplicity

Video Thumbnail

Reinforcement Learning from Human Feedback explained with math derivations and the PyTorch code.

Umar Jamil

Video Thumbnail

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

Umar Jamil

Umar Jamil

View Channel

About

No channel description available.

Latest Posts

Video Thumbnail

Attention is all you need (Transformer) - Model explanation (including math), Inference and Training

Umar Jamil

Video Thumbnail

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

Umar Jamil

Video Thumbnail

Distributed Training with PyTorch: complete tutorial with cloud infrastructure and code

Umar Jamil

Video Thumbnail

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

Umar Jamil

AI Assistant

Loading...