Sign In

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

Umar Jamil • July 25, 2023

Video Thumbnail

View Channel

About

No channel description available.

Latest Posts

Titans: Learning to Memorize at Test Time

Titans: Learning to Memorize at Test Time

Umar Jamil1 year ago

13559

Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Paper: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

Umar Jamil1 year ago

58775

Flash Attention derived and coded from first principles with Triton (Python)

Flash Attention derived and coded from first principles with Triton (Python)

Umar Jamil1 year ago

70025

Coding a Multimodal (Vision) Language Model from scratch in PyTorch with full explanation

Coding a Multimodal (Vision) Language Model from scratch in PyTorch with full explanation

Umar Jamil1 year ago

113998

Video Description

A full visual explanation of LoRA, with PyTorch code form scratch! Full code and slides are available on my GitHub: https://github.com/hkproj/pytorch-lora Chapters 00:00 - Introduction 00:47 - How neural networks work 01:48 - How fine tuning works 03:50 - LoRA 08:58 - Math intuition 10:25 - Math explanation 14:05 - PyTorch implementation from scratch

You May Also Like

ML Interpretability: feature visualization, adversarial example, interp. for language models

ML Interpretability: feature visualization, adversarial example, interp. for language models

Umar Jamil1 year ago

11544

Kolmogorov-Arnold Networks: MLP vs KAN, Math, B-Splines, Universal Approximation Theorem

Kolmogorov-Arnold Networks: MLP vs KAN, Math, B-Splines, Universal Approximation Theorem

Umar Jamil1 year ago

38331

Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math

Direct Preference Optimization (DPO) explained: Bradley-Terry model, log probabilities, math

Umar Jamil1 year ago

33076

Reinforcement Learning from Human Feedback explained with math derivations and the PyTorch code.

Reinforcement Learning from Human Feedback explained with math derivations and the PyTorch code.

Umar Jamil2 years ago

60282

Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion, Recurrent, Convolution, Math

Mamba and S4 Explained: Architecture, Parallel Scan, Kernel Fusion, Recurrent, Convolution, Math

Umar Jamil2 years ago

58115

Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer

Mistral / Mixtral Explained: Sliding Window Attention, Sparse Mixture of Experts, Rolling Buffer

Umar Jamil2 years ago

39155

Distributed Training with PyTorch: complete tutorial with cloud infrastructure and code

Distributed Training with PyTorch: complete tutorial with cloud infrastructure and code

Umar Jamil2 years ago

30816

How diffusion models work - explanation and code!

How diffusion models work - explanation and code!

Umar Jamil2 years ago

30368

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware Training

Quantization explained with PyTorch - Post-Training Quantization, Quantization-Aware Training

Umar Jamil2 years ago

47464

Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)

Retrieval Augmented Generation (RAG) Explained: Embedding, Sentence BERT, Vector Database (HNSW)

Umar Jamil2 years ago

80853

BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token

BERT explained: Training, Inference, BERT vs GPT/LLamA, Fine tuning, [CLS] token

Umar Jamil2 years ago

72005

Coding Stable Diffusion from scratch in PyTorch

Coding Stable Diffusion from scratch in PyTorch

Umar Jamil2 years ago

209397

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

Coding LLaMA 2 from scratch in PyTorch - KV Cache, Grouped Query Attention, Rotary PE, RMSNorm

Umar Jamil2 years ago

61776

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

LLaMA explained: KV-Cache, Rotary Positional Embedding, RMS Norm, Grouped Query Attention, SwiGLU

Umar Jamil2 years ago

106892

Segment Anything - Model explanation with code

Segment Anything - Model explanation with code

Umar Jamil2 years ago

36363

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

LoRA: Low-Rank Adaptation of Large Language Models - Explained visually + PyTorch code from scratch

Umar Jamil2 years ago

45831

AiChat Products Saved Account