400x Faster Embeddings! - Static & Distilled Embedding Models

Video Description

To try everything Brilliant has to offer—free—for a full 30 days, visit https://brilliant.org/AdamLucek/ You’ll also get 20% off an annual premium subscription! Resources: Code from Video: https://github.com/ALucek/static-embeddings My Model Trained: https://huggingface.co/AdamLucek/static-retrieval-mrl-MBERT-base-en-v1 Static Embeddings Blog: https://huggingface.co/blog/static-embeddings Model2Vec Blog Post: https://huggingface.co/blog/Pringled/model2vec Model2Vec Repo: https://github.com/MinishLab/model2vec/tree/main Andrej Karpathy Let’s Build the GPT Tokenizer - https://www.youtube.com/watch?v=zduSFxRajkE Chapters: 00:00 - Background Embeddings 03:01 - Brilliant! 04:22 - What are Static Embeddings 06:47 - W2V Training Example 10:01 - Tokenization & Vocabulary 13:30 - Pooling 14:38 - Static Embeddings In Action & Interpretation 19:35 - How Transformer Models Differ 22:18 - Modern Static Embedding Models 25:34 - Testing it Out 28:23 - Embedding Model Distillation 30:04 - Principle Component Analysis 31:30 - Token Level Weighting 33:09 - M2V in Action 35:37 - Discussion #ai #datascience #programming This video is sponsored by Brilliant