Find the BEST RAG Strategy with Domain Specific Evals

Video Description

Creating custom RAG chunking and embedding strategies with domain specific evaluation experiments Resources: Notebook Repo: https://github.com/ALucek/custom-rag-evals/tree/main Prior Chunking Breakdown: https://youtu.be/Pk2BeaGbcTE?si=pWK6hkXPnJCyb4Ux ChromaDB Repo: https://github.com/brandonstarxel/chunking_evaluation/tree/main ChromaDB Colab Notebook: https://colab.research.google.com/drive/1J5ALtDf0_RrswRz2fktjFVeFxe2jbXuJ?usp=sharing#scrollTo=yQiDRQKIf_cl ChromaDB Research Report: https://research.trychroma.com/evaluating-chunking Chapters: 00:00 - Why Measure Chunking & Embedding 03:44 - Creating a Custom Chunking Strategy 06:12 - Breaking Down Eval Metrics 08:00 - Metrics: Eval Dataset 09:17 - Metrics: Recall, Precision, IoU 13:27 - General Evals: Describing Test Set 14:32 - General Eval: Process & Running Test 17:58 - General Eval: Embedding Test 19:00 - Running Multiple Evals Across Strategies 20:36 - Multiple Evals: Interpreting Results 23:18 - Domain Specific Dataset Generation & Filtering 27:06 - Running Domain Specific Evals 29:32 - Final Thoughts #ai #machinelearning #datascience