MIPRO and DSPy with Krista Opsahl-Ong! - Weaviate Podcast #103

Video Description

Hey everyone! Thank you so much for watching the 103rd Weaviate Podcast with Krista Opsahl-Ong from Stanford University. Krista is the lead author of MIPRO, short for Multi-prompt Instruction Proposal Optimizer, and one of the leading developers and scientists behind DSPy! This was such a fun discussion beginning with the motivation of Automated Prompt Engineering, Multi-Layer Language Programs (also commonly referred to as Compound AI Systems), and their intersection. We then dove into the details of how MIPRO achieves this and miscellaneous topics in AI from Structured Outputs to Agents, DSPy for Code Generation, and more! I really hope you enjoy the podcast! As always, more than happy to answer any questions or discuss any ideas about the content in the podcast! Michael Ryan (MIPRO Co-Author): https://x.com/michaelryan207 Thank you to the DSPy team and DSPy community members for your support! Special thanks to Omar Khattab, Chris Potts, Matei Zaharia, Heather Miller, Arnav Singhvi, Herumb Shandilya, Sri Vardhamanan, Cyrus Nouroozi, Amir Mehr, Kyle Caverly, Keshav Santhanam, Thomas Ahle, Michael Ryan, Josh Purtell, Karel D'Oosterlinck, Eric Zhang, Shangyin Tan, Manish Shetty, Peter Zhong, Jasper Xian, Saron Samuel, Alberto Mancarella, Faraz Khoubsirat, Saiful Haq, Ashutosh Sharma, Rick Battle, Dhar Rawal, Insop Song, Tom Dorr, Igor Kotenkov, Corey Zumar, Lisa Li, David Hall, Ashwin Paranjape, Chris Manning, Avi Sil, and Chuyi Zhang. Helpful Links: MIPRO - https://arxiv.org/abs/2406.11695 MIPRO Animations from Michael Ryan - https://x.com/michaelryan207/status/1804189184988713065 DSPy for Code Generation - https://www.youtube.com/watch?v=yhYeDGxnuGY Compound AI Systems - https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/ DSPy - https://github.com/stanfordnlp/dspy/tree/main DSPy research paper - https://arxiv.org/abs/2310.03714 Large Language Models as Optimizers (OPRO) - https://arxiv.org/abs/2309.03409 The ImageNet Moment of DSPy from Professor Bo Wang’s Lab - https://x.com/lateinteraction/status/1783990747257360779 Unreasonable Effectiveness of Eccentric Prompting - https://arxiv.org/abs/2402.10949 BetterTogether - https://arxiv.org/abs/2311.01326 Thoughts from Weaviate Erika on the Automated Design of Agents - https://x.com/ecardenas300/status/1828085383734718723 Weaviate Recipes DSPy implementation of the Automated Design of Agents - https://github.com/weaviate/recipes/blob/main/integrations/llm-frameworks/dspy/Meta-Agent-Generation.ipynb Shreya Shankar - Who Validates the Validators? - https://arxiv.org/pdf/2404.12272 Eugene Yan - Evaluating the Effectiveness of LLM-Evaluators (aka LLM-as-Judge) - https://eugeneyan.com/writing/llm-evaluators/ Joao Moura (Crew AI) - Using agents to build an agent company - https://www.youtube.com/watch?v=Dc99-zTMyMg Chapters 0:00 Welcome Krista! 0:36 What lead you to work on DSPy? 2:32 What is Automated Prompt Engineering? 4:46 Multi-Stage Language Programs 7:10 Optimizing Multi-Stage Language Programs 9:30 The Proposal Problem 12:20 More examples of Multi-Stage Language Programs 14:25 Thoughts on Structured Outputs 18:35 Thoughts on Agents 20:20 MIPRO’s Credit Assignment 27:25 Many-Shot In-Context Learning 30:04 Automated Design of Agents 31:44 Making DSPy Easier to Use 34:24 Integrating Cheaper LMs 36:28 Deeper into Learning to Propose 44:12 DSPy for Code Generation 46:18 How much of your code is written by AI? 48:28 Multi-Agents and DSPy 51:18 AI User Interfaces 53:50 Generative Feedback Loops 56:30 What directions for the future excite you the most?