Python Tutorial: Build an AI-assisted Reddit Scraping Pipeline

Video Description

🚀 Sign up for Bright Data right now: https://brdta.com/cfe Automatically find and track topics you care about across Reddit posts. From camping to the latest in AI news, this course will show you how to build a powerful and resilient system in Python. The goal is of this course is to help you develop the skills you need to build a resilient data extraction platform using only a handful of tools and the latest in LLMs from Google. In addition to the new skills you'll learn, you'll also have rich data to help you better learn from what real people are experiencing all around the world. Topics: ✅ Easily download the latest Reddit conversations around topics you care about ✅ Ai-Powered Google search to extract relevant Reddit Communities (aka SERP) ✅ Build & ingest data through public webhooks (notifications that work software-to-software or app-to-app) ✅ Rapid prototype data scraping/extracting with Python & Jupyter Notebooks ✅ Use Gemini to run your Python functions based on plain english (aka Tool Calling) ✅ Store extracted data through the Django ORM and PostgreSQL ✅ Strict & structured data outputs for LLMs with Pydantic ✅ Fault-tolerant data downloads using background tasks & webhooks ✅ Configure serverless and serverfull worker managers (django-qstash & celery) ✅ and much more Resourses - My github: https://cfe.sh/github - Project Code Repo https://github.com/codingforentrepreneurs/Reddit-Content-Research-Agent - My Bright Data link - https://brdta.com/cfe (means more sign ups, more free courses) - Django QStash repo & docs https://djangoqstash.com - Django with Celery & Redit Blog Post: https://www.codingforentrepreneurs.com/blog/celery-redis-django Stack: ‣ Python ‣ Jupyter (rapid prototyping) ‣ Django (web app & automation coordinator) ‣ Postgres (database) ‣ Redis (caching & queues) ‣ Celery (background tasks) ‣ Django QStash (serverless background tasks) ‣ Bright Data Search Engine AI (SERP) ‣ Bright Data Crawl API (extract Reddit posts) ‣ LangChain (integration to Google Gemini LLM) ‣ LangGraph (easily unlock Tool Calling) ‣ Cloudflare Tunnels (public domain to your project to accept webhooks) Chapters 00:00:00 Welcome 00:03:46 Demo 00:12:03 Using Search Engine Results 00:14:16 Setup your Python Project 00:20:36 Load API Keys with Dotenv Files 00:24:26 Intro to LangChain 00:26:19 Bright Data Serp API with Python & LangChain 00:38:01 Strip Notebook Outputs for Security with pre-commit 00:42:56 Setup Google Gemini Models with LangChain 00:52:43 LLM with Structured Output 00:59:58 LLM Tool Calling The Hard Way 01:08:19 Tool Calling with LangGraph 01:23:41 Search & Format Reddit Communities via LLM and Bright Data 01:29:38 Scrape Reddit with the Bright Data Crawl API 01:41:58 Get Crawl API Snapshot Progress 01:47:00 Download Data from the Crawl API 01:54:53 Automating Data Pulls for Users 01:58:39 Install & Start the Django Project 02:02:31 Combine Django with Jupyter 02:05:23 Implement Postgres Database with Django 02:15:19 Setup Redis for Django & Caching 02:22:36 Getting Started with Celery & Django 02:33:51 Webhooks & Cloudflare Tunnels 02:36:47 Setup Cloudflare Tunnel with a Custom Domain 02:45:24 Django Qstash for Webhook-based Background Tasks 02:52:55 Bright Data to Django Model Part 1 03:02:16 Bright Data to Django Model Part 2 03:09:38 Store Bright Data Snapshots 03:17:38 Helper Functions for Scraping Events Part 1 03:24:56 Helper Functions for Scraping Events Part 2 03:32:52 Saving Snapshot Scraping Results 03:38:29 Configure Scraping as Background Tasks 03:49:48 Run Background Scraping Tasks 03:53:48 Poll Scrape Status as Background Task 04:02:04 Tracking Scrape Event Finished At Time 04:08:36 A Webhook Handler View in Django 04:16:11 Tracking Scraping Snapshots through Webhooks with Django 04:25:48 Improved Auth Key for Webhooks 04:30:42 Webhook Handler for Reddit Posts 04:38:32 Adjust Data to Scrape 04:50:47 Background Sync Snapshot Reddit Results 05:04:39 Storing Reddit Communities in Django 05:16:53 Reddit AI Agent into Django Project 05:26:04 Topic Extraction Agent 05:32:50 Fuzzy Query to Scraping 05:40:45 Auto Scrape Reddit Communities on Save 05:52:37 Scraping Workflow as a Service Function 06:00:17 Store Queries & Topics 06:09:24 Topics to Reddit Communities 06:16:41 Full Query Automation 06:23:09 Reddit Community Trackablity 06:28:19 Scheduled Background Task to Trigger Reddit Scraping 06:33:27 Django Management Command to Trigger Scraping 06:36:17 Final Query Commands 06:38:22 Thank you and next steps