End-to-End Healthcare Data Engineer Project | Azure Databricks + Data Factory

Video Description

🔥 Build an End-to-End Real-Time Healthcare Data Engineering Project on Azure! 🔥 In this video, I’ll Walk you through a **real-time patient flow data pipeline** using Azure Event Hubs, Azure Databricks (PySpark), Delta Lake, and Synapse Analytics. We’ll implement the **Medallion Architecture (Bronze, Silver, Gold)**, handle **schema evolution**, design a **Star Schema with Fact & Dimension tables**, and finally connect everything to Power BI for analytics. ✅ What You’ll Learn: • Real-time data streaming with **Azure Event Hubs + Kafka** • Ingestion & transformations with **Databricks + PySpark** • Implementing the **Medallion Architecture (Bronze, Silver, Gold)** • **Schema evolution** in Delta Lake • **SCD Type 2** handling in Gold layer • Building a **Star Schema** (Fact & Dimension tables) • Loading data into **Azure Synapse SQL Pool** • Connecting **Power BI** for reporting (coming soon) 📂 Project Repository: GitHub: https://github.com/Jay61616/hospital-patient-flow-analytics 🛠 Tools Used: • Azure Event Hubs • Azure Databricks (PySpark) • Azure Data Factory (for orchestration) • Azure Synapse Analytics • Power BI (for visualization) 📌Tools Links: Azure Free Trail: https://azure.microsoft.com/en-in/pricing/purchase-options/azure-account?msockid=1425ae24dd3c66113353bb8edce967df Git Bash: https://git-scm.com/downloads Power BI: https://www.microsoft.com/en-in/download/details.aspx?id=58494&msockid=1425ae24dd3c66113353bb8edce967df ⏱ Timestamps: 0:00 Intro 01:05 Client Requirements 07:19 Project Architecture 09:18 Producer Code (Raw Data) 17:12 Event Hub Initialize 22:55 Data Generation & stream 39:56 Databricks Initialize 41:18 Storage Account Initialize 48:11 Bronze Transformation 01:00:49 Key Vault Initialize 01:14:55 Silver Transformation 01:43:10 Gold Transformation 02:12:58 Data factory Initialize 02:19:30 Orchestration 02:36:26 Synapse Initialize 02:40:09 DDL for Data Warehouse 03:02:58 Power BI Connection 03:05:34 Git Push 📣 Stay Connected: LinkedIn: https://www.linkedin.com/in/jayachandrakadiveti/ GitHub: https://github.com/Jay61616 Website: https://jay.unaux.com/ Credits: Intro Music: https://socialbunnymarketing.com Icons: https://www.flaticon.com/ 👉 Hashtags: #azure #dataengineering #cloud #dataengineeringproject #healthcareanalytics #azureeventhub #databricks #pyspark #datafactory #synapse #realtimedata #portfolioproject #datafactory #azuredatabricks #databricks