data · Documented
Pharma Sales ETL
Extract, transform, and load pharmaceutical sales data into Azure for reporting.
Overview
A data-engineering project that moves pharmaceutical sales data through an Azure pipeline so it can be reported on. The architecture note and a worked demo are both public.
Problem
Sales extracts do not arrive analysis-ready. Without a repeatable load, every report starts from the file again.
Solution
An Azure pipeline: extract, transform, load into Azure SQL, with Python (SQLAlchemy, pyodbc) at the edges and Data Factory for orchestration.
Architecture
- Source extracts of pharmaceutical sales.
- Azure Data Factory for pipeline orchestration.
- Azure SQL as the landing database.
- Python services for transformation and database access.
- A written brief plus Demo 4.
Technology stack
- Python
- Azure Data Factory
- Azure SQL
- SQLAlchemy
- pyodbc
Features
- Repeatable load rather than a one-off spreadsheet.
- A short architecture brief meant to be read, not only a notebook.
Challenges
- Drawing a boundary between the pipeline and the report so each can change without the other.
Results
- Public brief, demo PDF, and a zip of the pack.
- A public GitHub repository is not listed for this project.
Lessons learned
- Write the architecture note while the pipeline still makes sense.