data · automation · Documented
Retail Data Warehouse ETL
Amazon Egypt scrape to quality checks, PostgreSQL / Snowflake, Airflow, and Power BI.
Overview
A warehouse project on Amazon Egypt retail data. The path is the point: scrape, quality checks, land in PostgreSQL and Snowflake, orchestrate with Airflow, read in Power BI.
Problem
A scrape folder is not a warehouse. Facts and dimensions have to be modelled before a refresh means anything.
Solution
A documented pipeline with quality checks before the warehouse, orchestration so the load is not manual, and a BI layer that reads the model rather than the scrape.
Architecture
- Amazon Egypt as the documented source.
- Quality checks before landing.
- PostgreSQL and Snowflake for the warehouse.
- Airflow for orchestration.
- Power BI for the reading layer.
- Public GitHub archive and Demo 2.
Technology stack
- Python
- PostgreSQL
- Snowflake
- Airflow
- Power BI
Features
- Staging, then a modelled warehouse.
- Orchestration rather than a notebook someone remembers to run.
Challenges
- Two warehouse targets — Postgres and Snowflake — only pay off if the model is the same idea in both.
Results
- Public source archive and demo PDF.
Lessons learned
- Quality checks belong before the fact table, not as a slide after the dashboard.