Data Engineer Roadmap

Build pipelines and warehouses that make data reliable and usable for analysts and data scientists.

7 milestones · ~235 hours total

1. 1

   SQL and data modelling

   ~40h

   Advanced SQL, normalisation, star schemas and designing tables for analytics.

   Practice this →
2. 2

   Python and tooling

   ~30h

   Python for ETL, working with files and APIs, and packaging scripts that run unattended.

   Practice this →
3. 3

   Batch pipelines

   ~35h

   Orchestration with Airflow or similar, scheduling, retries, idempotency and data-quality checks.

   Practice this →
4. 4

   Warehouses and lakes

   ~35h

   Columnar storage, partitioning, a cloud warehouse and transformation tools such as dbt.

   Practice this →
5. 5

   Streaming basics

   ~30h

   Message queues, Kafka concepts, event time and exactly-once versus at-least-once delivery.

   Practice this →
6. 6

   Cloud and cost

   ~25h

   Run pipelines on a cloud provider and keep storage and compute costs under control.

   Practice this →
7. 7

   Portfolio

   ~40h

   Build an end-to-end pipeline from a public source to a dashboard, with tests and documentation.

   Practice this →