Data Engineering Roadmap: Pipelines, Spark and dbt

Learn to move, clean and model data reliably: SQL, Python, batch and streaming, warehouses and dbt.

10 milestones · ~188 hours total

1. 1

   Advanced SQL

   ~30h

   Joins, window functions, CTEs, aggregations and query plans. This is the core skill of the role.

   Practice this →
2. 2

   Python for data work

   ~20h

   File formats like CSV, JSON and Parquet, pandas, generators and writing testable ETL scripts.

   Practice this →
3. 3

   Data modelling

   ~14h

   Normalisation, star schemas, facts and dimensions, slowly changing dimensions and OLTP vs OLAP.

   Practice this →
4. 4

   Warehouses and lakes

   ~14h

   How columnar storage works; try BigQuery, Snowflake trials or DuckDB; object storage and table formats.

   Practice this →
5. 5

   Batch processing with Apache Spark

   ~25h

   DataFrames, partitions, shuffles, joins and reading Parquet with PySpark. Learn to read the Spark UI.

   Practice this →
6. 6

   Workflow orchestration with Airflow

   ~15h

   DAGs, scheduling, retries, idempotent tasks and backfills.

   Practice this →
7. 7

   Transformations with dbt

   ~15h

   Models, tests, sources, documentation and incremental models on top of a warehouse.

   Practice this →
8. 8

   Streaming basics with Kafka

   ~15h

   Topics, partitions, consumer groups, delivery guarantees and a simple stream processing example.

   Practice this →
9. 9

   Data quality and observability

   ~10h

   Schema checks, freshness and volume tests with Great Expectations or dbt tests, and lineage.

   Practice this →
10. 10

    End-to-end portfolio pipeline

    ~30h

    Ingest a public dataset, orchestrate it, model it in a warehouse, test it and publish a dashboard.

    Practice this →