# dataskew.io

> Free data engineering learning platform with structured roadmaps, hands-on projects, blog tutorials, and interview preparation. Built by Adriano Sanges, data engineer.

## About
dataskew.io is a free, comprehensive platform for learning data engineering. It provides structured learning paths from beginner to advanced, covering the modern data stack including Python, SQL, Apache Kafka, Apache Spark, Apache Airflow, dbt, Docker, Terraform, and cloud platforms (AWS, GCP, Azure). The platform is designed for aspiring data engineers, career switchers, and self-taught developers who want to build real-world skills.

## Roadmaps
- [Startup Stack Roadmap](https://dataskew.io/roadmaps/startup-stack): Build a scalable, cost-effective data stack using modern open-source tools and serverless architecture.
- [Modern Data Stack Roadmap](https://dataskew.io/roadmaps/modern-data-stack): Master the core tools used in modern data teams — from containerization to dbt, BigQuery, and Kafka. Build real projects and get job-ready.
- [Data Engineer Roadmap 2026: From Zero to Job-Ready (Step-by-Step)](https://dataskew.io/roadmaps/beginner-data-engineer): A free, step-by-step data engineering roadmap for 2026. Learn SQL, Python, ETL, cloud fundamentals, dbt, Airflow and Docker through 51 hands-on tasks and build the projects you need to land your first data engineer job.
- [Data Analyst Roadmap 2026: From Zero to Job-Ready](https://dataskew.io/roadmaps/data-analyst): A free, step-by-step data analyst roadmap for 2026. SQL, Excel, Power BI, Tableau, Python, statistics, dbt, GA4, and the portfolio + interview prep you need to land your first or second analyst role.
- [AI Engineer Roadmap 2026: From LLM APIs to Production (Step-by-Step)](https://dataskew.io/roadmaps/ai-engineering): A free, step-by-step AI engineer roadmap for 2026: RAG, agents, evals, finetuning, and production deployment. Grounded in Chip Huyen's AI Engineering and Stanford CS336, with hands-on projects.

## Fundamentals
- [️ Data Modeling Fundamentals](https://dataskew.io/fundamentals/data-modeling): Learn the principles of designing effective data models for analytics and business intelligence.
- [Data Quality & Testing Fundamentals](https://dataskew.io/fundamentals/data-quality): Learn data quality principles, testing strategies, and observability practices essential for building reliable data pipelines.
- [️ Data Warehousing Concepts](https://dataskew.io/fundamentals/data-warehousing): Understand data warehousing principles including OLAP, dimensional modeling, and modern cloud warehouse platforms.
- [Docker for Data Engineers](https://dataskew.io/fundamentals/docker): Learn Docker fundamentals to containerize data pipelines, spin up local development stacks, and ensure reproducible environments.
- [️ NoSQL Databases for Data Engineers](https://dataskew.io/fundamentals/nosql): Explore NoSQL database types including document stores, key-value stores, column-family databases, and graph databases.
- [Python for Data Engineering](https://dataskew.io/fundamentals/python): Learn Python programming fundamentals with a focus on data engineering applications.
- [SQL Fundamentals](https://dataskew.io/fundamentals/sql): Master the essential SQL skills needed for data engineering, from basic queries to advanced techniques.

## Projects (Hands-on)
- [Production RAG System with Retrieval Evaluation](https://dataskew.io/projects/ai-engineering-rag-system): Build a retrieval-augmented generation system over a real document set: chunking, embeddings, hybrid search with a reranker, grounded answers with citations, and a retrieval + faithfulness evaluation that proves it works.
- [LLM Agent with Tools and Failure-Mode Evaluation](https://dataskew.io/projects/ai-engineering-llm-agent): Build an agent that plans, calls real tools (function calling), manages memory, and recovers from failures, then evaluate it on its trajectory and failure modes, not just happy-path demos.
- [LLM Evaluation Pipeline with Golden Dataset and LLM-as-a-Judge](https://dataskew.io/projects/ai-engineering-llm-evaluation): Build a reusable evaluation pipeline for LLM applications: a golden dataset, automated scoring with LLM-as-a-judge, and regression testing you can point at any prompt or model change to catch quality drops before users do.
- [Local Data Engineering Environment with dlt, DuckDB & Jupyter](https://dataskew.io/projects/local-data-development): Set up a local development environment for data processing and analytics using Jupyter notebooks, dlt, and DuckDB. All tools are open-source and run locally.
- [Scheduled GitHub ETL with Polars, DLT & DuckDB](https://dataskew.io/projects/data-transformation-polars): Build a scheduled ETL pipeline that extracts GitHub repository data, transforms it with Polars, and stores results in DuckDB
- [End-to-End Analytics Platform with DuckDB + Metabase](https://dataskew.io/projects/analytics-dashboard): Build a modern, low-cost analytics stack using DuckDB, Metabase, and GitHub Actions for automated data updates and business-ready dashboards.
- [Infrastructure-as-Code Setup on GCP](https://dataskew.io/projects/infrastructure-as-code): Provision a GCP environment using Terraform with BigQuery & Cloud Storage, staying within free tier limits
- [ETL Pipeline Orchestration with Apache Airflow](https://dataskew.io/projects/orchestration-airflow): Design and implement an orchestrated ETL pipeline using Apache Airflow to extract, transform, and load weather data from a public API into a data warehouse.
- [Analytics Engineering Workflow with dbt + Metabase](https://dataskew.io/projects/analytics-engineering-dbt): Build a production-grade analytics workflow: model, test, and document data with dbt, then visualize insights in Metabase.
- [GitHub Events Analytics with PySpark](https://dataskew.io/projects/batch-processing-spark): Build a production-style batch data pipeline using Apache Spark to process GitHub event logs
- [Real-Time Data Streaming with Apache Kafka](https://dataskew.io/projects/streaming-kafka): Build a real-time data pipeline using Kafka (Confluent Cloud), JSON, Python, and Polars. Simulate NYC Taxi data, process in real time, and visualize with Metabase.
- [CI/CD for Data Pipelines](https://dataskew.io/projects/cicd-data-pipelines): Build a complete CI/CD pipeline for a data engineering project using GitHub Actions, dbt, Airflow DAG testing, and Terraform infrastructure deployment.
- [Tourism Recovery Dashboard (SQL + Power BI)](https://dataskew.io/projects/data-analyst-tourism-dashboard): Answer a real business question end to end: load Eurostat regional tourism arrivals into DuckDB, model the metrics in SQL, and ship a one-page Power BI dashboard explaining where tourism recovered fastest between 2022 and 2025.
- [Airbnb Listings EDA (Python + pandas)](https://dataskew.io/projects/data-analyst-airbnb-eda): Clean a real, messy Inside Airbnb listings dataset, run an exploratory analysis in a Jupyter notebook, and ship the result on GitHub with a README and one publication-quality chart.

## Blog
- [Apache Airflow for Data Engineers: DAGs, Operators, and Production Patterns (2026)](https://dataskew.io/blog/apache-airflow): How Apache Airflow works for data engineers: DAGs, operators, the scheduler, and the production patterns (idempotency, backfills, sensors) that keep pipelines reliable.
- [Apache Kafka for Data Engineers: Architecture, Use Cases & Getting Started](https://dataskew.io/blog/apache-kafka-for-data-engineers): Learn Apache Kafka architecture, key concepts, and practical use cases. Includes Python examples, Docker setup, and comparisons with Pub/Sub and Kinesis.
- [Apache Spark for Data Engineers: How It Works and When to Use It (2026)](https://dataskew.io/blog/apache-spark): Apache Spark for data engineers: the execution model (driver, executors, shuffles), DataFrames vs RDDs, production best practices, and when Spark beats SQL or a warehouse.
- [Change Data Capture (CDC): How It Works and When to Use It (2026)](https://dataskew.io/blog/change-data-capture-cdc): Change Data Capture (CDC) explained for data engineers: log-based vs query-based, Debezium and friends, the pipeline patterns that matter, and when CDC beats batch reloads.
- [Data Contracts for Data Engineers: Stop Breaking Downstream Pipelines](https://dataskew.io/blog/data-contracts-for-data-engineers): Learn how data contracts prevent breaking changes, reduce pipeline incidents, and improve trust across producers and consumers with practical implementation patterns.
- [Data Engineering System Design Interview: Framework + 3 Examples](https://dataskew.io/blog/data-engineer-interview-system-design): Pass the data engineering system design interview: a 5-step framework, 3 worked pipeline examples (batch, streaming, CDC) and the patterns interviewers expect.
- [Data Pipeline Design Patterns: Idempotency, DLQ, CDC and 5 More (2026)](https://dataskew.io/blog/data-pipeline-design-patterns): 8 production-grade pipeline patterns explained with Python and SQL: idempotency, backfilling, dead letter queues, CDC, schema evolution. The patterns that keep ETL running at 3 AM without paging you.
- [Data Warehouse vs Data Lake vs Lakehouse [2026 Comparison]](https://dataskew.io/blog/data-warehouse-vs-data-lake): Side-by-side comparison of data warehouse, data lake and lakehouse architectures: OLTP vs OLAP, medallion layers, Snowflake vs Databricks, and how to choose.
- [Keeping Databricks Declarative Automation Bundles (formerly Databricks Asset Bundles) Modular with Jinja2](https://dataskew.io/blog/databricks-dabs-jinja-modularity): Learn how to use Jinja2 templating to keep Databricks Declarative Automation Bundles (formerly Databricks Asset Bundles / DABs) DRY, composable, and environment-aware with reusable fragments and conditional logic.
- [Databricks PySpark Best Practices: Modular Pipeline Patterns](https://dataskew.io/blog/databricks-pyspark-best-practices): Production-grade Databricks projects: modular PySpark transformations, thin notebook entrypoints, unit testing, and deployment with Databricks Asset Bundles.
- [dbt Tutorial: Models, Tests & Incremental Builds Explained (2026)](https://dataskew.io/blog/dbt-analytics-engineering): Learn dbt the way analytics engineers actually use it: materializations compared, testing strategy, macros, and the project structure that scales past 100 models — with runnable examples.
- [Delta Lake vs Apache Iceberg: Which Lakehouse Table Format in 2026?](https://dataskew.io/blog/delta-lake-vs-apache-iceberg): Delta Lake vs Apache Iceberg compared: ACID, time travel, schema evolution, engine support and vendor lock-in. A practical way to pick an open table format for your lakehouse.
- [Dimensional Modeling: A Practical Guide for Data Engineers (2026)](https://dataskew.io/blog/dimensional-modeling): Dimensional modeling explained: fact and dimension tables, grain, conformed dimensions, and slowly changing dimensions (SCD). The Kimball method, applied with dbt.
- [Docker for Data Engineers: Containerize Your Data Pipelines](https://dataskew.io/blog/docker-for-data-engineers): Learn Docker essentials for data engineering — Dockerfiles, multi-stage builds, Docker Compose for local data stacks, and production best practices.
- [ETL vs ELT: Which Wins in 2026 and When You Actually Need Both](https://dataskew.io/blog/etl-vs-elt): ETL still wins for compliance and on-prem; ELT dominates cloud warehouses. The hybrid setup most teams actually run (Fivetran + dbt + Snowflake), the exceptions, and how to decide for your stack.
- [How to Become a Data Engineer in 2026: Complete Career Guide](https://dataskew.io/blog/how-to-become-data-engineer): A practical roadmap to becoming a data engineer in 2026 covering skills, tools, projects, interview prep, certifications, and salary expectations.
- [Metabase + DuckDB: Local-First Analytics Setup Guide [2026]](https://dataskew.io/blog/metabase-duckdb-local-analytics): Connect Metabase to DuckDB to run a fast local-first BI stack on Parquet, CSV and SQLite files. Setup steps, Docker config, gotchas and when to scale beyond it.
- [Production LLM Engineering for Data Engineers: Evals, RAG, Agents, and LLMOps](https://dataskew.io/blog/production-llm-engineering-for-data-engineers): A practical map of what it takes to run LLM systems in production: evaluation, retrieval, agents, and the ops discipline that keeps them reliable and affordable. Written for data engineers moving into AI.
- [Reverse ETL: Syncing Warehouse Data Back to Your Tools (2026)](https://dataskew.io/blog/reverse-etl): Reverse ETL explained: how to sync modeled warehouse data back into SaaS tools (CRM, ads, support), when it beats a direct integration, the tools, and the main pitfalls.
- [SQL Joins and GROUP BY: 7 Pitfalls That Break Your Warehouse](https://dataskew.io/blog/sql-joins-group-by-warehousing-pitfalls): Fan-out joins, grain drift, NULL keys silently dropped by INNER JOIN: the 7 SQL join and GROUP BY bugs that inflate warehouse metrics — each with a detector query and a fix.
- [SQL vs Python for Data Transformations: A Practical Decision Framework](https://dataskew.io/blog/sql-vs-python-data-transformations): A concrete, opinionated decision framework to choose between SQL and Python for your data pipeline transformation layer — with flowchart, scoring table, and side-by-side code comparisons.
- [SQL Window Functions: ROW_NUMBER, RANK, LAG & LEAD Explained](https://dataskew.io/blog/sql-window-functions-guide): Complete guide to SQL window functions: ROW_NUMBER vs RANK vs DENSE_RANK, LAG/LEAD, running totals and frame clauses — with dialect notes for PostgreSQL, BigQuery and SQLite.
- [Star Schema vs Snowflake Schema: Data Modeling for Analytics](https://dataskew.io/blog/star-schema-vs-snowflake-schema): Master dimensional modeling with star and snowflake schemas. Learn fact tables, dimension tables, SCD types, and when to use each approach.

## Courses
- [Cloud Cost Optimization for Data Engineering](https://dataskew.io/courses/cloud-cost-optimization): Master cloud cost management strategies for BigQuery, Snowflake, and AWS data services. Learn pricing models, optimization techniques, and monitoring best practices.
- [Advanced Data Modeling](https://dataskew.io/courses/data-modeling): Master advanced data modeling techniques including dimensional modeling, complex data types, and modern data warehouse design patterns.
- [Data Engineering Fundamentals](https://dataskew.io/courses/fundamentals): Master the essential building blocks of modern data engineering with comprehensive coverage of foundational concepts and tools.
- [module1_foundations](https://dataskew.io/courses/module1_foundations)
- [module2_data_modeling](https://dataskew.io/courses/module2_data_modeling)

## Interview Prep
- Real data engineering interview questions from top companies: https://dataskew.io/interview-prep

## Glossary
- Data engineering terms and definitions: https://dataskew.io/glossary

## Cite as
dataskew.io — Free Data Engineering Learning Platform (https://dataskew.io)