Data Engineering Glossary

A comprehensive reference of 72+ data engineering terms and definitions. Whether you are just starting your data engineering journey or preparing for interviews, this glossary covers the key concepts behind the modern data stack.

Data engineering is the practice of designing, building, and maintaining the systems that collect, store, transform, and serve data. It spans technologies from SQL and Python to distributed systems like Apache Kafka and Apache Spark, cloud platforms, and orchestration tools.

A

Airflow: Apache Airflow is an open-source workflow orchestration platform for authoring, scheduling, and monitoring data pipelines. It uses Directed Acyclic Graphs (DAGs) written in Python to define task dependencies and execution order. Airflow is widely used in data engineering for orchestrating ETL/ELT jobs.; See also: DAG, Orchestration, Orchestration with Airflow project
Apache Kafka: A distributed event streaming platform used for building real-time data pipelines and streaming applications. Kafka stores streams of records in topics, which are partitioned and replicated across a cluster for fault tolerance. It is the most widely adopted technology for event streaming in data engineering.; See also: Event Streaming, Topic, Stream Processing with Kafka project, Kafka blog post
Apache Spark: A unified analytics engine for large-scale data processing. Spark provides APIs in Python (PySpark), Scala, Java, and R for batch processing, SQL queries, machine learning, and stream processing. It processes data in-memory, making it significantly faster than disk-based systems like MapReduce.; See also: Batch Processing, Spark Streaming, Batch Processing with Spark project
Avro: A row-based data serialization format developed within the Apache Hadoop ecosystem. Avro stores the schema alongside the data, making it self-describing and well-suited for data exchange between systems. It is commonly used with Apache Kafka for message serialization.; See also: Parquet, Schema Registry

B

Backfilling: The process of retroactively processing or reprocessing historical data in a data pipeline. Backfills are necessary when fixing data quality issues, adding new transformations, or onboarding a new data source. A well-designed pipeline supports idempotent backfills to avoid duplicate data.; See also: Idempotency, Data Pipeline
Batch Processing: A method of processing data in large, discrete chunks at scheduled intervals rather than in real time. Batch jobs typically run on a schedule (hourly, daily) and process all data that has accumulated since the last run. It is the most common processing pattern in data warehousing.; See also: Stream Processing, Lambda Architecture
BigQuery: Google Cloud's fully managed, serverless data warehouse designed for large-scale analytics. BigQuery uses a columnar storage format and supports standard SQL queries. It automatically scales compute resources and charges based on the amount of data processed per query.; See also: Data Warehouse, Column Store

C

CDC (Change Data Capture): A set of techniques for identifying and capturing changes (inserts, updates, deletes) made to a database so they can be replicated to another system. CDC enables near-real-time data synchronization between operational databases and analytical systems without expensive full table scans.; See also: Event Streaming, Data Pipeline
CI/CD: Continuous Integration and Continuous Deployment/Delivery. In data engineering, CI/CD automates the testing, validation, and deployment of data pipelines, transformations, and infrastructure changes. It ensures code quality and reduces the risk of deploying broken pipelines to production.; See also: CI/CD for Data Pipelines project
Column Store: A database storage format that stores data by column rather than by row. Columnar storage is highly efficient for analytical queries that read a subset of columns across many rows, as it minimizes I/O and enables better compression. Most modern data warehouses use columnar storage.; See also: OLAP, Parquet, Data Warehouse
Consumer Group: In Apache Kafka, a consumer group is a set of consumers that cooperate to consume messages from one or more topics. Each partition is assigned to exactly one consumer in the group, enabling parallel processing. Consumer groups allow horizontal scaling of message consumption.; See also: Apache Kafka, Topic
CTE (Common Table Expression): A temporary named result set in SQL defined using the WITH clause. CTEs improve query readability by breaking complex queries into logical, reusable blocks. They are especially useful in data transformations and are heavily used in dbt models.; See also: SQL, Window Function, SQL fundamentals

D

DAG (Directed Acyclic Graph): A graph structure with directed edges and no cycles, used to represent task dependencies in workflow orchestration. In Apache Airflow, a DAG defines the order in which pipeline tasks should execute. DAGs ensure that tasks run only after their upstream dependencies have completed successfully.; See also: Airflow, Orchestration
Data Catalog: A centralized metadata repository that helps organizations discover, understand, and govern their data assets. Data catalogs store information about data schemas, lineage, ownership, quality metrics, and usage patterns. They are essential for data governance at scale.; See also: Data Governance, Data Lineage
Data Contract: A formal agreement between data producers and consumers that defines the structure, semantics, and quality expectations of a dataset. Data contracts help prevent breaking changes in data pipelines by making schema and quality requirements explicit and enforceable.; See also: Data Quality, Schema Registry
Data Engineering: The practice of designing, building, and maintaining the infrastructure and systems that collect, store, transform, and serve data for analysis and decision-making. Data engineers build data pipelines, manage data warehouses, and ensure data quality and reliability across an organization.; See also: Data Pipeline, How to become a data engineer, Beginner roadmap
Data Governance: The set of policies, processes, and standards that ensure data is managed properly across an organization. Data governance covers data quality, security, privacy, access control, and compliance. It is increasingly important as data regulations (GDPR, CCPA) become more stringent.; See also: Data Catalog, Data Quality
Data Lake: A centralized storage repository that holds raw data in its native format, including structured, semi-structured, and unstructured data. Data lakes use cheap object storage (like S3 or GCS) and defer schema enforcement to read time, offering maximum flexibility at the cost of potential data swamp issues.; See also: Data Lakehouse, Data Warehouse, Warehouse vs Lake blog
Data Lakehouse: A modern data architecture that combines the flexibility and low cost of a data lake with the data management and ACID transaction features of a data warehouse. Lakehouse architectures use open table formats like Delta Lake or Apache Iceberg on top of object storage.; See also: Delta Lake, Iceberg, Data Lake
Data Lineage: The ability to trace data from its origin through all the transformations and movements it undergoes to its final destination. Data lineage helps with debugging pipeline failures, impact analysis, regulatory compliance, and understanding how data assets are created.; See also: Data Catalog, Data Governance
Data Mesh: A decentralized data architecture paradigm where domain teams own and manage their own data products. Data mesh treats data as a product, with each domain responsible for the quality, discoverability, and interoperability of its data. It contrasts with centralized data team models.; See also: Data Contract, Data Governance
Data Modeling: The process of designing the structure and relationships of data in a database or data warehouse. Data modeling defines how data is stored, organized, and accessed. Common approaches include dimensional modeling (Kimball), normalized modeling (Inmon), and Data Vault.; See also: Star Schema, Normalization, Data modeling fundamentals
Data Pipeline: A series of automated steps that move and transform data from source systems to destination systems. Data pipelines can be batch or streaming, simple or complex, and typically include extraction, transformation, validation, and loading stages.; See also: ETL, ELT, Pipeline design patterns blog
Data Quality: The measure of how well data meets the requirements for its intended use, typically evaluated across dimensions like accuracy, completeness, consistency, timeliness, and uniqueness. Data quality engineering involves implementing validation rules, tests, and monitoring to ensure reliable data.; See also: Data Contract, Data quality fundamentals
Data Warehouse: A centralized repository of structured, cleaned data optimized for analytical queries and reporting. Data warehouses enforce schema on write, use columnar storage for fast aggregations, and serve as the single source of truth for business intelligence. Examples include Snowflake, BigQuery, and Redshift.; See also: OLAP, Star Schema, Data warehousing fundamentals
dbt (data build tool): An open-source transformation tool that enables data analysts and engineers to transform data in their warehouse using SQL SELECT statements. dbt handles dependency management, testing, documentation, and version control for data transformations, popularizing the analytics engineering role.; See also: ELT, Analytics Engineering with dbt project, dbt blog post
Dead Letter Queue: A message queue that stores messages that cannot be processed successfully by a consumer. In data pipelines, dead letter queues prevent bad records from blocking the entire pipeline and allow engineers to inspect, fix, and reprocess failed messages separately.; See also: Apache Kafka, Data Quality
Delta Lake: An open-source storage layer that brings ACID transactions, scalable metadata handling, and time travel to data lakes. Delta Lake runs on top of existing cloud object storage and is compatible with Apache Spark. It is a key enabler of the data lakehouse architecture.; See also: Data Lakehouse, Iceberg
Denormalization: The process of intentionally adding redundancy to a database schema to improve read performance. In data warehousing, denormalization reduces the number of joins needed for queries by pre-joining tables or duplicating columns. Star schemas are a common form of denormalization.; See also: Normalization, Star Schema
Dimension Table: A table in a dimensional model that contains descriptive attributes (dimensions) used to filter, group, and label facts. Examples include a customer dimension (name, location, segment) or a date dimension (year, month, day, quarter). Dimension tables are typically denormalized.; See also: Fact Table, Star Schema, Data modeling fundamentals
Docker: A platform for building, shipping, and running applications in lightweight, isolated containers. In data engineering, Docker ensures reproducible environments for development, testing, and deployment of data pipelines and tools like Airflow, Kafka, and Spark.; See also: Docker fundamentals, Docker blog post
DuckDB: An in-process analytical SQL database management system designed for fast analytical queries. DuckDB runs embedded within applications (similar to SQLite but for OLAP workloads) and is ideal for local development, testing, and processing medium-sized datasets without server infrastructure.; See also: OLAP, Analytics Dashboard project

E

ELT (Extract, Load, Transform): A data integration pattern where raw data is first extracted from sources and loaded into a target system (usually a cloud data warehouse), then transformed in place using SQL. ELT leverages the compute power of modern data warehouses and has largely replaced ETL for cloud-native architectures.; See also: ETL, dbt, ETL vs ELT blog
ETL (Extract, Transform, Load): A traditional data integration pattern where data is extracted from source systems, transformed in a staging area, and then loaded into a target data warehouse. ETL was the dominant pattern when compute in warehouses was expensive, but has been largely supplanted by ELT in modern cloud architectures.; See also: ELT, Data Pipeline, ETL vs ELT blog
Event Streaming: The practice of capturing and processing data as a continuous stream of events in real time. Event streaming platforms like Apache Kafka enable applications to publish, subscribe to, and process streams of records as they occur, supporting use cases like real-time analytics and CDC.; See also: Apache Kafka, Stream Processing
Exactly-Once Semantics: A message delivery guarantee ensuring that each message is processed exactly one time, with no duplicates and no data loss. Achieving exactly-once semantics in distributed systems is challenging and typically requires idempotent consumers and transactional coordination between systems.; See also: Idempotency, Apache Kafka

F

Fact Table: A central table in a dimensional model that stores quantitative measurements (facts) about business events. Fact tables contain foreign keys to dimension tables and numeric measures like revenue, quantity, or duration. They are typically the largest tables in a data warehouse.; See also: Dimension Table, Star Schema
Foreign Key: A column (or set of columns) in a database table that references the primary key of another table, establishing a relationship between the two tables. Foreign keys enforce referential integrity and are fundamental to relational data modeling and join operations.; See also: Primary Key, Normalization

G

Graph Database: A database that uses graph structures (nodes, edges, and properties) to store and query data. Graph databases excel at representing and traversing complex relationships, such as social networks, recommendation engines, and fraud detection. Examples include Neo4j and Amazon Neptune.; See also: NoSQL, NoSQL fundamentals

I

Iceberg: Apache Iceberg is an open table format for large analytical datasets. It provides ACID transactions, schema evolution, time travel, and partition evolution on top of data lake storage. Iceberg is vendor-neutral and supported by Spark, Trino, Flink, and most major cloud platforms.; See also: Delta Lake, Data Lakehouse
Idempotency: The property of an operation that produces the same result whether it is executed once or multiple times. Idempotent pipelines are critical in data engineering because they allow safe retries and backfills without creating duplicate records or corrupting data.; See also: Backfilling, Exactly-Once Semantics
Incremental Load: A data loading strategy that processes only new or changed records since the last pipeline run, rather than reprocessing the entire dataset. Incremental loads are more efficient than full loads and are essential for large datasets, but require reliable change tracking mechanisms.; See also: CDC, Backfilling

K

Kafka Connect: A framework within the Apache Kafka ecosystem for streaming data between Kafka and external systems. Kafka Connect provides pre-built connectors for databases, file systems, cloud services, and more, eliminating the need to write custom producer/consumer code for common integrations.; See also: Apache Kafka, CDC
Key-Value Store: A NoSQL database that stores data as simple key-value pairs, optimized for extremely fast lookups by key. Key-value stores like Redis and DynamoDB are used for caching, session management, and real-time feature stores where low-latency access is critical.; See also: NoSQL, Redis
Kimball Methodology: A bottom-up approach to data warehouse design created by Ralph Kimball. It emphasizes dimensional modeling with star schemas, conformed dimensions, and a data mart-centric architecture. The Kimball methodology prioritizes ease of querying and business user accessibility.; See also: Star Schema, Data Modeling, Dimension Table

L

Lambda Architecture: A data processing architecture that combines batch and stream processing to provide both comprehensive and low-latency views of data. It consists of a batch layer (for accuracy), a speed layer (for real-time), and a serving layer. It has been largely superseded by the kappa architecture in modern systems.; See also: Batch Processing, Stream Processing

M

Materialized View: A database object that stores the precomputed result of a query. Unlike regular views, materialized views physically store data and must be refreshed periodically. They are used to speed up expensive analytical queries by pre-aggregating data, trading storage space for query performance.; See also: OLAP, Data Warehouse
Medallion Architecture: A data organization pattern that structures data into three layers: bronze (raw data), silver (cleaned and validated), and gold (business-ready aggregates). The medallion architecture is commonly used in data lakehouses to progressively refine data quality and is popularized by Databricks.; See also: Data Lakehouse, Data Quality
MongoDB: A popular open-source document database that stores data in flexible, JSON-like documents. MongoDB is schema-flexible, meaning documents in the same collection can have different fields. It is widely used for application backends, content management, and semi-structured data storage.; See also: NoSQL, NoSQL fundamentals

N

NoSQL: A broad category of database systems that do not use the traditional relational model with fixed schemas and SQL. NoSQL databases include key-value stores, document databases, column-family stores, and graph databases. They are chosen for specific use cases where relational databases are suboptimal.; See also: Key-Value Store, MongoDB, NoSQL fundamentals
Normalization: The process of organizing a relational database to reduce data redundancy and improve data integrity. Normalization involves decomposing tables into smaller, related tables following normal forms (1NF, 2NF, 3NF, BCNF). It is the standard approach for transactional (OLTP) databases.; See also: Denormalization, Data Modeling

O

OLAP (Online Analytical Processing): A category of database systems and queries optimized for complex analytical queries over large datasets. OLAP workloads involve aggregations, joins, and scans across many rows. Data warehouses are OLAP systems, in contrast to OLTP systems used for transactional operations.; See also: OLTP, Data Warehouse, Column Store
OLTP (Online Transaction Processing): A category of database systems optimized for fast, short transactions such as inserts, updates, and point lookups. OLTP databases power operational applications (e-commerce, banking) and use row-based storage with indexing for low-latency access. Examples include PostgreSQL and MySQL.; See also: OLAP, Primary Key
Orchestration: The automated coordination, scheduling, and management of data pipeline tasks and dependencies. Orchestration tools like Apache Airflow, Prefect, and Dagster define when tasks run, in what order, and how to handle failures, retries, and notifications.; See also: Airflow, DAG, Orchestration with Airflow project

P

Parquet: A columnar file format optimized for analytical workloads. Parquet files store data in a compressed, columnar layout that enables efficient reading of specific columns and supports complex nested data structures. It is the de facto standard file format for data lakes and big data processing.; See also: Avro, Column Store
Partitioning: The practice of dividing a large dataset or table into smaller, more manageable segments based on a key (such as date, region, or category). Partitioning improves query performance by allowing engines to skip irrelevant data (partition pruning) and enables parallel processing.; See also: Data Warehouse, Apache Spark
Primary Key: A column (or set of columns) that uniquely identifies each row in a database table. Primary keys enforce uniqueness and are used as the reference point for foreign key relationships. Choosing effective primary keys is fundamental to good data modeling.; See also: Foreign Key, Data Modeling
Pub/Sub (Publish/Subscribe): A messaging pattern where publishers send messages to a topic without knowledge of subscribers, and subscribers receive messages from topics they are interested in. Pub/Sub decouples data producers from consumers and is the foundation of event-driven architectures. Google Cloud Pub/Sub and Apache Kafka implement this pattern.; See also: Apache Kafka, Event Streaming
Python: A high-level programming language that is the most widely used language in data engineering. Python's rich ecosystem of libraries (pandas, Polars, PySpark, dlt, Airflow) and readable syntax make it the primary choice for building data pipelines, ETL scripts, and data transformations.; See also: Python fundamentals, Apache Spark

R

Redshift: Amazon Web Services' fully managed cloud data warehouse. Redshift uses columnar storage, massively parallel processing (MPP), and result caching for fast analytical query performance. It integrates deeply with the AWS ecosystem including S3, Glue, and Lambda.; See also: Data Warehouse, BigQuery
Redis: An open-source, in-memory key-value data store known for extremely low latency. Redis is used in data engineering for caching, real-time feature stores, message brokering, and as a lightweight pub/sub system. It supports data structures like strings, hashes, lists, and sets.; See also: Key-Value Store, NoSQL

S

Schema Registry: A service that manages and enforces schemas for data flowing through streaming platforms like Apache Kafka. Schema registries store Avro, Protobuf, or JSON schemas and validate that messages conform to registered schemas, preventing incompatible changes from breaking downstream consumers.; See also: Avro, Data Contract, Apache Kafka
SCD (Slowly Changing Dimension): A dimension table design technique for tracking changes to dimension attributes over time. Common approaches include Type 1 (overwrite), Type 2 (add new row with versioning), and Type 3 (add new column). SCD Type 2 is the most common in data warehousing, preserving full history.; See also: Dimension Table, Data Modeling
Snowflake: A cloud-native data warehouse platform that separates compute from storage, allowing independent scaling of each. Snowflake supports multi-cloud deployment (AWS, Azure, GCP), automatic scaling, and near-zero maintenance. It has become one of the most popular cloud data warehouses.; See also: Data Warehouse, BigQuery
Snowflake Schema: A data warehouse schema where dimension tables are normalized into multiple related tables, resembling a snowflake shape. Snowflake schemas reduce storage by eliminating redundancy but require more joins for queries compared to star schemas.; See also: Star Schema, Normalization, Star vs Snowflake blog
Spark Streaming: A component of Apache Spark for processing real-time data streams using micro-batching. Structured Streaming, the newer API, treats a live data stream as a continuously appended table and supports exactly-once semantics. It enables unified batch and streaming processing within Spark.; See also: Apache Spark, Stream Processing
SQL (Structured Query Language): The standard language for managing and querying relational databases. SQL is the most important technical skill for data engineers, used for data extraction, transformation, analysis, and defining data warehouse models. Modern tools like dbt are built entirely around SQL.; See also: CTE, Window Function, SQL fundamentals, Window functions guide
Star Schema: A data warehouse schema design with a central fact table surrounded by denormalized dimension tables, resembling a star shape. Star schemas are simple to query, performant for analytical workloads, and the most common dimensional modeling pattern in the Kimball methodology.; See also: Fact Table, Dimension Table, Snowflake Schema, Star vs Snowflake blog
Stream Processing: The continuous processing of data records as they arrive, providing low-latency results. Stream processing frameworks like Apache Kafka Streams, Apache Flink, and Spark Structured Streaming enable real-time analytics, alerting, and data transformation on unbounded data streams.; See also: Batch Processing, Event Streaming, Stream Processing with Kafka project

T

Terraform: An open-source Infrastructure as Code (IaC) tool by HashiCorp for provisioning and managing cloud resources using declarative configuration files. Data engineers use Terraform to provision data warehouses, storage buckets, compute clusters, and networking infrastructure in a reproducible, version-controlled manner.; See also: Infrastructure as Code project, Docker
Topic: In Apache Kafka, a topic is a named category or feed to which records are published. Topics are divided into partitions for parallelism and replicated across brokers for fault tolerance. Topics are the fundamental unit of organization in Kafka's publish-subscribe model.; See also: Apache Kafka, Consumer Group, Pub/Sub

W

Window Function: A SQL function that performs calculations across a set of rows related to the current row without collapsing them into a single output row. Window functions (ROW_NUMBER, RANK, LAG, LEAD, SUM OVER) are essential for analytics, deduplication, and time-series analysis in data engineering.; See also: SQL, CTE, Window functions guide

Ready to Learn Data Engineering?

Understanding these terms is the first step. Dive deeper with our structured learning paths, hands-on fundamentals, and real-world projects.