📊 Big Data & Data Engineering

Scale • Streaming • Warehousing • Analytics

Master the data ecosystem: Hadoop, Spark, Flink, Kafka, data lakes, lakehouse, and real-time pipelines — 45+ essential big data concepts.

46+

Big Data Technologies & Engineering Patterns

Beginner

📈

5 V's of Big Data

Volume, Velocity, Variety, Veracity, Value — core pillars.

HDFS (Hadoop FS)

Distributed storage, replication, NameNode/DataNode architecture.

MapReduce Paradigm

Shuffle, sort, reducer, YARN, classic batch processing.

Apache Spark

In-memory processing, RDDs, DataFrames, Catalyst optimizer.

Apache Hive

SQL-on-Hadoop, metastore, HQL, partitioning.

Apache Kafka

Distributed event streaming, topics, partitions, producer/consumer.

Apache Flink

True stream processing, event time, stateful functions, exactly-once.

Spark Streaming / Structured

Micro-batch, continuous processing, watermarks.

Apache Pulsar & Redpanda

Geo-replication, tiered storage, Kafka API compatible.

Data Lake vs Warehouse

Schema-on-read, raw storage, data swamp challenges.

Lakehouse (Delta / Iceberg)

ACID transactions, time travel, schema evolution, open table formats.

Columnar Formats

Parquet, ORC, Avro — efficient compression and predicate pushdown.

Apache Airflow

DAGs, operators, sensors, data pipeline orchestration.

Modern Orchestration (Dagster, Prefect)

Data assets, idempotency, observability.

Presto / Trino

Distributed SQL query engine, federated queries, low latency.

Cloud Warehouses

Snowflake, BigQuery, Redshift — separation of compute/storage.

NoSQL DB (HBase, Cassandra)

Wide-column stores, consistent hashing, tunable consistency.

Document DB (MongoDB)

Aggregation pipelines, sharding, flexible schema.

Elasticsearch & ELK

Full-text search, inverted index, log analytics.

Search & Observability

Analytics

Advanced

🔄

ETL / ELT Patterns

Extract, Transform, Load vs ELT, dbt, reverse ETL.

Data Mesh Architecture

Domain-oriented decentralization, data products, self-serve.

Data Fabric

Metadata-driven integration, active metadata, virtualized access.

ksqlDB & Materialize

Streaming SQL, materialized views, real-time analytics.

Data Quality & Great Expectations

Expectations, profiling, data validation pipelines.

Data Observability

Freshness, volume, schema drift, lineage (Monte Carlo, Soda).

MLlib & Distributed ML

Scalable algorithms, feature engineering at scale.

DVC & Data Versioning

Data version control, pipeline reproducibility.

ZooKeeper

Coordination service for distributed apps, leader election.

Apache Druid

Real-time analytics, OLAP, time-optimized storage.

ClickHouse

Columnar OLAP, vectorized execution, high compression.

Analytics

Real-time

🏗️ Data Engineering Roadmap

🌱 Data Fundamentals

HDFS, MapReduce, Hive basics, 5Vs, ZooKeeper

⚙️ Builder

Spark Core, Airflow, NoSQL, Data Lakes, Parquet

🔥 Architect

Flink, Kafka, Lakehouse (Delta/Iceberg), Streaming SQL, MPP

🏆 Strategist

Data Mesh, Data Fabric, Real-time ML, Advanced Observability

⚙️ Engineer Scalable Data Platforms

Hands-on labs, pipeline templates & cloud data projects — build production-ready skills.

Back to Home Data Services

📊 Big Data & Data Engineering

5 V's of Big Data

HDFS (Hadoop FS)

MapReduce Paradigm

Apache Spark

Apache Hive

Apache Kafka

Apache Flink

Spark Streaming / Structured

Apache Pulsar & Redpanda

Data Lake vs Warehouse

Lakehouse (Delta / Iceberg)

Columnar Formats

Apache Airflow

Modern Orchestration (Dagster, Prefect)

Presto / Trino

Cloud Warehouses

NoSQL DB (HBase, Cassandra)

Document DB (MongoDB)

Elasticsearch & ELK

ETL / ELT Patterns

Data Mesh Architecture

Data Fabric

ksqlDB & Materialize

Data Quality & Great Expectations

Data Observability

MLlib & Distributed ML

DVC & Data Versioning

ZooKeeper

Apache Druid

ClickHouse

🏗️ Data Engineering Roadmap

Related Computer Science Domains

⚙️ Engineer Scalable Data Platforms