🧠 Fundamentals

🎯 What are Columnar Formats?

Columnar formats store data by columns instead of rows. Instead of reading an entire row (all columns) to answer a query, columnar formats let you read only the columns you need. This makes them ideal for analytics, warehousing, and big data. The main columnar formats are Parquet and ORC, while Avro is row-based but often grouped with them for comparison.

Why Columnar Matters

Column pruning: Read only needed columns. Compression: Similar data compresses better. Encoding: Efficient per-column encodings. Predicate pushdown: Skip irrelevant data. Analytics: Perfect for OLAP. Cost: Less I/O, less cost.

📌 Column Storage📌 Compression📌 Analytics📌 Parquet/ORC

# Columnar Formats — the big picture
# Columnar formats store data by column, not row.

# CONCEPTUAL OVERVIEW
print("""
=== Row vs Column Storage ===

Row-based (CSV, Avro):
  Row 1: [Alice, 30, NYC, 50000]
  Row 2: [Bob,   25, LA,  60000]
  Row 3: [Carol, 35, NYC, 75000]

  → Stored as: Alice,30,NYC,50000,Bob,25,LA,60000,...

Columnar (Parquet, ORC):
  Name:  [Alice, Bob, Carol]
  Age:   [30, 25, 35]
  City:  [NYC, LA, NYC]
  Salary:[50000, 60000, 75000]

  → Stored as: Alice,Bob,Carol,30,25,35,NYC,LA,NYC,...
""")

# WHY COLUMNAR
print("=== Why Columnar? ===")
print("""
Analytics queries:
  SELECT AVG(salary) FROM users WHERE city = 'NYC';

Row storage:
  - Read entire rows
  - Read all columns
  - Waste I/O

Columnar storage:
  - Read only salary and city
  - Skip other columns
  - Less I/O
  - Faster

Benefits:
  - Column pruning (90% less I/O)
  - Better compression (same type)
  - Encoding (run-length, dictionary)
  - Predicate pushdown (skip data)
""")

# MAIN FORMATS
print("=== Main Columnar Formats ===")
print("""
1. APACHE PARQUET
   - Columnar
   - Spark-native
   - Widely used
   - Nested data
   - Open source

2. APACHE ORC
   - Columnar
   - Hive-native
   - Optimized
   - Indexes
   - ACID support

3. APACHE AVRO
   - Row-based
   - Schema evolution
   - Streaming
   - Kafka-friendly

4. OTHERS
   - Arrow (in-memory)
   - CarbonData
   - Kudu
""")

# REAL-WORLD USAGE
print("=== Real-World Usage ===")
print("""
Parquet:
  - Spark, Databricks, Presto
  - S3-based data lakes
  - ML pipelines

ORC:
  - Hive, Spark
  - HDFS-based warehouses
  - ACID tables

Avro:
  - Kafka
  - Streaming
  - Schema evolution
  - Row-oriented

All three are widely used.
Choice depends on engine, workload, and ecosystem.
""")

📐 Row vs Column

🎯 The Fundamental Difference

Row-based storage keeps all fields of a record together. Columnar storage keeps all values of a field together. This changes everything about how data is read, compressed, and queried.

Storage Layout

Row: Best for OLTP. Columnar: Best for OLAP. Row: Write-optimized. Columnar: Read-optimized. Row: All columns. Columnar: Only needed.

📐 Row Storage📐 Column Storage📐 OLTP📐 OLAP

# Row vs Column Storage
print("=== Visual Comparison ===")
print("""
Data:
  | id | name  | age | city | salary |
  |----|-------|-----|------|--------|
  | 1  | Alice | 30  | NYC  | 50000  |
  | 2  | Bob   | 25  | LA   | 60000  |
  | 3  | Carol | 35  | NYC  | 75000  |

ROW STORAGE (CSV, Avro):
  [1, Alice, 30, NYC, 50000]
  [2, Bob,   25, LA,  60000]
  [3, Carol, 35, NYC, 75000]

  Disk: 1,Alice,30,NYC,50000,2,Bob,25,LA,60000,...

COLUMNAR STORAGE (Parquet, ORC):
  id:     [1, 2, 3]
  name:   [Alice, Bob, Carol]
  age:    [30, 25, 35]
  city:   [NYC, LA, NYC]
  salary: [50000, 60000, 75000]

  Disk: [1,2,3], [Alice,Bob,Carol], [30,25,35], ...
""")

# QUERY EXAMPLE
print("=== Query Example ===")
print("""
Query: SELECT AVG(salary) FROM users WHERE city = 'NYC';

ROW STORAGE:
  - Read all rows
  - Read all columns
  - Read: id, name, age, city, salary
  - Waste: id, name, age

COLUMNAR STORAGE:
  - Read only city column
  - Read only salary column
  - Filter: city = NYC
  - Compute: AVG(salary)
  - Skip: id, name, age

I/O savings:
  - Row: 100% of data
  - Columnar: 40% of data (2/5 columns)
  - 60% less I/O
""")

print("\n=== Pros & Cons ===")
print(f"{'Aspect':<25} {'Row':<25} {'Columnar':<25}")
print("-" * 75)
print(f"{'Best for':<25} {'OLTP':<25} {'OLAP':<25}")
print(f"{'Write':<25} {'Fast':<25} {'Slower':<25}")
print(f"{'Read (single row)':<25} {'Fast':<25} {'Slow':<25}")
print(f"{'Read (analytics)':<25} {'Slow':<25} {'Fast':<25}")
print(f"{'Compression':<25} {'Good':<25} {'Excellent':<25}")
print(f"{'I/O':<25} {'All columns':<25} {'Only needed':<25}")
print(f"{'Examples':<25} {'CSV, Avro':<25} {'Parquet, ORC':<25}")

# WHEN TO USE
print("\n=== When to Use Which ===")
print("""
USE ROW STORAGE:
  ✓ OLTP
  ✓ Frequent writes
  ✓ Single-row lookups
  ✓ Small datasets
  ✓ Streaming

USE COLUMNAR:
  ✓ Analytics
  ✓ Large datasets
  ✓ Aggregations
  ✓ Wide tables
  ✓ BI/ML
""")

🟦 Apache Parquet

🎯 The Spark-Native Columnar Format

Apache Parquet is an open-source columnar file format designed for efficient data storage and retrieval. It's the de-facto standard in the Spark ecosystem and widely used in data lakes.

Parquet Characteristics

Columnar: By column. Nested: Complex types. Compression: Excellent. Encoding: Dictionary, RLE. Ecosystem: Spark, Presto, Hive. Open: Apache license.

🟦 Columnar🟦 Nested Data🟦 Spark-Native🟦 Open

# Apache Parquet — deep dive
print("=== What is Parquet? ===")
print("""
Apache Parquet:
  - Columnar file format
  - Open source (Apache)
  - Spark-native
  - Nested data
  - Efficient compression
  - Predicate pushdown

Created by:
  - Twitter + Cloudera (2013)
  - Apache (2015)

File structure:
  File
    ├── Row Group 1
    │   ├── Column Chunk (id)
    │   │   ├── Page 1
    │   │   └── Page 2
    │   ├── Column Chunk (name)
    │   └── ...
    ├── Row Group 2
    └── Footer (metadata)
""")

print("\n=== Parquet Features ===")
print("""
1. COLUMNAR STORAGE
   - Column pruning
   - Less I/O

2. NESTED DATA
   - Structs
   - Lists
   - Maps
   - Dremel encoding

3. COMPRESSION
   - Snappy (fast)
   - Gzip (better)
   - Zstd (best)
   - LZ4

4. ENCODING
   - Dictionary
   - Run-length
   - Delta
   - Bit-packing

5. PREDICATE PUSHDOWN
   - Row group statistics
   - Column statistics
   - Skip irrelevant data

6. SCHEMA
   - Self-describing
   - Column types
   - Nested schema
""")

print("\n=== Parquet Example ===")
print("""
# PySpark
# Write
df.write \\
    .mode("overwrite") \\
    .option("compression", "snappy") \\
    .parquet("s3://bucket/parquet/events")

# Read
df = spark.read.parquet("s3://bucket/parquet/events")

# With partitioning
df.write \\
    .partitionBy("year", "month") \\
    .parquet("s3://bucket/parquet/events")

# Hive
CREATE TABLE events (
    id INT,
    user_id INT,
    event_type STRING
)
STORED AS PARQUET;
""")

print("\n=== Parquet Pros & Cons ===")
print("""
PROS:
  ✓ Columnar
  ✓ Nested data
  ✓ Spark-native
  ✓ Widely supported
  ✓ Good compression
  ✓ Predicate pushdown
  ✓ Open source

CONS:
  ✗ Slower writes
  ✗ Not ideal for streaming
  ✗ Schema evolution limited
  ✗ Complex metadata
  ✗ Not ACID
""")

🟧 Apache ORC

🎯 The Hive-Optimized Format

Apache ORC (Optimized Row Columnar) is a columnar file format optimized for Hive. It has built-in indexes, statistics, and ACID support, making it ideal for data warehousing on Hadoop.

ORC Characteristics

Columnar: By column. Indexes: Built-in. Statistics: Rich. ACID: Hive support. Compression: Excellent. Hive: Native.

🟧 Columnar🟧 Indexes🟧 ACID🟧 Hive-Native

# Apache ORC — deep dive
print("=== What is ORC? ===")
print("""
Apache ORC:
  - Columnar file format
  - Optimized for Hive
  - Built-in indexes
  - Rich statistics
  - ACID support
  - Open source

Created by:
  - Hortonworks + Facebook (2013)
  - Apache (2015)

File structure:
  File
    ├── Stripe 1
    │   ├── Index Data
    │   ├── Row Data
    │   └── Stripe Footer
    ├── Stripe 2
    └── File Footer
""")

print("\n=== ORC Features ===")
print("""
1. COLUMNAR STORAGE
   - Column pruning
   - Less I/O

2. BUILT-IN INDEXES
   - Min/max per column
   - Bloom filters
   - Row-level indexes

3. STATISTICS
   - Column stats
   - Stripe stats
   - File stats

4. ACID SUPPORT
   - Hive ACID
   - Transactions
   - Updates/deletes

5. COMPRESSION
   - Snappy
   - Zlib
   - LZO
   - Zstd

6. PREDICATE PUSHDOWN
   - Index-based
   - Bloom filters
   - Skip data
""")

print("\n=== ORC Example ===")
print("""
-- Hive
CREATE TABLE events (
    id INT,
    user_id INT,
    event_type STRING
)
STORED AS ORC
TBLPROPERTIES ("orc.compress"="SNAPPY");

-- Insert
INSERT INTO events VALUES (1, 101, 'click');

-- Update (ACID)
UPDATE events SET event_type = 'view' WHERE id = 1;

-- Spark
df.write \\
    .mode("overwrite") \\
    .option("compression", "snappy") \\
    .orc("s3://bucket/orc/events")

df = spark.read.orc("s3://bucket/orc/events")
""")

print("\n=== ORC Pros & Cons ===")
print("""
PROS:
  ✓ Columnar
  ✓ Built-in indexes
  ✓ Rich statistics
  ✓ ACID support
  ✓ Hive-native
  ✓ Good compression
  ✓ Predicate pushdown

CONS:
  ✗ Hive-centric
  ✗ Less Spark-native
  ✗ Smaller ecosystem
  ✗ Fewer tools
  ✗ Not ideal for streaming
""")

🟩 Apache Avro

🎯 The Row-Based Schema Evolution Format

Apache Avro is a row-based data serialization format with rich schema evolution support. It's widely used in streaming (Kafka) and data pipelines where schemas change over time.

Avro Characteristics

Row-based: By row. Schema: JSON-based. Evolution: Excellent. Streaming: Kafka-friendly. Compact: Binary. Splittable: Yes.

🟩 Row-Based🟩 Schema Evolution🟩 Streaming🟩 Kafka

# Apache Avro — deep dive
print("=== What is Avro? ===")
print("""
Apache Avro:
  - Row-based format
  - Schema in JSON
  - Schema evolution
  - Compact binary
  - Splittable
  - Streaming-friendly

Created by:
  - Doug Cutting (Hadoop creator)
  - Apache (2009)

Schema example:
  {
    "type": "record",
    "name": "User",
    "fields": [
      {"name": "id", "type": "int"},
      {"name": "name", "type": "string"},
      {"name": "age", "type": "int"}
    ]
  }
""")

print("\n=== Avro Features ===")
print("""
1. ROW-BASED
   - All fields together
   - Fast writes
   - Fast single-row reads

2. SCHEMA EVOLUTION
   - Add fields (with defaults)
   - Remove fields
   - Rename (with aliases)
   - Type promotion

3. COMPACT
   - Binary encoding
   - No field names in data
   - Schema in header

4. SPLITTABLE
   - Can split for parallelism
   - Sync markers

5. STREAMING
   - Kafka-friendly
   - Schema Registry
   - Confluent ecosystem

6. LANGUAGE SUPPORT
   - Java, Python, C, C++
   - Many languages
""")

print("\n=== Avro Example ===")
print("""
# Schema definition (user.avsc)
{
  "type": "record",
  "name": "User",
  "fields": [
    {"name": "id", "type": "int"},
    {"name": "name", "type": "string"},
    {"name": "age", "type": ["int", "null"]}
  ]
}

# PySpark
df.write \\
    .format("avro") \\
    .mode("overwrite") \\
    .save("s3://bucket/avro/users")

df = spark.read.format("avro").load("s3://bucket/avro/users")

# Kafka
producer.send('users', value=avro_bytes)
""")

print("\n=== Avro Pros & Cons ===")
print("""
PROS:
  ✓ Row-based
  ✓ Schema evolution
  ✓ Compact binary
  ✓ Splittable
  ✓ Streaming-friendly
  ✓ Many languages
  ✓ Kafka integration

CONS:
  ✗ Not columnar
  ✗ Slower analytics
  ✗ No predicate pushdown
  ✗ Less compression
  ✗ Not ideal for BI
""")

🔢 Encoding

🎯 Making Data Smaller

Encoding (or encoding schemes) transforms data into a more compact representation before compression. Columnar formats use sophisticated encodings per column type.

Encoding Schemes

Dictionary: Replace with IDs. RLE: Run-length. Delta: Differences. Bit-packing: Fewer bits. Frame-of-reference: Offset. Boolean: Bitmap.

🔢 Dictionary🔢 RLE🔢 Delta🔢 Bit-Packing

# Encoding Schemes
print("=== Dictionary Encoding ===")
print("""
Dictionary:
  - Map unique values to IDs
  - Store dictionary + IDs
  - Great for low cardinality

Example:
  City column: [NYC, LA, NYC, NYC, SF, LA]
  
  Dictionary:
    NYC → 0
    LA  → 1
    SF  → 2
  
  Encoded: [0, 1, 0, 0, 2, 1]
  
  Savings: large for repetitive data
""")

print("\n=== Run-Length Encoding (RLE) ===")
print("""
RLE:
  - Store value + count
  - Great for repeated values

Example:
  Status column: [OK, OK, OK, OK, ERR, ERR, OK]
  
  Encoded: (OK, 4), (ERR, 2), (OK, 1)
  
  Savings: large for runs
""")

print("\n=== Delta Encoding ===")
print("""
Delta:
  - Store differences
  - Great for sorted values

Example:
  Timestamps: [1000, 1005, 1010, 1015, 1020]
  
  Deltas: [1000, 5, 5, 5, 5]
  
  Savings: large for sequential data
""")

print("\n=== Bit-Packing ===")
print("""
Bit-packing:
  - Use minimum bits
  - Great for small ranges

Example:
  Values: [1, 2, 3, 4, 5, 6, 7]
  Standard: 32 bits each = 224 bits
  Bit-packed: 3 bits each = 21 bits
  
  Savings: 90%
""")

print("\n=== Encoding Comparison ===")
print(f"{'Encoding':<25} {'Best For':<30} {'Example':<25}")
print("-" * 80)
print(f"{'Dictionary':<25} {'Low cardinality':<30} {'City, status':<25}")
print(f"{'RLE':<25} {'Repeated values':<30} {'Status, flags':<25}")
print(f"{'Delta':<25} {'Sorted values':<30} {'Timestamps, IDs':<25}")
print(f"{'Bit-packing':<25} {'Small ranges':<30} {'Counts, enums':<25}")
print(f"{'Frame-of-reference':<25} {'Clustered values':<30} {'Ages, prices':<25}")
print(f"{'Boolean':<25} {'True/false':<30} {'Flags':<25}")

print("\n=== Parquet Encodings ===")
print("""
Parquet supports:
  - Dictionary
  - Run-length
  - Delta (binary, int)
  - Bit-packing
  - Plain
  - Byte stream split

Selected automatically per column.
""")

print("\n=== ORC Encodings ===")
print("""
ORC supports:
  - Dictionary
  - Run-length
  - Delta
  - Bit-packing
  - Frame-of-reference
  - Direct

Selected automatically per column.
""")

🗜️ Compression

🎯 Reducing Data Size

Compression reduces the physical size of data. Columnar formats compress better than row formats because similar values are stored together. Common codecs: Snappy, Gzip, Zstd, LZ4.

Compression Codecs

Snappy: Fast, moderate. Gzip: Slower, better. Zstd: Best ratio. LZ4: Fastest. Brotli: Web. LZO: Hadoop.

🗜️ Snappy🗜️ Zstd🗜️ Gzip🗜️ LZ4

# Compression Codecs
print("=== Codec Comparison ===")
print(f"{'Codec':<15} {'Speed':<15} {'Ratio':<15} {'Use Case':<25}")
print("-" * 70)
print(f"{'Snappy':<15} {'Fast':<15} {'Good':<15} {'General purpose':<25}")
print(f"{'LZ4':<15} {'Fastest':<15} {'Good':<15} {'Low latency':<25}")
print(f"{'Gzip':<15} {'Slow':<15} {'Better':<15} {'Archives':<25}")
print(f"{'Zstd':<15} {'Medium':<15} {'Best':<15} {'Modern default':<25}")
print(f"{'Brotli':<15} {'Slow':<15} {'Best':<15} {'Web assets':<25}")
print(f"{'LZO':<15} {'Fast':<15} {'Moderate':<15} {'Hadoop legacy':<25}")

print("\n=== Compression in Parquet ===")
print("""
Codecs:
  - Snappy (default)
  - Gzip
  - Zstd
  - LZ4
  - Brotli
  - None

Configuration:
  df.write \\
      .option("compression", "snappy") \\
      .parquet(path)

  # Or Zstd
  df.write \\
      .option("compression", "zstd") \\
      .parquet(path)
""")

print("\n=== Compression in ORC ===")
print("""
Codecs:
  - Zlib (default)
  - Snappy
  - LZO
  - Zstd
  - None

Configuration:
  CREATE TABLE events (...)
  STORED AS ORC
  TBLPROPERTIES ("orc.compress"="SNAPPY");
""")

print("\n=== Compression in Avro ===")
print("""
Codecs:
  - null
  - deflate (gzip)
  - snappy
  - bzip2
  - xz
  - zstandard

Configuration:
  df.write \\
      .format("avro") \\
      .option("compression", "snappy") \\
      .save(path)
""")

print("\n=== Compression Comparison ===")
print(f"{'Codec':<15} {'Parquet':<15} {'ORC':<15} {'Avro':<15}")
print("-" * 60)
print(f"{'Snappy':<15} {'Yes':<15} {'Yes':<15} {'Yes':<15}")
print(f"{'Zstd':<15} {'Yes':<15} {'Yes':<15} {'Yes':<15}")
print(f"{'Gzip':<15} {'Yes':<15} {'Yes':<15} {'Yes':<15}")
print(f"{'LZ4':<15} {'Yes':<15} {'No':<15} {'No':<15}")
print(f"{'Brotli':<15} {'Yes':<15} {'No':<15} {'No':<15}")

print("\n=== Choosing a Codec ===")
print("""
For speed:
  → LZ4, Snappy

For balance:
  → Snappy, Zstd

For size:
  → Zstd, Gzip

For archives:
  → Gzip, Zstd

Modern recommendation:
  → Zstd (best balance)
""")

🔍 Predicate Pushdown

🎯 Skipping Irrelevant Data

Predicate pushdown allows the query engine to skip data that doesn't match the query filter. Columnar formats store statistics (min/max) per row group/stripe, enabling efficient skipping.

Predicate Pushdown

Statistics: Min/max per block. Skip: Irrelevant data. Parquet: Row group stats. ORC: Stripe + row indexes. Bloom: Membership. I/O: Massive savings.

🔍 Statistics🔍 Min/Max🔍 Bloom Filter🔍 Skip Data

# Predicate Pushdown
print("=== What is Predicate Pushdown? ===")
print("""
Predicate pushdown:
  - Push filter to storage
  - Skip irrelevant data
  - Less I/O
  - Faster queries

Without pushdown:
  1. Read all data
  2. Filter in memory
  3. Return results

With pushdown:
  1. Check statistics
  2. Skip non-matching blocks
  3. Read only matching data
  4. Filter in memory
  5. Return results
""")

print("\n=== How It Works ===")
print("""
Parquet:
  - Row group statistics
    - min value
    - max value
    - null count
  - Column statistics
  - Page statistics

ORC:
  - Stripe statistics
  - Row indexes
  - Bloom filters
  - More granular

Example:
  Query: WHERE age > 50
  Row group stats:
    Row group 1: age min=10, max=30 → SKIP
    Row group 2: age min=40, max=60 → READ
    Row group 3: age min=70, max=90 → READ
""")

print("\n=== Bloom Filters ===")
print("""
Bloom filter:
  - Probabilistic data structure
  - Membership test
  - No false negatives
  - Possible false positives

Use:
  Query: WHERE user_id = 12345
  Bloom filter: "Not in this block"
  → Skip block

ORC:
  - Bloom filters per column
  - Configurable

Parquet:
  - Bloom filters (newer)
  - Configurable
""")

print("\n=== Statistics Comparison ===")
print(f"{'Format':<15} {'Statistics':<25} {'Bloom':<15}")
print("-" * 55)
print(f"{'Parquet':<15} {'Row group + page':<25} {'Yes':<15}")
print(f"{'ORC':<15} {'Stripe + row':<25} {'Yes':<15}")
print(f"{'Avro':<15} {'None':<25} {'No':<15}")

print("\n=== Benefits ===")
print("""
I/O savings:
  - 10x for selective queries
  - 100x for point lookups
  - More for larger datasets

Example:
  1 TB table
  Query: WHERE id = 12345
  Statistics skip 99.9% of data
  Read: 1 GB
  Savings: 99.9%
""")

📦 Nested Data

🎯 Handling Complex Structures

Modern data is often nested: structs, arrays, maps. Columnar formats support nested data efficiently using techniques like Dremel (Parquet) and tree structures (ORC).

Nested Data

Structs: Nested records. Arrays: Lists. Maps: Key-value. Parquet: Dremel. ORC: Tree. Avro: Schema-based.

📦 Structs📦 Arrays📦 Maps📦 Dremel

# Nested Data
print("=== Complex Data Types ===")
print("""
Struct:
  {
    "user": {
      "id": 1,
      "name": "Alice",
      "address": {
        "city": "NYC",
        "zip": "10001"
      }
    }
  }

Array:
  {
    "id": 1,
    "tags": ["python", "spark", "sql"]
  }

Map:
  {
    "id": 1,
    "attributes": {
      "color": "red",
      "size": "M"
    }
  }
""")

print("\n=== Parquet Nested Support ===")
print("""
Parquet uses Dremel:
  - Repetition levels
  - Definition levels
  - Columnar nested storage

Example:
  message User {
    required int64 id;
    repeated string tags;
  }

Stored as:
  id:   [1, 2, 3]
  tags: columnar with levels
""")

print("\n=== ORC Nested Support ===")
print("""
ORC uses tree:
  - Struct type
  - List type
  - Map type
  - Union type

Example:
  CREATE TABLE users (
      id INT,
      tags ARRAY,
      attributes MAP
  )
  STORED AS ORC;
""")

print("\n=== Nested Support Comparison ===")
print(f"{'Feature':<20} {'Parquet':<20} {'ORC':<20} {'Avro':<20}")
print("-" * 80)
print(f"{'Struct':<20} {'Yes':<20} {'Yes':<20} {'Yes':<20}")
print(f"{'Array':<20} {'Yes':<20} {'Yes':<20} {'Yes':<20}")
print(f"{'Map':<20} {'Yes':<20} {'Yes':<20} {'Yes':<20}")
print(f"{'Union':<20} {'Yes':<20} {'Yes':<20} {'Yes':<20}")
print(f"{'Nested optimization':<20} {'Dremel':<20} {'Tree':<20} {'Schema':<20}")

print("\n=== When Nested Matters ===")
print("""
Nested data is common in:
  - JSON logs
  - Event data
  - ML features
  - IoT telemetry
  - Social media

Columnar formats handle nested data efficiently.
Avro is simpler but less optimized.
""")

📋 Schema Evolution

🎯 Changing Schemas Safely

Schema evolution lets you change table schemas over time without breaking existing data. Avro is the best at this, Parquet and ORC have limited support.

Schema Evolution

Avro: Excellent. Parquet: Limited. ORC: Limited. Add: Common. Rename: Avro only. Remove: Avro only.

📋 Add Column📋 Rename📋 Remove📋 Avro Best

# Schema Evolution
print("=== Schema Evolution Support ===")
print(f"{'Operation':<25} {'Parquet':<20} {'ORC':<20} {'Avro':<20}")
print("-" * 85)
print(f"{'Add column':<25} {'Limited':<20} {'Limited':<20} {'Excellent':<20}")
print(f"{'Rename column':<25} {'No':<20} {'No':<20} {'Yes':<20}")
print(f"{'Remove column':<25} {'No':<20} {'No':<20} {'Yes':<20}")
print(f"{'Type promotion':<25} {'Limited':<20} {'Limited':<20} {'Yes':<20}")
print(f"{'Reorder':<25} {'No':<20} {'No':<20} {'Yes':<20}")
print(f"{'Default values':<25} {'No':<20} {'No':<20} {'Yes':<20}")

print("\n=== Avro Schema Evolution ===")
print("""
Avro supports:
  - Add fields (with defaults)
  - Remove fields
  - Rename fields (with aliases)
  - Type promotion
  - Reorder fields

Example:
  v1:
    {"name": "id", "type": "int"}
    {"name": "name", "type": "string"}
  
  v2:
    {"name": "id", "type": "int"}
    {"name": "name", "type": "string"}
    {"name": "email", "type": ["string", "null"], "default": null}
  
  v1 data readable with v2 schema.
""")

print("\n=== Parquet Schema Evolution ===")
print("""
Parquet supports:
  - Add columns (with default)
  - Type promotion (limited)

Limitations:
  - No rename
  - No remove
  - No reorder
  - Schema embedded in each file

Workaround:
  - Use table formats (Delta, Iceberg)
  - For full schema evolution
""")

print("\n=== ORC Schema Evolution ===")
print("""
ORC supports:
  - Add columns
  - Type promotion (limited)

Limitations:
  - No rename
  - No remove
  - Schema embedded in each file

Workaround:
  - Use Hive ACID
  - Use table formats
""")

print("\n=== Best Practices ===")
print("""
1. Use Avro for streaming
   - Schema evolution
   - Kafka-friendly

2. Use Parquet/ORC for analytics
   - Columnar
   - Fast queries

3. Use table formats (Delta/Iceberg)
   - For full schema evolution
   - ACID + time travel
""")

⚖️ Full Comparison

🎯 Parquet vs ORC vs Avro

Complete comparison of the three main formats across all dimensions.

Comparison Table

Use this table as a quick reference for choosing the right format.

⚖️ Reference⚖️ Decision⚖️ Complete

# Full Comparison Table
print("=== Parquet vs ORC vs Avro ===")
print()

# Print header
print(f"{'Aspect':<25} {'Parquet':<20} {'ORC':<20} {'Avro':<20}")
print("-" * 85)

# Data
rows = [
    ('Storage', 'Columnar', 'Columnar', 'Row'),
    ('Origin', 'Twitter/Cloudera', 'Hortonworks/Facebook', 'Apache'),
    ('Year', '2013', '2013', '2009'),
    ('Nested data', 'Excellent', 'Excellent', 'Good'),
    ('Compression', 'Excellent', 'Excellent', 'Good'),
    ('Encoding', 'Dictionary, RLE, Delta', 'Dictionary, RLE, Delta', 'Basic'),
    ('Predicate pushdown', 'Row group stats', 'Stripe + row stats', 'None'),
    ('Bloom filters', 'Yes', 'Yes', 'No'),
    ('Schema evolution', 'Limited', 'Limited', 'Excellent'),
    ('ACID support', 'No', 'Yes (Hive)', 'No'),
    ('Splittable', 'Yes', 'Yes', 'Yes'),
    ('Streaming', 'No', 'No', 'Yes'),
    ('Kafka-friendly', 'No', 'No', 'Yes'),
    ('Spark support', 'Native', 'Good', 'Good'),
    ('Hive support', 'Good', 'Native', 'Good'),
    ('Presto/Trino', 'Native', 'Good', 'Limited'),
    ('Ecosystem', 'Largest', 'Hive-centric', 'Streaming'),
    ('Maturity', 'Mature', 'Mature', 'Mature'),
]

for row in rows:
    print(f"{row[0]:<25} {row[1]:<20} {row[2]:<20} {row[3]:<20}")

print()
print("=== Quick Guide ===")
print("""
Choose PARQUET when:
  - Spark ecosystem
  - Data lakes
  - Analytics
  - ML pipelines
  - S3-based

Choose ORC when:
  - Hive-centric
  - Data warehousing
  - ACID needed
  - HDFS-based
  - Optimized stats

Choose AVRO when:
  - Streaming
  - Kafka
  - Schema evolution
  - Row-based needs
  - Write-heavy
""")

print("=== Decision Flow ===")
print("""
Start
  │
  ├─ Streaming? ──Yes──→ Avro
  │                │
  │                No
  │                ↓
  ├─ Hive-centric? ──Yes──→ ORC
  │                  │
  │                  No
  │                  ↓
  ├─ Spark-centric? ──Yes──→ Parquet
  │                  │
  │                  No
  │                  ↓
  └─ Either works → Choose ecosystem
""")

🎯 Decision Guide

🎯 Choosing the Right Format

Use this guide to decide between Parquet, ORC, and Avro based on your needs.

Decision Framework

Questions: Workload? Engine? Evolution? Streaming? Answers: Lead to format. Trade-offs: Columnar vs row. Future: Table formats. Start: Parquet.

🎯 Questions🎯 Trade-offs🎯 Decision🎯 Future

# Decision Guide
print("=== Decision Questions ===")
print("""
1. WORKLOAD?
   - Analytics → Parquet/ORC
   - Streaming → Avro
   - OLTP → Row-based

2. ENGINE?
   - Spark → Parquet
   - Hive → ORC
   - Kafka → Avro

3. SCHEMA EVOLUTION?
   - Yes → Avro
   - Limited → Parquet/ORC

4. NESTED DATA?
   - Yes → Parquet/ORC
   - Simple → Any

5. COMPRESSION?
   - Important → Parquet/ORC
   - Less → Avro

6. ECOSYSTEM?
   - Spark → Parquet
   - Hive → ORC
   - Kafka → Avro
""")

print("\n=== Decision Matrix ===")
print(f"{'Requirement':<30} {'Best Choice':<20} {'Alternative':<20}")
print("-" * 70)
print(f"{'Spark analytics':<30} {'Parquet':<20} {'ORC':<20}")
print(f"{'Hive warehouse':<30} {'ORC':<20} {'Parquet':<20}")
print(f"{'Kafka streaming':<30} {'Avro':<20} {'—':<20}")
print(f"{'Schema evolution':<30} {'Avro':<20} {'—':<20}")
print(f"{'Nested data':<30} {'Parquet':<20} {'ORC':<20}")
print(f"{'Compression':<30} {'Parquet':<20} {'ORC':<20}")
print(f"{'ACID':<30} {'ORC (Hive)':<20} {'Parquet':<20}")
print(f"{'ML pipelines':<30} {'Parquet':<20} {'ORC':<20}")
print(f"{'Data lake':<30} {'Parquet':<20} {'ORC':<20}")

print("\n=== Decision Flowchart ===")
print("""
START
  │
  ├─ Streaming? ──Yes──→ Avro
  │                │
  │                No
  │                ↓
  ├─ Hive-centric? ──Yes──→ ORC
  │                  │
  │                  No
  │                  ↓
  ├─ Spark-centric? ──Yes──→ Parquet
  │                  │
  │                  No
  │                  ↓
  └─ Either works → Choose ecosystem
""")

print("\n=== Common Scenarios ===")
print("""
1. DATA LAKE ON S3
   → Parquet

2. HIVE WAREHOUSE
   → ORC

3. KAFKA PIPELINE
   → Avro

4. SPARK ML
   → Parquet

5. HDFS WAREHOUSE
   → ORC

6. STREAMING ETL
   → Avro

7. CLOUD ANALYTICS
   → Parquet

8. LEGACY HADOOP
   → ORC or Parquet
""")

💼 Applications

🎯 Real-World Use Cases

Each format excels in different industries and workloads.

Applications

Parquet: Data lakes, ML. ORC: Hive, warehouses. Avro: Kafka, streaming. Finance: Analytics. Retail: Recommendations. Healthcare: Records.

💼 Data Lakes💼 Warehouses💼 Streaming💼 Analytics

# Applications
print("=== Industry Use Cases ===")
print(f"{'Industry':<20} {'Format':<20} {'Use Case':<30}")
print("-" * 70)
print(f"{'Finance':<20} {'Parquet/ORC':<20} {'Risk, trading analytics':<30}")
print(f"{'Retail':<20} {'Parquet':<20} {'Recommendations, inventory':<30}")
print(f"{'Healthcare':<20} {'Parquet/ORC':<20} {'Records, compliance':<30}")
print(f"{'Media':<20} {'Parquet':<20} {'Streaming analytics':<30}")
print(f"{'Telecom':<20} {'Parquet':<20} {'Network analytics':<30}")
print(f"{'Tech':<20} {'Avro/Parquet':<20} {'Streaming, analytics':<30}")

print("\n=== Data Lake ===")
print("""
Use case:
  - Centralized storage
  - Analytics
  - ML
  - BI

Format:
  - Parquet (most common)

Why:
  - Columnar
  - Spark-native
  - Predicate pushdown
  - Nested data
""")

print("\n=== Data Warehouse ===")
print("""
Use case:
  - BI
  - Reporting
  - Dashboards

Format:
  - ORC (Hive)
  - Parquet (modern)

Why:
  - Columnar
  - Compression
  - Statistics
  - ACID (ORC)
""")

print("\n=== Streaming ===")
print("""
Use case:
  - Kafka
  - Real-time
  - Event pipelines

Format:
  - Avro

Why:
  - Row-based
  - Schema evolution
  - Kafka-friendly
  - Compact
""")

print("\n=== ML Pipelines ===")
print("""
Use case:
  - Feature engineering
  - Model training
  - Batch inference

Format:
  - Parquet

Why:
  - Columnar
  - Nested data
  - Spark-native
  - Fast reads
""")

🚀 Future

🎯 Where Columnar Formats Are Going

The future is convergence: table formats (Delta, Iceberg) add ACID to Parquet, open standards win, and AI drives optimization.

Future Directions

Table formats: ACID on Parquet. Open: Standards. AI: Auto-optimization. Cloud: Serverless. Real-time: Streaming. Unified: One format.

🚀 Table Formats🚀 Open🚀 AI🚀 Serverless

# Future of Columnar Formats
print("=== 1. Table Formats ===")
print("""
Trend: ACID on Parquet

Table formats:
  - Delta Lake
  - Apache Iceberg
  - Apache Hudi

What they add:
  - ACID transactions
  - Schema evolution
  - Time travel
  - Row-level updates

Future:
  - Parquet + table format = standard
  - Open formats
  - Multi-engine
""")

print("\n=== 2. Open Standards ===")
print("""
Trend: Open wins

Formats:
  - Parquet (Apache)
  - ORC (Apache)
  - Avro (Apache)
  - Arrow (Apache)

Table formats:
  - Iceberg (Apache)
  - Delta (Linux Foundation)
  - Hudi (Apache)

Future:
  - Standardization
  - Interoperability
  - No lock-in
""")

print("\n=== 3. AI and Automation ===")
print("""
Trend: AI-powered optimization

Automation:
  - Auto-compaction
  - Auto-partitioning
  - Auto-encoding
  - Auto-indexing

AI:
  - Query optimization
  - Storage optimization
  - Predictive caching

Future:
  - Self-optimizing
  - Self-tuning
  - AI-native
""")

print("\n=== 4. Serverless ===")
print("""
Trend: Serverless analytics

Services:
  - Athena (Parquet)
  - BigQuery
  - Snowflake
  - Databricks SQL

Benefits:
  - No infrastructure
  - Pay per use
  - Auto-scaling

Future:
  - Serverless default
  - Event-driven
  - Cost-effective
""")

print("\n=== 5. Real-Time ===")
print("""
Trend: Real-time columnar

Technologies:
  - Apache Arrow
  - Arrow Flight
  - Delta Live Tables

Benefits:
  - In-memory columnar
  - Zero-copy
  - Low latency

Future:
  - Real-time analytics
  - Streaming columnar
  - Unified batch + stream
""")

print("\n=== 6. Multi-Cloud ===")
print("""
Trend: Multi-cloud storage

Benefits:
  - No vendor lock-in
  - Best-of-breed
  - Disaster recovery

Formats:
  - Parquet on S3, GCS, ADLS
  - Table formats across clouds

Future:
  - Multi-cloud
  - Hybrid
  - Global
""")

# TIMELINE
print("\n=== Timeline ===")
print("""
Avro:
  2009: Created
  2010: Apache
  2024: Mature

Parquet:
  2013: Twitter + Cloudera
  2015: Apache
  2024: Dominant

ORC:
  2013: Hortonworks
  2015: Apache
  2024: Hive-native

Table formats:
  2019: Delta Lake
  2020: Iceberg
  2021: Hudi
  2024: Mainstream

Future:
  2030: One format?
  2035: AI-native?
  2040: New paradigms?
""")

print("\n=== Final Thoughts ===")
print("""
Columnar formats:
  - Parquet: Spark, lakes
  - ORC: Hive, warehouses
  - Avro: Kafka, streaming

The future is:
  - Table formats
  - Open standards
  - AI-driven
  - Serverless
  - Real-time

"Columnar is the foundation
 of modern analytics."

Keep exploring. Keep building.
""")