🧠 Druid Fundamentals

Q: What is Apache Druid?

Apache Druid is an open-source, distributed, real-time analytics database designed for fast slice-and-dice analytics on large datasets. It combines ideas from data warehouses, time-series databases, and search systems to deliver sub-second queries on streaming and batch data.

Q: What is the difference between Druid and ClickHouse?

Druid is optimized for real-time streaming ingestion, high concurrency, and pre-aggregation (rollup). ClickHouse is known for raw batch ingestion speed and broad SQL coverage. Druid excels at user-facing analytics at scale; ClickHouse is faster for batch-heavy analytical workloads.

Q: What is rollup in Druid?

Rollup is Druid's pre-aggregation feature. When enabled, rows with identical dimensions and timestamps are collapsed into a single row with aggregated metrics. This dramatically reduces storage and speeds up queries, at the cost of losing individual event details.

Q: Does Druid support exactly-once ingestion?

Yes. Druid provides exactly-once ingestion for Apache Kafka and Amazon Kinesis streams, with continuous backup into deep storage ensuring zero RPO (Recovery Point Objective).

Q: When should I use Apache Druid?

Use Druid for real-time analytics applications that require sub-second queries on streaming and batch data at high concurrency, such as clickstream analysis, IoT monitoring, network performance monitoring, and user-facing dashboards.

🎯 What is Apache Druid?

Apache Druid is a distributed, real-time analytics database built for speed and scale. It combines ideas from data warehouses, time-series databases, and search systems to deliver sub-second queries on streaming and batch data at high concurrency[citation:6][citation:18]. It powers user-facing analytics applications at companies like Airbnb, Cisco, and Salesforce[citation:9].

Why Druid Matters

Real-time: Sub-second queries on streaming data. Scale: Billions of rows, petabytes. Concurrency: Thousands of concurrent users. Ingestion: Kafka, Kinesis, batch. Rollup: Pre-aggregation for storage savings. Architecture: Microservices, independently scalable.

📌 Real-Time Analytics📌 Sub-Second Queries📌 Streaming + Batch📌 Rollup

# Apache Druid — the big picture
# Real-time analytics database for speed and scale.

# CONCEPTUAL OVERVIEW
print("""
=== Apache Druid ===

What it is:
  - Real-time analytics database
  - Distributed, columnar
  - Sub-second queries
  - Streaming + batch ingestion
  - Rollup (pre-aggregation)
  - High concurrency

Created by:
  - Metamarkets (2011)
  - Apache (2018)

Key features:
  - Kafka/Kinesis ingestion
  - SQL + JSON querying
  - Time-based partitioning
  - Inverted indexes
  - Deep storage (S3, HDFS)
  - Microservices architecture

What it's NOT:
  - Not a data warehouse (no complex joins)
  - Not for OLTP
  - Not for full-text search
  - Not for updates/deletes (append-only)

What it IS:
  - Real-time analytics
  - User-facing dashboards
  - Time-series analysis
  - Clickstream, IoT, APM
""")

# WHY IT MATTERS
print("=== Why Druid Matters ===")
print("""
Challenges:
  - Real-time data
  - High concurrency
  - Sub-second latency
  - Billions of rows
  - Streaming + batch

Druid solves:
  - Sub-second queries
  - High concurrency
  - Real-time ingestion
  - Scale
  - Storage efficiency (rollup)
""")

# REAL-WORLD
print("=== Real-World Scale ===")
print("""
Companies using Druid:
  - Airbnb
  - Alibaba
  - Cisco
  - Netflix
  - Salesforce
  - Shopify
  - Verizon
  - Nielsen

Scale examples:
  - The R&A: 25B+ rows, sub-second queries [citation:2]
  - 250M new rows every 15 minutes [citation:2]
  - 100% uptime during global broadcast [citation:2]
""")

🏗️ Architecture

🎯 Microservices Architecture

Druid's architecture is a disassembled database: each core service (ingestion, querying, coordination) can be independently deployed and scaled. Services can fail without impacting others[citation:7][citation:18].

Architecture

Master: Coordinator + Overlord. Query: Broker + Router. Data: Historical + MiddleManager + Peon. Deep Storage: S3, HDFS. Metadata: PostgreSQL, MySQL. Coordination: ZooKeeper.

🏗️ Master🏗️ Query🏗️ Data🏗️ Deep Storage

# Druid Architecture
print("=== Architecture Diagram ===")
print("""
  ┌─────────────────────────────────────────┐
  │              Clients                    │
  │         (BI, Apps, SQL)                 │
  └─────────────────┬───────────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
        ▼           ▼           ▼
  ┌─────────┐ ┌─────────┐ ┌─────────┐
  │ Broker  │ │ Router  │ │ Web UI  │
  │ (Query) │ │ (Proxy) │ │         │
  └────┬────┘ └────┬────┘ └─────────┘
       │           │
       └───────────┼───────────┐
                   │           │
                   ▼           ▼
  ┌─────────────────────────────────────────┐
  │         Coordinator / Overlord          │
  │         (Master Services)               │
  │                                         │
  │  - Segment management                   │
  │  - Task assignment                      │
  │  - Ingestion control                    │
  └─────────────────┬───────────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
        ▼           ▼           ▼
  ┌─────────┐ ┌─────────┐ ┌─────────┐
  │Historical│ │MiddleMgr│ │Indexer  │
  │(Data)   │ │(Ingest) │ │(Ingest) │
  └────┬────┘ └────┬────┘ └────┬────┘
       │           │           │
       └───────────┼───────────┘
                   │
                   ▼
  ┌─────────────────────────────────────────┐
  │           Deep Storage                  │
  │         (S3, HDFS, GCS)                 │
  └─────────────────────────────────────────┘
""")

print("\n=== Services ===")
print("""
MASTER SERVICES:
  Coordinator:
    - Manages data availability
    - Assigns segments to Historicals
    - Balances segments

  Overlord:
    - Controls ingestion workloads
    - Assigns tasks to MiddleManagers
    - Coordinates segment publishing

QUERY SERVICES:
  Broker:
    - Receives queries from clients
    - Routes to Data servers
    - Merges results

  Router:
    - API gateway
    - Routes to Brokers, Coordinators, Overlords
    - Runs web console

DATA SERVICES:
  Historical:
    - Stores queryable data
    - Downloads segments from deep storage
    - Responds to queries
    - No writes

  MiddleManager:
    - Handles ingestion
    - Spawns Peons
    - Reads from sources

  Peon:
    - Task execution engine
    - Single task per Peon
    - Runs on same host as MiddleManager

  Indexer (optional):
    - Alternative to MiddleManager + Peon
    - Tasks as threads in single JVM
    - Easier to configure
""")

print("\n=== Server Types ===")
print(f"{'Server':<20} {'Services':<40}")
print("-" * 60)
print(f"{'Master':<20} {'Coordinator + Overlord':<40}")
print(f"{'Query':<20} {'Broker + Router':<40}")
print(f"{'Data':<20} {'Historical + MiddleManager':<40}")

print("\n=== Deep Storage ===")
print("""
Deep Storage:
  - S3
  - HDFS
  - Google Cloud Storage
  - Azure Blob

Purpose:
  - Durable data storage
  - Backup
  - Segment handoff
  - Recovery

Metadata Store:
  - PostgreSQL
  - MySQL
  - Derby (dev only)

Coordination:
  - ZooKeeper
  - Consul (newer)
""")

📥 Ingestion

🎯 Getting Data In

Druid supports both streaming (Kafka, Kinesis) and batch (S3, HDFS, local) ingestion. Data is indexed, partitioned, and optionally pre-aggregated as it's ingested[citation:6][citation:17].

Ingestion

Streaming: Kafka, Kinesis. Batch: S3, HDFS, files. Exactly-once: Streaming. Formats: JSON, CSV, Parquet, ORC. Rollup: Pre-aggregation. SQL: INSERT, REPLACE.

📥 Streaming📥 Batch📥 Exactly-Once📥 Rollup

# Ingestion
print("=== Streaming Ingestion ===")
print("""
Kafka:
  - Native pull-based
  - Exactly-once
  - Zero RPO
  - Auto-scaling

Kinesis:
  - Native support
  - Exactly-once
  - AWS

Supervisor:
  - Manages tasks
  - Autoscaling
  - Handles failures

Example (Kafka):
  {
    "type": "kafka",
    "dataSchema": {
      "dataSource": "events",
      "timestampSpec": {"column": "timestamp", "format": "iso"},
      "dimensionsSpec": {
        "dimensions": ["user_id", "event_type"]
      },
      "metricsSpec": [
        {"name": "count", "type": "count"},
        {"name": "sum_amount", "type": "doubleSum", "fieldName": "amount"}
      ],
      "granularitySpec": {
        "type": "uniform",
        "segmentGranularity": "HOUR",
        "queryGranularity": "MINUTE",
        "rollup": true
      }
    },
    "ioConfig": {
      "topic": "events",
      "consumerProperties": {"bootstrap.servers": "localhost:9092"}
    }
  }
""")

print("\n=== Batch Ingestion ===")
print("""
Native Batch:
  - index_parallel task
  - S3, HDFS, local
  - JSON, CSV, Parquet, ORC
  - Parallelism via subtasks

SQL:
  - INSERT INTO ... SELECT
  - REPLACE
  - MSQ task engine

Hadoop:
  - index_hadoop
  - YARN
  - MapReduce

Example (SQL):
  INSERT INTO events
  SELECT * FROM TABLE(
    EXTERN(
      '{"type":"s3","uris":["s3://bucket/events/*.json"]}',
      '{"type":"json"}',
      '[{"name":"timestamp","type":"string"},{"name":"user_id","type":"long"}]'
    )
  )
""")

print("\n=== Ingestion Methods ===")
print(f"{'Method':<20} {'Source':<25} {'Format':<20}")
print("-" * 65)
print(f"{'Kafka':<20} {'Kafka':<25} {'JSON, Avro':<20}")
print(f"{'Kinesis':<20} {'Kinesis':<25} {'JSON':<20}")
print(f"{'Native Batch':<20} {'S3, HDFS, local':<25} {'JSON, CSV, Parquet':<20}")
print(f"{'SQL':<20} {'S3, HDFS, Druid':<25} {'Any':<20}")
print(f"{'Hadoop':<20} {'HDFS':<25} {'Any':<20}")

print("\n=== Rollup ===")
print("""
Rollup:
  - Pre-aggregation
  - Rows with same dimensions + timestamp collapsed
  - Metrics aggregated
  - Storage savings
  - Default: enabled

Example:
  Before:
    (12:00, user1, click, 1)
    (12:00, user1, click, 1)
    (12:00, user1, view, 1)

  After (rollup):
    (12:00, user1, click, 2)
    (12:00, user1, view, 1)

Savings:
  - Orders of magnitude
  - Depends on cardinality
  - Trade-off: no individual events
""")

print("\n=== Exactly-Once ===")
print("""
Streaming:
  - Exactly-once
  - Kafka offsets tracked
  - Continuous backup to deep storage
  - Zero RPO

Batch:
  - At-least-once
  - Idempotent writes
  - Versioned segments

Guarantees:
  - No data loss
  - No duplicates
  - Durable
""")

📦 Segments & Storage

🎯 How Data is Stored

Druid stores data in segments — column-oriented files containing up to a few million rows each. Segments are partitioned by time and optionally by other dimensions, compressed, and indexed for fast queries[citation:1][citation:6].

Segments

Columnar: Column-oriented. Compressed: Dictionary, bitmap. Partitioned: By time. Immutable: Append-only. Versioned: MVCC. Deep Storage: Backed up.

📦 Columnar📦 Compressed📦 Partitioned📦 Immutable

# Segments & Storage
print("=== What is a Segment? ===")
print("""
Segment:
  - Single file
  - Few million rows
  - Column-oriented
  - Compressed
  - Indexed
  - Immutable

Identifier:
  datasource_interval_version_partition

Example:
  clarity-cloud0_2018-05-21T16:00:00.000Z_2018-05-21T17:00:00.000Z_2018-05-21T15:56:09.909Z_1
""")

print("\n=== Segment Lifecycle ===")
print("""
1. CREATE
   - Task creates segment
   - Data stored in columnar format

2. DEEP STORAGE
   - Segment pushed to S3/HDFS
   - Durable backup

3. PUBLISH
   - Metadata stored in metadata store
   - Flag: used = true

4. AVAILABLE
   - Loaded by Historical
   - Queryable

5. VERSION
   - New version overwrites
   - Old version dropped

6. DELETE
   - Retention rules
   - Removed from cluster
""")

print("\n=== Storage Format ===")
print("""
Column-oriented:
  - Each column stored separately
  - Only needed columns read
  - Better compression

Encoding:
  - Dictionary (strings)
  - Raw (numbers)
  - Compressed

Indexes:
  - Inverted index (strings)
  - Bitmap index
  - Time index

Partitioning:
  - Time-based (primary)
  - Secondary (hash, range)
  - Dynamic
""")

print("\n=== Segment States ===")
print("""
sys.segments table:
  - is_published: metadata published
  - is_available: loaded for querying
  - is_realtime: only on realtime tasks
  - is_overshadowed: replaced by newer version

Query:
  SELECT * FROM sys.segments
  WHERE datasource = 'events';
""")

print("\n=== Deep Storage ===")
print("""
Purpose:
  - Durable storage
  - Backup
  - Segment handoff
  - Recovery

Options:
  - S3
  - HDFS
  - GCS
  - Azure Blob
  - Local (dev)

Config:
  druid.storage.type=s3
  druid.storage.bucket=my-bucket
  druid.storage.baseKey=druid/segments
""")

print("\n=== Retention ===")
print("""
Rules:
  - Time-based
  - Period: P30D (30 days)
  - Tier: hot, cold
  - Delete: auto

Example:
  {
    "type": "load",
    "tieredReplicants": {"hot": 2, "cold": 1},
    "period": "P30D"
  }

  {
    "type": "drop",
    "period": "P90D"
  }
""")

📊 Rollup & Pre-Aggregation

🎯 Storage Savings

Rollup is Druid's pre-aggregation feature. Rows with identical dimensions and timestamps are collapsed into a single row with aggregated metrics. This can reduce storage by orders of magnitude[citation:12].

Rollup

Pre-aggregate: At ingestion. Collapse: Same dims + time. Metrics: Sum, min, max, count. Savings: Orders of magnitude. Trade-off: No individual events. Default: Enabled.

📊 Pre-Aggregate📊 Collapse📊 Savings📊 Trade-off

# Rollup
print("=== What is Rollup? ===")
print("""
Rollup:
  - Pre-aggregation
  - At ingestion time
  - Rows with same dimensions + timestamp collapsed
  - Metrics aggregated

Example:
  Input:
    (12:00, user1, click, 1)
    (12:00, user1, click, 1)
    (12:00, user1, view, 1)
    (12:00, user2, click, 1)

  Rolled up:
    (12:00, user1, click, 2)
    (12:00, user1, view, 1)
    (12:00, user2, click, 1)

  Savings: 4 → 3 rows (25%)
  With more data: orders of magnitude
""")

print("\n=== Enable/Disable ===")
print("""
granularitySpec:
  rollup: true   # Default

To disable:
  rollup: false

Trade-off:
  Enabled: Less storage, faster queries
  Disabled: Full event detail
""")

print("\n=== Maximizing Rollup ===")
print("""
Tips:
  1. Fewer dimensions
     - Only what's needed

  2. Lower cardinality
     - Fewer unique values
     - Better rollup

  3. Sketches
     - Avoid high cardinality
     - Use theta sketches

  4. Right granularity
     - queryGranularity
     - Balance

Example:
  SELECT SUM("cnt") / COUNT(*) * 1.0
  FROM datasource;
  
  Higher ratio = better rollup
""")

print("\n=== Metrics ===")
print("""
Aggregators:
  - count
  - longSum
  - doubleSum
  - longMin
  - doubleMax
  - thetaSketch
  - HLLSketch

Example:
  metricsSpec:
    - name: count
      type: count
    - name: sum_amount
      type: doubleSum
      fieldName: amount
    - name: unique_users
      type: thetaSketch
      fieldName: user_id
""")

print("\n=== Benefits ===")
print("""
1. STORAGE
   - Less data
   - Cheaper

2. QUERIES
   - Faster
   - Less data to scan

3. MEMORY
   - Less RAM
   - More in cache

4. COST
   - Less storage
   - Less compute
""")

🔍 Querying

🎯 Fast Queries

Druid supports SQL and JSON-over-HTTP querying. Its scatter/gather architecture enables sub-second performance on billions of rows[citation:6][citation:18].

Querying

SQL: Standard SQL. JSON: Native. Scatter: Parallel. Gather: Merge. Pruning: Segment. Sub-second: Fast.

🔍 SQL🔍 JSON🔍 Scatter/Gather🔍 Sub-second

# Querying
print("=== SQL Querying ===")
print("""
Basic:
  SELECT * FROM events LIMIT 10;

Aggregation:
  SELECT
    event_type,
    COUNT(*) as count,
    SUM(amount) as revenue
  FROM events
  WHERE __time >= CURRENT_TIMESTAMP - INTERVAL '1' HOUR
  GROUP BY event_type
  ORDER BY count DESC;

Time-series:
  SELECT
    TIME_FLOOR(__time, 'PT1M') as minute,
    COUNT(*) as events
  FROM events
  GROUP BY 1
  ORDER BY 1;

Approximate:
  SELECT
    APPROX_COUNT_DISTINCT(user_id) as unique_users,
    APPROX_QUANTILE(latency, 0.95) as p95
  FROM events;
""")

print("\n=== Query Lifecycle ===")
print("""
1. Client → Broker
2. Broker → Segment pruning
   - Time intervals
   - Filters
   - Only relevant segments

3. Broker → Data nodes (scatter)
   - Parallel sub-queries
   - Historical + Realtime

4. Data nodes → Process
   - Columnar scan
   - Index lookup
   - Aggregation

5. Broker ← Results (gather)
   - Merge
   - Final aggregation
   - Return to client
""")

print("\n=== Query Types ===")
print(f"{'Type':<20} {'Use Case':<40}")
print("-" * 65)
print(f"{'Timeseries':<20} {'Time-based aggregations':<40}")
print(f"{'TopN':<20} {'Top N by metric':<40}")
print(f"{'GroupBy':<20} {'Multi-dimensional grouping':<40}")
print(f"{'Scan':<20} {'Raw data retrieval':<40}")
print(f"{'Search':<20} {'Full-text search':<40}")

print("\n=== SQL vs JSON ===")
print("""
SQL:
  - Standard
  - Familiar
  - Tool compatibility

JSON:
  - Native
  - Full control
  - Advanced features

Example JSON:
  {
    "queryType": "groupBy",
    "dataSource": "events",
    "granularity": "minute",
    "dimensions": ["event_type"],
    "aggregations": [
      {"type": "count", "name": "count"},
      {"type": "doubleSum", "name": "revenue", "fieldName": "amount"}
    ],
    "intervals": ["2024-01-01/2024-01-02"]
  }
""")

print("\n=== Caching ===")
print("""
Caching:
  - Query cache (Broker)
  - Segment cache (Historical)
  - Result cache

Benefits:
  - Faster repeated queries
  - Less compute
  - Lower cost

Cache:
  - Per segment
  - LRU eviction
  - Configurable
""")

print("\n=== Best Practices ===")
print("""
1. Use SQL for compatibility
2. Use JSON for advanced
3. Filter by time
4. Use approximate algorithms
5. Limit dimensions
6. Cache results
7. Monitor query performance
""")

🔢 Indexes

🎯 Fast Filtering

Druid builds specialized indexes for different column types: inverted indexes for strings, bitmap indexes for fast filtering, and time indexes for efficient time-range queries[citation:6].

Indexes

Inverted: String columns. Bitmap: Fast filtering. Time: Time ranges. Dictionary: Compression. Spatial: Geo. Range: Numeric.

🔢 Inverted🔢 Bitmap🔢 Time🔢 Dictionary

# Indexes
print("=== Index Types ===")
print("""
1. INVERTED INDEX
   - String columns
   - Term → row IDs
   - Fast equality
   - Fast search

2. BITMAP INDEX
   - Compressed bitmaps
   - Fast filtering
   - AND/OR operations
   - Memory efficient

3. TIME INDEX
   - Time-range queries
   - Partition pruning
   - Fast

4. DICTIONARY ENCODING
   - Strings → IDs
   - Compression
   - Fast equality

5. RANGE INDEX
   - Numeric columns
   - Min/max
   - Range pruning
""")

print("\n=== How Indexes Work ===")
print("""
Column: event_type

Values: [click, view, click, purchase, click]

Dictionary:
  click → 0
  view → 1
  purchase → 2

Encoded: [0, 1, 0, 2, 0]

Inverted Index:
  click → [0, 2, 4]
  view → [1]
  purchase → [3]

Query: WHERE event_type = 'click'
  → Bitmap [1, 0, 1, 0, 1]
  → Fast!
""")

print("\n=== Bitmap Index ===")
print("""
Bitmap:
  - One bit per row
  - 1 = matches
  - 0 = doesn't

Example:
  click → 10101
  view  → 01000

Query: WHERE type = 'click' AND type = 'view'
  → Impossible
  → Empty

Query: WHERE type = 'click' OR type = 'view'
  → 10101 OR 01000 = 11101
  → Fast!
""")

print("\n=== Configuring Indexes ===")
print("""
At ingestion:
  dimensionsSpec:
    dimensions:
      - name: event_type
        type: string
        createBitmapIndex: true

Note:
  - Bitmap index: default for strings
  - Disable for high cardinality
  - Memory trade-off
""")

print("\n=== Benefits ===")
print("""
1. SPEED
   - Fast filtering
   - Fast aggregation
   - Sub-second

2. COMPRESSION
   - Dictionary encoding
   - Bitmap compression
   - Less storage

3. MEMORY
   - Loaded in memory
   - Fast access

4. FLEXIBILITY
   - Ad-hoc queries
   - Multiple dimensions
""")

🚢 Deployment

🎯 Running Druid

Druid can be deployed in various configurations: single-server (dev), cluster (production), and Kubernetes[citation:4][citation:11].

Deployment

Quickstart: Single server. Cluster: Multiple servers. Kubernetes: Operator. Cloud: Managed. Docker: Compose. Imply: Managed.

🚢 Quickstart🚢 Cluster🚢 Kubernetes🚢 Managed

# Deployment
print("=== Quickstart (Single Server) ===")
print("""
Requirements:
  - 6 GiB RAM
  - Java 17
  - Linux/Mac (not Windows)

Install:
  tar -xzf apache-druid-37.0.0-bin.tar.gz
  cd apache-druid-37.0.0
  ./bin/start-druid

Access:
  http://localhost:8888/

Services started:
  - ZooKeeper
  - Broker
  - Router
  - Coordinator/Overlord
  - Historical
  - MiddleManager
""")

print("\n=== Docker Compose ===")
print("""
Files:
  - docker-compose.yml
  - environment

Run:
  docker-compose up -d

Services:
  - ZooKeeper
  - PostgreSQL (metadata)
  - 6 Druid containers

Memory:
  - 6 GiB minimum
  - 7 GiB per service

Single-node configs:
  - nano-quickstart
  - micro-quickstart
  - small
  - medium
  - large
  - xlarge
""")

print("\n=== Cluster Deployment ===")
print("""
Servers:
  Master:
    - Coordinator
    - Overlord

  Query:
    - Broker
    - Router

  Data:
    - Historical
    - MiddleManager

Sizing:
  - Coordinator: medium
  - Overlord: medium
  - Broker: large
  - Historical: large
  - MiddleManager: large

Scaling:
  - Add Historicals (query scale)
  - Add MiddleManagers (ingest scale)
  - Add Brokers (concurrency)
""")

print("\n=== Kubernetes ===")
print("""
Druid Operator:
  - Kubernetes controller
  - Manages Druid cluster
  - CRDs
  - Helm chart

Example:
  apiVersion: druid.apache.org/v1alpha1
  kind: Druid
  metadata:
    name: druid-cluster
  spec:
    nodes:
      brokers:
        replicas: 3
      historicals:
        replicas: 5
      middleManagers:
        replicas: 3
""")

print("\n=== Managed Services ===")
print("""
Imply:
  - Druid as a service
  - Managed
  - Free tier

Confluent:
  - Not Druid
  - Kafka

AWS:
  - Self-managed
  - EMR

GCP:
  - Self-managed

Azure:
  - Self-managed
""")

print("\n=== Best Practices ===")
print("""
1. Separate servers
   - Master
   - Query
   - Data

2. Sizing
   - RAM for segments
   - CPU for queries
   - Disk for cache

3. Deep storage
   - S3/HDFS
   - Durable

4. Metadata store
   - PostgreSQL
   - HA

5. ZooKeeper
   - 3+ nodes
   - Separate

6. Monitoring
   - Prometheus
   - Grafana

7. Backup
   - Deep storage
   - Metadata

8. Security
   - TLS
   - Auth
   - ACLs
""")

🔧 Operations

🎯 Running in Production

Operating Druid requires monitoring, tuning, and maintenance. Key areas: JVM tuning, segment management, and query optimization.

Operations

Monitoring: Metrics, logs. Tuning: JVM, memory. Segments: Compaction. Queries: Optimization. Backup: Deep storage. Upgrade: Rolling.

🔧 Monitoring🔧 Tuning🔧 Compaction🔧 Upgrade

# Operations
print("=== Monitoring ===")
print("""
Metrics:
  - Query latency
  - Query throughput
  - Ingestion rate
  - Segment count
  - JVM heap
  - GC pauses
  - Disk usage
  - Cache hit rate

Tools:
  - Prometheus
  - Grafana
  - Datadog
  - New Relic

Endpoints:
  - /status
  - /druid/coordinator/v1/loadstatus
  - /druid/broker/v1/loadstatus
""")

print("\n=== JVM Tuning ===")
print("""
Heap:
  - 50% of RAM
  - Historical: large
  - Broker: medium

Direct memory:
  - Segment cache
  - Query processing

Example:
  -Xms8g -Xmx8g
  -XX:MaxDirectMemorySize=16g
  -XX:+UseG1GC

GC:
  - G1GC
  - Tune pause
  - Monitor
""")

print("\n=== Segment Management ===")
print("""
Compaction:
  - Merge small segments
  - Reduce overhead
  - Improve queries

Auto-compaction:
  - Coordinator
  - Configurable

Manual:
  POST /druid/coordinator/v1/compact

Retention:
  - Load rules
  - Drop rules
  - Tiered storage
""")

print("\n=== Query Optimization ===")
print("""
1. Time filters
   - Always filter by time
   - Segment pruning

2. Limit dimensions
   - Fewer dimensions
   - Better performance

3. Approximate algorithms
   - Theta sketches
   - HLL sketches
   - Faster

4. Caching
   - Query cache
   - Segment cache
   - Result cache

5. Query priority
   - Druid supports
   - Priority levels
""")

print("\n=== Backup ===")
print("""
What to backup:
  - Deep storage (segments)
  - Metadata store
  - ZooKeeper
  - Config

Frequency:
  - Continuous (deep storage)
  - Daily (metadata)
  - On change (config)

Restore:
  - Restore files
  - Restart
  - Verify
""")

print("\n=== Upgrade ===")
print("""
Rolling upgrade:
  1. Backup
  2. Upgrade Coordinator/Overlord
  3. Upgrade Brokers
  4. Upgrade Historicals
  5. Upgrade MiddleManagers
  6. Verify

Or:
  - Blue/green
  - New cluster
  - Switch
""")

print("\n=== Best Practices ===")
print("""
1. Monitor everything
2. Alert on issues
3. Tune JVM
4. Compact segments
5. Retention rules
6. Backup
7. Test upgrades
8. Document
9. Capacity planning
10. Security
""")

💼 Use Cases

🎯 Where Druid Excels

Druid is ideal for real-time analytics applications that require sub-second queries at scale and high concurrency[citation:2][citation:8].

Use Cases

Clickstream: User behavior. IoT: Sensor data. APM: Network monitoring. Dashboards: User-facing. Fraud: Detection. Observability: Real-time.

💼 Clickstream💼 IoT💼 APM💼 Dashboards

# Use Cases
print("=== Clickstream Analytics ===")
print("""
Use case:
  - User behavior
  - Page views
  - Conversion
  - A/B testing

Data:
  - Events from web/mobile
  - Kafka streaming
  - Billions of events

Queries:
  - Real-time dashboards
  - Funnel analysis
  - User segmentation

Value:
  - Sub-second insights
  - High concurrency
  - User-facing
""")

print("\n=== IoT Monitoring ===")
print("""
Use case:
  - Sensor data
  - Device metrics
  - Anomaly detection

Data:
  - MQTT → Kafka
  - Millions of devices
  - Real-time

Queries:
  - Real-time monitoring
  - Alerts
  - Dashboards

Value:
  - Sub-second
  - High concurrency
  - Scalable
""")

print("\n=== Network Performance ===")
print("""
Use case:
  - Network traffic
  - Latency
  - Errors
  - Capacity

Data:
  - Flow logs
  - SNMP
  - Streaming

Queries:
  - Real-time monitoring
  - Root cause
  - Capacity planning

Value:
  - Sub-second
  - High concurrency
  - Time-series
""")

print("\n=== User-Facing Dashboards ===")
print("""
Use case:
  - Analytics dashboards
  - Customer-facing
  - Internal BI

Data:
  - Event streams
  - Batch imports

Queries:
  - Interactive
  - Ad-hoc
  - Scheduled

Value:
  - Sub-second
  - Many users
  - Fresh data
""")

print("\n=== Real-World Example ===")
print("""
The R&A (The Open Championship):
  - 25B+ rows
  - 250M rows per 15 minutes
  - Sub-second queries
  - 100% uptime
  - Millions of fans

Use case:
  - Real-time scoring
  - Leaderboards
  - Fan engagement
  - Broadcast feeds

Result:
  - Flawless coverage
  - Instant updates
  - Global scale
  [citation:2]
""")

print("\n=== When to Use Druid ===")
print("""
USE DRUID:
  ✓ Real-time analytics
  ✓ Sub-second queries
  ✓ High concurrency
  ✓ Streaming + batch
  ✓ Time-series
  ✓ User-facing dashboards
  ✓ Clickstream, IoT, APM

AVOID:
  ✗ Complex joins
  ✗ OLTP
  ✗ Full-text search
  ✗ Frequent updates/deletes
  ✗ Small data
""")

⚖️ vs Alternatives

🎯 Druid vs ClickHouse vs Pinot

Druid competes with ClickHouse and Pinot in the real-time OLAP space. Each has strengths and trade-offs[citation:3][citation:9][citation:15].

Comparison

Druid: Real-time, concurrency. ClickHouse: Batch speed, SQL. Pinot: Indexing, LinkedIn. Druid: Rollup, streaming. ClickHouse: Raw speed. Pinot: User-facing.

⚖️ Druid⚖️ ClickHouse⚖️ Pinot⚖️ Real-Time

Aspect	Apache Druid	ClickHouse	Apache Pinot
Origin	Metamarkets	Yandex	LinkedIn
Primary Use	Real-time analytics	Batch analytics	User-facing analytics
Streaming	Kafka, Kinesis (native)	Kafka (table engine)	Kafka (native)
Rollup	Built-in (default)	Materialized views	Pre-aggregation
Query Latency	Sub-second	Sub-second	Sub-second
Concurrency	Very high	High	Very high
SQL	Yes (partial)	Full SQL	Partial SQL
Joins	Limited	Full	Limited
Updates	No (append-only)	Limited (batch)	No (append-only)
Upserts	No	Limited	Yes (CDC)
Indexes	Inverted, bitmap	Sparse, skip	Inverted, star-tree
Storage	Segments, deep storage	MergeTree	Segments, deep storage
Architecture	Microservices	Monolithic	Microservices
Best For	Real-time, concurrency	Batch speed, SQL	User-facing, indexing

# When to Use Which
print("=== Use Druid WHEN ===")
print("""
✓ Real-time analytics
✓ Streaming ingestion
✓ High concurrency
✓ Sub-second queries
✓ Rollup (pre-aggregation)
✓ Time-series
✓ User-facing dashboards
✓ Clickstream, IoT, APM

Companies:
  - Airbnb
  - Cisco
  - Netflix
  - Salesforce
""")

print("\n=== Use ClickHouse WHEN ===")
print("""
✓ Batch analytics
✓ Full SQL
✓ Fast aggregation
✓ Raw speed
✓ Data warehousing
✓ Log analysis
✓ Large joins

Companies:
  - Yandex
  - Cloudflare
  - Uber
  - eBay
""")

print("\n=== Use Pinot WHEN ===")
print("""
✓ User-facing analytics
✓ High concurrency
✓ Single query shape
✓ LinkedIn scale
✓ Indexing
✓ Upserts (CDC)

Companies:
  - LinkedIn
  - Uber
  - Walmart
  - Microsoft Teams
""")

print("\n=== Decision Flow ===")
print("""
Start
  │
  ├─ Streaming? ──Yes──→ Druid
  │                │
  │                No
  │                ↓
  ├─ Full SQL? ──Yes──→ ClickHouse
  │              │
  │              No
  │              ↓
  ├─ User-facing? ──Yes──→ Pinot
  │                  │
  │                  No
  │                  ↓
  └─ Real-time? ──Yes──→ Druid
""")

print("\n=== Combined ===")
print("""
Many companies use multiple:
  - Druid: Real-time dashboards
  - ClickHouse: Batch analytics
  - Pinot: User-facing

Example:
  - Kafka → Druid (real-time)
  - S3 → ClickHouse (batch)
  - Both → Dashboards
""")

✅ Best Practices

🎯 Doing Druid Right

Follow these best practices for effective real-time analytics with Apache Druid.

Best Practices

Rollup: Enable. Dimensions: Minimize. Time: Filter. Metrics: Choose. Segments: Compact. Monitor.

✅ Rollup✅ Dimensions✅ Time Filter✅ Monitor

# Best Practices
print("=== Schema Design ===")
print("""
1. ENABLE ROLLUP
   - Default: true
   - Storage savings
   - Query speed

2. MINIMIZE DIMENSIONS
   - Only what's needed
   - Lower cardinality
   - Better rollup

3. CHOOSE METRICS
   - Count
   - Sum
   - Min/max
   - Sketches (approximate)

4. TIMESTAMP
   - Primary timestamp
   - __time column
   - Granularity

5. GRANULARITY
   - segmentGranularity
   - queryGranularity
   - Balance
""")

print("\n=== Ingestion ===")
print("""
1. STREAMING
   - Kafka/Kinesis
   - Exactly-once
   - Auto-scaling
   - Handoff

2. BATCH
   - S3/HDFS
   - Parallel
   - SQL or native
   - Idempotent

3. FORMATS
   - JSON
   - CSV
   - Parquet
   - ORC

4. VALIDATION
   - Schema
   - Types
   - Timestamps
""")

print("\n=== Querying ===")
print("""
1. FILTER BY TIME
   - Always
   - Segment pruning

2. LIMIT DIMENSIONS
   - Fewer = faster

3. APPROXIMATE
   - Theta sketches
   - HLL
   - Faster

4. CACHE
   - Query cache
   - Segment cache
   - Result cache

5. PRIORITIZE
   - Query priority
   - Important first
""")

print("\n=== Operations ===")
print("""
1. MONITOR
   - Metrics
   - Logs
   - Alerts

2. COMPACT
   - Merge small segments
   - Reduce overhead

3. RETENTION
   - Load rules
   - Drop rules

4. BACKUP
   - Deep storage
   - Metadata

5. UPGRADE
   - Rolling
   - Test
""")

print("\n=== Anti-Patterns ===")
print("""
AVOID:
  ✗ No rollup
  ✗ Too many dimensions
  ✗ No time filter
  ✗ Complex joins
  ✗ Small segments
  ✗ No monitoring
  ✗ No backup
  ✗ No retention
""")

print("\n=== Checklist ===")
print("""
✓ Rollup enabled
✓ Dimensions minimized
✓ Metrics chosen
✓ Time filters
✓ Approximate algorithms
✓ Caching
✓ Monitoring
✓ Compaction
✓ Retention
✓ Backup
✓ Security
✓ Documentation
""")

❓ FAQ

🎯 Frequently Asked Questions

Answers to common questions about Apache Druid.

What is Apache Druid?

Apache Druid is an open-source, distributed, real-time analytics database designed for fast slice-and-dice analytics on large datasets. It combines ideas from data warehouses, time-series databases, and search systems to deliver sub-second queries on streaming and batch data[citation:6][citation:18].

What is the difference between Druid and ClickHouse?

Druid is optimized for real-time streaming ingestion, high concurrency, and pre-aggregation (rollup). ClickHouse is known for raw batch ingestion speed and broad SQL coverage. Druid excels at user-facing analytics at scale; ClickHouse is faster for batch-heavy analytical workloads[citation:3][citation:9].

What is rollup in Druid?

Rollup is Druid's pre-aggregation feature. When enabled, rows with identical dimensions and timestamps are collapsed into a single row with aggregated metrics. This dramatically reduces storage and speeds up queries, at the cost of losing individual event details[citation:12].

Does Druid support exactly-once ingestion?

Yes. Druid provides exactly-once ingestion for Apache Kafka and Amazon Kinesis streams, with continuous backup into deep storage ensuring zero RPO (Recovery Point Objective)[citation:6].

When should I use Apache Druid?

Use Druid for real-time analytics applications that require sub-second queries on streaming and batch data at high concurrency, such as clickstream analysis, IoT monitoring, network performance monitoring, and user-facing dashboards[citation:6][citation:9].

What is a segment in Druid?

A segment is the fundamental storage unit in Druid — a column-oriented file containing up to a few million rows. Segments are partitioned by time, compressed, indexed, and stored in deep storage (S3, HDFS)[citation:1][citation:6].

What is the difference between Druid and Pinot?

Both are real-time OLAP databases. Druid emphasizes rollup, streaming ingestion, and broad use cases. Pinot emphasizes per-column indexing, single-query-shape optimization, and was built by LinkedIn for user-facing analytics[citation:3][citation:9].

How does Druid achieve sub-second queries?

Druid uses scatter/gather parallel processing, columnar storage with compression, bitmap indexes for fast filtering, and segment pruning based on time intervals and filters. Queries only read relevant columns and segments[citation:6].

Can Druid handle updates and deletes?

No, not in the traditional sense. Druid is append-only. You can overwrite data by reingesting with a higher version number, but individual row updates/deletes are not supported. This is a trade-off for performance[citation:1].

How many servers does Druid need?

Druid can run on a single server for development (quickstart). Production clusters typically use 3 server types: Master (Coordinator + Overlord), Query (Broker + Router), and Data (Historical + MiddleManager). Cluster size scales to thousands of nodes[citation:7].

# Quick Reference
print("=== Quick Answers ===")
print("""
Q: What is Druid?
A: Real-time analytics database.

Q: Sub-second queries?
A: Yes — designed for it.

Q: Streaming?
A: Kafka, Kinesis.

Q: Rollup?
A: Pre-aggregation, default on.

Q: Updates?
A: No — append-only.

Q: SQL?
A: Yes — partial support.

Q: Joins?
A: Limited.

Q: Kubernetes?
A: Yes — Druid Operator.

Q: Managed?
A: Imply.

Q: Alternative?
A: ClickHouse, Pinot.

Q: Best for?
A: Real-time, high concurrency.
""")

🚀 Future

🎯 Where Druid Is Going

The future is AI-augmented, cloud-native, Kubernetes-first, and more integrated with the modern data stack.

Future Directions

AI: Auto-optimization. Cloud: Managed. K8s: Native. Integration: Lakehouse. Performance: Faster. Open: Standards.

🚀 AI🚀 Cloud🚀 K8s🚀 Integration

# Future of Druid
print("=== 1. AI-Augmented ===")
print("""
Trend: AI in Druid

AI:
  - Auto-optimization
  - Query recommendations
  - Anomaly detection
  - Auto-scaling

Future:
  - Self-tuning
  - Predictive
  - Autonomous
""")

print("\n=== 2. Cloud-Native ===")
print("""
Trend: Cloud-first

Managed:
  - Imply
  - AWS
  - GCP
  - Azure

Serverless:
  - No infrastructure
  - Auto-scaling
  - Pay per use

Future:
  - Serverless default
  - Managed
  - Cost-effective
""")

print("\n=== 3. Kubernetes ===")
print("""
Trend: K8s-native

Druid Operator:
  - Kubernetes controller
  - CRDs
  - Helm

Features:
  - Dynamic scaling
  - Resource sharing
  - Easier deployment

Future:
  - K8s default
  - Operator
  - Cloud-native
""")

print("\n=== 4. Lakehouse Integration ===")
print("""
Trend: Druid + Lakehouse

Integration:
  - Iceberg
  - Delta Lake
  - Hudi

Features:
  - Read from lake
  - Write to lake
  - Unified

Future:
  - Lakehouse-native
  - Open formats
  - Unified analytics
""")

print("\n=== 5. Performance ===")
print("""
Trend: Faster

Features:
  - New segment format (v10)
  - Better compression
  - Faster queries
  - Lower memory

Future:
  - Sub-100ms
  - Higher concurrency
  - Lower cost
""")

print("\n=== 6. Open Standards ===")
print("""
Trend: Open wins

Standards:
  - SQL
  - Iceberg
  - OpenLineage
  - OpenTelemetry

Future:
  - Interoperability
  - No lock-in
  - Open
""")

# TIMELINE
print("\n=== Timeline ===")
print("""
Past:
  2011: Druid at Metamarkets
  2018: Apache top-level
  2020: Cloud services
  2022: Mature
  2024: Segment v10

Present:
  2025: Mature
  2026: AI-augmented

Future:
  2030: Serverless?
  2035: AI-native?
  2040: New paradigms?
""")

print("\n=== Final Thoughts ===")
print("""
Apache Druid is:
  - Real-time analytics
  - Sub-second queries
  - High concurrency
  - Streaming + batch
  - Rollup

The future is:
  - AI-augmented
  - Cloud-native
  - K8s-first
  - Lakehouse-integrated
  - Open

"Real-time insights at scale."

Keep exploring. Keep building.
""")