🧠 Data Observability Fundamentals

Q: What is Data Observability?

Data Observability is the ability to understand the health and state of your data systems by continuously monitoring data pipelines, detecting anomalies, and alerting on issues. It's like observability for software, but applied to data.

Q: What are the 5 pillars of Data Observability?

The five pillars are: freshness (is data up-to-date?), volume (is data complete?), schema (has structure changed?), lineage (where did it come from?), and quality (is it accurate?). Together they provide a complete picture of data health.

Q: What is the difference between Data Observability and Data Quality?

Data Quality focuses on correctness of data (accuracy, completeness, validity). Data Observability is broader — it monitors the entire data system, including freshness, volume, schema changes, and lineage, to detect and prevent issues before they impact consumers.

Q: What tools are used for Data Observability?

Popular tools include Monte Carlo, Datafold, Elementary, Soda, Acceldata, Bigeye, Anomalo, and open-source options like Great Expectations combined with Airflow, Dagster, and OpenLineage.

Q: Why does Data Observability matter?

Bad data costs organizations an average of $12.9M per year (Gartner). Data downtime — periods when data is missing, wrong, or stale — erodes trust and leads to bad decisions. Data Observability detects issues early, reduces downtime, and builds trust in data.

🎯 What is Data Observability?

Data Observability is the ability to understand the health and state of your data systems by continuously monitoring data pipelines, detecting anomalies, and alerting on issues. It's like software observability — but for data. It answers: Is my data fresh, complete, and trustworthy right now?

Why Data Observability Matters

Cost: $12.9M/year lost to bad data. Trust: Confidence in data. Downtime: Reduce incidents. Detection: Catch issues early. Blast radius: Limit impact. SLA: Meet commitments.

📌 Monitoring📌 Anomaly Detection📌 Alerting📌 Trust

# Data Observability — the big picture
# Understanding the health of your data systems.

# CONCEPTUAL OVERVIEW
print("""
=== Data Observability ===

What it is:
  - Monitor data health
  - Detect anomalies
  - Alert on issues
  - Understand pipelines
  - Build trust

Think of it as:
  - Software observability (logs, metrics, traces)
  - But for data
  - "Is my data healthy?"

Core questions:
  - Is data fresh?
  - Is data complete?
  - Has schema changed?
  - Where did it come from?
  - Is it correct?

5 Pillars:
  - Freshness
  - Volume
  - Schema
  - Lineage
  - Quality
""")

# WHY IT MATTERS
print("=== Why It Matters ===")
print("""
Cost of bad data:
  - $12.9M/year average (Gartner)
  - 60% of data projects fail
  - Lost revenue
  - Wasted time
  - Bad decisions

Data downtime:
  - Periods when data is missing
  - Wrong data
  - Stale data
  - Root cause hard to find

Data Observability:
  - Detects issues early
  - Reduces downtime
  - Builds trust
  - Saves money
""")

# REAL-WORLD
print("=== Real-World Impact ===")
print("""
Companies using Data Observability:
  - Netflix
  - Uber
  - Airbnb
  - Bloomberg
  - Fox
  - JetBlue

Use cases:
  - ETL monitoring
  - ML feature monitoring
  - BI dashboards
  - Data SLAs
  - Incident response
""")

📊 The 5 Pillars

🎯 The Five Pillars of Data Observability

Data Observability is defined by five pillars: freshness, volume, schema, lineage, and quality. Together they give a complete picture of data health.

The 5 Pillars

Freshness: Is data current? Volume: Is data complete? Schema: Has structure changed? Lineage: Where did it come from? Quality: Is it correct?

📊 Freshness📊 Volume📊 Schema📊 Lineage📊 Quality

# The 5 Pillars
print("=== 1. FRESHNESS ===")
print("""
Freshness:
  - Is data up-to-date?
  - When was it last updated?
  - Is it stale?

Questions:
  - Latest timestamp?
  - Expected refresh cadence?
  - Any delays?

Example:
  - Table last updated 3 hours ago
  - Expected: every hour
  - Alert: stale > 2 hours
""")

print("\n=== 2. VOLUME ===")
print("""
Volume:
  - Is data complete?
  - Expected row count?
  - Anomalies?

Questions:
  - Row count today?
  - Compared to yesterday?
  - Within expected range?

Example:
  - Daily rows: 1M ± 10%
  - Today: 100K
  - Alert: volume drop
""")

print("\n=== 3. SCHEMA ===")
print("""
Schema:
  - Has structure changed?
  - New columns?
  - Type changes?

Questions:
  - Columns added/removed?
  - Types changed?
  - Constraints broken?

Example:
  - Column 'email' renamed to 'email_address'
  - Downstream breaks
  - Alert: schema change
""")

print("\n=== 4. LINEAGE ===")
print("""
Lineage:
  - Where did data come from?
  - Where does it go?
  - What depends on it?

Questions:
  - Upstream sources?
  - Downstream consumers?
  - Impact of change?

Example:
  - Table 'orders' feeds 5 dashboards
  - Issue detected
  - Alert: 5 dashboards impacted
""")

print("\n=== 5. QUALITY ===")
print("""
Quality:
  - Is data correct?
  - Does it meet expectations?
  - Any anomalies?

Questions:
  - Values in range?
  - Nulls?
  - Duplicates?
  - Business rules?

Example:
  - 10% null emails
  - Expected: <1%
  - Alert: quality issue
""")

print("\n=== Summary Table ===")
print(f"{'Pillar':<15} {'Question':<35} {'Example':<25}")
print("-" * 75)
print(f"{'Freshness':<15} {'Is it current?':<35} {'Last update':<25}")
print(f"{'Volume':<15} {'Is it complete?':<35} {'Row count':<25}")
print(f"{'Schema':<15} {'Structure OK?':<35} {'Columns, types':<25}")
print(f"{'Lineage':<15} {'Where from/to?':<35} {'Upstream, down':<25}")
print(f"{'Quality':<15} {'Is it correct?':<35} {'Values, nulls':<25}")

⏰ Freshness

🎯 Is Data Current?

Freshness measures how up-to-date your data is. It answers: when was data last updated, and is it within the expected SLA?

Freshness

Last update: Timestamp. SLA: Expected cadence. Staleness: Age. Delay: Behind schedule. Alert: On threshold. Trend: Over time.

⏰ Last Update⏰ SLA⏰ Staleness⏰ Alert

# Freshness
print("=== What is Freshness? ===")
print("""
Freshness:
  - How current is data?
  - Time since last update
  - Compare to SLA

Metrics:
  - Last updated timestamp
  - Age (now - last update)
  - Expected cadence
  - SLA threshold

Example:
  - Table: users
  - Last update: 2 hours ago
  - Expected: every hour
  - SLA: < 2 hours
  - Status: ALERT (stale)
""")

print("\n=== How to Monitor ===")
print("""
# SQL
SELECT
    MAX(updated_at) as last_update,
    NOW() - MAX(updated_at) as age
FROM users;

# If age > expected:
#   Alert

# Python
import pandas as pd
from datetime import datetime, timedelta

def check_freshness(df, column, sla_hours=1):
    last = df[column].max()
    age = datetime.now() - last
    if age > timedelta(hours=sla_hours):
        return False, f"Stale: {age}"
    return True, f"Fresh: {age}"

# Great Expectations
validator.expect_column_max_to_be_between(
    "updated_at",
    min_value=datetime.now() - timedelta(hours=1),
    max_value=datetime.now(),
)
""")

print("\n=== Freshness SLA ===")
print("""
Common SLAs:
  - Real-time: < 1 minute
  - Streaming: < 5 minutes
  - Hourly: < 1 hour
  - Daily: < 24 hours
  - Weekly: < 7 days

Define per table:
  - Critical: strict
  - Non-critical: relaxed
  - Archive: no SLA
""")

print("\n=== Alerts ===")
print("""
Alert when:
  - Age > SLA
  - No update in expected window
  - Trend shows increasing delay

Alert channels:
  - Slack
  - Email
  - PagerDuty
  - Opsgenie

Include:
  - Table name
  - Last update
  - Age
  - SLA
  - Owner
""")

print("\n=== Root Cause ===")
print("""
When freshness alert:
  1. Check upstream
  2. Check pipeline
  3. Check source
  4. Check job schedule
  5. Check dependencies

Common causes:
  - Source delay
  - Job failure
  - Dependency failure
  - Schedule issue
  - Resource contention
""")

📦 Volume

🎯 Is Data Complete?

Volume monitors row counts and data size. Sudden drops or spikes indicate issues like pipeline failures, source problems, or upstream changes.

Volume

Row count: Total rows. Expected: Baseline. Anomaly: Deviation. Trend: Over time. Seasonality: Patterns. Alert: On anomaly.

📦 Row Count📦 Baseline📦 Anomaly📦 Trend

# Volume
print("=== What is Volume? ===")
print("""
Volume:
  - Row count
  - Data size
  - Completeness
  - Expected range

Metrics:
  - Today's rows
  - Yesterday's rows
  - 7-day average
  - Std deviation
  - Anomaly score

Example:
  - Yesterday: 1M rows
  - Today: 100K rows
  - Expected: 900K - 1.1M
  - Status: ALERT (drop)
""")

print("\n=== How to Monitor ===")
print("""
# SQL
SELECT COUNT(*) as row_count
FROM users
WHERE date = CURRENT_DATE;

# Compare to baseline
baseline = 1000000
threshold = 0.1  # 10%

if abs(row_count - baseline) / baseline > threshold:
    alert("Volume anomaly")

# Time series anomaly detection
from scipy import stats
z_score = (row_count - mean) / std
if abs(z_score) > 3:
    alert("Volume anomaly")
""")

print("\n=== Types of Anomalies ===")
print("""
1. DROP
   - Sudden decrease
   - Pipeline failure
   - Source issue

2. SPIKE
   - Sudden increase
   - Duplicate data
   - Backfill

3. TREND
   - Gradual change
   - Growth
   - Decay

4. SEASONALITY
   - Expected patterns
   - Weekday/weekend
   - Monthly

5. GAPS
   - Missing periods
   - Zero rows
   - Missing partitions
""")

print("\n=== Best Practices ===")
print("""
1. Establish baseline
   - 7-30 days of history
   - Account for seasonality

2. Set thresholds
   - % deviation
   - Z-score
   - IQR

3. Alert
   - Slack, PagerDuty
   - Include details

4. Investigate
   - Root cause
   - Fix
   - Postmortem

5. Monitor trend
   - Slow changes
   - Long-term patterns
""")

print("\n=== Example Alert ===")
print("""
Alert: Volume anomaly

Table: analytics.orders
Date: 2025-01-15
Rows today: 100,000
Expected: 900,000 - 1,100,000
Deviation: -90%
Severity: HIGH

Possible causes:
  - Pipeline failure
  - Source issue
  - Filter bug
  - Time zone

Owner: data-team
Runbook: /runbooks/volume
""")

📐 Schema

🎯 Has Structure Changed?

Schema monitoring detects changes to table structure: columns added/removed, types changed, constraints broken. Schema changes often break downstream consumers.

Schema

Columns: Added/removed. Types: Changed. Constraints: Broken. Order: Changed. Impact: Downstream. Versioning: Tracked.

📐 Columns📐 Types📐 Constraints📐 Impact

# Schema
print("=== What is Schema Monitoring? ===")
print("""
Schema:
  - Structure of data
  - Columns, types
  - Constraints
  - Order

Changes:
  - Column added
  - Column removed
  - Column renamed
  - Type changed
  - Order changed
  - Constraints changed

Impact:
  - Breaking changes
  - Downstream failures
  - Silent data issues
""")

print("\n=== How to Monitor ===")
print("""
# Capture schema
def get_schema(df):
    return {
        col: str(df[col].dtype)
        for col in df.columns
    }

# Compare
old_schema = get_schema(old_df)
new_schema = get_schema(new_df)

# Detect changes
added = set(new_schema) - set(old_schema)
removed = set(old_schema) - set(new_schema)
changed = {
    col for col in old_schema
    if col in new_schema and old_schema[col] != new_schema[col]
}

if added or removed or changed:
    alert("Schema change")
    print(f"Added: {added}")
    print(f"Removed: {removed}")
    print(f"Changed: {changed}")
""")

print("\n=== Types of Changes ===")
print("""
1. BREAKING
   - Column removed
   - Column renamed
   - Type narrowed
   - Constraint added

   Impact: Failures

2. NON-BREAKING
   - Column added (nullable)
   - Type widened
   - Constraint removed

   Impact: Usually safe

3. SILENT
   - Type coerced
   - Precision loss
   - Encoding change

   Impact: Silent bugs
""")

print("\n=== Schema Registry ===")
print("""
Schema Registry:
  - Central schema store
  - Version control
  - Compatibility check
  - Enforce

Tools:
  - Confluent Schema Registry
  - AWS Glue Schema Registry
  - Apicurio

Benefits:
  - Prevent breaking changes
  - Version history
  - Compatibility
  - Documentation
""")

print("\n=== Best Practices ===")
print("""
1. Version schema
   - SemVer
   - Track changes
   - Document

2. Compatibility
   - Backward
   - Forward
   - Full

3. Test
   - Schema changes
   - Downstream impact
   - CI/CD

4. Alert
   - Breaking changes
   - Owner notified
   - Downstream notified

5. Migrate
   - Deprecation period
   - Migration plan
   - Rollback
""")

print("\n=== Example Alert ===")
print("""
Alert: Breaking schema change

Table: analytics.users
Change: Column 'email' removed
Impact: 3 downstream tables

Consumers:
  - marketing.campaigns
  - sales.leads
  - analytics.user_360

Action: Contact owner
Owner: data-team
""")

🔗 Lineage

🎯 Where Did It Come From?

Lineage tracks data flow: sources, transformations, and consumers. It answers: where did this data come from, and what depends on it?

Lineage

Upstream: Sources. Downstream: Consumers. Transformations: How. Impact: Blast radius. Root cause: Trace back. Visual: Graph.

🔗 Upstream🔗 Downstream🔗 Impact🔗 Root Cause

# Lineage
print("=== What is Lineage? ===")
print("""
Lineage:
  - Data flow
  - Source → Target
  - Transformations
  - Dependencies

Types:
  - Table-level
  - Column-level
  - Job-level

Uses:
  - Root cause analysis
  - Impact analysis
  - Compliance
  - Debugging
""")

print("\n=== Example ===")
print("""
Source: Postgres (users)
  ↓
Job: dbt model (stg_users)
  ↓
Table: analytics.stg_users
  ↓
Job: dbt model (dim_users)
  ↓
Table: analytics.dim_users
  ↓
Dashboard: User Analytics

If source fails:
  - All downstream affected
  - 1 dashboard impacted
""")

print("\n=== Impact Analysis ===")
print("""
Scenario:
  Table 'orders' has issue

Question:
  What breaks?

Answer:
  Query lineage downstream:
  - orders → stg_orders → fct_orders
  - fct_orders → revenue_dashboard
  - fct_orders → ml_features

Impact:
  - 1 dashboard
  - 1 ML model
  - 3 tables

Action:
  - Alert owners
  - Prioritize fix
""")

print("\n=== Root Cause ===")
print("""
Scenario:
  Dashboard shows wrong numbers

Question:
  Why?

Answer:
  Query lineage upstream:
  - dashboard → fct_orders
  - fct_orders → stg_orders
  - stg_orders → orders
  - orders → source Postgres

Issue found:
  - Source Postgres had duplicates

Action:
  - Fix source
  - Reprocess
""")

print("\n=== Column-Level Lineage ===")
print("""
Column-level:
  - Track individual columns
  - More granular
  - Complex

Example:
  users.email
    → stg_users.email
    → dim_users.email
    → user_360.email

If users.email changes:
  - Track impact
  - Column-level
""")

print("\n=== Tools ===")
print("""
Lineage tools:
  - OpenLineage (open source)
  - DataHub
  - Amundsen
  - Apache Atlas
  - Monte Carlo
  - Datafold
  - dbt (built-in)

Integration:
  - Airflow
  - Spark
  - dbt
  - Dagster
  - Prefect
""")

print("\n=== Best Practices ===")
print("""
1. Capture automatically
   - Integrations
   - No manual

2. Column-level
   - More useful
   - More complex

3. Visualize
   - Graph
   - Interactive

4. Alert on impact
   - Notify consumers
   - Prioritize

5. Document
   - Ownership
   - SLA
   - Runbooks
""")

✅ Quality

🎯 Is Data Correct?

Quality monitoring checks that data meets expectations: values in range, no nulls where not allowed, no duplicates, business rules satisfied.

Quality

Values: In range. Nulls: No unexpected. Duplicates: No. Rules: Business. Anomalies: Detect. Expectations: Define.

✅ Values✅ Nulls✅ Duplicates✅ Rules

# Quality
print("=== What is Quality Monitoring? ===")
print("""
Quality:
  - Data correctness
  - Meets expectations
  - Business rules

Checks:
  - Values in range
  - No nulls
  - No duplicates
  - Format correct
  - Referential integrity
  - Business rules
""")

print("\n=== Quality Dimensions ===")
print("""
1. ACCURACY
   - Correct values
   - Match reality

2. COMPLETENESS
   - No missing
   - All present

3. CONSISTENCY
   - No contradictions
   - Same everywhere

4. VALIDITY
   - Conforms to rules
   - In range

5. UNIQUENESS
   - No duplicates
   - One record

6. TIMELINESS
   - Fresh
   - On time
""")

print("\n=== How to Monitor ===")
print("""
# Great Expectations
validator.expect_column_values_to_not_be_null("user_id")
validator.expect_column_values_to_be_unique("user_id")
validator.expect_column_values_to_be_between("age", 0, 150)
validator.expect_column_values_to_be_in_set("status", ["active", "inactive"])

# SQL checks
SELECT
    COUNT(*) as total,
    COUNT(user_id) as non_null,
    COUNT(DISTINCT user_id) as unique_count,
    SUM(CASE WHEN age < 0 OR age > 150 THEN 1 ELSE 0 END) as invalid_age
FROM users;

# Custom checks
def check_business_rule(df):
    # Rule: revenue >= 0
    violations = df[df['revenue'] < 0]
    if len(violations) > 0:
        alert(f"{len(violations)} violations")
""")

print("\n=== Anomaly Detection ===")
print("""
Types:
  - Statistical
  - ML-based
  - Rule-based

Statistical:
  - Z-score
  - IQR
  - MAD

ML:
  - Isolation Forest
  - Autoencoders
  - Time series

Rule:
  - Thresholds
  - Business rules
  - Custom
""")

print("\n=== Alerts ===")
print("""
Alert when:
  - Null % > threshold
  - Duplicates found
  - Values out of range
  - Business rule violated
  - Anomaly detected

Alert includes:
  - Table
  - Column
  - Issue
  - Sample values
  - Impact
  - Owner
""")

print("\n=== Best Practices ===")
print("""
1. Define expectations
   - Per table
   - Per column
   - Documented

2. Start critical
   - Important columns
   - Business rules
   - Iterate

3. Monitor continuously
   - Every run
   - Scheduled
   - Real-time

4. Alert meaningfully
   - Not too noisy
   - Actionable
   - Root cause

5. Fix and learn
   - Root cause
   - Prevent
   - Postmortem
""")

📡 Monitoring & Alerts

🎯 Detecting and Alerting

Effective Data Observability requires continuous monitoring, intelligent alerting, and clear incident response. The goal is to detect issues before consumers do.

Monitoring & Alerts

Continuous: Always on. Anomaly: Detect. Alert: Notify. Context: Rich. Actionable: Runbook. Feedback: Improve.

📡 Continuous📡 Anomaly📡 Alert📡 Runbook

# Monitoring & Alerts
print("=== Monitoring Approach ===")
print("""
1. BASELINE
   - Historical data
   - Expected patterns
   - Seasonality

2. DETECT
   - Thresholds
   - Anomalies
   - Trends

3. ALERT
   - Channel
   - Severity
   - Context

4. RESPOND
   - Runbook
   - Fix
   - Communicate

5. LEARN
   - Postmortem
   - Improve
   - Prevent
""")

print("\n=== Alert Design ===")
print("""
Good alert:
  - Actionable
  - Context-rich
  - Right severity
  - Right channel
  - Right time
  - Owner

Bad alert:
  - Noisy
  - Vague
  - Wrong severity
  - Wrong channel
  - No owner
  - No runbook
""")

print("\n=== Alert Channels ===")
print("""
Slack:
  - Team channel
  - Threads
  - Actionable

Email:
  - Formal
  - Record
  - Non-urgent

PagerDuty:
  - Critical
  - On-call
  - Escalation

Opsgenie:
  - Alternative
  - Similar

SMS:
  - Critical
  - Immediate

Webhook:
  - Custom
  - Integration
""")

print("\n=== Alert Example ===")
print("""
Slack Alert:

🚨 Data Observability Alert

Table: analytics.orders
Type: Freshness
Severity: HIGH

Issue: Stale data
Last update: 3 hours ago
SLA: 1 hour
Delay: 2 hours

Impact:
  - 3 dashboards
  - 1 ML model

Owner: @data-team
Runbook: /runbooks/freshness

Actions:
  - Check pipeline
  - Check source
  - Escalate if needed
""")

print("\n=== Runbooks ===")
print("""
Runbook:
  - Step-by-step
  - Common issues
  - Contacts
  - Escalation

Structure:
  - Symptom
  - Diagnosis
  - Fix
  - Verify
  - Prevent

Example:
  Freshness alert:
  1. Check pipeline status
  2. Check upstream
  3. Check source
  4. Restart if needed
  5. Verify
  6. Document
""")

print("\n=== Incident Response ===")
print("""
1. DETECT
   - Alert
   - Triage

2. ASSESS
   - Impact
   - Severity
   - Priority

3. RESPOND
   - Fix
   - Communicate
   - Escalate

4. RESOLVE
   - Verify
   - Close
   - Document

5. LEARN
   - Postmortem
   - Action items
   - Prevent
""")

print("\n=== Metrics ===")
print("""
Track:
  - MTTR (Mean Time To Resolve)
  - MTBF (Mean Time Between Failures)
  - Incidents per month
  - Alert volume
  - False positive rate
  - Coverage

Goals:
  - Reduce MTTR
  - Reduce incidents
  - Improve coverage
  - Reduce noise
""")

print("\n=== Best Practices ===")
print("""
1. RIGHT SEVERITY
   - Not everything critical

2. RIGHT CHANNEL
   - Match severity

3. RICH CONTEXT
   - Table, issue, owner

4. ACTIONABLE
   - Runbook
   - Clear next steps

5. FEEDBACK LOOP
   - Improve alerts
   - Reduce noise
   - Learn
""")

🛠️ Tools & Vendors

🎯 Data Observability Tools

The Data Observability ecosystem includes commercial platforms, open-source tools, and integrations with existing data stack components.

Tools

Commercial: Monte Carlo, Datafold, Bigeye. Open source: GX, OpenLineage. Native: dbt, Airflow. Custom: DIY. Free tier: Some.

🛠️ Monte Carlo🛠️ Datafold🛠️ Elementary🛠️ OpenLineage

# Tools & Vendors
print("=== Commercial Platforms ===")
print("""
1. MONTE CARLO
   - Market leader
   - ML-powered
   - End-to-end
   - Integrations

2. DATAFOLD
   - Data diff
   - Observability
   - CI/CD
   - Column-level

3. BIGEYE
   - Anomaly detection
   - Observability
   - Custom

4. ACCELDATA
   - Platform
   - Data observability
   - Multi-cloud

5. ANOMALO
   - Anomaly detection
   - ML-based
   - Automated

6. SODA
   - Data quality
   - Observability
   - YAML-based

7. ELEMENTARY
   - dbt-native
   - Open source
   - Observability
""")

print("\n=== Open Source ===")
print("""
1. GREAT EXPECTATIONS
   - Data quality
   - Expectations
   - Data docs

2. OPENLINEAGE
   - Lineage standard
   - Integrations
   - Open

3. DATAHUB
   - Metadata
   - Lineage
   - Discovery

4. AMUNDSEN
   - Discovery
   - Lineage
   - Metadata

5. SODA CORE
   - Data quality
   - Open source
   - SQL/YAML

6. DBT TESTS
   - Built-in
   - SQL-based
   - Simple
""")

print("\n=== Native Tools ===")
print("""
Airflow:
  - DAG monitoring
  - Task status
  - Logs
  - SLA misses

dbt:
  - Tests
  - Freshness
  - Documentation
  - Lineage

Dagster:
  - Asset checks
  - Observability
  - Lineage

Great Expectations:
  - Expectations
  - Validation
  - Data docs
""")

print("\n=== Comparison ===")
print(f"{'Tool':<20} {'Type':<20} {'Best For':<25}")
print("-" * 70)
print(f"{'Monte Carlo':<20} {'Commercial':<20} {'Enterprise':<25}")
print(f"{'Datafold':<20} {'Commercial':<20} {'Data diff':<25}")
print(f"{'Elementary':<20} {'Open source':<20} {'dbt shops':<25}")
print(f"{'Soda':<20} {'Both':<20} {'SQL quality':<25}")
print(f"{'Great Expectations':<20} {'Open source':<20} {'Python':<25}")
print(f"{'OpenLineage':<20} {'Open source':<20} {'Lineage':<25}")
print(f"{'DataHub':<20} {'Open source':<20} {'Catalog':<25}")

print("\n=== Choosing Tools ===")
print("""
Consider:
  - Budget
  - Team size
  - Data stack
  - Requirements
  - Time to value
  - Maintenance
  - Open source vs commercial

Start with:
  - Native tools (dbt tests)
  - Open source (GX)
  - Add commercial as scale
""")

⚖️ vs Data Quality

🎯 Two Related Concepts

Data Quality focuses on correctness. Data Observability is broader — it monitors the entire data system. They're complementary, not competing.

Comparison

Quality: Correctness. Observability: Health. Quality: Tests. Observability: Monitoring. Quality: Expectations. Observability: Anomalies.

⚖️ Quality⚖️ Observability⚖️ Tests⚖️ Monitoring

Aspect	Data Quality	Data Observability
Focus	Correctness	Health & state
Scope	Values, rules	Entire system
Approach	Testing	Monitoring
Pillars	Accuracy, completeness	5 pillars (incl. freshness, lineage)
Detection	Rules, thresholds	Anomalies, ML
Coverage	Column-level	System-wide
Timing	At validation	Continuous
Alerting	On failure	On anomaly
Tools	GX, dbt tests, Soda	Monte Carlo, Datafold, Elementary
Best for	Known rules	Unknown unknowns

# How They Work Together
print("=== Quality + Observability ===")
print("""
Data Quality:
  - Know rules
  - Test explicitly
  - Catch violations

Data Observability:
  - Don't know all rules
  - Monitor patterns
  - Catch anomalies

Together:
  - Comprehensive
  - Both covered
  - Trust

Example:
  Quality:
    - user_id not null
    - email format valid
    - age 0-150

  Observability:
    - Row count anomaly
    - Freshness delay
    - Schema change
    - Lineage impact
""")

print("\n=== When to Use Which ===")
print("""
USE DATA QUALITY:
  ✓ Known rules
  ✓ Explicit tests
  ✓ Business rules
  ✓ Compliance
  ✓ Documentation

USE DATA OBSERVABILITY:
  ✓ Unknown issues
  ✓ Anomalies
  ✓ System health
  ✓ Root cause
  ✓ Impact analysis

USE BOTH:
  ✓ Comprehensive
  ✓ Best practice
  ✓ Trust
""")

print("\n=== Modern Approach ===")
print("""
Modern data teams use:
  - dbt tests (Quality)
  - Great Expectations (Quality)
  - Elementary (Observability)
  - OpenLineage (Lineage)
  - Monte Carlo (Observability)

Integrated:
  - All sources
  - All consumers
  - All pipelines
  - End-to-end
""")

🚀 Implementation

🎯 Adopting Data Observability

Adopting Data Observability is a journey. Start small, prove value, and scale.

Roadmap

Assess: Current state. Pilot: One use case. Expand: More tables. Automate: Alerts. Optimize: Iterate. Scale: Org-wide.

🚀 Assess🚀 Pilot🚀 Expand🚀 Scale

# Implementation Roadmap
print("=== Phase 1: ASSESS (Weeks 1-2) ===")
print("""
Goals:
  - Understand current state
  - Identify gaps
  - Prioritize

Activities:
  - Inventory tables
  - Identify critical data
  - Review incidents
  - Assess tools

Deliverables:
  - Critical table list
  - Gap analysis
  - Priorities
""")

print("\n=== Phase 2: PILOT (Weeks 3-6) ===")
print("""
Goals:
  - Prove value
  - Learn
  - Iterate

Activities:
  - Pick 3-5 critical tables
  - Implement freshness, volume
  - Set up alerts
  - Measure

Deliverables:
  - Working monitoring
  - Alerts
  - Results
""")

print("\n=== Phase 3: EXPAND (Weeks 7-12) ===")
print("""
Goals:
  - More tables
  - More pillars
  - More coverage

Activities:
  - Add tables
  - Add schema, lineage
  - Add quality
  - Integrate

Deliverables:
  - 20-50 tables
  - 5 pillars
  - Integration
""")

print("\n=== Phase 4: AUTOMATE (Months 4-6) ===")
print("""
Goals:
  - Automate
  - Scale
  - Optimize

Activities:
  - Auto-discovery
  - Auto-baseline
  - Auto-alert
  - Runbooks
  - Self-service

Deliverables:
  - Automated
  - Scaled
  - Documented
""")

print("\n=== Phase 5: SCALE (Months 6+) ===")
print("""
Goals:
  - Org-wide
  - Culture
  - Continuous

Activities:
  - All tables
  - All teams
  - Training
  - Best practices
  - Continuous

Deliverables:
  - Full coverage
  - Culture
  - Reliability
""")

print("\n=== Success Metrics ===")
print("""
1. COVERAGE
   - % tables monitored
   - % pillars covered

2. DETECTION
   - Issues detected
   - Time to detect

3. RESOLUTION
   - MTTR
   - Incidents

4. TRUST
   - Consumer confidence
   - SLA compliance

5. BUSINESS
   - Cost saved
   - Revenue impact
""")

print("\n=== Pitfalls ===")
print("""
AVOID:
  ✗ Big bang
  ✗ Too many alerts
  ✗ No runbooks
  ✗ No ownership
  ✗ Only tech
  ✗ Ignoring culture

BEST PRACTICES:
  ✓ Start small
  ✓ Critical tables first
  ✓ Right alerts
  ✓ Runbooks
  ✓ Ownership
  ✓ Iterate
""")

✅ Best Practices

🎯 Doing Data Observability Right

Follow these best practices for effective Data Observability.

Best Practices

Start small: Critical tables. Right alerts: Actionable. Ownership: Clear. Runbooks: Documented. Iterate: Improve. Culture: Trust.

✅ Start Small✅ Right Alerts✅ Ownership✅ Runbooks

# Best Practices
print("=== Start Small ===")
print("""
1. CRITICAL TABLES
   - Not everything
   - Top 5-10
   - Business-critical

2. FEW ALERTS
   - High signal
   - Actionable
   - Not noisy

3. ITERATE
   - Add more
   - Refine
   - Remove false positives

4. MEASURE
   - Coverage
   - MTTR
   - Incidents
""")

print("\n=== Alert Design ===")
print("""
1. RIGHT SEVERITY
   - Critical: page
   - High: Slack
   - Medium: email
   - Low: dashboard

2. RICH CONTEXT
   - Table
   - Issue
   - Impact
   - Owner
   - Runbook

3. ACTIONABLE
   - Clear next steps
   - Runbook
   - Escalation

4. FEEDBACK
   - Was it useful?
   - Fix noise
   - Improve
""")

print("\n=== Ownership ===")
print("""
1. CLEAR OWNER
   - Per table
   - Per dataset
   - Per domain

2. RESPONSIBILITY
   - Fix issues
   - Improve quality
   - Communicate

3. ESCALATION
   - On-call
   - Manager
   - Executive

4. ACCOUNTABILITY
   - SLAs
   - Metrics
   - Reviews
""")

print("\n=== Runbooks ===")
print("""
1. DOCUMENT
   - Common issues
   - Steps
   - Contacts

2. STRUCTURE
   - Symptom
   - Diagnosis
   - Fix
   - Verify

3. MAINTAIN
   - Update
   - Review
   - Share

4. TEST
   - Practice
   - Train
   - Improve
""")

print("\n=== Culture ===")
print("""
1. TRUST
   - Data teams trusted
   - Consumers confident
   - Reliability valued

2. TRANSPARENCY
   - Status visible
   - Issues shared
   - Learning culture

3. ACCOUNTABILITY
   - Owners
   - SLAs
   - Metrics

4. CONTINUOUS
   - Improve
   - Iterate
   - Learn
""")

print("\n=== Anti-Patterns ===")
print("""
AVOID:
  ✗ Too many alerts
  ✗ No owner
  ✗ No runbook
  ✗ Ignoring alerts
  ✗ Manual everything
  ✗ No metrics
  ✗ Tech-only
  ✗ Big bang
""")

print("\n=== Checklist ===")
print("""
✓ Start with critical tables
✓ Right severity alerts
✓ Rich context
✓ Clear ownership
✓ Runbooks
✓ Metrics
✓ Feedback loop
✓ Culture
✓ Iterate
✓ Scale
""")

❓ FAQ

🎯 Frequently Asked Questions

Answers to common questions about Data Observability.

What is Data Observability?

What are the 5 pillars of Data Observability?

The five pillars are: freshness (is data up-to-date?), volume (is data complete?), schema (has structure changed?), lineage (where did it come from?), and quality (is it accurate?). Together they provide a complete picture of data health.

What is the difference between Data Observability and Data Quality?

Data Quality focuses on correctness of data (accuracy, completeness, validity). Data Observability is broader — it monitors the entire data system, including freshness, volume, schema changes, and lineage, to detect and prevent issues before they impact consumers.

What tools are used for Data Observability?

Popular tools include Monte Carlo, Datafold, Elementary, Soda, Acceldata, Bigeye, Anomalo, and open-source options like Great Expectations combined with Airflow, Dagster, and OpenLineage.

Why does Data Observability matter?

Bad data costs organizations an average of $12.9M per year (Gartner). Data downtime — periods when data is missing, wrong, or stale — erodes trust and leads to bad decisions. Data Observability detects issues early, reduces downtime, and builds trust in data.

How do I get started with Data Observability?

Start with 3-5 critical tables. Implement freshness and volume monitoring first. Set up alerts in Slack. Then expand to schema, lineage, and quality. Use open-source tools initially, then scale with commercial platforms if needed.

What is the difference between Data Observability and monitoring?

Monitoring is a subset of observability. Monitoring collects metrics and checks thresholds. Observability goes further: it correlates signals across systems, understands context, and helps you answer "why" not just "what." Observability includes monitoring, lineage, anomaly detection, and root cause analysis.

Can Data Observability be done with open source tools?

Yes. You can build effective Data Observability with open-source tools: Great Expectations for quality, OpenLineage for lineage, DataHub for metadata, dbt tests for transformations, and Airflow for orchestration. Commercial platforms add scale, ML, and managed services.

What are the most common Data Observability issues?

Common issues include: stale data (freshness), missing rows (volume), breaking schema changes, duplicate data (quality), null values, downstream impact (lineage), and unknown unknowns (anomalies).

How much does Data Observability cost?

Costs vary widely. Open-source: free (but requires engineering time). Commercial platforms: $50K-$500K+/year depending on scale. Managed services: typically priced per table, per source, or per volume. Consider total cost: tool + implementation + maintenance.

# Quick Reference
print("=== Quick Answers ===")
print("""
Q: What are the 5 pillars?
A: Freshness, volume, schema, lineage, quality.

Q: Is it different from Data Quality?
A: Yes — broader, system-wide.

Q: Best tool?
A: Depends. Monte Carlo, Elementary, GX.

Q: How to start?
A: 3-5 tables, freshness + volume.

Q: Open source possible?
A: Yes — GX + OpenLineage + DataHub.

Q: ROI?
A: Reduces downtime, saves $M.

Q: Culture needed?
A: Yes — trust, ownership, transparency.
""")

🔮 Future

🎯 Where Data Observability Is Going

The future is AI-native, automated, real-time, and unified. Data Observability is becoming a core part of every data platform.

Future Directions

AI: ML-powered. Automated: Self-healing. Real-time: Streaming. Unified: One platform. Open: Standards. Predictive: Prevent.

🔮 AI🔮 Automated🔮 Real-Time🔮 Unified

# Future of Data Observability
print("=== 1. AI-Native ===")
print("""
Trend: AI at core

AI:
  - Anomaly detection
  - Root cause
  - Predictions
  - Auto-remediation

Future:
  - AI-first
  - Autonomous
  - Self-healing
""")

print("\n=== 2. Automated Remediation ===")
print("""
Trend: Self-healing

Features:
  - Detect
  - Diagnose
  - Fix
  - Verify

Examples:
  - Auto-retry
  - Auto-quarantine
  - Auto-scale
  - Auto-rollback

Future:
  - Zero manual
  - Autonomous
  - Reliable
""")

print("\n=== 3. Real-Time ===")
print("""
Trend: Streaming obs

Features:
  - Real-time
  - Continuous
  - Low latency
  - Live

Future:
  - Real-time standard
  - Stream-native
  - Immediate
""")

print("\n=== 4. Unified Platform ===")
print("""
Trend: One platform

Features:
  - Quality
  - Observability
  - Lineage
  - Catalog
  - Governance

Future:
  - Unified
  - One view
  - Complete
""")

print("\n=== 5. Open Standards ===")
print("""
Trend: Open wins

Standards:
  - OpenLineage
  - OpenMetadata
  - Data Contracts
  - OpenTelemetry for data

Future:
  - Interoperability
  - No lock-in
  - Open
""")

print("\n=== 6. Predictive ===")
print("""
Trend: Prevent issues

Features:
  - Predict failures
  - Forecast anomalies
  - Proactive alerts
  - Prevent

Future:
  - Zero incidents
  - Proactive
  - Reliable
""")

# TIMELINE
print("\n=== Timeline ===")
print("""
Past:
  2019: Monte Carlo
  2020: Growing market
  2021: Datafold, Elementary
  2022: Mainstream
  2023: AI features

Present:
  2024: Mature market
  2025: AI-native

Future:
  2030: Autonomous?
  2035: Standard?
  2040: New paradigms?
""")

print("\n=== Final Thoughts ===")
print("""
Data Observability:
  - 5 pillars
  - Monitoring
  - Alerting
  - Trust

The future is:
  - AI-native
  - Automated
  - Real-time
  - Unified
  - Predictive

"Trust your data.
 Monitor everything."

Keep exploring. Keep building.
""")