📦 DVC
HOME

🧠 DVC Fundamentals

🎯 What is DVC?

DVC (Data Version Control) is an open-source tool for versioning datasets, ML models, and pipelines. It works alongside Git, storing lightweight .dvc metadata files in Git and the actual data in remote storage (S3, GCS, Azure, SSH, local). It's "Git for data".

Why DVC Matters

Reproducible: Same data + code = same model. Versioned: Track every change. Collaborative: Share data via Git. Storage: Any remote. Pipelines: DAGs as code. Experiments: Compare runs.

📌 Git for Data📌 Reproducible ML📌 Remote Storage📌 Pipelines
# DVC & Data Versioning — the big picture # Version control for data, models, and ML pipelines. # CONCEPTUAL OVERVIEW print(""" === DVC (Data Version Control) === What it is: - Version control for data - Works with Git - Small .dvc files in Git - Actual data in remote storage - Pipelines and experiments Why it matters: - Reproducible ML - Track data changes - Collaborate - Roll back - Audit How it works: - git tracks code (small) - dvc tracks data (large) - .dvc files point to data - Remote storage holds data Analogy: - Git = code version control - DVC = data version control - Together = full ML reproducibility """) # WHY IT MATTERS print("=== The Problem ===") print(""" Without DVC: - Data not versioned - Hard to reproduce - No audit trail - Can't roll back - Collaboration hard - Models not tracked - "Which data trained this model?" With DVC: - Data versioned - Reproducible - Audit trail - Roll back - Collaborate - Models tracked - Full lineage """) # REAL-WORLD ADOPTION print("=== Who Uses DVC ===") print(""" Companies: - Iterative (creators) - Many ML teams - Startups to enterprise Use cases: - ML reproducibility - Data versioning - Model versioning - Pipeline orchestration - Experiment tracking - MLOps """)

🎯 Why Version Data

🎯 The Case for Data Versioning

Machine learning is 80% data work. Without versioning, you can't reproduce experiments, debug models, or collaborate. Data versioning solves this.

Why Version Data?

Reproducibility: Same results. Debugging: What changed? Compliance: Audit trail. Collaboration: Share safely. Rollback: Revert issues. Lineage: Track provenance.

🎯 Reproducibility🎯 Debugging🎯 Compliance🎯 Collaboration
# Why Version Data print("=== The ML Reproducibility Crisis ===") print(""" Problems: - "Which data trained this model?" - "Can we reproduce last month's result?" - "What changed between runs?" - "Why did accuracy drop?" - "Who changed the dataset?" Impact: - Wasted time - Lost trust - Can't audit - Can't roll back - Can't debug - Team friction Root cause: - Data not versioned - Code versioned (Git) - Data unversioned (chaos) """) print("\n=== Benefits ===") print(""" 1. REPRODUCIBILITY - Same data + code = same result - Time travel - Deterministic 2. DEBUGGING - What changed? - When? - By whom? - Diff data 3. COMPLIANCE - Audit trail - Provenance - Lineage 4. COLLABORATION - Share data - Track changes - Review 5. ROLLBACK - Revert bad data - Restore old version - Safe experimentation 6. LINEAGE - Data → Model - Model → Predictions - Full graph 7. STORAGE - Deduplication - Efficient - Versioned """) print("\n=== Use Cases ===") print(""" 1. ML EXPERIMENTS - Track hyperparameters - Track data versions - Compare runs 2. MODEL DEPLOYMENT - Which data trained model? - Reproduce - Roll back 3. DATA PIPELINES - Track transformations - Reproduce - Debug 4. TEAM COLLABORATION - Share datasets - Review changes - Version control 5. COMPLIANCE - Audit trail - Provenance - Explainability """) print("\n=== Real-World Example ===") print(""" Scenario: Model accuracy dropped Without DVC: - Don't know what changed - Data? Code? Env? - Days of debugging - Maybe never find With DVC: - Check data version - Check code version - Diff - Find cause - Roll back - Fix Time saved: Hours to minutes """)

📦 DVC Deep Dive

🎯 How DVC Works

DVC stores small .dvc files in Git and the actual data in remote storage. The .dvc file contains a hash and metadata that DVC uses to retrieve the data.

DVC Architecture

Git: Code + .dvc files. DVC: Data + models. Remote: Storage. Cache: Local. Hash: Content-addressable. Pointer: .dvc file.

📦 Git + DVC📦 Pointer Files📦 Remote Storage📦 Local Cache
# DVC Deep Dive print("=== How DVC Works ===") print(""" 1. TRACK DATA $ dvc add data.csv Creates: - data.csv.dvc (pointer) - .gitignore (adds data.csv) - .dvc/cache/ (stores data) 2. COMMIT TO GIT $ git add data.csv.dvc .gitignore $ git commit -m "Add dataset" 3. PUSH TO REMOTE $ dvc remote add -d myremote s3://bucket/dvc $ dvc push 4. CLONE AND PULL $ git clone repo $ dvc pull """) print("\n=== Pointer File ===") print(""" # data.csv.dvc outs: - md5: a1b2c3d4e5f6... size: 10485760 hash: md5 path: data.csv Small file (~100 bytes) Stored in Git Points to actual data """) print("\n=== Directory Structure ===") print(""" project/ ├── .git/ # Git ├── .dvc/ # DVC config │ ├── config # Remote config │ └── cache/ # Local cache ├── .gitignore # Data ignored ├── data.csv.dvc # Data pointer ├── model.pkl.dvc # Model pointer ├── dvc.yaml # Pipeline ├── dvc.lock # Pipeline lock ├── params.yaml # Hyperparameters ├── src/ # Code │ └── train.py └── data/ # Actual data (gitignored) └── data.csv """) print("\n=== Content-Addressable Storage ===") print(""" DVC uses MD5 hashes: - Content-addressable - Deduplication - Cache by hash - Efficient Example: data.csv → hash a1b2c3... Stored at: .dvc/cache/a1/b2c3... Same content → same hash → No duplication """) print("\n=== What DVC Tracks ===") print(""" - Datasets - Models - Metrics - Plots - Pipelines - Experiments - Artifacts What DVC Does NOT Track: - Code (Git does) - Small files - Config (unless via dvc.yaml) """) print("\n=== Workflow ===") print(""" 1. Init $ git init $ dvc init 2. Track data $ dvc add data.csv 3. Git commit $ git add . $ git commit -m "Add data" 4. Set remote $ dvc remote add -d storage s3://bucket 5. Push $ dvc push $ git push 6. Pull (teammate) $ git clone repo $ dvc pull 7. Update data $ dvc add data.csv # New version $ git add data.csv.dvc $ git commit -m "Update data" $ dvc push """)

⌨️ DVC Commands

🎯 Essential DVC Commands

DVC has a rich CLI. Here are the commands you'll use every day.

Commands

init: Setup. add: Track. push/pull: Sync. repro: Run pipeline. exp: Experiments. metrics: Compare.

⌨️ init⌨️ add⌨️ push⌨️ repro
# Essential DVC Commands print("=== Setup ===") print(""" # Initialize DVC (inside Git repo) dvc init # Configure user dvc config core.analytics false """) print("\n=== Data Tracking ===") print(""" # Track file dvc add data.csv # Track directory dvc add data/ # Track multiple dvc add data1.csv data2.csv # Remove from DVC dvc remove data.csv.dvc # Stop tracking dvc unprotect data.csv """) print("\n=== Remote Storage ===") print(""" # Add remote dvc remote add -d storage s3://my-bucket/dvc # Add without default dvc remote add myremote gs://my-bucket/dvc # Configure dvc remote modify storage region us-east-1 dvc remote modify storage profile my-profile # List dvc remote list # Push/pull dvc push dvc pull # Push specific dvc push data.csv.dvc dvc pull data.csv.dvc # Push to specific remote dvc push -r myremote """) print("\n=== Data Management ===") print(""" # Check status dvc status # List tracked dvc list . # Get info dvc list-url s3://bucket/dvc # Data versioning git log data.csv.dvc git diff HEAD~1 data.csv.dvc # Restore old version git checkout HEAD~1 data.csv.dvc dvc checkout """) print("\n=== Pipelines ===") print(""" # Create stage dvc stage add -n train \\ -d src/train.py \\ -d data/train.csv \\ -p learning_rate,n_estimators \\ -o models/model.pkl \\ -M metrics.json \\ python src/train.py # Run pipeline dvc repro # Reproduce specific stage dvc repro train # Visualize DAG dvc dag """) print("\n=== Experiments ===") print(""" # Run experiment dvc exp run # Run with parameters dvc exp run -S learning_rate=0.01 # List experiments dvc exp list # Show experiment dvc exp show # Compare dvc exp diff exp-1 exp-2 # Apply experiment to workspace dvc exp apply exp-1 # Push experiments dvc exp push origin exp-1 # Branch from experiment dvc exp branch exp-1 my-branch """) print("\n=== Metrics & Plots ===") print(""" # Show metrics dvc metrics show # Diff metrics dvc metrics diff # Show plots dvc plots show # Diff plots dvc plots diff # Compare to previous dvc metrics diff HEAD~1 """) print("\n=== Cache ===") print(""" # Check cache dvc cache dir # Verify cache dvc cache verify # Clear cache dvc cache clear # Garbage collection dvc gc """) print("\n=== Utility ===") print(""" # Version dvc version # Doctor dvc doctor # List all dvc list-repo . # Checkout data dvc checkout """) print("\n=== Quick Reference ===") print(f"{'Command':<20} {'Purpose':<40}") print("-" * 60) print(f"{'dvc init':<20} {'Initialize DVC':<40}") print(f"{'dvc add':<20} {'Track data':<40}") print(f"{'dvc push':<20} {'Upload to remote':<40}") print(f"{'dvc pull':<20} {'Download from remote':<40}") print(f"{'dvc status':<20} {'Check status':<40}") print(f"{'dvc repro':<20} {'Run pipeline':<40}") print(f"{'dvc exp':<20} {'Experiments':<40}") print(f"{'dvc metrics':<20} {'Show metrics':<40}") print(f"{'dvc plots':<20} {'Show plots':<40}") print(f"{'dvc dag':<20} {'Visualize DAG':<40}")

☁️ Remote Storage

🎯 Where Data Lives

DVC remotes store actual data outside Git. You can use cloud storage (S3, GCS, Azure), SSH servers, or local/NFS.

Remote Storage

S3: AWS. GCS: Google. Azure: Microsoft. SSH: Server. Local: Filesystem. HDFS: Hadoop.

☁️ S3☁️ GCS☁️ Azure☁️ SSH
# Remote Storage print("=== Supported Remotes ===") print(""" Cloud: - AWS S3 - Google Cloud Storage - Azure Blob Storage - Alibaba OSS - DigitalOcean Spaces On-prem: - SSH/SFTP - Local filesystem - NFS - HDFS - WebDAV Other: - Google Drive - Azure Data Lake - Custom """) print("\n=== Configure S3 ===") print(""" # Add remote dvc remote add -d storage s3://my-bucket/dvc-store # Configure credentials dvc remote modify storage access_key_id AKIA... dvc remote modify storage secret_access_key secret... # Or use profile dvc remote modify storage profile my-profile # Region dvc remote modify storage region us-east-1 # Endpoint (for MinIO) dvc remote modify storage endpointurl http://minio:9000 """) print("\n=== Configure GCS ===") print(""" # Add remote dvc remote add -d storage gs://my-bucket/dvc-store # Configure dvc remote modify storage credentialpath /path/to/creds.json # Or env var export GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json """) print("\n=== Configure Azure ===") print(""" # Add remote dvc remote add -d storage azure://my-container/dvc-store # Configure dvc remote modify storage account_name myaccount dvc remote modify storage account_key mykey dvc remote modify storage tenant_id mytenant dvc remote modify storage client_id myclient dvc remote modify storage client_secret mysecret """) print("\n=== Configure SSH ===") print(""" # Add remote dvc remote add -d storage ssh://user@host/path/to/dvc # Configure dvc remote modify storage port 22 dvc remote modify storage user myuser dvc remote modify storage keyfile /path/to/key dvc remote modify storage password mypass """) print("\n=== Configure Local ===") print(""" # Add remote dvc remote add -d storage /mnt/shared/dvc-store # Or NFS dvc remote add -d storage nfs://server/path """) print("\n=== Configure HDFS ===") print(""" # Add remote dvc remote add -d storage hdfs://namenode:8020/dvc-store # Configure dvc remote modify storage user hdfs """) print("\n=== Multiple Remotes ===") print(""" # Add multiple dvc remote add local /mnt/dvc dvc remote add cloud s3://bucket/dvc dvc remote add archive s3://archive/dvc # Set default dvc remote default cloud # Push to specific dvc push -r local dvc pull -r cloud # List dvc remote list """) print("\n=== Best Practices ===") print(""" 1. Use cloud for team 2. Use local for dev 3. Encrypt sensitive data 4. Lifecycle policies 5. Versioning enabled 6. Access control 7. Monitor costs 8. Backup Example config: dvc remote add -d storage s3://bucket/dvc dvc remote modify storage version_aware true dvc remote modify storage sse SSE-KMS dvc remote modify storage sse_kms_key_id key-id """)

🔗 DVC Pipelines

🎯 DAGs as Code

DVC pipelines are declarative DAGs (dvc.yaml) that define stages, dependencies, and outputs. DVC rebuilds only what changed and caches intermediate results.

DVC Pipelines

dvc.yaml: Pipeline definition. Stage: Unit of work. Deps: Inputs. Outs: Outputs. Repro: Rebuild. Cache: Skip unchanged.

🔗 dvc.yaml🔗 Stages🔗 Deps🔗 Cache
# DVC Pipelines print("=== What is a DVC Pipeline? ===") print(""" DVC Pipeline: - Declarative DAG - dvc.yaml file - Stages with commands - Dependencies (inputs) - Outputs - Parameters - Metrics Benefits: - Reproducible - Incremental - Cached - Lineage - Parallel """) print("\n=== dvc.yaml Example ===") print(""" # dvc.yaml stages: prepare: cmd: python src/prepare.py deps: - src/prepare.py - data/raw.csv params: - prepare.test_size - prepare.random_state outs: - data/train.csv - data/test.csv featurize: cmd: python src/featurize.py deps: - src/featurize.py - data/train.csv - data/test.csv outs: - data/train_features.pkl - data/test_features.pkl train: cmd: python src/train.py deps: - src/train.py - data/train_features.pkl params: - train.n_estimators - train.max_depth - train.learning_rate outs: - models/model.pkl metrics: - metrics.json: cache: false plots: - plots/roc.json: cache: false evaluate: cmd: python src/evaluate.py deps: - src/evaluate.py - models/model.pkl - data/test_features.pkl metrics: - eval_metrics.json: cache: false """) print("\n=== params.yaml ===") print(""" # params.yaml prepare: test_size: 0.2 random_state: 42 train: n_estimators: 100 max_depth: 10 learning_rate: 0.1 evaluate: threshold: 0.5 """) print("\n=== Create Stage via CLI ===") print(""" # Create stage dvc stage add -n train \\ -d src/train.py \\ -d data/train.pkl \\ -p train.n_estimators,train.max_depth \\ -o models/model.pkl \\ -M metrics.json \\ python src/train.py """) print("\n=== Run Pipeline ===") print(""" # Run all dvc repro # Run specific stage dvc repro train # Force re-run dvc repro --force # Dry run dvc repro --dry # Pipeline only dvc repro --pipeline """) print("\n=== Visualize ===") print(""" # DAG dvc dag # Output: ┌─────────┐ │ prepare │ └────┬────┘ │ ▼ ┌───────────┐ │ featurize │ └─────┬─────┘ │ ▼ ┌───────┐ │ train │ └───┬───┘ │ ▼ ┌──────────┐ │ evaluate │ └──────────┘ """) print("\n=== Lock File ===") print(""" # dvc.lock (auto-generated) schema: '2.0' stages: prepare: cmd: python src/prepare.py deps: - path: data/raw.csv md5: a1b2c3... params: params.yaml: prepare.test_size: 0.2 outs: - path: data/train.csv md5: d4e5f6... """) print("\n=== Benefits ===") print(""" 1. REPRODUCIBLE - Same inputs → same outputs 2. INCREMENTAL - Only rebuild changed 3. CACHED - Skip unchanged - Fast 4. LINEAGE - Track dependencies - Debug 5. PARALLEL - Independent stages - Faster 6. VERSIONED - dvc.yaml in Git - dvc.lock in Git - Data in remote """) print("\n=== Best Practices ===") print(""" 1. Small stages 2. Clear names 3. Define params 4. Track metrics 5. Commit dvc.yaml 6. Commit dvc.lock 7. Use params.yaml 8. Document """)

🧪 DVC Experiments

🎯 Compare ML Runs

DVC Experiments let you run, track, and compare ML experiments without polluting Git history. You can change hyperparameters, data, or code, and compare results.

Experiments

Run: exp run. Compare: exp show. Apply: exp apply. Branch: exp branch. Push: exp push. Track: Params, metrics.

🧪 Run🧪 Compare🧪 Apply🧪 Branch
# DVC Experiments print("=== What are DVC Experiments? ===") print(""" DVC Experiments: - Track ML runs - Compare metrics - No Git pollution - Reproduce - Versioned Use cases: - Hyperparameter tuning - Model comparison - A/B testing - Research - AutoML """) print("\n=== Run Experiment ===") print(""" # Run with defaults dvc exp run # Override params dvc exp run -S train.n_estimators=200 dvc exp run -S train.learning_rate=0.01 # Multiple dvc exp run -S train.n_estimators=100,200,500 # Set name dvc exp run --name my-experiment # Queue experiments dvc exp run --queue -S train.n_estimators=100 dvc exp run --queue -S train.n_estimators=200 dvc exp run --run-all # Parallel dvc exp run --jobs 4 """) print("\n=== Show Experiments ===") print(""" # List dvc exp list # Show table dvc exp show # Output: ┏━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━┓ ┃ Experiment ┃ Created ┃ metric1 ┃ metric2 ┃ ┡━━━━━━━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━┩ │ workspace │ - │ 0.92 │ 0.88 │ │ main │ Jan 15 │ 0.90 │ 0.85 │ │ ├── exp-1 │ 10:00 │ 0.91 │ 0.86 │ │ ├── exp-2 │ 10:05 │ 0.93 │ 0.89 │ │ └── exp-3 │ 10:10 │ 0.89 │ 0.84 │ └───────────────┴────────────┴─────────┴─────────┘ """) print("\n=== Compare ===") print(""" # Diff two experiments dvc exp diff exp-1 exp-2 # Diff with workspace dvc exp diff exp-1 # Output shows: - Metrics changed - Params changed - Plots diff """) print("\n=== Apply Experiment ===") print(""" # Apply to workspace dvc exp apply exp-1 # Push dvc exp push origin exp-1 # Pull dvc exp pull origin exp-1 # Branch from experiment dvc exp branch exp-1 my-branch git checkout my-branch """) print("\n=== Remove ===") print(""" # Remove specific dvc exp remove exp-1 # Remove all dvc exp remove --all # Garbage collect dvc gc """) print("\n=== Workflow ===") print(""" 1. BASELINE dvc exp run --name baseline 2. ITERATE dvc exp run -S train.n_estimators=200 --name exp-200 dvc exp run -S train.n_estimators=500 --name exp-500 3. COMPARE dvc exp show 4. CHOOSE BEST dvc exp apply exp-200 5. PROMOTE dvc exp branch exp-200 best-model git checkout best-model """) print("\n=== Example: Hyperparameter Tuning ===") print(""" # Grid search for n in 100 200 500; do for lr in 0.01 0.1 0.5; do dvc exp run --queue \\ -S train.n_estimators=$n \\ -S train.learning_rate=$lr done done dvc exp run --run-all --jobs 4 dvc exp show --sort-by avg_precision """) print("\n=== Benefits ===") print(""" 1. NO GIT POLLUTION - No commits - Clean history 2. FAST - Cache - Parallel 3. COMPARABLE - Table - Plots 4. REPRODUCIBLE - Any experiment - Any time 5. SHAREABLE - Push/pull - Team 6. SCALABLE - Queue - Parallel """)

📊 Metrics & Plots

🎯 Track What Matters

DVC tracks metrics (numbers) and plots (charts) across experiments. Compare runs visually and pick the best.

Metrics & Plots

Metrics: JSON/YAML. Plots: JSON/CSV. Show: Table. Diff: Compare. HTML: Reports. Automate: CI.

📊 Metrics📊 Plots📊 Diff📊 Reports
# Metrics & Plots print("=== Metrics ===") print(""" Metrics: - JSON, YAML, CSV - Numbers - Tracked per stage - Compared Example metrics.json: { "accuracy": 0.92, "precision": 0.90, "recall": 0.88, "f1": 0.89 } Define in dvc.yaml: metrics: - metrics.json """) print("\n=== Show Metrics ===") print(""" # Show all dvc metrics show # Show specific dvc metrics show metrics.json # JSON output dvc metrics show --json # All branches dvc metrics show --all-branches # All commits dvc metrics show --all-commits # All tags dvc metrics show --all-tags """) print("\n=== Diff Metrics ===") print(""" # Current vs HEAD dvc metrics diff # Between commits dvc metrics diff HEAD~1 HEAD # Between branches dvc metrics diff main experiment # Between experiments dvc metrics diff exp-1 exp-2 # Output: Path Metric Old New Change metrics.json accuracy 0.90 0.92 +0.02 metrics.json precision 0.88 0.90 +0.02 """) print("\n=== Plots ===") print(""" Plots: - JSON, CSV, TSV - Charts - HTML reports - Interactive Example plots/roc.json: { "fpr": [0, 0.1, 0.2, ...], "tpr": [0, 0.5, 0.7, ...] } Define in dvc.yaml: plots: - plots/roc.json: template: simple x: fpr y: tpr """) print("\n=== Show Plots ===") print(""" # Show all dvc plots show # Specific dvc plots show plots/roc.json # All branches dvc plots show --all-branches # Output HTML dvc plots show --html plots.html # Open in browser open plots.html """) print("\n=== Diff Plots ===") print(""" # Diff dvc plots diff # Between commits dvc plots diff HEAD~1 HEAD # Between experiments dvc plots diff exp-1 exp-2 # Output HTML dvc plots diff --html diff.html """) print("\n=== Plot Templates ===") print(""" Templates: - linear (default) - simple - scatter - confusion - confusion_normalized - smooth - custom (Vega-Lite) Example: plots: - plots/confusion.json: template: confusion x: actual y: predicted - plots/roc.json: template: simple x: fpr y: tpr """) print("\n=== Custom Plots ===") print(""" # Vega-Lite spec plots: - plots/data.json: template: custom spec: mark: bar encoding: x: {field: category, type: nominal} y: {field: value, type: quantitative} """) print("\n=== CI Integration ===") print(""" GitHub Actions: - Run DVC pipeline - Generate plots - Comment on PR - Compare metrics Example: - name: Run DVC run: dvc repro - name: Metrics run: dvc metrics diff --md >> $GITHUB_STEP_SUMMARY - name: Plots run: dvc plots diff --html plots.html """) print("\n=== Best Practices ===") print(""" 1. Track key metrics - Accuracy - Precision - Recall - F1 - AUC 2. Track plots - ROC - PR curve - Confusion matrix - Loss curves 3. Version metrics - Alongside code - Alongside data 4. Compare often - Every experiment - Every PR 5. Automate - CI/CD - Reports - Alerts """)

🗂️ Data Registry

🎯 Catalog Your Data

A Data Registry catalogs datasets and models. DVC supports registries via Git tags, branches, and DVC Studio to publish and consume versioned artifacts.

Data Registry

Versioned: Tags, branches. Discoverable: Search. Documented: Metadata. Importable: Reuse. Governed: Access. Integrated: CI/CD.

🗂️ Versioned🗂️ Discoverable🗂️ Importable🗂️ Governed
# Data Registry print("=== What is a Data Registry? ===") print(""" Data Registry: - Catalog of datasets/models - Versioned - Discoverable - Documented - Reusable - Governed Purpose: - Find data - Reuse - Version control - Collaboration - Governance DVC approaches: - Git tags - Git branches - DVC Studio - dvc import - dvc import-url """) print("\n=== Git Tags as Registry ===") print(""" # Tag a version git tag -a v1.0 -m "Dataset v1.0" dvc push git push --tags # Another version git tag -a v1.1 -m "Dataset v1.1" dvc push git push --tags # List git tag # Checkout git checkout v1.0 dvc checkout """) print("\n=== Import from Registry ===") print(""" # Import DVC data from another repo dvc import https://github.com/org/data-repo data/dataset.csv # Import URL dvc import-url s3://bucket/data.csv data/local.csv # Update import dvc update data/dataset.csv.dvc """) print("\n=== Import Structure ===") print(""" # After import project/ ├── data/ │ ├── dataset.csv.dvc # Import definition │ └── dataset.csv # Actual file (gitignored) └── .dvc/ # dataset.csv.dvc content: deps: - path: data/dataset.csv repo: url: https://github.com/org/data-repo rev_lock: abc123... outs: - md5: def456... size: 10485760 path: dataset.csv """) print("\n=== DVC Studio ===") print(""" DVC Studio: - Web UI for DVC - Visualize experiments - Compare runs - Share results - Team collaboration - Model registry Features: - Dashboard - Experiment tracking - Model registry - Data registry - Sharing - Integrations URL: studio.iterative.ai """) print("\n=== Model Registry ===") print(""" Model registry: - Versioned models - Metadata - Lineage - Approval - Deployment With DVC: - Track models with dvc add - Tag versions - Push to remote - Reference in code Example: dvc add models/model.pkl git add models/model.pkl.dvc git commit -m "Model v1.0" git tag -a model-v1.0 dvc push git push --tags """) print("\n=== Registry Workflow ===") print(""" 1. PUBLISH - Create dataset - dvc add - git commit - git tag - dvc push - git push --tags 2. DISCOVER - Browse registry - Read docs - Check metrics 3. CONSUME - dvc import - Use data - Track lineage 4. UPDATE - Check for updates - dvc update - Review changes 5. VERSION - Pin to tag - Or track latest """) print("\n=== Best Practices ===") print(""" 1. Semantic versioning - v1.0.0 - Major.minor.patch 2. Document - README - Schema - Provenance - License 3. Tag releases - git tag - dvc push 4. Access control - Private repos - Cloud IAM 5. Monitor - Usage - Cost - Quality 6. Update - Regular - Reviewed - Tested """)

⚖️ vs Alternatives

🎯 DVC vs Others

DVC competes with Git LFS, lakeFS, Pachyderm, Dolt, and others. Each has strengths.

Alternatives

Git LFS: Large files. lakeFS: Data lake versioning. Pachyderm: Data pipelines. Dolt: Versioned SQL. Delta: Table format. Iceberg: Table format.

⚖️ Git LFS⚖️ lakeFS⚖️ Pachyderm⚖️ Dolt
Tool Focus Best For
DVCML data versioningML teams, reproducibility
Git LFSLarge files in GitSimple large file storage
lakeFSData lake versioningS3/GCS with Git semantics
PachydermData pipelinesKubernetes-native data pipelines
DoltVersioned SQLGit for databases
Delta LakeTable formatACID tables in data lake
IcebergTable formatOpen table format, time travel
# DVC vs Git LFS print("=== DVC vs Git LFS ===") print(f"{'Aspect':<25} {'DVC':<25} {'Git LFS':<25}") print("-" * 75) print(f"{'Focus':<25} {'ML data':<25} {'Large files':<25}") print(f"{'Storage':<25} {'Any remote':<25} {'LFS server':<25}") print(f"{'Pipelines':<25} {'Yes':<25} {'No':<25}") print(f"{'Experiments':<25} {'Yes':<25} {'No':<25}") print(f"{'Metrics':<25} {'Yes':<25} {'No':<25}") print(f"{'Pointer files':<25} {'Yes':<25} {'Yes':<25}") print(f"{'Cost':<25} {'Pay storage':<25} {'Pay LFS':<25}") print(f"{'Best for':<25} {'ML':<25} {'Simple':<25}") print("\n=== DVC vs lakeFS ===") print(f"{'Aspect':<25} {'DVC':<25} {'lakeFS':<25}") print("-" * 75) print(f"{'Focus':<25} {'ML data':<25} {'Data lake':<25}") print(f"{'Storage':<25} {'Any':<25} {'S3/GCS':<25}") print(f"{'Versioning':<25} {'Git-like':<25} {'Git-like':<25}") print(f"{'Branches':<25} {'Git':<25} {'Native':<25}") print(f"{'Scale':<25} {'Small-medium':<25} {'Large':<25}") print(f"{'Setup':<25} {'Simple':<25} {'Complex':<25}") print(f"{'Best for':<25} {'ML':<25} {'Data lake':<25}") print("\n=== When to Use Which ===") print(""" USE DVC: ✓ ML workflows ✓ Small-medium data ✓ Git-based ✓ Reproducible ML ✓ Simple setup USE GIT LFS: ✓ Large files in Git ✓ Simple ✓ No ML features USE LAKEFS: ✓ Data lake ✓ Large scale ✓ S3/GCS ✓ Git semantics USE PACHYDERM: ✓ K8s-native ✓ Data pipelines ✓ Scale USE DOLT: ✓ Versioned SQL ✓ Git for DB USE DELTA/ICEBERG: ✓ Table format ✓ ACID ✓ Time travel """) print("\n=== Modern Stack ===") print(""" Common combinations: - DVC + Git + S3 - lakeFS + S3 + Spark - Pachyderm + K8s - Delta + Spark + Databricks - Iceberg + Trino + S3 Choose based on: - Scale - Use case - Team - Existing stack """)

✅ Best Practices

🎯 Doing DVC Right

Follow these best practices for reproducible, collaborative ML with DVC.

Best Practices

Git: Commit often. DVC: Push data. Pipelines: Declare. Params: Externalize. Metrics: Track. Document.

✅ Git Commit✅ DVC Push✅ Pipelines✅ Document
# Best Practices print("=== Repository Structure ===") print(""" project/ ├── .dvc/ # DVC config ├── .github/workflows/ # CI/CD ├── data/ │ ├── raw/ # Raw data (DVC) │ ├── interim/ # Intermediate (DVC) │ └── processed/ # Processed (DVC) ├── models/ # Models (DVC) ├── src/ │ ├── data/ │ ├── features/ │ ├── models/ │ └── visualization/ ├── notebooks/ ├── tests/ ├── params.yaml # Hyperparameters ├── dvc.yaml # Pipeline ├── dvc.lock # Lock file ├── requirements.txt ├── README.md └── .gitignore """) print("\n=== Git Hygiene ===") print(""" 1. Commit .dvc files git add data.csv.dvc git commit 2. Commit dvc.yaml - Pipeline definition - Always 3. Commit dvc.lock - Exact versions - Always 4. Commit .gitignore - DVC manages - Don't edit manually 5. Commit params.yaml - Hyperparameters - Version 6. Never commit data - Too large - DVC handles """) print("\n=== Workflow ===") print(""" Daily: 1. git pull 2. dvc pull 3. Make changes 4. dvc repro 5. dvc exp run (test) 6. dvc push 7. git add . && git commit 8. git push Weekly: - Review experiments - Update docs - Clean cache - Review metrics """) print("\n=== Params Management ===") print(""" 1. Externalize params # params.yaml train: n_estimators: 100 max_depth: 10 2. Reference in stages params: - train.n_estimators - train.max_depth 3. Override in experiments dvc exp run -S train.n_estimators=200 4. Version in Git git add params.yaml 5. Track changes dvc metrics diff """) print("\n=== Pipelines ===") print(""" 1. Small stages - One job per stage - Easy to debug - Cacheable 2. Clear names - prepare - featurize - train - evaluate 3. Declare deps - Files - Params - Stages 4. Declare outs - Files - Metrics - Plots 5. Use params - Externalize - Version 6. Document - Comments - README 7. Test - Small sample - CI """) print("\n=== Remote Storage ===") print(""" 1. Cloud storage - S3/GCS/Azure - Reliable - Scalable 2. Encryption - At rest - In transit 3. Versioning - Bucket versioning - Safety 4. Access control - IAM - Least privilege 5. Lifecycle - Archive old - Delete temp 6. Monitor - Cost - Usage 7. Backup - Cross-region - Multi-cloud 8. Document - Where - How - Who """) print("\n=== CI/CD ===") print(""" GitHub Actions: name: ML Pipeline on: [push] jobs: test: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: iterative/setup-dvc@v1 - uses: actions/setup-python@v4 - run: pip install -r requirements.txt - run: dvc pull - run: dvc repro - run: dvc metrics diff --md - run: dvc push """) print("\n=== Anti-Patterns ===") print(""" AVOID: ✗ Committing data to Git ✗ Not committing .dvc files ✗ Manual .gitignore edits ✗ Hardcoded params ✗ Monolithic stages ✗ No pipeline ✗ No remote ✗ No docs ✗ No CI/CD """) print("\n=== Checklist ===") print(""" ✓ Git + DVC initialized ✓ Remote configured ✓ Data tracked ✓ Pipelines defined ✓ Params externalized ✓ Metrics tracked ✓ Plots tracked ✓ Docs written ✓ CI/CD configured ✓ Team trained ✓ Backup configured ✓ Cost monitored """)

❓ FAQ

🎯 Frequently Asked Questions

Answers to common questions about DVC and data versioning.

What is DVC?

DVC (Data Version Control) is an open-source tool for versioning datasets, machine learning models, and pipelines. It works alongside Git, storing lightweight metadata in Git and the actual data in remote storage (S3, GCS, Azure, local).

Why do I need data versioning?

Data versioning lets you reproduce ML experiments, track which data trained which model, roll back to previous versions, and collaborate with teammates. Without it, you can't reliably reproduce results or audit decisions.

What is the difference between DVC and Git LFS?

Git LFS stores large files directly in Git's LFS server. DVC stores only small .dvc pointer files in Git and the actual data in any remote storage (S3, GCS, Azure, SSH, local). DVC also adds pipelines, metrics, and experiments.

What is a DVC pipeline?

A DVC pipeline (dvc.yaml) is a declarative definition of your ML workflow: stages, commands, dependencies, and outputs. DVC tracks the lineage, rebuilds only changed stages, and caches intermediate results.

What are DVC experiments?

DVC experiments let you run and track multiple ML experiments (hyperparameters, data, code) without polluting your Git history. You can compare metrics, visualize results, and reproduce any experiment.

Is DVC free?

Yes. DVC is open source (Apache 2.0). There's also DVC Studio (free tier) for team collaboration and a paid tier for larger teams. You pay only for your cloud storage.

What storage does DVC support?

DVC supports S3, Google Cloud Storage, Azure Blob, SSH/SFTP, local/NFS, HDFS, Google Drive, Alibaba OSS, DigitalOcean Spaces, and more. You can also have multiple remotes.

How does DVC handle large files?

DVC uses content-addressable storage with MD5 hashes. Files are stored in a cache by their hash, enabling deduplication and efficient versioning. Only the .dvc pointer file goes into Git.

Can I use DVC without Git?

Not recommended. DVC is designed to work alongside Git. You need Git for version control of code and DVC metadata (.dvc files). DVC without Git is possible but loses most benefits.

What is the difference between DVC and lakeFS?

DVC is best for ML workflows, small-medium data, and Git-based collaboration. lakeFS is best for data lakes at scale, providing Git-like semantics (branches, commits, merges) directly on S3/GCS objects. Both are versioning, but at different scales and use cases.

# Quick Reference print("=== Quick Answers ===") print(""" Q: Is DVC free? A: Yes — Apache 2.0. Q: Do I need Git? A: Yes — DVC works with Git. Q: Where is data stored? A: Remote (S3, GCS, etc.). Q: What's in Git? A: Only .dvc pointer files. Q: Can I version models? A: Yes — same as data. Q: Can I compare experiments? A: Yes — dvc exp show. Q: Can I use it with Python? A: Yes — DVC + Python. Q: Can I use it in CI/CD? A: Yes — GitHub Actions, GitLab CI. Q: Does it handle big data? A: Yes — up to PB. Q: Is it production-ready? A: Yes — used by many companies. """)

🚀 Future

🎯 Where DVC Is Going

The future is MLOps-native, cloud-first, AI-augmented, and integrated with the modern data stack.

Future Directions

MLOps: Integrated. Cloud: Native. AI: Automated. Scale: PB. Standards: Open. Democratize: Accessible.

🚀 MLOps🚀 Cloud🚀 AI🚀 Standards
# Future of DVC print("=== 1. MLOps Integration ===") print(""" Trend: DVC in MLOps stack Integration: - MLflow - Kubeflow - Airflow - Dagster - Weights & Biases - Neptune Future: - Standard for ML versioning - Integrated with orchestrators - End-to-end MLOps """) print("\n=== 2. Cloud-Native ===") print(""" Trend: Cloud-first Features: - Managed service - Serverless - Auto-scaling - Multi-cloud Future: - DVC Cloud - No setup - Integrated """) print("\n=== 3. AI-Augmented ===") print(""" Trend: AI in DVC AI: - Auto-detect data drift - Recommend experiments - Predict metrics - Optimize pipeline Future: - AI assistant - AutoML - Automated """) print("\n=== 4. Larger Scale ===") print(""" Trend: PB scale Features: - Distributed cache - Parallel transfer - Efficient storage Future: - PB scale - Exabyte? - Faster """) print("\n=== 5. Open Standards ===") print(""" Trend: Open wins Standards: - DVC format - Git integration - Remote protocols Future: - Standard - Interoperable - No lock-in """) print("\n=== 6. Democratization ===") print(""" Trend: Accessible Features: - Simple - Free - Open source - Docs Future: - Standard for ML - Every ML team - Easy to learn """) # TIMELINE print("\n=== Timeline ===") print(""" Past: 2017: DVC created 2018: Iterative 2020: DVC 1.0 2021: DVC 2.0 2022: Experiments 2023: DVC Studio Present: 2024: Mature 2025: MLOps standard Future: 2030: Standard? 2035: AI-native? 2040: New paradigms? """) print("\n=== Final Thoughts ===") print(""" DVC is: - Git for data - ML versioning - Pipelines - Experiments - Open source The future is: - MLOps-integrated - Cloud-native - AI-augmented - Standard - Accessible "Version your data. Reproduce your ML." Keep exploring. Keep building. """)

🛠️ Ecosystem

🎯 DVC Ecosystem

DVC integrates with a rich ecosystem of ML tools: cloud storage, orchestrators, experiment trackers, and MLOps platforms.

Ecosystem

Storage: S3, GCS, Azure. CI/CD: GitHub, GitLab. Orchestration: Airflow, Dagster. Tracking: MLflow, W&B. Platforms: Studio, Cloud. Languages: Python, CLI.

🛠️ Storage🛠️ Orchestration🛠️ Tracking🛠️ Platforms
# DVC Ecosystem print("=== Storage ===") print(""" Cloud: - AWS S3 - Google Cloud Storage - Azure Blob - Alibaba OSS - DigitalOcean Spaces On-prem: - SSH/SFTP - NFS - HDFS - Local filesystem Other: - Google Drive - Azure Data Lake """) print("\n=== Orchestration ===") print(""" Airflow: - DVC operator - Run pipeline - Track Dagster: - DVC integration - Assets Prefect: - DVC tasks Kubeflow: - DVC in pipelines """) print("\n=== Experiment Tracking ===") print(""" MLflow: - Track experiments - Compare with DVC Weights & Biases: - Visualization - Compare - DVC data Neptune: - Metadata - DVC integration DVC Studio: - Native DVC - Free tier """) print("\n=== CI/CD ===") print(""" GitHub Actions: - iterative/setup-dvc - Run pipeline - Cache GitLab CI: - DVC support - Pipeline Jenkins: - DVC plugin CircleCI: - DVC support """) print("\n=== Languages ===") print(""" Python: - dvc package - Python API CLI: - dvc command - Scripting Other: - Any language - Via CLI """) print("\n=== Platforms ===") print(""" DVC Studio: - Web UI - Experiments - Models - Data Iterative Cloud: - Managed - Team - Scale Self-hosted: - Open source - Your infra """) print("\n=== Integrations ===") print(f"{'Tool':<25} {'Purpose':<40}") print("-" * 65) print(f"{'S3/GCS/Azure':<25} {'Storage':<40}") print(f"{'GitHub/GitLab':<25} {'CI/CD':<40}") print(f"{'Airflow/Dagster':<25} {'Orchestration':<40}") print(f"{'MLflow/W&B':<25} {'Tracking':<40}") print(f"{'Studio':<25} {'Platform':<40}") print(f"{'Python':<25} {'API':<40}") print("\n=== Example Stack ===") print(""" Modern ML Stack with DVC: Code: Git + GitHub Data: DVC + S3 Pipeline: dvc.yaml Params: params.yaml Experiments: DVC Experiments Tracking: DVC Studio / MLflow Orchestration: Airflow Compute: AWS / GCP / Azure CI/CD: GitHub Actions Deployment: SageMaker / Vertex AI All versioned together. """) print("\n=== Getting Started ===") print(""" 1. Install DVC pip install dvc[s3] 2. Init git init dvc init 3. Track data dvc add data.csv git add . && git commit -m "Add data" 4. Configure remote dvc remote add -d storage s3://bucket/dvc 5. Push dvc push git push 6. Build pipeline dvc stage add ... 7. Run experiments dvc exp run 8. Collaborate dvc push git push """)