🧠 Evaluation Fundamentals

🎯 Why Model Evaluation Matters

Model evaluation is the process of measuring how well a machine learning model performs on unseen data. It's the bridge between training and deployment. Without proper evaluation, you can't know if your model is good, if it will generalize, or if it's ready for production.

Core Evaluation Concepts

Generalization: Performance on unseen data. Overfitting: Great on training, poor on test. Underfitting: Poor on both. Bias-variance tradeoff: Balance complexity. Metrics: Choose appropriate measures. Validation: Estimate true performance.

📌 Generalization📌 Metrics📌 Validation📌 Bias-Variance

# Model evaluation fundamentals
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Evaluate on TRAINING data
train_pred = model.predict(X_train)
train_acc = accuracy_score(y_train, train_pred)
print(f"Training accuracy: {train_acc:.4f}")

# Evaluate on TEST data
test_pred = model.predict(X_test)
test_acc = accuracy_score(y_test, test_pred)
print(f"Test accuracy: {test_acc:.4f}")

# The gap tells you about generalization
gap = train_acc - test_acc
print(f"\nGeneralization gap: {gap:.4f}")

if gap > 0.1:
    print("⚠️ Large gap: possible overfitting")
elif gap > 0.05:
    print("⚠️ Moderate gap: some overfitting")
else:
    print("✅ Small gap: good generalization")

# Three scenarios
print("\n=== Three Scenarios ===")
print("""
Underfitting:
  Train accuracy: LOW
  Test accuracy: LOW
  Gap: SMALL
  Fix: More complex model, more features

Good fit:
  Train accuracy: HIGH
  Test accuracy: HIGH
  Gap: SMALL
  Fix: None needed

Overfitting:
  Train accuracy: HIGH
  Test accuracy: LOW
  Gap: LARGE
  Fix: More data, regularization, simpler model
""")

✂️ Train/Val/Test Split

🎯 The Three-Way Split

Proper data splitting is critical for honest evaluation. You need: training set (fit model), validation set (tune hyperparameters), and test set (final evaluation). Never touch the test set until the very end.

Split Ratios

Typical: 60/20/20 or 70/15/15. Large data: 98/1/1. Small data: Use cross-validation. Stratified: Preserve class ratios. Temporal: For time series, split by time. Group: Keep related samples together.

📊 Train🎯 Val✅ Test⚖️ Stratified

# Data splitting strategies
import numpy as np
from sklearn.model_selection import train_test_split, StratifiedKFold, GroupKFold, TimeSeriesSplit
from sklearn.datasets import load_breast_cancer
import pandas as pd

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

print(f"Total samples: {len(X)}")
print(f"Class distribution: {np.bincount(y)}")

# 1. SIMPLE SPLIT (train/test)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
print(f"\n=== Simple Split ===")
print(f"Train: {len(X_train)}, Test: {len(X_test)}")

# 2. THREE-WAY SPLIT (train/val/test)
X_temp, X_test, y_temp, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_temp, y_temp, test_size=0.25, random_state=42  # 0.25 × 0.8 = 0.2
)
print(f"\n=== Three-Way Split ===")
print(f"Train: {len(X_train)}, Val: {len(X_val)}, Test: {len(X_test)}")

# 3. STRATIFIED SPLIT (preserve class ratios)
X_train_s, X_test_s, y_train_s, y_test_s = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42
)
print(f"\n=== Stratified Split ===")
print(f"Original class distribution: {np.bincount(y) / len(y)}")
print(f"Train class distribution:    {np.bincount(y_train_s) / len(y_train_s)}")
print(f"Test class distribution:     {np.bincount(y_test_s) / len(y_test_s)}")

# 4. TIME SERIES SPLIT
print(f"\n=== Time Series Split ===")
# Create synthetic time series
n = 100
X_ts = np.arange(n).reshape(-1, 1)
y_ts = np.arange(n) + np.random.randn(n)

tscv = TimeSeriesSplit(n_splits=5)
for fold, (train_idx, test_idx) in enumerate(tscv.split(X_ts)):
    print(f"Fold {fold+1}: Train [{train_idx[0]}-{train_idx[-1]}], "
          f"Test [{test_idx[0]}-{test_idx[-1]}]")

# 5. GROUP SPLIT (keep related samples together)
print(f"\n=== Group Split ===")
# Example: multiple images per patient
groups = np.repeat(np.arange(20), 5)  # 20 patients, 5 samples each
X_grouped = np.random.randn(100, 5)
y_grouped = np.random.randint(0, 2, 100)

gkf = GroupKFold(n_splits=5)
for fold, (train_idx, test_idx) in enumerate(gkf.split(X_grouped, y_grouped, groups)):
    train_groups = np.unique(groups[train_idx])
    test_groups = np.unique(groups[test_idx])
    print(f"Fold {fold+1}: Train groups={len(train_groups)}, "
          f"Test groups={len(test_groups)}")

# 6. COMMON MISTAKES
print(f"\n=== Common Mistakes ===")
print("""
❌ Splitting AFTER preprocessing
   Fix: Split first, then preprocess (fit on train only)

❌ Using test set for hyperparameter tuning
   Fix: Use validation set or cross-validation

❌ Random split for time series
   Fix: Use TimeSeriesSplit (past → future)

❌ Ignoring class imbalance in split
   Fix: Use stratify=y

❌ Splitting related samples
   Fix: Use GroupKFold

❌ Not setting random_state
   Fix: Set seed for reproducibility
""")

# 7. Correct preprocessing order
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer

print("=== Correct Preprocessing Order ===")
print("""
# WRONG ❌
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # Uses ALL data!
X_train, X_test = train_test_split(X_scaled, ...)

# CORRECT ✅
X_train, X_test = train_test_split(X, ...)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)  # Fit on train only
X_test = scaler.transform(X_test)          # Transform test
""")

# Using Pipeline to avoid mistakes
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000))
])

# Fit on train, evaluate on test
pipeline.fit(X_train, y_train)
test_acc = pipeline.score(X_test, y_test)
print(f"Pipeline test accuracy: {test_acc:.4f}")
print("Pipeline ensures no data leakage!")

🔄 Cross-Validation

🎯 Better Performance Estimation

Cross-validation splits data into K folds, trains K models, and averages results. It provides a more robust estimate of performance and uses data efficiently. K-fold is the most common; stratified preserves class ratios.

CV Variants

K-Fold: Standard K-fold CV. Stratified: Preserves class ratios. Leave-One-Out: K=N (expensive). Repeated: Multiple runs for stability. Time Series: For temporal data. Nested: For hyperparameter tuning.

🔄 K-Fold⚖️ Stratified📊 Repeated⏱️ Time Series

# Cross-validation in practice
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import (
    cross_val_score, cross_validate, KFold, 
    StratifiedKFold, LeaveOneOut, RepeatedStratifiedKFold
)
from sklearn.metrics import accuracy_score, precision_score, recall_score

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

# 1. BASIC K-FOLD CV
print("=== Basic K-Fold CV ===")
model = LogisticRegression(max_iter=1000)

for k in [3, 5, 10]:
    scores = cross_val_score(model, X, y, cv=k, scoring='accuracy')
    print(f"K={k}: accuracy = {scores.mean():.4f} (±{scores.std():.4f})")

# 2. STRATIFIED K-FOLD (default for classifiers)
print("\n=== Stratified K-Fold ===")
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(model, X, y, cv=skf, scoring='accuracy')
print(f"Stratified CV: {scores.mean():.4f} (±{scores.std():.4f})")

# 3. MULTIPLE METRICS
print("\n=== Multiple Metrics ===")
scoring = {
    'accuracy': 'accuracy',
    'precision': 'precision',
    'recall': 'recall',
    'f1': 'f1',
    'roc_auc': 'roc_auc'
}

results = cross_validate(model, X, y, cv=5, scoring=scoring)
print("Cross-validation results:")
for metric in scoring:
    scores = results[f'test_{metric}']
    print(f"  {metric}: {scores.mean():.4f} (±{scores.std():.4f})")

# 4. COMPARE MODELS
print("\n=== Model Comparison ===")
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'Decision Tree': DecisionTreeClassifier(random_state=42),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42)
}

for name, model in models.items():
    scores = cross_val_score(model, X, y, cv=5, scoring='accuracy')
    print(f"{name}: {scores.mean():.4f} (±{scores.std():.4f})")

# 5. REPEATED CV
print("\n=== Repeated Stratified CV ===")
rskf = RepeatedStratifiedKFold(n_splits=5, n_repeats=10, random_state=42)
scores = cross_val_score(model, X, y, cv=rskf, scoring='accuracy')
print(f"Repeated CV: {scores.mean():.4f} (±{scores.std():.4f})")
print(f"Total fits: {len(scores)}")

# 6. LEAVE-ONE-OUT (for small data)
print("\n=== Leave-One-Out CV ===")
# Use small subset for LOO
X_small = X[:50]
y_small = y[:50]

loo = LeaveOneOut()
scores = cross_val_score(model, X_small, y_small, cv=loo, scoring='accuracy')
print(f"LOO accuracy: {scores.mean():.4f}")
print(f"Number of fits: {len(scores)} (expensive!)")

# 7. NESTED CV (for hyperparameter tuning)
print("\n=== Nested CV ===")
from sklearn.model_selection import GridSearchCV

inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

param_grid = {'C': [0.1, 1, 10]}

# Inner CV for hyperparameter tuning
clf = GridSearchCV(
    LogisticRegression(max_iter=1000),
    param_grid,
    cv=inner_cv
)

# Outer CV for unbiased performance estimate
nested_scores = cross_val_score(clf, X, y, cv=outer_cv, scoring='accuracy')
print(f"Nested CV: {nested_scores.mean():.4f} (±{nested_scores.std():.4f})")

# 8. WHEN TO USE WHAT
print("\n=== Choosing CV Strategy ===")
print("""
Small dataset (< 1000):
  → Leave-One-Out or 10-fold CV

Medium dataset (1000-100k):
  → 5-fold or 10-fold CV

Large dataset (> 100k):
  → 3-fold CV or holdout

Imbalanced classes:
  → Stratified K-Fold

Time series:
  → TimeSeriesSplit (respects order)

Grouped data:
  → GroupKFold (keep groups together)

Hyperparameter tuning:
  → Nested CV for unbiased estimate
""")

🎯 Classification Metrics

🎯 Measuring Classification

Classification has many metrics, each capturing a different aspect: accuracy, precision, recall, F1, AUC. The right choice depends on the problem.

Key Metrics

Accuracy: Correct / total. Good for balanced classes. Precision: TP / (TP + FP). Minimize false positives. Recall: TP / (TP + FN). Minimize false negatives. F1: Harmonic mean of precision and recall. AUC: Area under ROC curve. Log loss: Penalizes confident wrong predictions.

✅ Accuracy🎯 Precision📞 Recall⚖️ F1

# Classification metrics in detail
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    classification_report, confusion_matrix, log_loss,
    balanced_accuracy_score, matthews_corrcoef, cohen_kappa_score
)

# Load and split data
data = load_breast_cancer()
X, y = data.data, data.target
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

# Predictions
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

# 1. ACCURACY
acc = accuracy_score(y_test, y_pred)
print(f"Accuracy: {acc:.4f}")
print("  Correct predictions / total predictions")

# 2. PRECISION
prec = precision_score(y_test, y_pred)
print(f"\nPrecision: {prec:.4f}")
print("  Of predicted positives, how many are correct?")
print("  TP / (TP + FP)")
print("  Use when false positives are costly (spam filter)")

# 3. RECALL
rec = recall_score(y_test, y_pred)
print(f"\nRecall: {rec:.4f}")
print("  Of actual positives, how many did we find?")
print("  TP / (TP + FN)")
print("  Use when false negatives are costly (cancer detection)")

# 4. F1 SCORE
f1 = f1_score(y_test, y_pred)
print(f"\nF1 Score: {f1:.4f}")
print("  Harmonic mean of precision and recall")
print("  2 * (P * R) / (P + R)")
print("  Use when you need balance")

# 5. BALANCED ACCURACY
bal_acc = balanced_accuracy_score(y_test, y_pred)
print(f"\nBalanced Accuracy: {bal_acc:.4f}")
print("  Average of sensitivity and specificity")
print("  Better for imbalanced classes")

# 6. MATTHEWS CORRELATION COEFFICIENT
mcc = matthews_corrcoef(y_test, y_pred)
print(f"\nMCC: {mcc:.4f}")
print("  Range: -1 to +1")
print("  Good for imbalanced data")

# 7. COHEN'S KAPPA
kappa = cohen_kappa_score(y_test, y_pred)
print(f"\nCohen's Kappa: {kappa:.4f}")
print("  Agreement above chance")
print("  Range: -1 to +1")

# 8. LOG LOSS
logloss = log_loss(y_test, y_proba)
print(f"\nLog Loss: {logloss:.4f}")
print("  Penalizes confident wrong predictions")
print("  Lower is better")

# 9. COMPLETE REPORT
print("\n=== Classification Report ===")
print(classification_report(y_test, y_pred, target_names=data.target_names))

# 10. CONFUSION MATRIX
print("=== Confusion Matrix ===")
cm = confusion_matrix(y_test, y_pred)
print(cm)
print("\nStructure:")
print(f"  TN={cm[0,0]}, FP={cm[0,1]}")
print(f"  FN={cm[1,0]}, TP={cm[1,1]}")

# 11. WHEN TO USE WHAT
print("\n=== Metric Selection Guide ===")
print("""
Accuracy:
  ✓ Balanced classes
  ✓ All errors equally costly
  ✗ Imbalanced classes

Precision:
  ✓ False positives costly
  ✓ Spam filter (marking ham as spam)
  ✓ Criminal conviction

Recall:
  ✓ False negatives costly
  ✓ Cancer detection (miss disease)
  ✓ Fraud detection

F1 Score:
  ✓ Balance of precision and recall
  ✓ Imbalanced classes
  ✓ General purpose

AUC-ROC:
  ✓ Ranking quality
  ✓ Imbalanced classes
  ✓ Comparing models

Log Loss:
  ✓ Need calibrated probabilities
  ✓ Ranking confidence matters

MCC:
  ✓ Imbalanced classes
  ✓ Single-number summary
  ✓ Range [-1, 1]
""")

# 12. Example scenarios
print("=== Example Scenarios ===")
print("""
Cancer Screening:
  Recall is critical (don't miss cases)
  Some false positives acceptable
  → Optimize recall, then precision

Spam Filter:
  Precision is critical (don't block real email)
  Some spam may slip through
  → Optimize precision

Fraud Detection:
  Balance needed (both errors costly)
  F1 or AUC
  → Optimize F1

Loan Approval:
  Regulatory requirements
  Need explainability
  → Depends on regulation

Search Ranking:
  Ranking quality matters
  → NDCG, MRR, MAP
""")

📈 Regression Metrics

🎯 Measuring Regression

Regression metrics measure how close predictions are to actual values: MSE, RMSE, MAE, MAPE, R². The right choice depends on what you care about.

Key Metrics

MSE: Mean squared error — penalizes large errors. RMSE: Root MSE — same units as target. MAE: Mean absolute error — robust to outliers. MAPE: Mean absolute percentage error — scale-free. R²: Coefficient of determination — fraction of variance explained.

📊 MSE📈 RMSE📉 MAE🎯 R²

# Regression metrics in detail
import numpy as np
from sklearn.datasets import fetch_california_housing
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression, Ridge
from sklearn.ensemble import RandomForestRegressor
from sklearn.metrics import (
    mean_squared_error, mean_absolute_error, 
    r2_score, mean_absolute_percentage_error,
    median_absolute_error, explained_variance_score
)

# Load data
data = fetch_california_housing()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train model
model = RandomForestRegressor(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# 1. MSE — Mean Squared Error
mse = mean_squared_error(y_test, y_pred)
print(f"MSE: {mse:.4f}")
print("  Average of squared errors")
print("  Penalizes large errors heavily")
print("  Units: squared target units")

# 2. RMSE — Root Mean Squared Error
rmse = np.sqrt(mse)
print(f"\nRMSE: {rmse:.4f}")
print("  Square root of MSE")
print("  Same units as target")
print("  Most interpretable")

# 3. MAE — Mean Absolute Error
mae = mean_absolute_error(y_test, y_pred)
print(f"\nMAE: {mae:.4f}")
print("  Average of absolute errors")
print("  Robust to outliers")
print("  Same units as target")

# 4. MAPE — Mean Absolute Percentage Error
mape = mean_absolute_percentage_error(y_test, y_pred)
print(f"\nMAPE: {mape:.4f} ({mape*100:.2f}%)")
print("  Average percentage error")
print("  Scale-free")
print("  Problematic when y close to 0")

# 5. Median Absolute Error
med_ae = median_absolute_error(y_test, y_pred)
print(f"\nMedian Absolute Error: {med_ae:.4f}")
print("  Median of absolute errors")
print("  Very robust to outliers")

# 6. R² — Coefficient of Determination
r2 = r2_score(y_test, y_pred)
print(f"\nR² Score: {r2:.4f}")
print("  Proportion of variance explained")
print("  Range: -∞ to 1")
print("  1 = perfect, 0 = same as mean")
print(f"  {r2*100:.2f}% of variance explained")

# 7. Explained Variance
ev = explained_variance_score(y_test, y_pred)
print(f"\nExplained Variance: {ev:.4f}")
print("  Similar to R² but doesn't penalize bias")

# 8. Comparison of models
print("\n=== Model Comparison ===")
models = {
    'Linear Regression': LinearRegression(),
    'Ridge': Ridge(),
    'Random Forest': RandomForestRegressor(n_estimators=100, random_state=42)
}

print(f"{'Model':<20} {'RMSE':<10} {'MAE':<10} {'R²':<10}")
print("-" * 50)

for name, model in models.items():
    model.fit(X_train, y_train)
    y_pred = model.predict(X_test)
    
    rmse = np.sqrt(mean_squared_error(y_test, y_pred))
    mae = mean_absolute_error(y_test, y_pred)
    r2 = r2_score(y_test, y_pred)
    
    print(f"{name:<20} {rmse:<10.4f} {mae:<10.4f} {r2:<10.4f}")

# 9. Metric selection guide
print("\n=== When to Use What ===")
print("""
MSE:
  ✓ Mathematical convenience
  ✓ Differentiable (for optimization)
  ✗ Hard to interpret (squared units)
  ✗ Sensitive to outliers

RMSE:
  ✓ Same units as target
  ✓ Most interpretable
  ✓ Penalizes large errors
  ✗ Sensitive to outliers

MAE:
  ✓ Robust to outliers
  ✓ Same units as target
  ✓ Interpretable
  ✗ Doesn't penalize large errors enough

MAPE:
  ✓ Scale-free (percentage)
  ✓ Comparable across datasets
  ✗ Breaks when y ≈ 0
  ✗ Asymmetric

R²:
  ✓ Standard metric
  ✓ Comparable across models
  ✓ Range [−∞, 1]
  ✗ Can be negative (worse than mean)

Median AE:
  ✓ Very robust
  ✗ Ignores most data
""")

# 10. Handling outliers
print("=== Handling Outliers ===")

# Add outliers to test data
y_test_outlier = y_test.copy()
y_test_outlier[:10] = 100  # Add some outliers

y_pred_outlier = model.predict(X_test)
y_pred_outlier[:10] = 50  # Predictions also affected

mse_clean = mean_squared_error(y_test, y_pred)
mse_dirty = mean_squared_error(y_test_outlier, y_pred_outlier)

mae_clean = mean_absolute_error(y_test, y_pred)
mae_dirty = mean_absolute_error(y_test_outlier, y_pred_outlier)

print(f"MSE: {mse_clean:.4f} → {mse_dirty:.4f} ({(mse_dirty/mse_clean):.1f}x)")
print(f"MAE: {mae_clean:.4f} → {mae_dirty:.4f} ({(mae_dirty/mae_clean):.1f}x)")
print("\nMSE is much more sensitive to outliers!")

🔢 Confusion Matrix

🎯 The Complete Picture

The confusion matrix shows all four outcomes of binary classification: True Positives, True Negatives, False Positives, False Negatives. It's the foundation for precision, recall, and many other metrics.

The Four Outcomes

TP: Predicted positive, actually positive. TN: Predicted negative, actually negative. FP: Predicted positive, actually negative (Type I error). FN: Predicted negative, actually positive (Type II error). Multiclass: Extends to K×K matrix. Normalized: Shows rates instead of counts.

✅ TP✅ TN❌ FP❌ FN

# Confusion matrix in depth
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import (
    confusion_matrix, ConfusionMatrixDisplay,
    classification_report, precision_score, recall_score,
    f1_score, accuracy_score, specificity_score
)

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)

# 1. BASIC CONFUSION MATRIX
cm = confusion_matrix(y_test, y_pred)
print("=== Confusion Matrix ===")
print(cm)

TN, FP, FN, TP = cm.ravel()
print(f"\nTP = {TP}: Predicted positive, actually positive")
print(f"TN = {TN}: Predicted negative, actually negative")
print(f"FP = {FP}: Predicted positive, actually negative (Type I)")
print(f"FN = {FN}: Predicted negative, actually positive (Type II)")

# 2. DERIVED METRICS
print("\n=== Derived Metrics ===")

# Accuracy
accuracy = (TP + TN) / (TP + TN + FP + FN)
print(f"Accuracy: {accuracy:.4f} = (TP+TN) / Total")

# Precision
precision = TP / (TP + FP)
print(f"Precision: {precision:.4f} = TP / (TP+FP)")

# Recall (Sensitivity, TPR)
recall = TP / (TP + FN)
print(f"Recall: {recall:.4f} = TP / (TP+FN)")

# Specificity (TNR)
specificity = TN / (TN + FP)
print(f"Specificity: {specificity:.4f} = TN / (TN+FP)")

# F1
f1 = 2 * precision * recall / (precision + recall)
print(f"F1: {f1:.4f} = 2PR / (P+R)")

# NPV (Negative Predictive Value)
npv = TN / (TN + FN)
print(f"NPV: {npv:.4f} = TN / (TN+FN)")

# FPR (False Positive Rate)
fpr = FP / (FP + TN)
print(f"FPR: {fpr:.4f} = FP / (FP+TN)")

# FNR (False Negative Rate)
fnr = FN / (FN + TP)
print(f"FNR: {fnr:.4f} = FN / (FN+TP)")

# 3. NORMALIZED CONFUSION MATRIX
print("\n=== Normalized Confusion Matrix ===")
cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
print(cm_norm.round(4))
print("Rows sum to 1 (percentages)")

# 4. VISUALIZATION
fig, axes = plt.subplots(1, 3, figsize=(18, 5))

# Raw counts
disp1 = ConfusionMatrixDisplay(cm, display_labels=data.target_names)
disp1.plot(ax=axes[0], cmap='Blues', colorbar=False)
axes[0].set_title('Confusion Matrix (Counts)')

# Normalized
disp2 = ConfusionMatrixDisplay(cm_norm, display_labels=data.target_names)
disp2.plot(ax=axes[1], cmap='Blues', values_format='.2%', colorbar=False)
axes[1].set_title('Normalized by True Label')

# All metrics
metrics_text = f"""
Accuracy: {accuracy:.3f}
Precision: {precision:.3f}
Recall: {recall:.3f}
Specificity: {specificity:.3f}
F1: {f1:.3f}
NPV: {npv:.3f}
"""
axes[2].text(0.1, 0.5, metrics_text, fontsize=14, family='monospace')
axes[2].axis('off')
axes[2].set_title('Derived Metrics')

plt.tight_layout()
plt.show()

# 5. MULTICLASS CONFUSION MATRIX
print("\n=== Multiclass Example ===")
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier

iris = load_iris()
X_iris, y_iris = iris.data, iris.target

X_train, X_test, y_train, y_test = train_test_split(
    X_iris, y_iris, test_size=0.3, random_state=42, stratify=y_iris
)

model_iris = RandomForestClassifier(n_estimators=100, random_state=42)
model_iris.fit(X_train, y_train)
y_pred_iris = model_iris.predict(X_test)

cm_iris = confusion_matrix(y_test, y_pred_iris)
print("Iris Confusion Matrix:")
print(cm_iris)

# 6. CLASSIFICATION REPORT
print("\n=== Classification Report ===")
print(classification_report(y_test, y_pred_iris, 
                           target_names=iris.target_names))

# 7. WHEN TO USE WHAT
print("=== Interpreting Confusion Matrix ===")
print(f"""
Medical Screening (Disease Detection):
  Class 1 = Disease
  TP = {TP}: Correctly detected disease
  FN = {FN}: Missed disease (dangerous!)
  FP = {FP}: False alarm (anxiety, cost)
  TN = {TN}: Correctly identified healthy
  
  Priority: Minimize FN (high recall)

Spam Filter:
  Class 1 = Spam
  TP = {TP}: Correctly blocked spam
  FP = {FP}: Legitimate email blocked (bad!)
  FN = {FN}: Spam in inbox (annoying)
  TN = {TN}: Legitimate email delivered
  
  Priority: Minimize FP (high precision)
""")

# 8. COST-SENSITIVE ANALYSIS
print("=== Cost-Sensitive Analysis ===")

# Different costs for FP and FN
costs = {
    'FP cost = 1, FN cost = 1': (1, 1),
    'FP cost = 1, FN cost = 10': (1, 10),
    'FP cost = 10, FN cost = 1': (10, 1),
}

for desc, (cost_fp, cost_fn) in costs.items():
    total_cost = FP * cost_fp + FN * cost_fn
    print(f"{desc}: Total cost = {total_cost}")

# 9. THRESHOLD TUNING
print("\n=== Threshold Tuning ===")
y_proba = model.predict_proba(X_test)[:, 1]

thresholds = [0.3, 0.5, 0.7]
for threshold in thresholds:
    y_pred_t = (y_proba > threshold).astype(int)
    cm_t = confusion_matrix(y_test, y_pred_t)
    TN, FP, FN, TP = cm_t.ravel()
    
    precision_t = TP / (TP + FP) if (TP + FP) > 0 else 0
    recall_t = TP / (TP + FN) if (TP + FN) > 0 else 0
    
    print(f"Threshold {threshold}: Precision={precision_t:.3f}, "
          f"Recall={recall_t:.3f}, TP={TP}, FP={FP}, FN={FN}")

📉 ROC & AUC

🎯 Ranking Quality

The ROC curve plots True Positive Rate vs False Positive Rate at all thresholds. AUC (Area Under Curve) summarizes the curve as a single number. It measures how well the model ranks positives above negatives.

Understanding ROC

TPR: True Positive Rate (Recall, Sensitivity). FPR: False Positive Rate = 1 - Specificity. AUC = 0.5: Random guessing. AUC = 1.0: Perfect classifier. AUC = 0.8: Good. AUC = 0.7: Fair. AUC = 0.6: Poor.

📈 TPR📉 FPR🎯 AUC⚖️ Threshold

# ROC curve and AUC
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer, make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.metrics import (
    roc_curve, roc_auc_score, auc,
    RocCurveDisplay, precision_recall_curve,
    average_precision_score
)

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# 1. TRAIN MODELS
models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'SVM': SVC(probability=True, random_state=42)
}

# Get probabilities
trained_models = {}
for name, model in models.items():
    model.fit(X_train, y_train)
    trained_models[name] = model

# 2. ROC CURVE
fig, ax = plt.subplots(figsize=(10, 8))

for name, model in trained_models.items():
    y_proba = model.predict_proba(X_test)[:, 1]
    fpr, tpr, thresholds = roc_curve(y_test, y_proba)
    auc_score = roc_auc_score(y_test, y_proba)
    
    ax.plot(fpr, tpr, label=f'{name} (AUC = {auc_score:.3f})', linewidth=2)
    
    print(f"\n{name}:")
    print(f"  AUC: {auc_score:.4f}")
    print(f"  FPR range: {fpr.min():.4f} to {fpr.max():.4f}")
    print(f"  TPR range: {tpr.min():.4f} to {tpr.max():.4f}")

# Diagonal (random)
ax.plot([0, 1], [0, 1], 'k--', label='Random (AUC = 0.5)', linewidth=1)

ax.set_xlabel('False Positive Rate (1 - Specificity)', fontsize=12)
ax.set_ylabel('True Positive Rate (Sensitivity)', fontsize=12)
ax.set_title('ROC Curves', fontsize=14)
ax.legend(loc='lower right')
ax.grid(alpha=0.3)
ax.set_xlim([0, 1])
ax.set_ylim([0, 1.02])

plt.tight_layout()
plt.show()

# 3. UNDERSTANDING AUC
print("\n=== Understanding AUC ===")
print("""
AUC = Probability that model ranks a random positive
      higher than a random negative

Interpretation:
  1.0   = Perfect ranking
  0.9+  = Excellent
  0.8-0.9 = Good
  0.7-0.8 = Fair
  0.6-0.7 = Poor
  0.5   = Random
  <0.5  = Worse than random
""")

# 4. THRESHOLD ANALYSIS
print("=== Threshold Analysis ===")
model = trained_models['Logistic Regression']
y_proba = model.predict_proba(X_test)[:, 1]

fpr, tpr, thresholds = roc_curve(y_test, y_proba)

# Find optimal thresholds
# Youden's J statistic
j_scores = tpr - fpr
optimal_idx = np.argmax(j_scores)
optimal_threshold = thresholds[optimal_idx]

print(f"Optimal threshold (Youden's J): {optimal_threshold:.4f}")
print(f"  TPR: {tpr[optimal_idx]:.4f}")
print(f"  FPR: {fpr[optimal_idx]:.4f}")
print(f"  J-score: {j_scores[optimal_idx]:.4f}")

# Threshold for high recall
recall_target = 0.95
recall_idx = np.where(tpr >= recall_target)[0]
if len(recall_idx) > 0:
    high_recall_idx = recall_idx[0]
    print(f"\nThreshold for 95% recall: {thresholds[high_recall_idx]:.4f}")
    print(f"  TPR: {tpr[high_recall_idx]:.4f}")
    print(f"  FPR: {fpr[high_recall_idx]:.4f}")

# 5. PRECISION-RECALL CURVE
print("\n=== Precision-Recall Curve ===")
fig, axes = plt.subplots(1, 2, figsize=(16, 6))

for name, model in trained_models.items():
    y_proba = model.predict_proba(X_test)[:, 1]
    precision, recall, _ = precision_recall_curve(y_test, y_proba)
    ap = average_precision_score(y_test, y_proba)
    
    axes[0].plot(recall, precision, label=f'{name} (AP = {ap:.3f})', linewidth=2)
    print(f"{name}: Average Precision = {ap:.4f}")

axes[0].set_xlabel('Recall')
axes[0].set_ylabel('Precision')
axes[0].set_title('Precision-Recall Curves')
axes[0].legend()
axes[0].grid(alpha=0.3)

# 6. COMPARING ROC AND PR
print("\n=== ROC vs PR Curves ===")
print("""
ROC Curve:
  ✓ Good for balanced classes
  ✓ Threshold-independent
  ✓ Well-understood
  ✗ Can be optimistic for imbalanced data

PR Curve:
  ✓ Better for imbalanced classes
  ✓ Focuses on positive class
  ✓ More informative when positives are rare
  ✗ Harder to interpret

Use ROC when:
  - Classes are balanced
  - Both classes equally important

Use PR when:
  - Classes are imbalanced
  - Positive class is more important
  - False positives are costly
""")

# 7. IMBALANCED DATA EXAMPLE
print("=== Imbalanced Data Demo ===")
X_imb, y_imb = make_classification(
    n_samples=1000, n_features=20, 
    weights=[0.95, 0.05], random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X_imb, y_imb, test_size=0.2, random_state=42
)

model_imb = LogisticRegression(max_iter=1000, class_weight='balanced')
model_imb.fit(X_train, y_train)
y_proba_imb = model_imb.predict_proba(X_test)[:, 1]

roc_auc = roc_auc_score(y_test, y_proba_imb)
ap = average_precision_score(y_test, y_proba_imb)

print(f"Class distribution: {np.bincount(y_test)}")
print(f"ROC-AUC: {roc_auc:.4f}")
print(f"Average Precision: {ap:.4f}")
print("\nNotice how AUC can look good even for imbalanced data!")

# 8. MULTICLASS AUC
print("\n=== Multiclass AUC ===")
from sklearn.datasets import load_iris

iris = load_iris()
X_iris, y_iris = iris.data, iris.target

X_train, X_test, y_train, y_test = train_test_split(
    X_iris, y_iris, test_size=0.2, random_state=42, stratify=y_iris
)

model_iris = LogisticRegression(max_iter=1000, multi_class='ovr')
model_iris.fit(X_train, y_train)
y_proba_iris = model_iris.predict_proba(X_test)

# OVR AUC
auc_ovr = roc_auc_score(y_test, y_proba_iris, multi_class='ovr', average='macro')
# OVO AUC
auc_ovo = roc_auc_score(y_test, y_proba_iris, multi_class='ovo', average='macro')

print(f"Multiclass AUC (OVR macro): {auc_ovr:.4f}")
print(f"Multiclass AUC (OVO macro): {auc_ovo:.4f}")

# 9. CONFIDENCE INTERVALS
print("\n=== AUC Confidence Interval ===")
from sklearn.utils import resample

def bootstrap_auc(y_true, y_score, n_bootstrap=1000, ci=0.95):
    """Bootstrap confidence interval for AUC."""
    aucs = []
    for _ in range(n_bootstrap):
        idx = resample(np.arange(len(y_true)))
        if len(np.unique(y_true[idx])) < 2:
            continue
        aucs.append(roc_auc_score(y_true[idx], y_score[idx]))
    
    lower = np.percentile(aucs, (1-ci)/2 * 100)
    upper = np.percentile(aucs, (1+ci)/2 * 100)
    return np.mean(aucs), lower, upper

mean_auc, lower, upper = bootstrap_auc(y_test, y_proba, n_bootstrap=500)
print(f"AUC: {mean_auc:.4f} [{lower:.4f}, {upper:.4f}] (95% CI)")

# 10. KEY TAKEAWAYS
print("\n=== Key Takeaways ===")
print("""
1. AUC measures RANKING quality, not calibration
   - Tells you if positives score higher than negatives
   - Doesn't tell you if probabilities are meaningful

2. AUC is threshold-independent
   - Good for comparing models
   - Doesn't tell you which threshold to use

3. AUC can be misleading for imbalanced data
   - PR curve often more informative
   - Consider average precision

4. AUC confidence intervals matter
   - Report them in papers
   - Use bootstrap for estimation

5. Different averaging for multiclass
   - Macro: unweighted average
   - Weighted: by class frequency
   - Micro: global average
""")

📊 Precision-Recall Curve

🎯 Focus on Positives

The Precision-Recall curve plots precision vs recall at all thresholds. It's especially useful for imbalanced datasets where the positive class is rare and we care most about finding it.

PR vs ROC

PR curve: Better for imbalanced data. ROC curve: Better for balanced data. Average Precision: Area under PR curve. Baseline: Fraction of positives. Focus: Positive class only. Threshold: Still threshold-dependent.

📊 Precision📞 Recall⚖️ Imbalanced🎯 AP

# Precision-Recall curve in depth
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    precision_recall_curve, average_precision_score,
    roc_auc_score, precision_score, recall_score, f1_score
)

# Create imbalanced data
X, y = make_classification(
    n_samples=10000, n_features=20,
    weights=[0.99, 0.01],  # 1% positives
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42, stratify=y
)

print(f"Training set: {np.bincount(y_train)}")
print(f"Test set: {np.bincount(y_test)}")
print(f"Positive rate: {y_test.mean():.4f}")

# Train model
model = RandomForestClassifier(n_estimators=100, random_state=42)
model.fit(X_train, y_train)
y_proba = model.predict_proba(X_test)[:, 1]

# 1. PRECISION-RECALL CURVE
precision, recall, thresholds = precision_recall_curve(y_test, y_proba)

# Average Precision
ap = average_precision_score(y_test, y_proba)
print(f"\nAverage Precision: {ap:.4f}")

# Baseline (random classifier)
baseline = y_test.mean()
print(f"Baseline (random): {baseline:.4f}")

# 2. ROC CURVE FOR COMPARISON
from sklearn.metrics import roc_curve

fpr, tpr, _ = roc_curve(y_test, y_proba)
roc_auc = roc_auc_score(y_test, y_proba)
print(f"ROC-AUC: {roc_auc:.4f}")

# 3. VISUALIZATION
fig, axes = plt.subplots(1, 2, figsize=(16, 6))

# PR curve
axes[0].plot(recall, precision, linewidth=2, color='blue')
axes[0].axhline(y=baseline, color='r', linestyle='--', 
                label=f'Baseline (AP = {baseline:.3f})')
axes[0].set_xlabel('Recall', fontsize=12)
axes[0].set_ylabel('Precision', fontsize=12)
axes[0].set_title(f'Precision-Recall Curve (AP = {ap:.3f})', fontsize=14)
axes[0].legend()
axes[0].grid(alpha=0.3)
axes[0].set_xlim([0, 1])
axes[0].set_ylim([0, 1.05])

# ROC curve
axes[1].plot(fpr, tpr, linewidth=2, color='green')
axes[1].plot([0, 1], [0, 1], 'k--', label='Random')
axes[1].set_xlabel('False Positive Rate', fontsize=12)
axes[1].set_ylabel('True Positive Rate', fontsize=12)
axes[1].set_title(f'ROC Curve (AUC = {roc_auc:.3f})', fontsize=14)
axes[1].legend()
axes[1].grid(alpha=0.3)

plt.tight_layout()
plt.show()

# 4. FINDING OPTIMAL THRESHOLD
print("\n=== Optimal Thresholds ===")

# F1-optimal
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-10)
f1_optimal_idx = np.argmax(f1_scores[:-1])  # Skip last
f1_optimal_threshold = thresholds[f1_optimal_idx]

print(f"F1-optimal threshold: {f1_optimal_threshold:.4f}")
print(f"  Precision: {precision[f1_optimal_idx]:.4f}")
print(f"  Recall: {recall[f1_optimal_idx]:.4f}")
print(f"  F1: {f1_scores[f1_optimal_idx]:.4f}")

# High recall
recall_target = 0.9
recall_idx = np.where(recall >= recall_target)[0]
if len(recall_idx) > 0:
    high_recall_idx = recall_idx[-1]  # Highest precision at that recall
    print(f"\nThreshold for {recall_target:.0%} recall: {thresholds[high_recall_idx]:.4f}")
    print(f"  Precision: {precision[high_recall_idx]:.4f}")
    print(f"  Recall: {recall[high_recall_idx]:.4f}")

# High precision
precision_target = 0.9
prec_idx = np.where(precision >= precision_target)[0]
if len(prec_idx) > 0:
    high_prec_idx = prec_idx[0]
    print(f"\nThreshold for {precision_target:.0%} precision: {thresholds[high_prec_idx]:.4f}")
    print(f"  Precision: {precision[high_prec_idx]:.4f}")
    print(f"  Recall: {recall[high_prec_idx]:.4f}")

# 5. INTERPOLATED PRECISION
print("\n=== Interpolated Precision ===")
print("""
Interpolated precision smooths the curve:
  p_interp(r) = max_{r' >= r} p(r')

This makes the curve monotonic and easier to compare.
Used in older papers and evaluation metrics.
""")

# Compute interpolated precision
def interpolate_precision(precision, recall, n_points=11):
    """Compute 11-point interpolated precision."""
    interp_precision = []
    for r in np.linspace(0, 1, n_points):
        # Find precision at recall >= r
        mask = recall >= r
        if mask.any():
            interp_precision.append(precision[mask].max())
        else:
            interp_precision.append(0)
    return interp_precision

interp_prec = interpolate_precision(precision, recall)
print(f"11-point interpolated precision: {np.array(interp_prec).round(3)}")

# 6. COMPARING MODELS
print("\n=== Model Comparison ===")
models = {
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'LR Balanced': LogisticRegression(max_iter=1000, class_weight='balanced')
}

fig, ax = plt.subplots(figsize=(10, 8))

results = {}
for name, model in models.items():
    model.fit(X_train, y_train)
    y_proba = model.predict_proba(X_test)[:, 1]
    
    precision, recall, _ = precision_recall_curve(y_test, y_proba)
    ap = average_precision_score(y_test, y_proba)
    auc = roc_auc_score(y_test, y_proba)
    
    results[name] = {'ap': ap, 'auc': auc}
    
    ax.plot(recall, precision, label=f'{name} (AP = {ap:.3f})', linewidth=2)
    print(f"{name}:")
    print(f"  Average Precision: {ap:.4f}")
    print(f"  ROC-AUC: {auc:.4f}")

ax.axhline(y=baseline, color='k', linestyle='--', 
           label=f'Baseline ({baseline:.3f})')
ax.set_xlabel('Recall', fontsize=12)
ax.set_ylabel('Precision', fontsize=12)
ax.set_title('PR Curves — Model Comparison', fontsize=14)
ax.legend()
ax.grid(alpha=0.3)

plt.tight_layout()
plt.show()

# 7. WHEN TO USE PR VS ROC
print("\n=== When to Use PR vs ROC ===")
print("""
Use PR Curve When:
  ✓ Highly imbalanced classes
  ✓ Positive class is rare
  ✓ False positives are costly
  ✓ You care about ranking positives
  ✓ You want to detect positive cases

Examples:
  - Fraud detection (1% fraud)
  - Disease screening (rare disease)
  - Information retrieval
  - Anomaly detection
  - Click-through rate prediction

Use ROC Curve When:
  ✓ Balanced classes
  ✓ Both classes equally important
  ✓ You want threshold-independent measure
  ✓ You care about both TP and TN
  ✓ Comparing across datasets

Examples:
  - Medical diagnosis (balanced)
  - Image classification
  - Binary sentiment analysis
  - General model comparison

Rule of Thumb:
  If positive rate < 10% → Use PR curve
  If positive rate > 30% → Use ROC curve
  In between → Use both, report both
""")

# 8. AREA UNDER PR CURVE (AP)
print("=== Average Precision (AP) ===")
print("""
AP = ∫ precision(recall) d(recall)

Interpretation:
  1.0    = Perfect classifier
  0.9+   = Excellent
  0.7+   = Good
  0.5+   = Fair
  ≈ baseline = No better than random

AP is more informative than AUC for imbalanced data.

Baseline AP = positive rate (e.g., 0.01 for 1% positives)

A good AP is significantly higher than baseline.
""")

# 9. FINAL EXAMPLE
print("\n=== Complete Example ===")

# Simulate a fraud detection scenario
X_fraud, y_fraud = make_classification(
    n_samples=50000, n_features=30,
    weights=[0.998, 0.002],  # 0.2% fraud
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X_fraud, y_fraud, test_size=0.2, random_state=42, stratify=y_fraud
)

model = RandomForestClassifier(
    n_estimators=100, 
    class_weight='balanced',
    random_state=42
)
model.fit(X_train, y_train)

y_proba = model.predict_proba(X_test)[:, 1]

# Metrics
ap = average_precision_score(y_test, y_proba)
roc_auc = roc_auc_score(y_test, y_proba)

precision, recall, thresholds = precision_recall_curve(y_test, y_proba)

print(f"Class distribution: {np.bincount(y_test)}")
print(f"Positive rate: {y_test.mean():.4f}")
print(f"\nMetrics:")
print(f"  ROC-AUC: {roc_auc:.4f} (looks good)")
print(f"  Average Precision: {ap:.4f} (more honest)")
print(f"  Baseline AP: {y_test.mean():.4f}")
print(f"  AP improvement: {ap / y_test.mean():.1f}x baseline")

# Business impact
print(f"\nBusiness Impact:")
threshold = 0.5
y_pred = (y_proba > threshold).astype(int)
tp = ((y_test == 1) & (y_pred == 1)).sum()
fp = ((y_test == 0) & (y_pred == 1)).sum()
fn = ((y_test == 1) & (y_pred == 0)).sum()

print(f"  At threshold {threshold}:")
print(f"    True Positives (caught fraud): {tp}")
print(f"    False Positives (false alarms): {fp}")
print(f"    False Negatives (missed fraud): {fn}")
print(f"    Precision: {tp / (tp + fp):.4f}")
print(f"    Recall: {tp / (tp + fn):.4f}")

⚖️ Imbalanced Data

🎯 When One Class Dominates

Imbalanced data occurs when one class is much rarer than another (e.g., 1% fraud, 0.1% disease). Standard metrics and methods can mislead. Specialized techniques are needed.

Handling Imbalance

Metrics: PR curve, F1, MCC, balanced accuracy. Resampling: Oversample minority, undersample majority. SMOTE: Synthetic minority samples. Class weights: Penalize minority errors more. Threshold tuning: Don't use 0.5. Ensemble: Balanced bagging, EasyEnsemble.

📊 Resampling🧬 SMOTE⚖️ Weights🎯 Threshold

# Handling imbalanced data
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    classification_report, confusion_matrix,
    roc_auc_score, average_precision_score,
    f1_score, balanced_accuracy_score
)

# Create imbalanced dataset
X, y = make_classification(
    n_samples=10000, n_features=20,
    weights=[0.95, 0.05],
    random_state=42
)

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

print(f"Training distribution: {np.bincount(y_train)}")
print(f"Test distribution: {np.bincount(y_test)}")
print(f"Imbalance ratio: {(y_train == 0).sum() / (y_train == 1).sum():.1f}:1")

# 1. BASELINE (no handling)
print("\n=== Baseline (No Handling) ===")
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
y_proba = model.predict_proba(X_test)[:, 1]

print(f"Accuracy: {(y_pred == y_test).mean():.4f}")
print(f"Balanced Accuracy: {balanced_accuracy_score(y_test, y_pred):.4f}")
print(f"F1: {f1_score(y_test, y_pred):.4f}")
print(f"AUC: {roc_auc_score(y_test, y_proba):.4f}")
print(f"AP: {average_precision_score(y_test, y_proba):.4f}")
print("\nNotice: High accuracy but poor minority class performance!")

print("\nClassification Report:")
print(classification_report(y_test, y_pred, target_names=['Majority', 'Minority']))

# 2. CLASS WEIGHTS
print("\n=== Class Weights ===")
model_balanced = LogisticRegression(
    max_iter=1000, class_weight='balanced'
)
model_balanced.fit(X_train, y_train)
y_pred_bal = model_balanced.predict(X_test)

print(f"Accuracy: {(y_pred_bal == y_test).mean():.4f}")
print(f"Balanced Accuracy: {balanced_accuracy_score(y_test, y_pred_bal):.4f}")
print(f"F1: {f1_score(y_test, y_pred_bal):.4f}")

print("\nClassification Report:")
print(classification_report(y_test, y_pred_bal, target_names=['Majority', 'Minority']))

# 3. RANDOM OVERSAMPLING
print("\n=== Random Oversampling ===")
from imblearn.over_sampling import RandomOverSampler

ros = RandomOverSampler(random_state=42)
X_train_ros, y_train_ros = ros.fit_resample(X_train, y_train)
print(f"After oversampling: {np.bincount(y_train_ros)}")

model_ros = LogisticRegression(max_iter=1000)
model_ros.fit(X_train_ros, y_train_ros)
y_pred_ros = model_ros.predict(X_test)

print(f"F1: {f1_score(y_test, y_pred_ros):.4f}")

# 4. RANDOM UNDERSAMPLING
print("\n=== Random Undersampling ===")
from imblearn.under_sampling import RandomUnderSampler

rus = RandomUnderSampler(random_state=42)
X_train_rus, y_train_rus = rus.fit_resample(X_train, y_train)
print(f"After undersampling: {np.bincount(y_train_rus)}")

model_rus = LogisticRegression(max_iter=1000)
model_rus.fit(X_train_rus, y_train_rus)
y_pred_rus = model_rus.predict(X_test)

print(f"F1: {f1_score(y_test, y_pred_rus):.4f}")

# 5. SMOTE (Synthetic Minority Oversampling)
print("\n=== SMOTE ===")
from imblearn.over_sampling import SMOTE

smote = SMOTE(random_state=42)
X_train_smote, y_train_smote = smote.fit_resample(X_train, y_train)
print(f"After SMOTE: {np.bincount(y_train_smote)}")

model_smote = LogisticRegression(max_iter=1000)
model_smote.fit(X_train_smote, y_train_smote)
y_pred_smote = model_smote.predict(X_test)

print(f"F1: {f1_score(y_test, y_pred_smote):.4f}")

# 6. COMPARISON TABLE
print("\n=== Method Comparison ===")
methods = {
    'Baseline': y_pred,
    'Class Weights': y_pred_bal,
    'Oversampling': y_pred_ros,
    'Undersampling': y_pred_rus,
    'SMOTE': y_pred_smote
}

print(f"{'Method':<20} {'Accuracy':<12} {'Bal Acc':<12} {'F1':<10}")
print("-" * 55)

for name, pred in methods.items():
    acc = (pred == y_test).mean()
    bal_acc = balanced_accuracy_score(y_test, pred)
    f1 = f1_score(y_test, pred)
    print(f"{name:<20} {acc:<12.4f} {bal_acc:<12.4f} {f1:<10.4f}")

# 7. THRESHOLD TUNING
print("\n=== Threshold Tuning ===")
y_proba = model.predict_proba(X_test)[:, 1]

thresholds = [0.1, 0.2, 0.3, 0.5, 0.7]
print(f"{'Threshold':<12} {'Precision':<12} {'Recall':<12} {'F1':<10}")
print("-" * 50)

best_f1 = 0
best_threshold = 0.5

from sklearn.metrics import precision_score, recall_score

for thresh in thresholds:
    y_pred_t = (y_proba > thresh).astype(int)
    prec = precision_score(y_test, y_pred_t, zero_division=0)
    rec = recall_score(y_test, y_pred_t, zero_division=0)
    f1 = f1_score(y_test, y_pred_t, zero_division=0)
    
    if f1 > best_f1:
        best_f1 = f1
        best_threshold = thresh
    
    print(f"{thresh:<12} {prec:<12.4f} {rec:<12.4f} {f1:<10.4f}")

print(f"\nBest threshold: {best_threshold} (F1 = {best_f1:.4f})")

# 8. EVALUATION METRICS FOR IMBALANCED DATA
print("\n=== Metrics for Imbalanced Data ===")
print("""
❌ AVOID:
  - Accuracy (misleading)
  - ROC-AUC (can be optimistic)

✅ USE:
  - Precision-Recall curve
  - Average Precision
  - F1 Score (or F-beta)
  - Balanced Accuracy
  - Matthews Correlation Coefficient (MCC)
  - Cohen's Kappa
  - Brier Score

✅ ALSO CONSIDER:
  - Cost-sensitive metrics
  - Business-relevant metrics
  - Confusion matrix analysis
  - Per-class metrics
""")

# 9. MCC — THE BEST SINGLE METRIC?
print("=== Matthews Correlation Coefficient ===")
from sklearn.metrics import matthews_corrcoef

for name, pred in methods.items():
    mcc = matthews_corrcoef(y_test, pred)
    print(f"{name:<20} MCC = {mcc:.4f}")

print("""
MCC ranges from -1 to +1:
  +1 = Perfect prediction
   0 = Random prediction
  -1 = Total disagreement

Why MCC is great for imbalanced data:
  - Considers all four confusion matrix cells
  - Balanced regardless of class sizes
  - Single number summary
  - Well-defined even for extreme imbalance
""")

# 10. PUTTING IT ALL TOGETHER
print("\n=== Best Practice Pipeline ===")
print("""
1. SPLIT first (before any resampling)
   - Resample only on training data!
   - Never resample test set

2. CHOOSE appropriate metric
   - F1, PR-AUC, MCC
   - Based on business needs

3. APPLY class weights or resampling
   - Start with class_weight='balanced'
   - Try SMOTE if needed
   - Combine methods if helpful

4. TUNE threshold
   - Don't use default 0.5
   - Optimize for chosen metric
   - Consider business costs

5. VALIDATE properly
   - Use stratified CV
   - Multiple metrics
   - Confidence intervals

6. MONITOR in production
   - Data drift
   - Class distribution shifts
   - Model performance over time
""")

# 11. COMMON MISTAKES
print("=== Common Mistakes ===")
print("""
❌ Resampling before splitting
   → Data leakage, overly optimistic results

❌ Using accuracy as primary metric
   → Misleading for imbalanced data

❌ Oversampling the test set
   → Inflates performance

❌ Using SMOTE with categorical features
   → Creates invalid synthetic samples

❌ Ignoring business costs
   → Wrong optimization target

❌ Not monitoring class drift
   → Model degrades as distribution changes
""")

🎯 Probability Calibration

🎯 Making Probabilities Meaningful

Calibration ensures that predicted probabilities match actual frequencies. If a model says "70% chance of rain," it should rain 70% of the time. Many models (SVM, Random Forest) are poorly calibrated.

Calibration Techniques

Platt scaling: Sigmoid fit on scores. Isotonic: Non-parametric, monotonic. Temperature scaling: For neural networks. Reliability diagram: Visualize calibration. Brier score: Overall calibration + accuracy. Log loss: Penalizes miscalibration.

📊 Sigmoid📈 Isotonic🌡️ Temperature📉 Brier

# Probability calibration
import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.calibration import (
    CalibratedClassifierCV, calibration_curve
)
from sklearn.metrics import brier_score_loss, log_loss

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Split for calibration
X_train_cal, X_cal, y_train_cal, y_cal = train_test_split(
    X_train, y_train, test_size=0.3, random_state=42, stratify=y_train
)

# 1. UNCALIBRATED MODELS
print("=== Uncalibrated Models ===")

models = {
    'Logistic Regression': LogisticRegression(max_iter=1000),
    'Random Forest': RandomForestClassifier(n_estimators=100, random_state=42),
    'SVM': SVC(probability=True, random_state=42)
}

fig, axes = plt.subplots(2, 3, figsize=(18, 12))

for i, (name, model) in enumerate(models.items()):
    # Train
    model.fit(X_train_cal, y_train_cal)
    y_proba = model.predict_proba(X_test)[:, 1]
    
    # Reliability diagram
    prob_true, prob_pred = calibration_curve(y_test, y_proba, n_bins=10)
    
    axes[0, i].plot(prob_pred, prob_true, 's-', label=name)
    axes[0, i].plot([0, 1], [0, 1], 'k--', label='Perfect')
    axes[0, i].set_xlabel('Predicted probability')
    axes[0, i].set_ylabel('Actual probability')
    axes[0, i].set_title(f'{name} (Uncalibrated)')
    axes[0, i].legend()
    axes[0, i].grid(alpha=0.3)
    
    # Brier score
    brier = brier_score_loss(y_test, y_proba)
    logloss = log_loss(y_test, y_proba)
    
    print(f"{name}:")
    print(f"  Brier score: {brier:.4f}")
    print(f"  Log loss: {logloss:.4f}")

# 2. CALIBRATED MODELS
print("\n=== Calibrated Models (Sigmoid) ===")

for i, (name, model) in enumerate(models.items()):
    # Calibrate with sigmoid
    calibrated = CalibratedClassifierCV(
        model, method='sigmoid', cv=5
    )
    calibrated.fit(X_train_cal, y_train_cal)
    y_proba_cal = calibrated.predict_proba(X_test)[:, 1]
    
    # Reliability diagram
    prob_true, prob_pred = calibration_curve(y_test, y_proba_cal, n_bins=10)
    
    axes[1, i].plot(prob_pred, prob_true, 's-', label=f'{name} (calibrated)')
    axes[1, i].plot([0, 1], [0, 1], 'k--', label='Perfect')
    axes[1, i].set_xlabel('Predicted probability')
    axes[1, i].set_ylabel('Actual probability')
    axes[1, i].set_title(f'{name} (Calibrated)')
    axes[1, i].legend()
    axes[1, i].grid(alpha=0.3)
    
    brier = brier_score_loss(y_test, y_proba_cal)
    logloss = log_loss(y_test, y_proba_cal)
    
    print(f"{name}:")
    print(f"  Brier score: {brier:.4f}")
    print(f"  Log loss: {logloss:.4f}")

plt.tight_layout()
plt.show()

# 3. COMPARING CALIBRATION METHODS
print("\n=== Calibration Methods ===")

# Base model
base_model = SVC(probability=True, random_state=42)

# Different calibration methods
calibration_methods = {
    'Uncalibrated': base_model,
    'Sigmoid (Platt)': CalibratedClassifierCV(base_model, method='sigmoid', cv=5),
    'Isotonic': CalibratedClassifierCV(base_model, method='isotonic', cv=5)
}

fig, ax = plt.subplots(figsize=(10, 8))

for name, model in calibration_methods.items():
    model.fit(X_train_cal, y_train_cal)
    y_proba = model.predict_proba(X_test)[:, 1]
    
    prob_true, prob_pred = calibration_curve(y_test, y_proba, n_bins=10)
    brier = brier_score_loss(y_test, y_proba)
    
    ax.plot(prob_pred, prob_true, 's-', label=f'{name} (Brier={brier:.4f})')

ax.plot([0, 1], [0, 1], 'k--', label='Perfect')
ax.set_xlabel('Predicted probability')
ax.set_ylabel('Actual probability')
ax.set_title('Calibration Comparison')
ax.legend()
ax.grid(alpha=0.3)

plt.tight_layout()
plt.show()

# 4. BRIER SCORE DECOMPOSITION
print("=== Brier Score Decomposition ===")
print("""
Brier = Reliability - Resolution + Uncertainty

Reliability: How close predictions are to actual rates
  (Lower is better)

Resolution: How much predictions vary from base rate
  (Higher is better)

Uncertainty: Inherent variability in data
  (Fixed for a given dataset)

Good models have:
  - Low reliability (well calibrated)
  - High resolution (discriminative)
""")

# 5. TEMPERATURE SCALING (for neural networks)
print("=== Temperature Scaling ===")
print("""
Temperature scaling is popular for neural networks:

  p_calibrated = softmax(logits / T)

Where T is a learned temperature parameter:
  T > 1: Softens probabilities (less confident)
  T < 1: Sharpens probabilities (more confident)
  T = 1: No change

Advantages:
  - Single parameter
  - Doesn't change accuracy
  - Easy to fit

Implementation:
  - Hold out validation set
  - Optimize T to minimize NLL
  - Apply to test predictions
""")

# Simulate temperature scaling
def temperature_scale(logits, T):
    """Apply temperature scaling to logits."""
    return logits / T

def softmax(x):
    """Numerically stable softmax."""
    exp_x = np.exp(x - np.max(x, axis=-1, keepdims=True))
    return exp_x / np.sum(exp_x, axis=-1, keepdims=True)

# Example logits
logits = np.array([[2.0, 0.5], [0.1, 1.5], [-0.5, 1.0]])

for T in [0.5, 1.0, 2.0, 5.0]:
    scaled = temperature_scale(logits, T)
    probs = softmax(scaled)
    print(f"T = {T}: {probs[:, 1].round(4)}")

# 6. WHEN TO CALIBRATE
print("\n=== When to Calibrate ===")
print("""
Calibrate when:
  ✓ You need accurate probabilities
  ✓ Downstream decisions use probabilities
  ✓ Combining with other probability estimates
  ✓ Risk assessment applications
  ✓ Medical or financial predictions

Don't need calibration when:
  ✗ Only using predicted labels
  ✗ Ranking is all that matters
  ✗ Model is already calibrated (Logistic Regression)

Models that typically need calibration:
  - SVM
  - Random Forest
  - Gradient Boosting (sometimes)
  - Naive Bayes
  - Neural Networks (often)

Models that are usually calibrated:
  - Logistic Regression (with proper fitting)
  - Bayesian models
""")

# 7. BEST PRACTICES
print("\n=== Calibration Best Practices ===")
print("""
1. Always use a HELD-OUT set for calibration
   - Don't calibrate on training data
   - Don't calibrate on test data
   - Use 20-30% of training for calibration

2. Choose method based on data size
   - Sigmoid: Small data, works well
   - Isotonic: Large data, more flexible

3. Evaluate with proper metrics
   - Brier score
   - Log loss
   - Reliability diagram
   - ECE (Expected Calibration Error)

4. Consider cost-sensitive calibration
   - Asymmetric costs
   - Business needs

5. Monitor calibration over time
   - Data drift affects calibration
   - Recalibrate periodically

6. Report calibration
   - Show reliability diagram
   - Report Brier score
   - Discuss limitations
""")

# 8. ECE (Expected Calibration Error)
def expected_calibration_error(y_true, y_proba, n_bins=10):
    """Compute Expected Calibration Error."""
    bin_boundaries = np.linspace(0, 1, n_bins + 1)
    bin_lowers = bin_boundaries[:-1]
    bin_uppers = bin_boundaries[1:]
    
    ece = 0
    for bin_lower, bin_upper in zip(bin_lowers, bin_uppers):
        # Find samples in this bin
        in_bin = (y_proba > bin_lower) & (y_proba <= bin_upper)
        prop_in_bin = in_bin.mean()
        
        if prop_in_bin > 0:
            # Accuracy in this bin
            accuracy_in_bin = y_true[in_bin].mean()
            # Average confidence in this bin
            avg_confidence = y_proba[in_bin].mean()
            # Add to ECE
            ece += np.abs(avg_confidence - accuracy_in_bin) * prop_in_bin
    
    return ece

# Test ECE
model = LogisticRegression(max_iter=1000)
model.fit(X_train_cal, y_train_cal)
y_proba = model.predict_proba(X_test)[:, 1]

ece = expected_calibration_error(y_test, y_proba)
print(f"\nECE: {ece:.4f}")
print("Lower ECE = better calibration")

⚙️ Hyperparameter Tuning

🎯 Finding the Best Configuration

Hyperparameters control model behavior but aren't learned from data. Tuning finds the best configuration for your problem. Methods: grid search, random search, Bayesian optimization, and AutoML.

Tuning Methods

Grid search: Try all combinations. Random search: Sample randomly — often better. Bayesian: Model-guided search. Hyperband: Early stopping. AutoML: Automated everything. Nested CV: Unbiased evaluation.

🔍 Grid🎲 Random🧠 Bayesian⚡ Hyperband

# Hyperparameter tuning
import numpy as np
import time
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (
    train_test_split, GridSearchCV, RandomizedSearchCV,
    cross_val_score, StratifiedKFold
)
from sklearn.ensemble import RandomForestClassifier
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

# 1. GRID SEARCH
print("=== Grid Search ===")

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [5, 10, 20],
    'min_samples_split': [2, 5, 10]
}

# Total combinations: 3 × 3 × 3 = 27
# × 5 folds = 135 fits

grid_search = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=cv,
    scoring='accuracy',
    n_jobs=-1,
    verbose=1
)

t0 = time.time()
grid_search.fit(X_train, y_train)
grid_time = time.time() - t0

print(f"\nGrid search time: {grid_time:.2f}s")
print(f"Best params: {grid_search.best_params_}")
print(f"Best score: {grid_search.best_score_:.4f}")
print(f"Test score: {grid_search.score(X_test, y_test):.4f}")

# 2. RANDOM SEARCH
print("\n=== Random Search ===")

from scipy.stats import randint, uniform

param_dist = {
    'n_estimators': randint(50, 500),
    'max_depth': randint(3, 30),
    'min_samples_split': randint(2, 20),
    'min_samples_leaf': randint(1, 10),
    'max_features': uniform(0.1, 0.9)
}

random_search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42),
    param_dist,
    n_iter=50,  # Try 50 random combinations
    cv=cv,
    scoring='accuracy',
    n_jobs=-1,
    random_state=42,
    verbose=1
)

t0 = time.time()
random_search.fit(X_train, y_train)
random_time = time.time() - t0

print(f"\nRandom search time: {random_time:.2f}s")
print(f"Best params: {random_search.best_params_}")
print(f"Best score: {random_search.best_score_:.4f}")
print(f"Test score: {random_search.score(X_test, y_test):.4f}")

# 3. BAYESIAN OPTIMIZATION
print("\n=== Bayesian Optimization ===")

try:
    import optuna
    optuna.logging.set_verbosity(optuna.logging.WARNING)
    
    def objective(trial):
        params = {
            'n_estimators': trial.suggest_int('n_estimators', 50, 500),
            'max_depth': trial.suggest_int('max_depth', 3, 30),
            'min_samples_split': trial.suggest_int('min_samples_split', 2, 20),
            'min_samples_leaf': trial.suggest_int('min_samples_leaf', 1, 10),
        }
        
        model = RandomForestClassifier(**params, random_state=42)
        scores = cross_val_score(model, X_train, y_train, cv=cv, scoring='accuracy')
        return scores.mean()
    
    study = optuna.create_study(direction='maximize')
    
    t0 = time.time()
    study.optimize(objective, n_trials=50, show_progress_bar=False)
    optuna_time = time.time() - t0
    
    print(f"\nOptuna time: {optuna_time:.2f}s")
    print(f"Best params: {study.best_params}")
    print(f"Best score: {study.best_value:.4f}")
    
    # Evaluate on test
    best_model = RandomForestClassifier(**study.best_params, random_state=42)
    best_model.fit(X_train, y_train)
    print(f"Test score: {best_model.score(X_test, y_test):.4f}")
    
except ImportError:
    print("Optuna not installed. Install with: pip install optuna")

# 4. COMPARISON
print("\n=== Method Comparison ===")
print(f"{'Method':<20} {'Time':<10} {'Best Score':<12}")
print("-" * 45)
print(f"{'Grid Search':<20} {grid_time:<10.2f} {grid_search.best_score_:<12.4f}")
print(f"{'Random Search':<20} {random_time:<10.2f} {random_search.best_score_:<12.4f}")

# 5. NESTED CV FOR UNBIASED ESTIMATE
print("\n=== Nested Cross-Validation ===")

inner_cv = StratifiedKFold(n_splits=3, shuffle=True, random_state=42)
outer_cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

nested_clf = GridSearchCV(
    RandomForestClassifier(random_state=42),
    param_grid,
    cv=inner_cv,
    scoring='accuracy'
)

nested_scores = cross_val_score(
    nested_clf, X_train, y_train, cv=outer_cv, scoring='accuracy'
)

print(f"Nested CV score: {nested_scores.mean():.4f} (±{nested_scores.std():.4f})")
print("This is the unbiased estimate of generalization!")

# 6. LEARNING CURVES
print("\n=== Learning Curves ===")

from sklearn.model_selection import learning_curve

train_sizes, train_scores, val_scores = learning_curve(
    RandomForestClassifier(n_estimators=100, random_state=42),
    X_train, y_train,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=cv,
    scoring='accuracy',
    n_jobs=-1
)

print(f"{'Train Size':<15} {'Train Score':<15} {'Val Score':<15}")
print("-" * 45)
for i in range(len(train_sizes)):
    print(f"{train_sizes[i]:<15} {train_scores[i].mean():<15.4f} {val_scores[i].mean():<15.4f}")

# 7. VALIDATION CURVES
print("\n=== Validation Curves ===")

from sklearn.model_selection import validation_curve

param_range = [1, 5, 10, 20, 50]
train_scores, val_scores = validation_curve(
    RandomForestClassifier(n_estimators=100, random_state=42),
    X_train, y_train,
    param_name='max_depth',
    param_range=param_range,
    cv=cv,
    scoring='accuracy'
)

print(f"{'Max Depth':<12} {'Train':<12} {'Val':<12}")
print("-" * 36)
for i, d in enumerate(param_range):
    print(f"{d:<12} {train_scores[i].mean():<12.4f} {val_scores[i].mean():<12.4f}")

# 8. TUNING STRATEGY
print("\n=== Tuning Strategy ===")
print("""
1. START with defaults
   - Understand baseline performance
   - Don't tune blindly

2. TUNE important hyperparameters first
   - Random Forest: n_estimators, max_depth
   - SVM: C, gamma
   - Neural Nets: learning rate, architecture

3. USE random search
   - Better than grid for high dimensions
   - Finds good values faster
   - Sample widely initially

4. REFINE with Bayesian optimization
   - When random search finds good region
   - More efficient use of trials
   - Learn from past evaluations

5. CONSIDER Hyperband / ASHA
   - Early stopping for bad configs
   - Efficient for expensive models
   - Great for neural networks

6. VALIDATE properly
   - Nested CV for unbiased estimate
   - Hold out test set
   - Report confidence intervals

7. DON'T over-tune
   - Diminishing returns
   - Risk of overfitting to validation
   - Consider simple models
""")

# 9. AUTO-ML
print("=== AutoML Tools ===")
print("""
Tools that automate hyperparameter tuning:

Open Source:
  - Optuna: Flexible, modern
  - Ray Tune: Distributed tuning
  - Hyperopt: Classic Bayesian
  - Scikit-Optimize: sklearn integration
  - FLAML: Fast, lightweight

Commercial:
  - Google Vertex AI Vizier
  - AWS SageMaker Autopilot
  - Azure AutoML
  - H2O Driverless AI
  - DataRobot

AutoML Pipelines:
  - Auto-sklearn
  - TPOT
  - AutoKeras
  - PyCaret

When to use AutoML:
  - Limited ML expertise
  - Time constraints
  - Baseline establishment
  - Many models to compare

When NOT to use AutoML:
  - Need full control
  - Custom objectives
  - Domain constraints
  - Interpretability critical
""")

🚨 Data Leakage

🎯 The Silent Killer

Data leakage occurs when information from outside the training set leaks into the model, causing overly optimistic evaluation. It's the #1 cause of ML models that look great in development but fail in production.

Types of Leakage

Target leakage: Features contain the target. Train-test contamination: Test info in training. Temporal leakage: Using future data. Preprocessing leakage: Fit on all data. Group leakage: Same entity in train and test. Duplicate leakage: Duplicates across sets.

🎯 Target⏱️ Temporal🔧 Preprocessing👥 Group

# Data leakage — types and detection
import numpy as np
import pandas as pd
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# 1. TARGET LEAKAGE
print("=== Target Leakage ===")
print("When a feature is a proxy for the target")

# Example: Predicting if someone will get a loan
# Feature: "loan_approved" (which IS the target!)
np.random.seed(42)
n = 1000

data = pd.DataFrame({
    'income': np.random.randn(n) * 20000 + 50000,
    'age': np.random.randint(22, 65, n),
    'credit_score': np.random.randint(300, 850, n),
    'loan_approved': np.random.randint(0, 2, n)  # TARGET
})

# Sneaky: "approved_amount" is only set if approved
data['approved_amount'] = data['loan_approved'] * np.random.randn(n) * 5000 + 10000
data['approved_amount'] = data['approved_amount'].fillna(0)

# Train with the leaky feature
X = data[['income', 'age', 'credit_score', 'approved_amount']]
y = data['loan_approved']

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

leaky_model = RandomForestClassifier(n_estimators=100, random_state=42)
leaky_model.fit(X_train, y_train)

print(f"With leakage: {leaky_model.score(X_test, y_test):.4f}")

# Train without the leaky feature
X_clean = data[['income', 'age', 'credit_score']]
X_train, X_test, y_train, y_test = train_test_split(
    X_clean, y, test_size=0.2, random_state=42
)

clean_model = RandomForestClassifier(n_estimators=100, random_state=42)
clean_model.fit(X_train, y_train)

print(f"Without leakage: {clean_model.score(X_test, y_test):.4f}")
print("Notice the suspiciously high accuracy with leakage!")

# 2. PREPROCESSING LEAKAGE
print("\n=== Preprocessing Leakage ===")

# Generate data
X, y = make_classification(n_samples=1000, n_features=20, random_state=42)

# WRONG: Scale before split
print("\n❌ WRONG — Scale before split:")
scaler = StandardScaler()
X_scaled_wrong = scaler.fit_transform(X)  # Uses ALL data!

X_train, X_test, y_train, y_test = train_test_split(
    X_scaled_wrong, y, test_size=0.2, random_state=42
)

model = LogisticRegression(max_iter=1000)
wrong_scores = cross_val_score(model, X_scaled_wrong, y, cv=5)
print(f"  CV score (wrong): {wrong_scores.mean():.4f}")

# CORRECT: Split then scale
print("\n✅ CORRECT — Split then scale:")
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Fit on train only
X_test_scaled = scaler.transform(X_test)         # Transform test

model = LogisticRegression(max_iter=1000)
model.fit(X_train_scaled, y_train)
correct_score = model.score(X_test_scaled, y_test)
print(f"  Test score (correct): {correct_score:.4f}")

# The difference may seem small here, but can be huge in practice!

# 3. TEMPORAL LEAKAGE
print("\n=== Temporal Leakage ===")

# WRONG: Random split for time series
dates = pd.date_range('2020-01-01', periods=1000, freq='D')
values = np.cumsum(np.random.randn(1000)) + 100

ts_data = pd.DataFrame({
    'date': dates,
    'value': values,
    'target': (values > values.mean()).astype(int)
})

# ❌ Random split — leaks future
random_train = ts_data.sample(frac=0.7, random_state=42)
random_test = ts_data.drop(random_train.index)

print("❌ Random split for time series:")
print(f"  Train dates: {random_train['date'].min()} to {random_train['date'].max()}")
print(f"  Test dates: {random_test['date'].min()} to {random_test['date'].max()}")
print("  Test set has data BEFORE some training data!")

# ✅ Chronological split
split_idx = int(len(ts_data) * 0.7)
chrono_train = ts_data.iloc[:split_idx]
chrono_test = ts_data.iloc[split_idx:]

print("\n✅ Chronological split:")
print(f"  Train dates: {chrono_train['date'].min()} to {chrono_train['date'].max()}")
print(f"  Test dates: {chrono_test['date'].min()} to {chrono_test['date'].max()}")
print("  All test data comes after training data")

# 4. GROUP LEAKAGE
print("\n=== Group Leakage ===")
print("When related samples are in both train and test")

# Example: Multiple images per patient
np.random.seed(42)
patient_ids = np.repeat(np.arange(100), 5)  # 100 patients, 5 samples each
X_medical = np.random.randn(500, 10)
y_medical = np.random.randint(0, 2, 500)

# ❌ Random split
X_train, X_test, y_train, y_test = train_test_split(
    X_medical, y_medical, test_size=0.2, random_state=42
)

# Check overlap
patient_ids_train = patient_ids[:len(X_train)]
patient_ids_test = patient_ids[len(X_train):]
overlap = set(patient_ids_train) & set(patient_ids_test)

print(f"❌ Random split: {len(overlap)} patients in both train and test")
print("  Model may memorize patient-specific features")

# ✅ Group split
from sklearn.model_selection import GroupShuffleSplit

gss = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train_idx, test_idx = next(gss.split(X_medical, y_medical, groups=patient_ids))

patient_ids_train = patient_ids[train_idx]
patient_ids_test = patient_ids[test_idx]
overlap = set(patient_ids_train) & set(patient_ids_test)

print(f"\n✅ Group split: {len(overlap)} patients in both train and test")
print("  No leakage!")

# 5. DUPLICATE LEAKAGE
print("\n=== Duplicate Leakage ===")

# Create data with duplicates
X_dup = np.random.randn(100, 5)
y_dup = np.random.randint(0, 2, 100)

# Duplicate some samples
X_dup = np.vstack([X_dup, X_dup[:20]])
y_dup = np.concatenate([y_dup, y_dup[:20]])

# Random split — duplicates may end up in both sets
X_train, X_test, y_train, y_test = train_test_split(
    X_dup, y_dup, test_size=0.3, random_state=42
)

# Check for exact duplicates
from scipy.spatial.distance import cdist
distances = cdist(X_test, X_train)
duplicates = (distances < 1e-6).any(axis=1)

print(f"Test samples with duplicate in train: {duplicates.sum()}")
print(f"Percentage: {duplicates.mean() * 100:.1f}%")

if duplicates.sum() > 0:
    print("❌ Duplicate leakage!")
    print("Fix: Remove duplicates BEFORE splitting")

# 6. DETECTING LEAKAGE
print("\n=== Detecting Leakage ===")
print("""
Warning signs:
  1. Accuracy is "too good to be true"
  2. Huge gap between CV and test performance
  3. Feature importance dominated by one feature
  4. Model fails in production
  5. Simple model outperforms complex one

Diagnostic steps:
  1. Review feature engineering
  2. Check for temporal order
  3. Look for IDs/groups
  4. Remove duplicates
  5. Audit preprocessing order
  6. Check feature-target correlations
  7. Test on truly held-out data
""")

# 7. PREVENTION
print("=== Prevention Checklist ===")
print("""
✅ ALWAYS split BEFORE any preprocessing
✅ Fit scalers, encoders ONLY on training data
✅ Use Pipeline to enforce correct order
✅ Use GroupKFold for grouped data
✅ Use TimeSeriesSplit for temporal data
✅ Remove duplicates before splitting
✅ Check feature-target correlations
✅ Think about how features were collected
✅ Validate on truly unseen data
✅ Document data collection process
✅ Review with domain experts
✅ Test in production-like conditions
""")

# 8. CORRECT PIPELINE
print("=== Correct Pipeline Example ===")
print("""
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.feature_selection import SelectKBest
from sklearn.linear_model import LogisticRegression

# ❌ WRONG — manual preprocessing
X_imputed = imputer.fit_transform(X)  # Leak!
X_scaled = scaler.fit_transform(X_imputed)  # Leak!
X_selected = selector.fit_transform(X_scaled, y)  # Leak!
X_train, X_test = train_test_split(X_selected, ...)

# ✅ CORRECT — use Pipeline
pipeline = Pipeline([
    ('imputer', SimpleImputer()),
    ('scaler', StandardScaler()),
    ('selector', SelectKBest()),
    ('clf', LogisticRegression())
])

# Fit on training data only
pipeline.fit(X_train, y_train)
pipeline.score(X_test, y_test)
""")

# Test the correct pipeline
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer

X, y = make_classification(n_samples=1000, n_features=20, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='mean')),
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000))
])

# Cross-validate the pipeline
scores = cross_val_score(pipeline, X_train, y_train, cv=5)
print(f"Pipeline CV score: {scores.mean():.4f} (±{scores.std():.4f})")

pipeline.fit(X_train, y_train)
print(f"Pipeline test score: {pipeline.score(X_test, y_test):.4f}")

✅ Best Practices

🎯 Evaluation Done Right

Proper model evaluation is both a science and an art. Following best practices ensures your results are reliable, reproducible, and trustworthy.

Key Principles

Split properly: Always hold out test data. Use CV: For robust estimates. Multiple metrics: No single number. Confidence intervals: Quantify uncertainty. Consider costs: Business impact. Reproducibility: Set random seeds.

📊 Multiple Metrics🔄 Cross-Validation📈 Confidence🎯 Business

# Best practices for model evaluation
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import (
    train_test_split, StratifiedKFold, cross_validate,
    GridSearchCV
)
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline
from sklearn.linear_model import LogisticRegression
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import (
    accuracy_score, precision_score, recall_score, f1_score,
    roc_auc_score, average_precision_score, confusion_matrix,
    classification_report
)
import warnings
warnings.filterwarnings('ignore')

# 1. SET RANDOM SEEDS
print("=== 1. Reproducibility ===")
RANDOM_STATE = 42
np.random.seed(RANDOM_STATE)

# 2. LOAD AND SPLIT
print("\n=== 2. Data Splitting ===")
data = load_breast_cancer()
X, y = data.data, data.target

# Hold out test set FIRST
X_train_full, X_test, y_train_full, y_test = train_test_split(
    X, y, test_size=0.2, random_state=RANDOM_STATE, stratify=y
)

# Split training into train/val
X_train, X_val, y_train, y_val = train_test_split(
    X_train_full, y_train_full, test_size=0.2, 
    random_state=RANDOM_STATE, stratify=y_train_full
)

print(f"Train: {len(X_train)} ({len(X_train)/len(X):.1%})")
print(f"Val:   {len(X_val)} ({len(X_val)/len(X):.1%})")
print(f"Test:  {len(X_test)} ({len(X_test)/len(X):.1%})")

# 3. USE PIPELINES
print("\n=== 3. Preprocessing Pipeline ===")

pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', LogisticRegression(max_iter=1000, random_state=RANDOM_STATE))
])

# 4. CROSS-VALIDATE
print("\n=== 4. Cross-Validation ===")

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=RANDOM_STATE)

scoring = {
    'accuracy': 'accuracy',
    'precision': 'precision',
    'recall': 'recall',
    'f1': 'f1',
    'roc_auc': 'roc_auc',
    'average_precision': 'average_precision'
}

cv_results = cross_validate(pipeline, X_train, y_train, cv=cv, scoring=scoring)

print("Cross-validation results:")
for metric in scoring:
    scores = cv_results[f'test_{metric}']
    print(f"  {metric:<20} {scores.mean():.4f} ± {scores.std():.4f}")

# 5. MULTIPLE METRICS ON VALIDATION
print("\n=== 5. Validation Metrics ===")

pipeline.fit(X_train, y_train)
y_val_pred = pipeline.predict(X_val)
y_val_proba = pipeline.predict_proba(X_val)[:, 1]

val_metrics = {
    'Accuracy': accuracy_score(y_val, y_val_pred),
    'Precision': precision_score(y_val, y_val_pred),
    'Recall': recall_score(y_val, y_val_pred),
    'F1': f1_score(y_val, y_val_pred),
    'ROC-AUC': roc_auc_score(y_val, y_val_proba),
    'Avg Precision': average_precision_score(y_val, y_val_proba)
}

for metric, value in val_metrics.items():
    print(f"  {metric:<20} {value:.4f}")

# 6. CONFUSION MATRIX
print("\n=== 6. Confusion Matrix ===")
cm = confusion_matrix(y_val, y_val_pred)
print(cm)
print("\nClassification Report:")
print(classification_report(y_val, y_val_pred, target_names=data.target_names))

# 7. CONFIDENCE INTERVALS
print("=== 7. Confidence Intervals ===")

from scipy import stats

def confidence_interval(scores, confidence=0.95):
    """Compute confidence interval for scores."""
    n = len(scores)
    mean = np.mean(scores)
    se = stats.sem(scores)
    h = se * stats.t.ppf((1 + confidence) / 2, n - 1)
    return mean - h, mean + h

print("95% Confidence Intervals (from CV):")
for metric in ['accuracy', 'roc_auc', 'f1']:
    scores = cv_results[f'test_{metric}']
    lower, upper = confidence_interval(scores)
    print(f"  {metric:<20} {scores.mean():.4f} [{lower:.4f}, {upper:.4f}]")

# 8. HYPERPARAMETER TUNING (using validation)
print("\n=== 8. Hyperparameter Tuning ===")

param_grid = {
    'clf__C': [0.01, 0.1, 1, 10],
    'clf__penalty': ['l2']
}

grid_search = GridSearchCV(
    pipeline,
    param_grid,
    cv=cv,
    scoring='roc_auc',
    n_jobs=-1
)

grid_search.fit(X_train, y_train)
print(f"Best params: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.4f}")

# 9. FINAL EVALUATION ON TEST SET
print("\n=== 9. Final Test Evaluation ===")
print("Only do this ONCE!")

best_model = grid_search.best_estimator_
y_test_pred = best_model.predict(X_test)
y_test_proba = best_model.predict_proba(X_test)[:, 1]

test_metrics = {
    'Accuracy': accuracy_score(y_test, y_test_pred),
    'Precision': precision_score(y_test, y_test_pred),
    'Recall': recall_score(y_test, y_test_pred),
    'F1': f1_score(y_test, y_test_pred),
    'ROC-AUC': roc_auc_score(y_test, y_test_proba),
    'Avg Precision': average_precision_score(y_test, y_test_proba)
}

print("Final test metrics:")
for metric, value in test_metrics.items():
    print(f"  {metric:<20} {value:.4f}")

# 10. COMPARE TO VALIDATION
print("\n=== 10. Val vs Test Comparison ===")
print(f"{'Metric':<20} {'Validation':<15} {'Test':<15} {'Gap':<10}")
print("-" * 60)

for metric in ['Accuracy', 'Precision', 'Recall', 'F1', 'ROC-AUC']:
    val_val = val_metrics[metric]
    test_val = test_metrics[metric]
    gap = abs(val_val - test_val)
    print(f"{metric:<20} {val_val:<15.4f} {test_val:<15.4f} {gap:<10.4f}")

# 11. BASELINE COMPARISON
print("\n=== 11. Baseline Comparison ===")

# Majority class baseline
from sklearn.dummy import DummyClassifier

baseline = DummyClassifier(strategy='most_frequent')
baseline.fit(X_train, y_train)
baseline_acc = baseline.score(X_test, y_test)

print(f"Majority class baseline: {baseline_acc:.4f}")
print(f"Our model: {test_metrics['Accuracy']:.4f}")
print(f"Improvement: {(test_metrics['Accuracy'] - baseline_acc) * 100:.1f}%")

# 12. DOCUMENT EVERYTHING
print("\n=== 12. Document Everything ===")

documentation = {
    'Dataset': 'Breast Cancer Wisconsin',
    'Split': '60/20/20 stratified train/val/test',
    'Random State': RANDOM_STATE,
    'Model': 'Logistic Regression',
    'Best Params': grid_search.best_params_,
    'CV Strategy': '5-fold StratifiedKFold',
    'Primary Metric': 'ROC-AUC',
    'Test ROC-AUC': f"{test_metrics['ROC-AUC']:.4f}",
    'Baseline': f"{baseline_acc:.4f}",
    'Notes': 'Standardized features, L2 regularization'
}

print("Model Documentation:")
for key, value in documentation.items():
    print(f"  {key}: {value}")

# 13. BEST PRACTICES CHECKLIST
print("\n\n=== Best Practices Checklist ===")
print("""
DATA:
  ✅ Set random seeds for reproducibility
  ✅ Split data BEFORE any preprocessing
  ✅ Use stratified splits for classification
  ✅ Use temporal splits for time series
  ✅ Use group splits for grouped data
  ✅ Hold out test set and DON'T touch it

PREPROCESSING:
  ✅ Use Pipelines to prevent leakage
  ✅ Fit transformers on training data only
  ✅ Handle missing values properly
  ✅ Document all preprocessing steps

MODELING:
  ✅ Start with simple baselines
  ✅ Cross-validate for robust estimates
  ✅ Use nested CV for hyperparameter tuning
  ✅ Try multiple algorithms

EVALUATION:
  ✅ Report multiple metrics
  ✅ Choose metrics appropriate to the problem
  ✅ Include confidence intervals
  ✅ Analyze confusion matrix
  ✅ Compare to baselines
  ✅ Test on truly held-out data
  ✅ Check for overfitting (train vs val gap)

COMMUNICATION:
  ✅ Document everything
  ✅ Report limitations
  ✅ Discuss assumptions
  ✅ Share code and data (when possible)
  ✅ Explain in business terms
  ✅ Visualize results

PRODUCTION:
  ✅ Monitor performance over time
  ✅ Watch for data drift
  ✅ Retrain when needed
  ✅ A/B test changes
  ✅ Have rollback plans
""")

# 14. COMMON PITFALLS
print("\n=== Common Pitfalls ===")
print("""
❌ Using accuracy for imbalanced data
❌ Tuning hyperparameters on the test set
❌ Not using cross-validation
❌ Ignoring confidence intervals
❌ Preprocessing before splitting
❌ Forgetting to set random seeds
❌ Comparing models on different test sets
❌ Not using pipelines
❌ Overfitting to the validation set
❌ Ignoring business context
❌ Not documenting the process
❌ Trusting a single metric
""")

🚀 Future: AutoML & Beyond

🎯 The Next Frontier

Model evaluation is evolving: AutoML, continuous evaluation, fairness-aware metrics, uncertainty quantification, and foundation model evaluation are pushing the boundaries.

Future Directions

AutoML: Automated model selection and tuning. Continuous: Real-time monitoring. Fairness: Multi-objective evaluation. Uncertainty: Quantify what we don't know. Foundation models: New evaluation paradigms. Human-in-loop: Combining metrics and judgment.

🤖 AutoML📊 Continuous⚖️ Fairness🎯 Uncertainty

# Future of model evaluation
import numpy as np

# 1. AUTOMATED EVALUATION
print("=== Automated Evaluation ===")

class AutoEval:
    """Automated model evaluation."""
    
    def __init__(self, task_type):
        self.task_type = task_type
        self.metrics = self.select_metrics()
    
    def select_metrics(self):
        """Automatically select appropriate metrics."""
        if self.task_type == 'binary_classification':
            return ['accuracy', 'precision', 'recall', 'f1', 
                   'roc_auc', 'average_precision', 'mcc']
        elif self.task_type == 'multiclass_classification':
            return ['accuracy', 'macro_f1', 'weighted_f1', 
                   'cohen_kappa', 'mcc']
        elif self.task_type == 'regression':
            return ['mse', 'rmse', 'mae', 'r2', 'mape']
        elif self.task_type == 'ranking':
            return ['ndcg', 'mrr', 'map', 'precision_at_k']
        return []
    
    def evaluate(self, y_true, y_pred, y_proba=None):
        """Comprehensive evaluation."""
        results = {}
        # Compute all metrics
        # Detect data issues
        # Flag anomalies
        # Suggest improvements
        return results

# 2. CONTINUOUS EVALUATION
print("\n=== Continuous Evaluation ===")
print("""
Traditional: Evaluate once at deployment
Future: Continuous evaluation in production

Components:
1. Real-time metrics
   - Accuracy, precision, recall on live data
   - When labels become available

2. Data drift detection
   - Feature distributions
   - Label distributions
   - Concept drift

3. Performance monitoring
   - Alerts when performance drops
   - Automated rollback

4. Shadow deployment
   - Run new model alongside old
   - Compare on live traffic

5. A/B testing
   - Controlled experiments
   - Statistical significance
""")

# 3. FAIRNESS-AWARE EVALUATION
print("=== Fairness-Aware Evaluation ===")

class FairnessEval:
    """Multi-objective fairness evaluation."""
    
    def __init__(self, sensitive_attributes):
        self.sensitive_attributes = sensitive_attributes
        self.fairness_metrics = {
            'demographic_parity': 'P(ŷ=1|A=0) = P(ŷ=1|A=1)',
            'equal_opportunity': 'TPR(A=0) = TPR(A=1)',
            'equalized_odds': 'TPR and FPR equal across groups',
            'predictive_parity': 'Precision equal across groups',
            'individual_fairness': 'Similar individuals treated similarly'
        }
    
    def evaluate(self, y_true, y_pred, sensitive):
        """Evaluate fairness across groups."""
        results = {}
        for attr_name, attr_values in sensitive.items():
            for metric_name, description in self.fairness_metrics.items():
                # Compute metric
                results[f'{attr_name}_{metric_name}'] = self.compute(
                    y_true, y_pred, attr_values, metric_name
                )
        return results
    
    def compute(self, y_true, y_pred, groups, metric):
        """Compute a specific fairness metric."""
        return 0.0  # placeholder

# 4. UNCERTAINTY QUANTIFICATION
print("\n=== Uncertainty Quantification ===")
print("""
Types of uncertainty:
1. Aleatoric: Inherent noise in data
2. Epistemic: Model's lack of knowledge

Methods:
- Bayesian neural networks
- Monte Carlo dropout
- Deep ensembles
- Conformal prediction
- Quantile regression

Why it matters:
- Know when model is uncertain
- Abstain when uncertain
- Better decisions
- Safety-critical applications

Metrics:
- Calibration error
- Sharpness
- Coverage probability
- Interval width
""")

# Monte Carlo Dropout example
print("\n--- Monte Carlo Dropout ---")
print("""
At test time:
1. Keep dropout ON
2. Run forward pass N times
3. Compute mean and std of predictions

predictions = [model(x) for _ in range(N)]
mean = np.mean(predictions, axis=0)
std = np.std(predictions, axis=0)

High std → high uncertainty
Use for confidence estimation
""")

# 5. FOUNDATION MODEL EVALUATION
print("=== Foundation Model Evaluation ===")
print("""
Traditional metrics don't work for LLMs:

Challenges:
- No single correct answer
- Open-ended generation
- Context-dependent quality
- Multi-task capabilities
- Emergent behaviors

New evaluation approaches:
1. Benchmark suites
   - MMLU, BIG-bench, HELM
   - Multiple tasks

2. Human evaluation
   - Pairwise comparisons
   - Likert scales
   - Elo ratings

3. LLM-as-judge
   - Use GPT-4 to evaluate
   - Automated but biased
   - Needs validation

4. Task-specific metrics
   - BLEU, ROUGE (translation, summarization)
   - Exact match (QA)
   - Pass@k (code)

5. Red-teaming
   - Adversarial testing
   - Safety evaluation
   - Jailbreak attempts

6. Capability elicitation
   - Few-shot prompting
   - Chain-of-thought
   - Tool use

Best practices:
- Multiple benchmarks
- Diverse tasks
- Human + automated
- Report limitations
- Consider safety
""")

# 6. MULTI-OBJECTIVE EVALUATION
print("\n=== Multi-Objective Evaluation ===")

class MultiObjectiveEval:
    """Evaluate models on multiple objectives."""
    
    def __init__(self, objectives):
        self.objectives = objectives  # {'accuracy': 0.4, 'fairness': 0.3, 'latency': 0.3}
    
    def evaluate(self, model_results):
        """Weighted multi-objective score."""
        score = 0
        for obj, weight in self.objectives.items():
            score += weight * model_results[obj]
        return score
    
    def pareto_frontier(self, models):
        """Find Pareto-optimal models."""
        # Models not dominated on any objective
        pareto = []
        for model in models:
            dominated = False
            for other in models:
                if (all(other[obj] >= model[obj] for obj in self.objectives) and
                    any(other[obj] > model[obj] for obj in self.objectives)):
                    dominated = True
                    break
            if not dominated:
                pareto.append(model)
        return pareto

# Example
objectives = {'accuracy': 0.5, 'fairness': 0.3, 'latency': 0.2}
evaluator = MultiObjectiveEval(objectives)

model_results = {
    'accuracy': 0.92,
    'fairness': 0.85,
    'latency': 0.95
}

score = evaluator.evaluate(model_results)
print(f"Multi-objective score: {score:.4f}")

# 7. HUMAN-IN-THE-LOOP EVALUATION
print("\n=== Human-in-the-Loop Evaluation ===")
print("""
Metrics alone aren't enough. Humans provide:

1. Qualitative assessment
   - Are outputs useful?
   - Are they appropriate?
   - Do they make sense?

2. Edge case evaluation
   - Rare scenarios
   - Adversarial inputs
   - Distribution shift

3. Context understanding
   - Business relevance
   - Cultural sensitivity
   - Ethical implications

4. Continuous feedback
   - User ratings
   - Error reports
   - Improvement suggestions

5. Expert review
   - Domain experts
   - Quality assurance
   - Safety review

Approach: Combine automated + human
- Automated: Fast, consistent, scalable
- Human: Nuanced, contextual, flexible
""")

# 8. FUTURE METRICS
print("=== Emerging Metrics ===")
print("""
For LLMs and Generative AI:

1. Factuality
   - TruthfulQA
   - FactScore
   - Hallucination detection

2. Reasoning
   - GSM8K (math)
   - BBH (reasoning)
   - ARC (science)

3. Bias and Fairness
   - BOLD (bias)
   - WinoBias
   - StereoSet

4. Robustness
   - Adversarial NLI
   - ANLI
   - CheckList

5. Toxicity
   - Perspective API
   - Hate speech detection
   - Safety classifiers

6. Efficiency
   - Tokens/second
   - Energy per query
   - Carbon footprint

7. Alignment
   - HH-RLHF
   - TruthfulQA
   - Instruction following
""")

# 9. FUTURE OF EVALUATION
print("\n=== Future of Evaluation ===")
print("""
Trends:
1. Automated evaluation
   - LLM-as-judge
   - Agentic evaluation
   - Continuous monitoring

2. Multi-modal
   - Text, image, audio, video
   - Cross-modal consistency
   - Unified benchmarks

3. Multi-objective
   - Accuracy + fairness + robustness
   - Pareto frontier
   - Trade-off analysis

4. Real-world focused
   - Deployment-specific metrics
   - Business impact
   - User satisfaction

5. Interpretable
   - Why did model fail?
   - Error analysis
   - Failure modes

6. Standardized
   - Industry benchmarks
   - Regulatory compliance
   - Reproducible results

7. Fast
   - Rapid iteration
   - Small-scale proxies
   - Efficient sampling
""")

# 10. RECOMMENDATIONS
print("=== Recommendations ===")
print("""
For practitioners:
1. Start with simple baselines
2. Use cross-validation
3. Report multiple metrics
4. Include confidence intervals
5. Consider fairness
6. Monitor in production
7. Iterate continuously

For researchers:
1. Use standardized benchmarks
2. Report all metrics
3. Include failure analysis
4. Consider fairness and bias
5. Be transparent about limitations
6. Share code and data
7. Peer review results

For organizations:
1. Define success metrics
2. Align with business goals
3. Consider all stakeholders
4. Monitor post-deployment
5. Iterate based on feedback
6. Document everything
7. Invest in evaluation infrastructure

Key insight:
Model evaluation is not a one-time task.
It's an ongoing process throughout the ML lifecycle.
""")

# 11. FINAL THOUGHTS
print("\n=== Final Thoughts ===")
print("""
Model evaluation has evolved from:
  Simple accuracy → Multi-metric → Multi-objective
  
And will continue to evolve:
  Automated → Continuous → Human-AI collaborative

The goal remains the same:
  Reliable, trustworthy, and useful models
  That create value for society

Best advice:
  1. Always evaluate properly
  2. Consider the full context
  3. Report honestly
  4. Iterate based on evidence
  5. Think about impact

"All models are wrong, but some are useful."
  — George Box
  
Good evaluation helps you find the useful ones.
""")