🧠 XGBoost Fundamentals

🎯 What is XGBoost?

XGBoost (eXtreme Gradient Boosting) is a scalable, distributed gradient boosting library that has dominated machine learning competitions since 2014. It implements gradient boosted decision trees with regularization, parallel processing, missing value handling, and built-in cross-validation. It's the go-to algorithm for tabular data.

Why XGBoost Matters

State-of-the-art: Wins most Kaggle competitions on tabular data. Fast: Parallel and distributed training. Regularized: Prevents overfitting. Robust: Handles missing values natively. Flexible: Custom objectives, metrics, and eval functions. Production-ready: Used by industry leaders.

📌 Gradient Boosting📌 Regularization📌 Speed📌 Accuracy

# XGBoost fundamentals (conceptual)
import xgboost as xgb
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# 1. Load data
data = load_breast_cancer()
X_train, X_test, y_train, y_test = train_test_split(
    data.data, data.target, test_size=0.2, random_state=42
)

# 2. Train XGBoost classifier
model = xgb.XGBClassifier(
    n_estimators=100,      # Number of boosting rounds
    max_depth=6,           # Maximum tree depth
    learning_rate=0.1,     # Step size shrinkage
    objective='binary:logistic',  # Loss function
    use_label_encoder=False,
    eval_metric='logloss'
)

model.fit(X_train, y_train)

# 3. Predict
y_pred = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.4f}")

# 4. Feature importance
importance = model.feature_importances_
top_features = np.argsort(importance)[-5:]
print(f"\nTop 5 features:")
for idx in top_features[::-1]:
    print(f"  {data.feature_names[idx]}: {importance[idx]:.4f}")

# Why XGBoost wins:
# 1. Regularization (L1, L2) prevents overfitting
# 2. Parallel tree construction
# 3. Approximate split finding (weighted quantile sketch)
# 4. Native missing value handling
# 5. Cache-aware and out-of-core computation

📈 Gradient Boosting

🎯 The Foundation

Gradient boosting is an ensemble technique that builds models sequentially, where each new model corrects the errors of the previous ones. It's like a team of weak learners (shallow trees) working together to form a strong learner.

How Gradient Boosting Works

1. Start: Initial prediction (mean for regression). 2. Compute residuals: Actual - Predicted. 3. Fit tree: Predict residuals. 4. Update: Add tree's predictions (scaled by learning rate). 5. Repeat: For N iterations. Result: Sum of all trees = final prediction.

🎯 Weak Learners🔄 Sequential📉 Residuals➕ Additive

# Gradient boosting from scratch
import numpy as np
from sklearn.tree import DecisionTreeRegressor

class GradientBoosting:
    def __init__(self, n_estimators=100, learning_rate=0.1, max_depth=3):
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate
        self.max_depth = max_depth
        self.trees = []
        self.initial_prediction = None
    
    def fit(self, X, y):
        # 1. Initial prediction (mean for regression)
        self.initial_prediction = y.mean()
        current_predictions = np.full_like(y, self.initial_prediction, dtype=float)
        
        # 2. Boosting rounds
        for i in range(self.n_estimators):
            # Compute residuals (negative gradient)
            residuals = y - current_predictions
            
            # Fit a tree to residuals
            tree = DecisionTreeRegressor(max_depth=self.max_depth)
            tree.fit(X, residuals)
            
            # Update predictions
            update = tree.predict(X)
            current_predictions += self.learning_rate * update
            
            self.trees.append(tree)
    
    def predict(self, X):
        predictions = np.full(X.shape[0], self.initial_prediction)
        for tree in self.trees:
            predictions += self.learning_rate * tree.predict(X)
        return predictions

# Example
from sklearn.datasets import make_regression
from sklearn.metrics import mean_squared_error

X, y = make_regression(n_samples=1000, n_features=10, noise=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Train gradient boosting
gb = GradientBoosting(n_estimators=100, learning_rate=0.1, max_depth=3)
gb.fit(X_train, y_train)
y_pred = gb.predict(X_test)
print(f"Custom GB MSE: {mean_squared_error(y_test, y_pred):.4f}")

# Compare with XGBoost
xgb_model = xgb.XGBRegressor(
    n_estimators=100, learning_rate=0.1, max_depth=3
)
xgb_model.fit(X_train, y_train)
xgb_pred = xgb_model.predict(X_test)
print(f"XGBoost MSE: {mean_squared_error(y_test, xgb_pred):.4f}")

# Comparison of boosting algorithms:
# - AdaBoost: Reweights misclassified samples
# - Gradient Boosting: Fits residuals (gradients)
# - XGBoost: Regularized gradient boosting
# - LightGBM: Histogram-based, faster
# - CatBoost: Handles categorical features

🌳 Decision Trees

🎯 The Building Blocks

XGBoost uses decision trees as base learners. Each tree splits data based on feature thresholds, creating a hierarchical decision structure. Trees are usually shallow (max_depth 3-10) to prevent overfitting.

Tree Structure

Root node: Top of the tree, initial split. Internal nodes: Decision points. Leaves: Final predictions. Split: Feature + threshold. Depth: Number of levels. CART: Classification and Regression Trees.

🌳 Nodes✂️ Splits🍃 Leaves📊 CART

# Decision trees in XGBoost
import xgboost as xgb
import numpy as np

# Tree parameters that control complexity
tree_params = {
    'max_depth': 6,              # Maximum depth of each tree
    'min_child_weight': 1,       # Minimum sum of instance weight in child
    'gamma': 0,                  # Minimum loss reduction for split
    'subsample': 1.0,            # Fraction of samples per tree
    'colsample_bytree': 1.0,     # Fraction of features per tree
    'colsample_bylevel': 1.0,    # Fraction of features per level
    'colsample_bynode': 1.0,     # Fraction of features per node
    'max_leaves': 0,             # Maximum number of leaves (0 = no limit)
    'max_bin': 256,              # Number of bins for histogram
    'tree_method': 'hist',       # 'auto', 'exact', 'approx', 'hist'
    'grow_policy': 'depthwise',  # 'depthwise', 'lossguide'
}

# Tree visualization
from xgboost import plot_tree
import matplotlib.pyplot as plt

# Train a small tree
X, y = make_classification(n_samples=100, n_features=5, random_state=42)
model = xgb.XGBClassifier(n_estimators=1, max_depth=3)
model.fit(X, y)

# Visualize (requires graphviz)
# plot_tree(model, num_trees=0)
# plt.show()

# Manual tree inspection
booster = model.get_booster()
tree_dump = booster.get_dump(with_stats=True)
print("Tree structure:")
print(tree_dump[0])

# Tree depth vs performance
depths = [1, 2, 3, 5, 8, 12, 20]
results = []

for depth in depths:
    model = xgb.XGBClassifier(max_depth=depth, n_estimators=100)
    model.fit(X_train, y_train)
    train_acc = accuracy_score(y_train, model.predict(X_train))
    test_acc = accuracy_score(y_test, model.predict(X_test))
    results.append((depth, train_acc, test_acc))
    print(f"Depth {depth}: Train={train_acc:.4f}, Test={test_acc:.4f}")

# Typically:
# - depth 1-3: Underfitting
# - depth 4-8: Sweet spot
# - depth 10+: Overfitting
print("\nGuidelines:")
print("  Shallow trees (3-6): Less overfitting, need more trees")
print("  Deep trees (8-12): More overfitting, need regularization")

🎯 Objective Function

🎯 What XGBoost Optimizes

The objective function in XGBoost has two parts: a loss function (measuring prediction error) and a regularization term (penalizing model complexity). Together they prevent overfitting.

Objective = Loss + Regularization

Formula: Obj = Σ L(y_i, ŷ_i) + Σ Ω(f_k). Loss: Depends on task (MSE, logloss, etc.). Regularization: Ω(f) = γT + ½λ||w||². T: Number of leaves. w: Leaf weights. Second-order: Uses both gradient and Hessian.

📉 Loss🛡️ Regularization📊 Gradient🔢 Hessian

# XGBoost objective functions
import xgboost as xgb
import numpy as np

# Available objectives
OBJECTIVES = {
    # Regression
    'reg:squarederror': 'MSE for regression',
    'reg:squaredlogerror': 'Squared log error',
    'reg:logistic': 'Logistic regression',
    'reg:pseudohubererror': 'Huber loss (robust)',
    'reg:absoluteerror': 'MAE (L1)',
    'reg:quantileerror': 'Quantile regression',
    
    # Classification
    'binary:logistic': 'Binary classification with probabilities',
    'binary:logitraw': 'Binary classification (raw scores)',
    'binary:hinge': 'SVM hinge loss',
    'multi:softmax': 'Multiclass classification',
    'multi:softprob': 'Multiclass with probabilities',
    
    # Ranking
    'rank:pairwise': 'Pairwise ranking',
    'rank:ndcg': 'NDCG ranking',
    'rank:map': 'MAP ranking',
    
    # Survival
    'survival:cox': 'Cox proportional hazards',
    'survival:aft': 'Accelerated failure time',
    
    # Count
    'count:poisson': 'Poisson regression',
    'reg:tweedie': 'Tweedie regression',
}

# Custom objective function
def custom_mse_objective(preds, dtrain):
    """Custom objective: MSE with gradient and hessian."""
    labels = dtrain.get_label()
    # Gradient: 2 * (preds - labels)
    grad = 2 * (preds - labels)
    # Hessian: 2
    hess = 2 * np.ones_like(preds)
    return grad, hess

# Custom metric
def custom_rmse(preds, dtrain):
    """Custom evaluation metric: RMSE."""
    labels = dtrain.get_label()
    return 'custom_rmse', np.sqrt(np.mean((preds - labels) ** 2))

# Train with custom objective
dtrain = xgb.DMatrix(X_train, label=y_train)
dtest = xgb.DMatrix(X_test, label=y_test)

params = {
    'max_depth': 6,
    'eta': 0.1,
    'verbosity': 0,
}

# Custom objective
bst = xgb.train(
    params,
    dtrain,
    num_boost_round=100,
    obj=custom_mse_objective,
    feval=custom_rmse,
    evals=[(dtest, 'test')],
    verbose_eval=10
)

# Objective function components
class XGBoostObjective:
    """XGBoost objective from scratch."""
    def __init__(self, reg_lambda=1.0, reg_alpha=0.0, gamma=0.0):
        self.reg_lambda = reg_lambda  # L2 regularization
        self.reg_alpha = reg_alpha    # L1 regularization
        self.gamma = gamma            # Minimum split loss
    
    def compute_loss(self, y, pred):
        """MSE loss."""
        return np.mean((y - pred) ** 2)
    
    def compute_gradient(self, y, pred):
        """First derivative of MSE."""
        return 2 * (pred - y)
    
    def compute_hessian(self, y, pred):
        """Second derivative of MSE."""
        return 2 * np.ones_like(pred)
    
    def compute_leaf_weight(self, gradients, hessians):
        """Optimal leaf weight: -G / (H + λ)."""
        G = gradients.sum()
        H = hessians.sum()
        return -G / (H + self.reg_lambda)
    
    def compute_split_gain(self, G_L, H_L, G_R, H_R, G, H):
        """Gain from splitting a node."""
        gain = 0.5 * (
            (G_L ** 2) / (H_L + self.reg_lambda) +
            (G_R ** 2) / (H_R + self.reg_lambda) -
            (G ** 2) / (H + self.reg_lambda)
        ) - self.gamma
        return gain

print("\n=== XGBoost Objective ===")
print("Obj = Σ L(y_i, ŷ_i) + Σ Ω(f_k)")
print("Ω(f) = γT + ½λ||w||²")
print("\nWhere:")
print("  L: Loss function")
print("  T: Number of leaves")
print("  w: Leaf weights")
print("  γ: Minimum split loss")
print("  λ: L2 regularization")

🛡️ Regularization

🎯 Preventing Overfitting

XGBoost's key innovation is its regularization — a penalty on model complexity built into the objective function. This prevents overfitting and makes XGBoost more robust than vanilla gradient boosting.

Regularization Techniques

L2 (λ): Penalizes large leaf weights. L1 (α): Encourages sparse weights. Gamma (γ): Minimum loss reduction for split. Max depth: Limits tree complexity. Min child weight: Minimum samples per leaf. Subsample: Row sampling. Colsample: Feature sampling.

📉 L2📉 L1✂️ Gamma📊 Sampling

# XGBoost regularization parameters
import xgboost as xgb
from sklearn.datasets import make_classification
from sklearn.model_selection import cross_val_score
import numpy as np

X, y = make_classification(n_samples=1000, n_features=20, 
                            n_informative=10, random_state=42)

# 1. L2 Regularization (lambda)
print("=== L2 Regularization (reg_lambda) ===")
for reg_lambda in [0, 0.1, 1, 10, 100]:
    model = xgb.XGBClassifier(
        reg_lambda=reg_lambda,
        n_estimators=100,
        max_depth=6
    )
    scores = cross_val_score(model, X, y, cv=5)
    print(f"  reg_lambda={reg_lambda:>5}: CV accuracy = {scores.mean():.4f}")

# 2. L1 Regularization (alpha)
print("\n=== L1 Regularization (reg_alpha) ===")
for reg_alpha in [0, 0.1, 1, 10, 100]:
    model = xgb.XGBClassifier(
        reg_alpha=reg_alpha,
        n_estimators=100,
        max_depth=6
    )
    scores = cross_val_score(model, X, y, cv=5)
    print(f"  reg_alpha={reg_alpha:>5}: CV accuracy = {scores.mean():.4f}")

# 3. Gamma (minimum split loss)
print("\n=== Gamma (min_split_loss) ===")
for gamma in [0, 0.1, 0.5, 1, 5]:
    model = xgb.XGBClassifier(
        gamma=gamma,
        n_estimators=100,
        max_depth=6
    )
    scores = cross_val_score(model, X, y, cv=5)
    print(f"  gamma={gamma:>5}: CV accuracy = {scores.mean():.4f}")

# 4. Min child weight
print("\n=== Min Child Weight ===")
for mcw in [1, 3, 5, 10, 20]:
    model = xgb.XGBClassifier(
        min_child_weight=mcw,
        n_estimators=100,
        max_depth=6
    )
    scores = cross_val_score(model, X, y, cv=5)
    print(f"  min_child_weight={mcw:>5}: CV accuracy = {scores.mean():.4f}")

# 5. Subsample
print("\n=== Subsample ===")
for subsample in [0.5, 0.7, 0.8, 0.9, 1.0]:
    model = xgb.XGBClassifier(
        subsample=subsample,
        n_estimators=100,
        max_depth=6
    )
    scores = cross_val_score(model, X, y, cv=5)
    print(f"  subsample={subsample}: CV accuracy = {scores.mean():.4f}")

# 6. Colsample_bytree
print("\n=== Colsample by tree ===")
for colsample in [0.5, 0.7, 0.8, 0.9, 1.0]:
    model = xgb.XGBClassifier(
        colsample_bytree=colsample,
        n_estimators=100,
        max_depth=6
    )
    scores = cross_val_score(model, X, y, cv=5)
    print(f"  colsample_bytree={colsample}: CV accuracy = {scores.mean():.4f}")

# Recommended starting point:
default_params = {
    'n_estimators': 100,
    'max_depth': 6,
    'learning_rate': 0.1,
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'reg_lambda': 1.0,
    'reg_alpha': 0.0,
    'gamma': 0.0,
    'min_child_weight': 1,
}

print("\n=== Recommended Starting Point ===")
for k, v in default_params.items():
    print(f"  {k}: {v}")

✂️ Split Finding Algorithm

🎯 How XGBoost Finds the Best Split

XGBoost uses an approximate split finding algorithm based on weighted quantile sketch. This makes it fast on large datasets while maintaining accuracy.

Split Finding Methods

Exact: Try all possible splits (slow). Approximate: Use quantiles of feature distribution. Histogram: Bucket continuous features. Weighted quantile: Considers Hessian weights. Global vs local: Propose splits once or per level. Parallel: Features processed in parallel.

🎯 Exact📊 Approximate📈 Histogram⚡ Parallel

# Split finding algorithms in XGBoost
import xgboost as xgb
import numpy as np
import time

# Generate large dataset
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=100000, n_features=50, random_state=42)

# 1. Exact split finding (slow but accurate)
print("=== Exact Split Finding ===")
t0 = time.time()
model_exact = xgb.XGBClassifier(
    tree_method='exact',
    n_estimators=100,
    max_depth=6
)
model_exact.fit(X, y)
exact_time = time.time() - t0
print(f"Time: {exact_time:.2f}s")

# 2. Approximate split finding
print("\n=== Approximate Split Finding ===")
t0 = time.time()
model_approx = xgb.XGBClassifier(
    tree_method='approx',
    n_estimators=100,
    max_depth=6
)
model_approx.fit(X, y)
approx_time = time.time() - t0
print(f"Time: {approx_time:.2f}s")

# 3. Histogram-based (fastest)
print("\n=== Histogram-based Split Finding ===")
t0 = time.time()
model_hist = xgb.XGBClassifier(
    tree_method='hist',
    n_estimators=100,
    max_depth=6,
    max_bin=256
)
model_hist.fit(X, y)
hist_time = time.time() - t0
print(f"Time: {hist_time:.2f}s")

# 4. GPU histogram
print("\n=== GPU Histogram ===")
try:
    t0 = time.time()
    model_gpu = xgb.XGBClassifier(
        tree_method='hist',
        device='cuda',
        n_estimators=100,
        max_depth=6
    )
    model_gpu.fit(X, y)
    gpu_time = time.time() - t0
    print(f"Time: {gpu_time:.2f}s")
except Exception as e:
    print(f"GPU not available: {e}")

# Weighted quantile sketch (the core algorithm)
class WeightedQuantileSketch:
    """Approximate quantile sketch for split finding."""
    def __init__(self, max_bins=256):
        self.max_bins = max_bins
    
    def build(self, feature_values, hessians):
        """Build histogram of feature values weighted by Hessian."""
        # 1. Sort by feature value
        sorted_idx = np.argsort(feature_values)
        sorted_values = feature_values[sorted_idx]
        sorted_hessians = hessians[sorted_idx]
        
        # 2. Compute cumulative weight
        cumsum = np.cumsum(sorted_hessians)
        total_weight = cumsum[-1]
        
        # 3. Find quantile boundaries
        boundaries = []
        target = total_weight / self.max_bins
        current_target = target
        
        for i in range(len(sorted_values)):
            if cumsum[i] >= current_target:
                boundaries.append(sorted_values[i])
                current_target += target
        
        return np.array(boundaries)
    
    def propose_splits(self, feature_values, hessians):
        """Propose candidate split points."""
        boundaries = self.build(feature_values, hessians)
        # Midpoints between boundaries
        splits = (boundaries[:-1] + boundaries[1:]) / 2
        return splits

# Example
feature = np.random.randn(1000)
hessians = np.random.rand(1000) + 0.5

sketch = WeightedQuantileSketch(max_bins=10)
splits = sketch.propose_splits(feature, hessians)
print(f"\nProposed splits: {splits.round(3)}")

print("\n=== Split Finding Comparison ===")
print(f"Exact:       {exact_time:.2f}s (slowest, most accurate)")
print(f"Approx:      {approx_time:.2f}s (medium)")
print(f"Hist:        {hist_time:.2f}s (fastest)")
print(f"GPU Hist:    {gpu_time if 'gpu_time' in dir() else 'N/A'} (fastest on GPU)")

❓ Missing Values Handling

🎯 Native Support

XGBoost handles missing values natively — you don't need to impute them. During tree construction, XGBoost learns the optimal direction to send missing values (left or right) at each split.

Missing Value Strategy

Sparsity-aware: Treats missing as a separate category. Optimal direction: Learns which way to go. Two passes: Try both directions, pick better. Handles NaN: Works with numpy NaN. No imputation: Saves preprocessing. Real-world: Common in tabular data.

🎯 Sparsity🔄 Auto Direction✅ No Imputation📊 Native

# Missing value handling in XGBoost
import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

# Create dataset with missing values
np.random.seed(42)
n_samples = 1000
n_features = 10

X = np.random.randn(n_samples, n_features)
y = (X[:, 0] + X[:, 1] > 0).astype(int)

# Add 20% missing values randomly
missing_mask = np.random.rand(*X.shape) < 0.2
X_missing = X.copy()
X_missing[missing_mask] = np.nan

print(f"Total values: {X.size}")
print(f"Missing values: {missing_mask.sum()} ({missing_mask.mean()*100:.1f}%)")

# 1. XGBoost handles missing natively
print("\n=== XGBoost with native NaN handling ===")
X_train, X_test, y_train, y_test = train_test_split(
    X_missing, y, test_size=0.2, random_state=42
)

model = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    missing=np.nan  # Explicitly specify missing value
)
model.fit(X_train, y_train)
y_pred = model.predict(X_test)
print(f"Accuracy with NaN: {accuracy_score(y_test, y_pred):.4f}")

# 2. Compare with imputation
from sklearn.impute import SimpleImputer

print("\n=== With mean imputation ===")
imputer = SimpleImputer(strategy='mean')
X_imputed = imputer.fit_transform(X_missing)

X_train_imp, X_test_imp, y_train_imp, y_test_imp = train_test_split(
    X_imputed, y, test_size=0.2, random_state=42
)

model_imp = xgb.XGBClassifier(n_estimators=100, max_depth=6)
model_imp.fit(X_train_imp, y_train_imp)
y_pred_imp = model_imp.predict(X_test_imp)
print(f"Accuracy with imputation: {accuracy_score(y_test_imp, y_pred_imp):.4f}")

# 3. Different missing value indicators
print("\n=== Custom missing value ===")
X_custom = X.copy()
X_custom[missing_mask] = -999  # Custom sentinel value

X_train_c, X_test_c, y_train_c, y_test_c = train_test_split(
    X_custom, y, test_size=0.2, random_state=42
)

model_custom = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    missing=-999  # Tell XGBoost what to treat as missing
)
model_custom.fit(X_train_c, y_train_c)
y_pred_c = model_custom.predict(X_test_c)
print(f"Accuracy with custom missing: {accuracy_score(y_test_c, y_pred_c):.4f}")

# 4. Missing value direction learned per split
print("\n=== How XGBoost handles missing ===")
booster = model.get_booster()
tree_dump = booster.get_dump(with_stats=True)

# Check first split of first tree
print("First tree's first split:")
first_tree = tree_dump[0]
first_lines = first_tree.split('\n')[:3]
for line in first_lines:
    print(f"  {line}")

# XGBoost's approach:
# - At each split, tries sending missing to left or right
# - Picks direction that gives higher gain
# - This is learned from data
# - No imputation needed

print("\n=== Key Benefits ===")
print("1. No need to impute missing values")
print("2. XGBoost learns optimal direction for missing")
print("3. Works well with sparse data")
print("4. Saves preprocessing time")
print("5. Often outperforms imputation")

⚙️ Hyperparameters

🎯 Tuning XGBoost

XGBoost has many hyperparameters. They fall into three categories: general parameters (which booster), booster parameters (tree-specific), and learning task parameters (objective, metric).

Hyperparameter Groups

General: booster, nthread, verbosity. Tree: max_depth, min_child_weight, gamma, subsample, colsample. Regularization: reg_alpha, reg_lambda. Learning: eta (learning_rate), n_estimators. Task: objective, eval_metric, seed. Advanced: tree_method, grow_policy, max_bin.

🌳 Tree🛡️ Regularization📈 Learning Rate🎯 Objective

# Complete XGBoost hyperparameters guide
import xgboost as xgb

# All key hyperparameters with defaults and ranges
HYPERPARAMETERS = {
    # General parameters
    'booster': {
        'default': 'gbtree',
        'options': ['gbtree', 'gblinear', 'dart'],
        'description': 'Which booster to use'
    },
    'nthread': {
        'default': 'all',
        'range': 'integer',
        'description': 'Number of parallel threads'
    },
    'verbosity': {
        'default': 1,
        'options': [0, 1, 2, 3],
        'description': 'Logging level'
    },
    
    # Tree booster parameters
    'eta': {
        'default': 0.3,
        'range': [0.01, 0.3],
        'description': 'Learning rate (step size shrinkage)'
    },
    'gamma': {
        'default': 0,
        'range': [0, 5],
        'description': 'Minimum loss reduction for split'
    },
    'max_depth': {
        'default': 6,
        'range': [3, 12],
        'description': 'Maximum tree depth'
    },
    'min_child_weight': {
        'default': 1,
        'range': [1, 20],
        'description': 'Minimum sum of instance weight in child'
    },
    'subsample': {
        'default': 1.0,
        'range': [0.5, 1.0],
        'description': 'Fraction of samples per tree'
    },
    'colsample_bytree': {
        'default': 1.0,
        'range': [0.5, 1.0],
        'description': 'Fraction of features per tree'
    },
    'colsample_bylevel': {
        'default': 1.0,
        'range': [0.5, 1.0],
        'description': 'Fraction of features per level'
    },
    'colsample_bynode': {
        'default': 1.0,
        'range': [0.5, 1.0],
        'description': 'Fraction of features per node'
    },
    'lambda': {
        'default': 1.0,
        'range': [0, 10],
        'description': 'L2 regularization on weights'
    },
    'alpha': {
        'default': 0.0,
        'range': [0, 10],
        'description': 'L1 regularization on weights'
    },
    'tree_method': {
        'default': 'auto',
        'options': ['auto', 'exact', 'approx', 'hist', 'gpu_hist'],
        'description': 'Tree construction algorithm'
    },
    'scale_pos_weight': {
        'default': 1.0,
        'range': [1, 100],
        'description': 'Balance positive/negative weights'
    },
    
    # Learning task parameters
    'objective': {
        'default': 'reg:squarederror',
        'options': ['binary:logistic', 'multi:softmax', 'reg:squarederror'],
        'description': 'Loss function'
    },
    'eval_metric': {
        'default': 'rmse',
        'options': ['rmse', 'mae', 'logloss', 'auc', 'error'],
        'description': 'Evaluation metric'
    },
    'seed': {
        'default': 0,
        'description': 'Random seed'
    },
}

# Print formatted guide
print("=" * 70)
print("XGBoost Hyperparameters Guide")
print("=" * 70)

for param, info in HYPERPARAMETERS.items():
    print(f"\n{param}")
    print(f"  Default: {info.get('default', 'N/A')}")
    if 'range' in info:
        print(f"  Range:   {info['range']}")
    if 'options' in info:
        print(f"  Options: {info['options']}")
    print(f"  {info['description']}")

# Tuning strategy
print("\n\n=== Tuning Strategy ===")
print("""
1. Start with defaults:
   - eta = 0.1, max_depth = 6, subsample = 0.8
   
2. Tune max_depth and min_child_weight (most important)
   - Grid search: [3, 5, 7, 9] and [1, 3, 5]
   
3. Tune gamma (regularization)
   - Grid search: [0, 0.1, 0.5, 1, 5]
   
4. Tune subsample and colsample_bytree
   - Grid search: [0.6, 0.8, 1.0]
   
5. Tune reg_alpha and reg_lambda
   - Grid search: [0, 0.1, 1, 10]
   
6. Lower learning rate, increase n_estimators
   - eta = 0.01, n_estimators = 5000, early_stopping = 100
""")

# Practical example
print("\n=== Practical Example ===")

# Baseline
baseline_params = {
    'n_estimators': 100,
    'max_depth': 6,
    'learning_rate': 0.1,
    'subsample': 0.8,
    'colsample_bytree': 0.8
}

# Tuned
tuned_params = {
    'n_estimators': 500,
    'max_depth': 4,
    'learning_rate': 0.05,
    'subsample': 0.7,
    'colsample_bytree': 0.7,
    'reg_lambda': 2.0,
    'reg_alpha': 0.1,
    'gamma': 0.2,
    'min_child_weight': 3
}

print("Baseline:", baseline_params)
print("\nTuned:", tuned_params)

🎓 Training & Tuning

🎯 Getting the Best Model

Training XGBoost effectively requires: cross-validation, early stopping, hyperparameter tuning, and learning curves. These techniques help you find the sweet spot between underfitting and overfitting.

Training Best Practices

Cross-validation: 5-fold or 10-fold CV. Early stopping: Stop when validation metric stops improving. Learning curves: Plot train vs validation. Hyperparameter search: Grid, random, or Bayesian. Learning rate: Lower = better but slower. N_estimators: Find with early stopping.

🔄 CV⏹️ Early Stop📈 Curves🔍 Tuning

# XGBoost training with best practices
import xgboost as xgb
import numpy as np
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split, GridSearchCV, RandomizedSearchCV, cross_val_score
from sklearn.metrics import accuracy_score, roc_auc_score
import matplotlib.pyplot as plt

# Generate data
X, y = make_classification(n_samples=5000, n_features=30, 
                            n_informative=15, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
X_train, X_val, y_train, y_val = train_test_split(
    X_train, y_train, test_size=0.2, random_state=42
)

# 1. Training with early stopping
print("=== Training with Early Stopping ===")
dtrain = xgb.DMatrix(X_train, label=y_train)
dval = xgb.DMatrix(X_val, label=y_val)
dtest = xgb.DMatrix(X_test, label=y_test)

params = {
    'max_depth': 6,
    'eta': 0.1,
    'objective': 'binary:logistic',
    'eval_metric': 'logloss',
    'subsample': 0.8,
    'colsample_bytree': 0.8,
    'seed': 42
}

# Train with early stopping
evals = [(dtrain, 'train'), (dval, 'val')]
bst = xgb.train(
    params,
    dtrain,
    num_boost_round=1000,
    evals=evals,
    early_stopping_rounds=50,
    verbose_eval=50
)

print(f"\nBest iteration: {bst.best_iteration}")
print(f"Best score: {bst.best_score:.4f}")

# 2. Cross-validation
print("\n=== Cross-Validation ===")
cv_results = xgb.cv(
    params,
    dtrain,
    num_boost_round=1000,
    nfold=5,
    early_stopping_rounds=50,
    metrics='logloss',
    seed=42,
    verbose_eval=50
)

print(f"\nCV results shape: {cv_results.shape}")
print(f"Best CV logloss: {cv_results['test-logloss-mean'].min():.4f}")

# 3. Grid search
print("\n=== Grid Search ===")
param_grid = {
    'max_depth': [3, 5, 7],
    'learning_rate': [0.05, 0.1, 0.2],
    'subsample': [0.7, 0.8, 0.9]
}

model = xgb.XGBClassifier(n_estimators=100, eval_metric='logloss')

grid_search = GridSearchCV(
    model, param_grid, cv=3, scoring='roc_auc', n_jobs=-1, verbose=1
)
grid_search.fit(X_train, y_train)

print(f"\nBest parameters: {grid_search.best_params_}")
print(f"Best CV score: {grid_search.best_score_:.4f}")

# 4. Random search (faster for large grids)
print("\n=== Random Search ===")
from scipy.stats import randint, uniform

param_dist = {
    'max_depth': randint(3, 12),
    'learning_rate': uniform(0.01, 0.3),
    'subsample': uniform(0.5, 0.5),
    'colsample_bytree': uniform(0.5, 0.5),
    'reg_lambda': uniform(0, 5),
    'reg_alpha': uniform(0, 2),
    'min_child_weight': randint(1, 10)
}

random_search = RandomizedSearchCV(
    model, param_dist, n_iter=50, cv=3, 
    scoring='roc_auc', n_jobs=-1, random_state=42, verbose=1
)
random_search.fit(X_train, y_train)

print(f"\nBest parameters: {random_search.best_params_}")
print(f"Best CV score: {random_search.best_score_:.4f}")

# 5. Learning curves
print("\n=== Learning Curves ===")
from sklearn.model_selection import learning_curve

train_sizes, train_scores, val_scores = learning_curve(
    xgb.XGBClassifier(n_estimators=100, max_depth=6),
    X_train, y_train,
    train_sizes=np.linspace(0.1, 1.0, 10),
    cv=5,
    scoring='roc_auc'
)

print(f"Train sizes: {train_sizes}")
print(f"Train scores: {train_scores.mean(axis=1).round(4)}")
print(f"Val scores: {val_scores.mean(axis=1).round(4)}")

# 6. Train final model with best params
print("\n=== Final Model ===")
best_params = random_search.best_params_
best_params['n_estimators'] = 500

final_model = xgb.XGBClassifier(**best_params)
final_model.fit(X_train, y_train)

y_pred = final_model.predict(X_test)
y_proba = final_model.predict_proba(X_test)[:, 1]

print(f"Test Accuracy: {accuracy_score(y_test, y_pred):.4f}")
print(f"Test ROC-AUC: {roc_auc_score(y_test, y_proba):.4f}")

# 7. Save model
final_model.save_model('xgboost_model.json')
print("\nModel saved to xgboost_model.json")

📊 Feature Importance

🎯 Understanding the Model

XGBoost provides multiple ways to measure feature importance: gain (average gain when feature is used), weight (number of times used), cover (average coverage), and total gain. These help identify which features matter most.

Importance Types

Gain: Average gain when feature is used in splits. Weight: Number of times feature is used. Cover: Average coverage (samples affected). Total gain: Sum of gains. Permutation: Shuffle feature, measure drop. SHAP: Game-theoretic attribution.

📈 Gain⚖️ Weight📊 Cover🎯 SHAP

# Feature importance in XGBoost
import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# Load data
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train model
model = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    random_state=42
)
model.fit(X_train, y_train)

# 1. Built-in feature importance (gain)
print("=== Feature Importance (Gain) ===")
importance_gain = model.get_booster().get_score(importance_type='gain')
sorted_gain = sorted(importance_gain.items(), key=lambda x: x[1], reverse=True)

for feature, score in sorted_gain[:10]:
    print(f"  {feature}: {score:.2f}")

# 2. Weight importance
print("\n=== Feature Importance (Weight) ===")
importance_weight = model.get_booster().get_score(importance_type='weight')
sorted_weight = sorted(importance_weight.items(), key=lambda x: x[1], reverse=True)

for feature, score in sorted_weight[:10]:
    print(f"  {feature}: {score:.0f}")

# 3. Cover importance
print("\n=== Feature Importance (Cover) ===")
importance_cover = model.get_booster().get_score(importance_type='cover')
sorted_cover = sorted(importance_cover.items(), key=lambda x: x[1], reverse=True)

for feature, score in sorted_cover[:10]:
    print(f"  {feature}: {score:.2f}")

# 4. All types in one DataFrame
importance_df = pd.DataFrame({
    'feature': list(importance_gain.keys()),
    'gain': [importance_gain.get(f, 0) for f in importance_gain.keys()],
    'weight': [importance_weight.get(f, 0) for f in importance_gain.keys()],
    'cover': [importance_cover.get(f, 0) for f in importance_gain.keys()],
})
importance_df = importance_df.sort_values('gain', ascending=False)
print("\n=== Top 10 Features ===")
print(importance_df.head(10).to_string(index=False))

# 5. Permutation importance
from sklearn.inspection import permutation_importance

print("\n=== Permutation Importance ===")
perm_importance = permutation_importance(
    model, X_test, y_test, n_repeats=10, random_state=42
)

perm_df = pd.DataFrame({
    'feature': X.columns,
    'importance': perm_importance.importances_mean,
    'std': perm_importance.importances_std
}).sort_values('importance', ascending=False)

print(perm_df.head(10).to_string(index=False))

# 6. SHAP values
try:
    import shap
    
    print("\n=== SHAP Values ===")
    explainer = shap.TreeExplainer(model)
    shap_values = explainer.shap_values(X_test)
    
    # Summary plot
    # shap.summary_plot(shap_values, X_test, show=False)
    # plt.tight_layout()
    # plt.savefig('shap_summary.png')
    
    # Feature importance from SHAP
    shap_importance = np.abs(shap_values).mean(axis=0)
    shap_df = pd.DataFrame({
        'feature': X.columns,
        'shap_importance': shap_importance
    }).sort_values('shap_importance', ascending=False)
    
    print(shap_df.head(10).to_string(index=False))
    
except ImportError:
    print("\nSHAP not installed. Install with: pip install shap")

# 7. Native plot importance
from xgboost import plot_importance
# plot_importance(model, max_num_features=15)
# plt.tight_layout()
# plt.savefig('feature_importance.png')

print("\n=== Importance Types Summary ===")
print("""
Gain:    Average gain when feature is used for splitting
         → Best for understanding feature contribution
         
Weight:  Number of times feature is used in trees
         → Biased toward high-cardinality features
         
Cover:   Average number of samples affected by splits
         → Measures how many samples use the feature
         
Permutation: Shuffle feature, measure performance drop
             → Model-agnostic, but slower
             
SHAP:    Game-theoretic feature attribution
         → Most accurate, but computationally expensive
""")

🌐 Distributed & GPU Training

🎯 Scaling XGBoost

XGBoost scales to massive datasets through distributed training (across multiple machines) and GPU acceleration (using CUDA). This enables training on billions of samples.

Scaling Options

GPU Hist: Fastest single-GPU training. Multi-GPU: Data-parallel across GPUs. Distributed: Spark, Dask, Kubernetes. Out-of-core: Data too large for RAM. External memory: Read from disk in chunks. Communication: Allreduce for gradients.

🚀 GPU🌐 Distributed💾 Out-of-core📊 Scale

# Distributed and GPU XGBoost
import xgboost as xgb
import numpy as np
import time

# Generate large dataset
n_samples = 1_000_000
n_features = 100
X = np.random.randn(n_samples, n_features).astype(np.float32)
y = (X[:, 0] + X[:, 1] > 0).astype(np.float32)

print(f"Dataset: {X.shape}, {X.nbytes / 1e9:.2f} GB")

# 1. CPU training
print("\n=== CPU Training ===")
t0 = time.time()
dtrain = xgb.DMatrix(X, label=y)
params_cpu = {
    'max_depth': 6,
    'eta': 0.1,
    'tree_method': 'hist',
    'nthread': -1
}
bst_cpu = xgb.train(params_cpu, dtrain, num_boost_round=100)
cpu_time = time.time() - t0
print(f"CPU time: {cpu_time:.2f}s")

# 2. GPU training
print("\n=== GPU Training ===")
try:
    t0 = time.time()
    params_gpu = {
        'max_depth': 6,
        'eta': 0.1,
        'tree_method': 'hist',
        'device': 'cuda'
    }
    bst_gpu = xgb.train(params_gpu, dtrain, num_boost_round=100)
    gpu_time = time.time() - t0
    print(f"GPU time: {gpu_time:.2f}s")
    print(f"Speedup: {cpu_time / gpu_time:.1f}x")
except Exception as e:
    print(f"GPU not available: {e}")

# 3. Multi-GPU (if available)
print("\n=== Multi-GPU Training ===")
try:
    params_multi = {
        'max_depth': 6,
        'eta': 0.1,
        'tree_method': 'hist',
        'device': 'cuda:0',  # Uses all available GPUs by default
    }
    bst_multi = xgb.train(params_multi, dtrain, num_boost_round=100)
    print("Multi-GPU training complete")
except Exception as e:
    print(f"Multi-GPU not available: {e}")

# 4. Out-of-core training (data larger than RAM)
print("\n=== Out-of-core Training ===")
# Save data to disk
dtrain.save_binary('train.buffer')

# Load with external memory
dtrain_ext = xgb.DMatrix('train.buffer#train.buffer.cache')
params_ext = {
    'max_depth': 6,
    'eta': 0.1,
    'tree_method': 'hist',
    'max_bin': 256
}
bst_ext = xgb.train(params_ext, dtrain_ext, num_boost_round=100)
print("Out-of-core training complete")

# 5. Distributed training with Dask
print("\n=== Distributed Training with Dask ===")
try:
    import dask.array as da
    import dask.distributed
    from xgboost import dask as dxgb
    
    # Create Dask cluster
    client = dask.distributed.Client(n_workers=4)
    
    # Convert to Dask arrays
    X_dask = da.from_array(X, chunks=(100000, n_features))
    y_dask = da.from_array(y, chunks=100000)
    
    # Train
    dtrain_dask = dxgb.DaskDMatrix(client, X_dask, y_dask)
    params_dask = {
        'max_depth': 6,
        'eta': 0.1,
        'tree_method': 'hist'
    }
    output = dxgb.train(
        client,
        params_dask,
        dtrain_dask,
        num_boost_round=100
    )
    print("Dask training complete")
    client.close()
    
except ImportError:
    print("Dask not installed")

# 6. Distributed training with Spark
print("\n=== Distributed Training with Spark ===")
print("""
For Spark, use the XGBoost4J-Spark library:

from pyspark.ml.feature import VectorAssembler
from pyspark.sql import SparkSession
from xgboost.spark import SparkXGBClassifier

spark = SparkSession.builder.appName("XGBoost").getOrCreate()

# Prepare data
assembler = VectorAssembler(inputCols=feature_cols, outputCol="features")
train_df = assembler.transform(train_data)

# Train
xgb_classifier = SparkXGBClassifier(
    features_col="features",
    label_col="label",
    num_workers=4,
    max_depth=6,
    n_estimators=100
)
model = xgb_classifier.fit(train_df)
""")

# Scaling comparison
print("\n=== Scaling Comparison ===")
print(f"{'Method':<20} {'Time':<12} {'Speedup':<10}")
print("-" * 45)
print(f"{'CPU (single)':<20} {cpu_time:<12.2f} {'1.0x':<10}")
if 'gpu_time' in dir():
    print(f"{'GPU (single)':<20} {gpu_time:<12.2f} {cpu_time/gpu_time:.1f}x")
print(f"{'Multi-GPU':<20} {'-':<12} {'-':<10}")
print(f"{'Distributed':<20} {'-':<12} {'-':<10}")

# GPU performance tips
print("\n=== GPU Performance Tips ===")
print("""
1. Use hist tree method (not exact)
2. Use float32 instead of float64
3. Increase max_bin for accuracy (default 256)
4. Set nthread appropriately
5. Use DMatrix to avoid re-conversion
6. Consider memory usage: GPU RAM is limited
7. Multi-GPU works best with large data
8. Use quantile sketch for large datasets
""")

🔍 Interpretability (SHAP)

🎯 Understanding Predictions

SHAP (SHapley Additive exPlanations) is the gold standard for interpreting XGBoost models. It assigns each feature a contribution to each prediction based on cooperative game theory.

SHAP Values

Local: Explain individual predictions. Global: Overall feature importance. Direction: Positive or negative impact. Interaction: Feature interactions. Additive: Sum of SHAP values = prediction - baseline. Consistent: Guaranteed to match model.

🎯 Local📊 Global➕ Additive🔬 Fair

# SHAP for XGBoost interpretability
import xgboost as xgb
import numpy as np
import pandas as pd
import shap
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split

# Load data
data = load_breast_cancer()
X = pd.DataFrame(data.data, columns=data.feature_names)
y = data.target

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train XGBoost
model = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=6,
    learning_rate=0.1,
    random_state=42
)
model.fit(X_train, y_train)

# 1. Create SHAP explainer
print("=== Creating SHAP Explainer ===")
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)

print(f"SHAP values shape: {shap_values.shape}")
print(f"Base value: {explainer.expected_value}")

# 2. Global feature importance
print("\n=== Global Feature Importance (SHAP) ===")
shap_importance = np.abs(shap_values).mean(axis=0)
shap_df = pd.DataFrame({
    'feature': X.columns,
    'shap_importance': shap_importance
}).sort_values('shap_importance', ascending=False)

print(shap_df.head(10).to_string(index=False))

# 3. Summary plot (beeswarm)
# shap.summary_plot(shap_values, X_test, show=False)

# 4. Bar plot
# shap.summary_plot(shap_values, X_test, plot_type="bar", show=False)

# 5. Individual prediction explanation
print("\n=== Individual Prediction ===")
sample_idx = 0
sample = X_test.iloc[sample_idx]
sample_shap = shap_values[sample_idx]

print(f"Sample prediction: {model.predict_proba(X_test)[sample_idx]}")
print(f"\nFeature contributions:")
contributions = pd.DataFrame({
    'feature': X.columns,
    'value': sample.values,
    'shap': sample_shap
}).sort_values('shap', key=abs, ascending=False)

print(contributions.head(10).to_string(index=False))

# 6. Waterfall plot (for single prediction)
# shap.plots.waterfall(shap.Explanation(
#     values=sample_shap,
#     base_values=explainer.expected_value,
#     data=sample.values,
#     feature_names=X.columns.tolist()
# ))

# 7. Force plot
# shap.force_plot(
#     explainer.expected_value,
#     sample_shap,
#     sample,
#     matplotlib=True
# )

# 8. Dependence plot
# shap.dependence_plot("mean radius", shap_values, X_test)

# 9. Interaction values
print("\n=== SHAP Interaction Values ===")
shap_interaction = explainer.shap_interaction_values(X_test[:100])
print(f"Interaction values shape: {shap_interaction.shape}")

# 10. SHAP for different tasks
print("\n=== SHAP for Different Tasks ===")

# Regression
from sklearn.datasets import make_regression
X_reg, y_reg = make_regression(n_samples=500, n_features=10, random_state=42)
X_train_r, X_test_r, y_train_r, y_test_r = train_test_split(
    X_reg, y_reg, test_size=0.2, random_state=42
)

reg_model = xgb.XGBRegressor(n_estimators=100, random_state=42)
reg_model.fit(X_train_r, y_train_r)

reg_explainer = shap.TreeExplainer(reg_model)
reg_shap = reg_explainer.shap_values(X_test_r)
print(f"Regression SHAP shape: {reg_shap.shape}")

# Multiclass
from sklearn.datasets import load_iris
iris = load_iris()
X_mc, y_mc = iris.data, iris.target
X_train_mc, X_test_mc, y_train_mc, y_test_mc = train_test_split(
    X_mc, y_mc, test_size=0.2, random_state=42
)

mc_model = xgb.XGBClassifier(n_estimators=100, random_state=42)
mc_model.fit(X_train_mc, y_train_mc)

mc_explainer = shap.TreeExplainer(mc_model)
mc_shap = mc_explainer.shap_values(X_test_mc)
print(f"Multiclass SHAP: {len(mc_shap)} arrays of shape {mc_shap[0].shape}")

# SHAP best practices
print("\n=== SHAP Best Practices ===")
print("""
1. Use TreeExplainer for tree-based models (fast, exact)
2. Use KernelExplainer for any model (slow, approximate)
3. Use DeepExplainer for neural networks
4. Use LinearExplainer for linear models

For XGBoost specifically:
- TreeExplainer is the fastest option
- Handles missing values
- Supports all objectives
- Provides interaction values

Interpretation:
- Positive SHAP → pushes prediction higher
- Negative SHAP → pushes prediction lower
- Magnitude → importance
- Sum of SHAP + base = prediction
""")

# Save SHAP values
np.save('shap_values.npy', shap_values)
print("\nSHAP values saved to shap_values.npy")

💼 Real-World Applications

🎯 Where XGBoost Shines

XGBoost dominates tabular data problems: finance, healthcare, e-commerce, advertising, and more. It's the algorithm of choice when you need accuracy, speed, and interpretability on structured data.

Industry Use Cases

Finance: Credit scoring, fraud detection, algorithmic trading. Healthcare: Disease prediction, drug discovery. E-commerce: Recommendation, churn prediction. Advertising: CTR prediction, bid optimization. Insurance: Risk assessment, claims. Manufacturing: Predictive maintenance.

💰 Finance🏥 Healthcare🛒 E-commerce📢 Ads

# Real-world XGBoost applications
import xgboost as xgb
import numpy as np
import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.metrics import (accuracy_score, roc_auc_score, 
                              mean_squared_error, precision_score, recall_score)

# 1. CREDIT SCORING
print("=== 1. Credit Scoring ===")
from sklearn.datasets import make_classification

X_credit, y_credit = make_classification(
    n_samples=10000, n_features=20, n_informative=10,
    weights=[0.9, 0.1], random_state=42  # Imbalanced
)
X_train, X_test, y_train, y_test = train_test_split(
    X_credit, y_credit, test_size=0.2, random_state=42
)

# Use scale_pos_weight for imbalanced data
scale = (y_train == 0).sum() / (y_train == 1).sum()

credit_model = xgb.XGBClassifier(
    n_estimators=200,
    max_depth=5,
    learning_rate=0.05,
    scale_pos_weight=scale,
    eval_metric='auc'
)
credit_model.fit(X_train, y_train)

y_proba = credit_model.predict_proba(X_test)[:, 1]
y_pred = (y_proba > 0.5).astype(int)

print(f"AUC-ROC: {roc_auc_score(y_test, y_proba):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall: {recall_score(y_test, y_pred):.4f}")

# 2. FRAUD DETECTION
print("\n=== 2. Fraud Detection ===")
# Highly imbalanced dataset
X_fraud, y_fraud = make_classification(
    n_samples=50000, n_features=30, n_informative=15,
    weights=[0.99, 0.01], random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X_fraud, y_fraud, test_size=0.2, random_state=42
)

# Fraud detection needs high recall
fraud_model = xgb.XGBClassifier(
    n_estimators=200,
    max_depth=6,
    learning_rate=0.1,
    scale_pos_weight=100,
    eval_metric='aucpr'  # Precision-Recall AUC
)
fraud_model.fit(X_train, y_train)

y_proba = fraud_model.predict_proba(X_test)[:, 1]
y_pred = (y_proba > 0.3).astype(int)  # Lower threshold for recall

print(f"AUC-PR: {roc_auc_score(y_test, y_proba):.4f}")
print(f"Precision: {precision_score(y_test, y_pred):.4f}")
print(f"Recall: {recall_score(y_test, y_pred):.4f}")

# 3. CUSTOMER CHURN
print("\n=== 3. Customer Churn ===")
X_churn, y_churn = make_classification(
    n_samples=5000, n_features=15, n_informative=8,
    weights=[0.7, 0.3], random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X_churn, y_churn, test_size=0.2, random_state=42
)

churn_model = xgb.XGBClassifier(
    n_estimators=150,
    max_depth=5,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8
)
churn_model.fit(X_train, y_train)

y_proba = churn_model.predict_proba(X_test)[:, 1]
print(f"AUC: {roc_auc_score(y_test, y_proba):.4f}")

# 4. HOUSE PRICE PREDICTION
print("\n=== 4. House Price Prediction ===")
from sklearn.datasets import fetch_california_housing

housing = fetch_california_housing()
X_house = housing.data
y_house = housing.target

X_train, X_test, y_train, y_test = train_test_split(
    X_house, y_house, test_size=0.2, random_state=42
)

house_model = xgb.XGBRegressor(
    n_estimators=200,
    max_depth=6,
    learning_rate=0.05,
    subsample=0.8
)
house_model.fit(X_train, y_train)

y_pred = house_model.predict(X_test)
rmse = np.sqrt(mean_squared_error(y_test, y_pred))
print(f"RMSE: {rmse:.4f}")

# 5. CLICK-THROUGH RATE (CTR) PREDICTION
print("\n=== 5. CTR Prediction ===")
X_ctr, y_ctr = make_classification(
    n_samples=100000, n_features=50, n_informative=25,
    weights=[0.95, 0.05], random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X_ctr, y_ctr, test_size=0.2, random_state=42
)

ctr_model = xgb.XGBClassifier(
    n_estimators=300,
    max_depth=7,
    learning_rate=0.05,
    subsample=0.8,
    colsample_bytree=0.8,
    scale_pos_weight=19,
    tree_method='hist'
)
ctr_model.fit(X_train, y_train)

y_proba = ctr_model.predict_proba(X_test)[:, 1]
print(f"AUC: {roc_auc_score(y_test, y_proba):.4f}")

# 6. DISEASE PREDICTION
print("\n=== 6. Disease Prediction ===")
X_disease, y_disease = make_classification(
    n_samples=5000, n_features=25, n_informative=12,
    weights=[0.85, 0.15], random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X_disease, y_disease, test_size=0.2, random_state=42
)

disease_model = xgb.XGBClassifier(
    n_estimators=200,
    max_depth=4,
    learning_rate=0.05,
    reg_lambda=2.0,
    eval_metric='auc'
)
disease_model.fit(X_train, y_train)

y_proba = disease_model.predict_proba(X_test)[:, 1]
print(f"AUC: {roc_auc_score(y_test, y_proba):.4f}")

# 7. PREDICTIVE MAINTENANCE
print("\n=== 7. Predictive Maintenance ===")
X_maint, y_maint = make_classification(
    n_samples=10000, n_features=30, n_informative=15,
    weights=[0.9, 0.1], random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
    X_maint, y_maint, test_size=0.2, random_state=42
)

maint_model = xgb.XGBClassifier(
    n_estimators=200,
    max_depth=6,
    learning_rate=0.05,
    scale_pos_weight=9,
    eval_metric='auc'
)
maint_model.fit(X_train, y_train)

y_proba = maint_model.predict_proba(X_test)[:, 1]
print(f"AUC: {roc_auc_score(y_test, y_proba):.4f}")

# Application summary
print("\n\n=== XGBoost Application Summary ===")
applications = {
    'Credit Scoring': 'Binary classification, imbalanced, AUC',
    'Fraud Detection': 'Extreme imbalance, precision-recall',
    'Customer Churn': 'Binary classification, feature importance',
    'House Prices': 'Regression, RMSE',
    'CTR Prediction': 'Large scale, imbalanced, AUC',
    'Disease Prediction': 'Healthcare, interpretability',
    'Predictive Maintenance': 'Time series, imbalanced',
    'Recommendation': 'Ranking, NDCG',
    'Risk Assessment': 'Insurance, regression',
    'Sentiment Analysis': 'NLP + tabular features',
}

for app, desc in applications.items():
    print(f"  {app:<25}: {desc}")

🚀 Future: AutoML & Beyond

🎯 Where XGBoost is Heading

XGBoost continues to evolve: AutoML, federated learning, deep integration with deep learning, and new algorithms like LightGBM and CatBoost.

Future Directions

AutoML: Automatic hyperparameter tuning. Federated: Train across devices without sharing data. Deep learning: Hybrid models (GBDT + NN). New algorithms: LightGBM, CatBoost, NGBoost. Explainability: Better interpretability. Integration: Cloud ML platforms.

🤖 AutoML🔒 Federated🧠 Hybrid☁️ Cloud

# Future of gradient boosting
import xgboost as xgb
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score
import time

# 1. AutoML with Optuna
print("=== AutoML with Optuna ===")
try:
    import optuna
    
    def objective(trial):
        params = {
            'n_estimators': trial.suggest_int('n_estimators', 100, 1000),
            'max_depth': trial.suggest_int('max_depth', 3, 12),
            'learning_rate': trial.suggest_float('learning_rate', 0.01, 0.3, log=True),
            'subsample': trial.suggest_float('subsample', 0.5, 1.0),
            'colsample_bytree': trial.suggest_float('colsample_bytree', 0.5, 1.0),
            'reg_lambda': trial.suggest_float('reg_lambda', 0, 10),
            'reg_alpha': trial.suggest_float('reg_alpha', 0, 10),
            'min_child_weight': trial.suggest_int('min_child_weight', 1, 10),
        }
        
        X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2)
        
        model = xgb.XGBClassifier(**params, eval_metric='logloss')
        model.fit(X_train, y_train, 
                  eval_set=[(X_val, y_val)],
                  early_stopping_rounds=20,
                  verbose=False)
        
        return accuracy_score(y_val, model.predict(X_val))
    
    # Run optimization
    study = optuna.create_study(direction='maximize')
    study.optimize(objective, n_trials=20, show_progress_bar=False)
    
    print(f"Best params: {study.best_params}")
    print(f"Best accuracy: {study.best_value:.4f}")
    
except ImportError:
    print("Optuna not installed. Install with: pip install optuna")

# 2. Federated Learning with XGBoost
print("\n=== Federated Learning ===")
print("""
Federated XGBoost:
- Multiple parties train on local data
- Only gradients/statistics are shared
- Privacy-preserving

Implementation:
1. Each party computes local histograms
2. Encrypted aggregation of histograms
3. Global split finding
4. Repeat for each tree

Libraries:
- FATE (WeBank)
- NVIDIA FLARE
- Flower
- PySyft
""")

# 3. Hybrid Deep Learning + XGBoost
print("\n=== Hybrid Models ===")
try:
    import torch
    import torch.nn as nn
    
    class GBDT_NN_Hybrid(nn.Module):
        """Hybrid GBDT + Neural Network."""
        def __init__(self, input_dim, hidden_dim=64):
            super().__init__()
            self.nn = nn.Sequential(
                nn.Linear(input_dim, hidden_dim),
                nn.ReLU(),
                nn.Dropout(0.3),
                nn.Linear(hidden_dim, 32),
                nn.ReLU(),
                nn.Linear(32, 1),
                nn.Sigmoid()
            )
        
        def forward(self, x):
            return self.nn(x)
    
    # Get XGBoost predictions as features
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
    
    xgb_model = xgb.XGBClassifier(n_estimators=100, max_depth=6)
    xgb_model.fit(X_train, y_train)
    
    # Use leaf indices as new features
    leaf_indices_train = xgb_model.apply(X_train)
    leaf_indices_test = xgb_model.apply(X_test)
    
    print(f"XGBoost leaf features: {leaf_indices_train.shape}")
    
except ImportError:
    print("PyTorch not installed")

# 4. New Gradient Boosting Algorithms
print("\n=== New Algorithms ===")
algorithms = {
    'LightGBM': {
        'features': ['Histogram-based', 'Leaf-wise growth', 'GOSS', 'EFB'],
        'speed': 'Fastest',
        'accuracy': 'Excellent',
    },
    'CatBoost': {
        'features': ['Categorical support', 'Ordered boosting', 'GPU'],
        'speed': 'Fast',
        'accuracy': 'Excellent',
    },
    'XGBoost': {
        'features': ['Regularization', 'Parallel', 'GPU'],
        'speed': 'Fast',
        'accuracy': 'Excellent',
    },
    'NGBoost': {
        'features': ['Probabilistic', 'Uncertainty'],
        'speed': 'Slow',
        'accuracy': 'Good',
    },
}

for name, info in algorithms.items():
    print(f"\n{name}:")
    for k, v in info.items():
        print(f"  {k}: {v}")

# 5. XGBoost in Cloud Platforms
print("\n=== Cloud Platform Integration ===")
clouds = {
    'AWS SageMaker': 'Built-in XGBoost container',
    'Google Vertex AI': 'XGBoost training and serving',
    'Azure ML': 'XGBoost pipelines',
    'Databricks': 'XGBoost on Spark',
    'Domino Data Lab': 'Enterprise ML',
    'H2O.ai': 'Distributed XGBoost',
}

for cloud, desc in clouds.items():
    print(f"  {cloud:<25}: {desc}")

# 6. Future trends
print("\n=== Future Trends ===")
trends = [
    "AutoML: Automatic model selection and tuning",
    "Federated: Privacy-preserving distributed learning",
    "Hybrid: Combining GBDT with neural networks",
    "Explainability: Better interpretation methods",
    "Efficiency: 10x faster training with new algorithms",
    "Integration: Unified ML pipelines",
    "Uncertainty: Probabilistic predictions (NGBoost)",
    "Causal: Causal inference with boosting",
]

for i, trend in enumerate(trends, 1):
    print(f"  {i}. {trend}")

# 7. Practical recommendations
print("\n\n=== Practical Recommendations ===")
recommendations = """
For most tabular data problems:
1. Start with XGBoost or LightGBM
2. Tune max_depth, learning_rate, n_estimators
3. Use early stopping with validation set
4. Handle imbalanced data with scale_pos_weight
5. Use SHAP for interpretability
6. Consider CatBoost for categorical features
7. Use GPU for large datasets
8. Ensemble multiple boosting models

When to use what:
- XGBoost: General purpose, robust, well-documented
- LightGBM: Fastest, large datasets, leaf-wise growth
- CatBoost: Categorical features, ordered boosting
- NGBoost: Need uncertainty estimates

For production:
- Save models in JSON or UBJ format
- Use ONNX for cross-platform
- Monitor for drift
- Retrain periodically
- Use SHAP for explanations
"""

print(recommendations)