🎯 SVM Core Philosophy

🎯 Margin Maximization

Support Vector Machine (SVM) is a powerful supervised learning algorithm that finds the hyperplane maximizing the margin between classes. It's effective in high-dimensional spaces and can handle non-linear boundaries using the kernel trick.

Why SVM?

Maximum margin principle — SVM focuses on the most challenging points (support vectors) to define the decision boundary. This leads to excellent generalization. The kernel trick allows SVM to implicitly map data to high-dimensional feature spaces without explicit computation.

📌 Margin Maximization📌 Support Vectors📌 Kernel Trick📌 Generalization

from sklearn.svm import SVC
from sklearn.datasets import make_classification

X, y = make_classification(n_samples=100, n_features=2, n_redundant=0, random_state=42)
svm = SVC(kernel='linear', C=1.0)
svm.fit(X, y)
print(f"Support vectors: {len(svm.support_vectors_)}")
print(f"Intercept: {svm.intercept_}")
print(f"Coefficients: {svm.coef_}")

📐 Maximal Margin & Support Vectors

🎯 The Decision Boundary

Hyperplane: w·x + b = 0. Margin = 2/||w||. Support vectors are data points closest to the hyperplane.

Support Vector Intuition

Only support vectors matter for the final model. Removing non-support vectors doesn't change the decision boundary. This makes SVM memory-efficient.

📐 Margin = 2/||w||🔍 Support Vectors

# Visualize margin (conceptual)
# svm.support_vectors_ contains the points that define the margin
print(f"Support vectors: {svm.support_vectors_[:3]}")
print(f"Support vector indices: {svm.support_}")

🧮 Dual Formulation & Optimization

🎯 Primal vs Dual

Primal: minimize ½||w||². Dual: maximize Σαᵢ - ½Σαᵢαⱼyᵢyⱼxᵢ·xⱼ subject to 0 ≤ αᵢ ≤ C, Σαᵢyᵢ = 0.

Why Dual?

The dual formulation is convex and enables the kernel trick. αᵢ are Lagrange multipliers — only support vectors have αᵢ > 0.

📊 Convex Optimization🧮 Lagrange Multipliers

# SVM dual coefficients (alpha)
# In sklearn, dual_coef_ stores αᵢyᵢ
print(f"Dual coefficients shape: {svm.dual_coef_.shape}")
print(f"Number of support vectors: {len(svm.support_)}")

🌀 Kernel Trick

🎯 Implicit Feature Mapping

Kernel function K(xᵢ, xⱼ) = φ(xᵢ)·φ(xⱼ). Linear, Polynomial, RBF, Sigmoid.

RBF Kernel Magic

RBF (Gaussian) kernel maps data to infinite-dimensional space, allowing highly flexible decision boundaries. γ controls the influence of each point.

🌀 RBF Kernel📈 Polynomial📐 Linear

# Different kernels
svm_linear = SVC(kernel='linear')
svm_poly = SVC(kernel='poly', degree=3)
svm_rbf = SVC(kernel='rbf', gamma='scale')
# RBF kernel function: K(x, x') = exp(-γ||x - x'||²)

⚖️ Soft Margin & C Parameter

🎯 Balancing Margin vs Misclassification

C controls the trade-off: large C → hard margin (low bias, high variance). small C → soft margin (high bias, low variance).

C Parameter Tuning

C is the penalty for misclassification. High C tries to classify all training points correctly, risking overfitting. Low C allows more misclassifications for a wider margin.

⚖️ C Trade-off🛡️ Overfitting Control

# C parameter effects
svm_hard = SVC(C=100, kernel='linear')  # hard margin
svm_soft = SVC(C=0.01, kernel='linear')  # soft margin
# Use GridSearchCV to find optimal C

📉 Hinge Loss & Regularization

🎯 Loss = max(0, 1 - yᵢ(w·xᵢ + b))

Objective: Σ max(0, 1 - yᵢ(w·xᵢ + b)) + λ||w||².

Convex Surrogate

Hinge loss is a convex approximation to the 0-1 loss. It's zero when margin ≥ 1, linear otherwise.

📉 Convex Loss🔍 Regularization

import numpy as np
def hinge_loss(y_true, y_pred):
    return np.maximum(0, 1 - y_true * y_pred)
y_true = np.array([1, -1, 1])
y_pred = np.array([0.5, -0.8, 1.2])
print(f"Hinge loss: {hinge_loss(y_true, y_pred)}")

📈 SVM for Regression (SVR)

🎯 ε-insensitive Tube

SVR tries to fit a tube of width ε around the regression line. Points outside the tube incur penalty.

SVR Parameters

ε controls the tube width. C controls penalty for points outside the tube. Larger ε → less sensitive to noise.

📈 ε-Tube⚖️ ε Parameter

from sklearn.svm import SVR
X_reg = np.random.rand(100, 1) * 10
y_reg = np.sin(X_reg).ravel() + np.random.normal(0, 0.1, 100)
svr = SVR(kernel='rbf', C=1.0, epsilon=0.2)
svr.fit(X_reg, y_reg)
print(f"SVR R²: {svr.score(X_reg, y_reg):.3f}")

🔀 Multiclass with SVM

🎯 One-vs-One (OvO) vs One-vs-Rest (OvR)

OvO: train K(K-1)/2 classifiers. OvR: train K classifiers.

Strategy Comparison

OvO is computationally expensive for large K but more accurate. OvR is faster. sklearn uses OvO by default for SVC.

⚔️ OvO⚖️ OvR

from sklearn.datasets import load_iris
iris = load_iris()
svm_multi = SVC(kernel='rbf', decision_function_shape='ovo')
svm_multi.fit(iris.data, iris.target)
print(f"Accuracy: {svm_multi.score(iris.data, iris.target):.3f}")

📊 SVM & Dimensionality

🎯 High-Dimensional Performance

SVM is effective even when n_features > n_samples, thanks to the kernel trick and regularization.

Feature Reduction

PCA can be used before SVM to reduce dimensionality while preserving variance.

📉 High-Dim Friendly🔍 PCA + SVM

from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_pca = pca.fit_transform(X)
svm_pca = SVC(kernel='rbf').fit(X_pca, y)

📐 Feature Scaling Importance

🎯 Scale Matters for SVM

Features with large magnitudes dominate the kernel computation. Always standardize features.

Standardization

Use StandardScaler or MinMaxScaler before SVM. RBF kernel is particularly sensitive to scale.

📊 StandardScaler📏 MinMaxScaler

from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)
svm_scaled = SVC().fit(X_scaled, y)
print(f"Scaled accuracy: {svm_scaled.score(X_scaled, y):.3f}")

⚖️ Imbalanced Data & Class Weights

🎯 Class Weight Parameter

class_weight='balanced' automatically adjusts weights inversely proportional to class frequencies.

Handling Imbalance

Penalize misclassifications of minority class more heavily. Use class_weight dict or 'balanced'.

⚖️ class_weight📊 Balanced

svm_balanced = SVC(class_weight='balanced')
# Or custom weights
svm_custom = SVC(class_weight={0: 1.0, 1: 5.0})

🧠 Kernel Selection Guide

🎯 Choosing the Right Kernel

Linear: high-dimensional, text data. RBF: non-linear, default choice. Poly: polynomial relationships.

Heuristics

Start with RBF. If n_features > n_samples, use linear. If features are discrete, use linear.

📐 Linear🌀 RBF📈 Polynomial

# GridSearch for kernel and parameters
params = {'kernel': ['linear', 'rbf', 'poly'], 'C': [0.1, 1, 10], 'gamma': ['scale', 'auto']}
# GridSearchCV(svc, params, cv=5)

⚙️ sklearn API & Pipelines

🎯 SVC, SVR, LinearSVC

Unified API, GridSearchCV, Pipelines.

Production Setup

Combine scaling and SVM in a Pipeline with hyperparameter tuning.

🔧 Pipeline🔎 GridSearch

from sklearn.pipeline import Pipeline
pipe = Pipeline([('scaler', StandardScaler()), ('svm', SVC())])
params = {'svm__C': [0.1, 1, 10], 'svm__gamma': ['scale', 'auto']}
grid = GridSearchCV(pipe, params, cv=5)
grid.fit(X, y)
print(f"Best params: {grid.best_params_}")

🚀 Deployment & Performance

🎯 Model Serving

Pickle, ONNX, MLflow. SVM can be slow for large datasets.

Optimization Strategies

Use LinearSVC for linear kernels. For large datasets, consider SGDClassifier with hinge loss.

📦 Pickle⚡ LinearSVC

import joblib
joblib.dump(svm, 'svm_model.joblib')
loaded_svm = joblib.load('svm_model.joblib')
print(loaded_svm.predict(X[:1]))