๐ Logistic Core Philosophy
๐ฏ Probabilistic Classification
Logistic Regression is a generalized linear model for binary classification. It models the probability that an instance belongs to a class via the logistic (sigmoid) function. Despite its name, it's a classification algorithm that outputs probabilities.
Why "Regression"?
It's a regression of the log-odds โ the model predicts the log-odds of the event, which are linear in the features. The sigmoid transforms this into a probability. Key assumptions: linearity in log-odds, no multicollinearity, independent errors.
import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=100, n_features=4, random_state=42)
model = LogisticRegression()
model.fit(X, y)
print(f"Coef: {model.coef_}, Intercept: {model.intercept_}")
print(f"Probabilities (first 3): {model.predict_proba(X[:3])}")๐งฎ Odds, Logits & Sigmoid
๐ฏ The Sigmoid Transformation
Logistic function: ฯ(z) = 1/(1+e^(-z)). Logit: ln(p/(1-p)) = ฮฒโ + ฮฒโxโ + ... + ฮฒโxโ. Odds = p/(1-p).
From Linear to Probability
The linear combination of features gives the log-odds. The sigmoid squashes this to [0,1], yielding a probability. This monotonic transformation ensures that higher log-odds โ higher probability.
import numpy as np
def sigmoid(z): return 1/(1+np.exp(-z))
z = np.array([-2, 0, 2])
print(f"Sigmoid: {sigmoid(z)}")
# odds: p/(1-p)
p = 0.8
print(f"Odds: {p/(1-p):.2f}, Logit: {np.log(p/(1-p)):.2f}")๐ Loss: Log-Likelihood & Cross-Entropy
๐ฏ Maximum Likelihood Estimation
Minimize negative log-likelihood: -ฮฃ [yแตข log(pแตข) + (1-yแตข) log(1-pแตข)].
Cross-Entropy & MLE
Cross-entropy loss is convex, ensuring a unique global minimum. It penalizes confident wrong predictions heavily.
from sklearn.metrics import log_loss
y_true = [1, 0, 1, 1]
y_pred = [0.9, 0.1, 0.8, 0.7]
print(f"Log Loss: {log_loss(y_true, y_pred):.4f}")โก Gradient Descent & Newton
๐ฏ Solvers
Gradient descent (batch, mini-batch, stochastic), Newton-CG, LBFGS.
Convergence & Learning Rate
Log-loss is convex, so gradient descent converges to global minimum. Newton methods use second-order Hessian for faster convergence.
# sklearn defaults: lbfgs
model = LogisticRegression(solver='lbfgs', max_iter=1000)
model.fit(X, y)
print(f"Iterations: {model.n_iter_}")๐งฉ L1, L2 & ElasticNet
๐ฏ Shrinkage & Feature Selection
L1 (Lasso) shrinks coefficients to zero, L2 (Ridge) penalizes squared magnitude.
Bias-Variance Tradeoff
Regularization reduces overfitting. L1 yields sparse models โ good for high-dimensional data. ElasticNet combines L1 and L2.
model_l1 = LogisticRegression(penalty='l1', solver='saga', C=0.1)
model_l1.fit(X, y)
print(f"L1 Coef: {model_l1.coef_}")๐ Multinomial & One-vs-Rest
๐ฏ Extending to K Classes
Multinomial logistic (softmax) or OvR (one-vs-rest).
Softmax vs OvR
Softmax outputs probabilities that sum to 1, suitable for mutually exclusive classes. OvR trains K binary classifiers.
model_multi = LogisticRegression(multi_class='multinomial', solver='lbfgs')
from sklearn.datasets import load_iris
iris = load_iris()
model_multi.fit(iris.data, iris.target)
print(f"Multinomial coef shape: {model_multi.coef_.shape}")๐ Feature Engineering & Interactions
๐ฏ Polynomials, Interactions, Transformations
Logistic regression is linear in log-odds, so feature engineering is critical.
Capturing Nonlinearity
Add polynomial terms, interaction features, or use splines to capture nonlinear relationships.
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2, include_bias=False)
X_poly = poly.fit_transform(X)
model_poly = LogisticRegression().fit(X_poly, y)๐ Evaluation: ROC, AUC, Calibration
๐ฏ Threshold & Performance
ROC curve, AUC, Brier score, calibration plots.
Beyond Accuracy
AUC measures ranking quality. Calibration ensures predicted probabilities match empirical frequencies.
from sklearn.metrics import roc_auc_score, brier_score_loss
y_pred_prob = model.predict_proba(X)[:,1]
print(f"AUC: {roc_auc_score(y, y_pred_prob):.3f}")
print(f"Brier: {brier_score_loss(y, y_pred_prob):.3f}")๐ Odds Ratio & Interpretation
๐ฏ Exponentiate Coefficients
Odds ratio = e^(ฮฒ). For a one-unit increase in x, odds multiply by e^(ฮฒ).
Clinical & Business Interpretation
Interpret coefficients in terms of odds ratios โ crucial for medical, marketing, and risk analytics.
import numpy as np
coef = model.coef_[0]
odds_ratio = np.exp(coef)
print(f"Odds ratios: {odds_ratio}")๐ฏ Decision Boundary & Threshold
๐ฏ Default threshold = 0.5
Adjust threshold to balance sensitivity/specificity.
Threshold Tuning
Lower threshold โ more positives (higher recall). Use ROC to find optimal threshold for your cost function.
threshold = 0.3
y_pred = (y_pred_prob >= threshold).astype(int)
print(f"Precision/Recall at threshold 0.3")โ๏ธ sklearn API & Pipelines
๐ฏ Scikit-Learn Integration
LogisticRegression, Pipeline, GridSearchCV.
Production-Ready Pipelines
StandardScaler + LogisticRegression + GridSearch for hyperparameter tuning.
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
pipe = Pipeline([('scaler', StandardScaler()), ('clf', LogisticRegression())])
pipe.fit(X, y)๐ Statistical Inference & P-values
๐ฏ Wald, Likelihood Ratio, Score
P-values for coefficients, confidence intervals.
Hypothesis Testing
Test if coefficient is significantly different from zero. Use statsmodels for detailed summary.
import statsmodels.api as sm
X_sm = sm.add_constant(X)
model_sm = sm.Logit(y, X_sm).fit()
print(model_sm.summary())๐ง Bayesian Logistic Regression
๐ฏ Posterior over coefficients
Regularization as prior (L2 = Gaussian prior).
Uncertainty Quantification
Bayesian approach gives credible intervals, naturally handles uncertainty.
# Using PyMC (pseudo)
# with pm.Model() as model:
# ฮฒ = pm.Normal('ฮฒ', mu=0, sigma=10, shape=X.shape[1])
# p = pm.math.sigmoid(X @ ฮฒ)
# y_obs = pm.Bernoulli('y_obs', p, observed=y)๐ Deployment & Monitoring
๐ฏ Model Serving & Drift
Pickle, ONNX, MLflow. Monitor prediction drift and data drift.
CI/CD for Models
Log models with MLflow, serve via FastAPI, monitor calibration drift.
import joblib
joblib.dump(model, 'logreg.joblib')
loaded_model = joblib.load('logreg.joblib')
print(loaded_model.predict_proba(X[:1]))