🧬 Autoencoders: Foundations

🎯 The Information Bottleneck

Autoencoders are neural networks that learn to copy their input to their output through a compressed latent space. They consist of an encoder (compression) and a decoder (reconstruction). The bottleneck forces the model to learn the most salient features of the data.

Deep Explanation

The autoencoder learns an identity function under constraints — the latent space has lower dimensionality than the input. This forces the network to capture the underlying structure and discard noise. Autoencoders are the foundation of representation learning.

Why it matters: Autoencoders are used for dimensionality reduction, anomaly detection, denoising, feature extraction, and generative modeling (VAEs). They are unsupervised learning workhorses.

📌 Bottleneck📌 Unsupervised📌 Representation Learning📌 Reconstruction

import torch
import torch.nn as nn

class Autoencoder(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Linear(input_dim, 128),
            nn.ReLU(),
            nn.Linear(128, latent_dim)
        )
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim)
        )

    def forward(self, x):
        z = self.encoder(x)
        return self.decoder(z), z

🏗️ Architecture & Components

🎯 Encoder, Bottleneck, Decoder

Encoder: maps input to latent representation. Bottleneck: the compressed latent space (lower dimensionality). Decoder: reconstructs input from latent representation.

Design Choices

The bottleneck dimension determines the compression level. Too small → information loss. Too large → learning trivial identity. Activation functions: ReLU, LeakyReLU, sigmoid (for binary data).

📐 Latent Dimension⚡ Activation Functions🏗️ Bottleneck

# Encoder with dropout for regularization
class Encoder(nn.Module):
    def __init__(self, input_dim, latent_dim, hidden_dim=256):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(input_dim, hidden_dim),
            nn.ReLU(),
            nn.Dropout(0.2),
            nn.Linear(hidden_dim, hidden_dim//2),
            nn.ReLU(),
            nn.Linear(hidden_dim//2, latent_dim)
        )

📉 Reconstruction Loss

🎯 Measuring Reconstruction Quality

Common losses: Mean Squared Error (MSE) for continuous data, Binary Cross-Entropy (BCE) for binary data. The loss is the difference between input and reconstructed output.

Loss Functions

MSE: (x - x')². BCE: -[x log(x') + (1-x) log(1-x')]. The choice depends on the data distribution.

📊 MSE📉 BCE🎯 Reconstruction Error

import torch.nn.functional as F

def reconstruction_loss(x, x_hat, loss_type='mse'):
    if loss_type == 'mse':
        return F.mse_loss(x_hat, x, reduction='mean')
    elif loss_type == 'bce':
        return F.binary_cross_entropy(x_hat, x, reduction='mean')
    else:
        return F.l1_loss(x_hat, x, reduction='mean')

🔽 Undercomplete Autoencoders

🎯 Latent Dim < Input Dim

The classic autoencoder where the latent space has lower dimensionality than the input. It learns a compressed representation and is used for dimensionality reduction (similar to PCA but non-linear).

Non-linear PCA

With non-linear activations, autoencoders can learn more complex representations than linear PCA. They can capture non-linear manifolds in data.

📉 Dimensionality Reduction🔽 Non-linear📊 Feature Extraction

# Undercomplete autoencoder example
input_dim = 784  # MNIST
latent_dim = 32  # compressed
model = Autoencoder(input_dim, latent_dim)
# The encoder output shape: (batch, 32)

🧮 Regularized Autoencoders

🎯 Preventing Trivial Identity

Regularized autoencoders add constraints to the loss function (beyond reconstruction) to learn useful representations. Examples: sparse, denoising, contractive, and variational autoencoders.

Why Regularize?

Without regularization, autoencoders can learn to copy inputs exactly (if the bottleneck is large enough). Regularization encourages the model to learn robust, meaningful features.

🧮 L1 Regularization🧹 Denoising🔒 Contractive

# Sparse autoencoder with L1 penalty
def sparse_loss(x, x_hat, z, lambda_sparse=0.001):
    recon_loss = F.mse_loss(x_hat, x)
    sparse_penalty = lambda_sparse * torch.mean(torch.abs(z))
    return recon_loss + sparse_penalty

🎲 Variational Autoencoders (VAEs)

🎯 Probabilistic Latent Space

VAEs (Kingma & Welling, 2014) learn a distribution over the latent space (typically Gaussian). They combine reconstruction loss with KL divergence to make the latent space continuous and smooth.

Generative Power

The probabilistic latent space allows sampling — you can generate new data by sampling from the prior distribution and decoding. VAEs are powerful generative models.

🎲 Probabilistic📊 KL Divergence🎨 Generative

class VAE(nn.Module):
    def __init__(self, input_dim, latent_dim):
        super().__init__()
        self.encoder = nn.Linear(input_dim, 128)
        self.mu = nn.Linear(128, latent_dim)
        self.logvar = nn.Linear(128, latent_dim)
        self.decoder = nn.Sequential(
            nn.Linear(latent_dim, 128),
            nn.ReLU(),
            nn.Linear(128, input_dim)
        )

    def reparameterize(self, mu, logvar):
        std = torch.exp(0.5 * logvar)
        eps = torch.randn_like(std)
        return mu + eps * std

    def forward(self, x):
        h = F.relu(self.encoder(x))
        mu = self.mu(h)
        logvar = self.logvar(h)
        z = self.reparameterize(mu, logvar)
        return self.decoder(z), mu, logvar

📐 KL Divergence & Reparameterization

🎯 The VAE Core

KL divergence: D_KL(q(z|x) || p(z)) = -½Σ(1 + log(σ²) - μ² - σ²). It forces the latent distribution to be close to the prior (standard normal). Reparameterization enables backpropagation through stochastic sampling.

Why It Works

Reparameterization rewrites z = μ + σ·ε, where ε ~ N(0,1). This makes the sampling operation differentiable, allowing gradient-based optimization.

📐 KL Divergence🔄 Reparameterization📊 Differentiable

def vae_loss(x, x_hat, mu, logvar, beta=1.0):
    recon_loss = F.binary_cross_entropy(x_hat, x, reduction='sum')
    kl_loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return (recon_loss + beta * kl_loss) / x.size(0)

🖼️ Convolutional Autoencoders

🎯 For Image Data

Use convolutional layers for the encoder and transposed convolutions for the decoder. They preserve spatial structure and are more parameter-efficient for images.

Architecture Details

Encoder: Conv2D → ReLU → MaxPool. Decoder: ConvTranspose2D → ReLU → Upsample. Use batch normalization for stability.

🖼️ Conv2D📈 Transposed Conv🏗️ Spatial Structure

class ConvAutoencoder(nn.Module):
    def __init__(self):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(16, 32, 3, stride=2, padding=1),
            nn.ReLU(),
            nn.Conv2d(32, 64, 3, stride=2, padding=1),
            nn.ReLU()
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(64, 32, 3, stride=2, padding=1, output_padding=1),
            nn.ReLU(),
            nn.ConvTranspose2d(32, 16, 3, stride=2, padding=1, output_padding=1),
            nn.ReLU(),
            nn.ConvTranspose2d(16, 1, 3, stride=2, padding=1, output_padding=1),
            nn.Sigmoid()
        )

🧹 Denoising Autoencoders

🎯 Robust Reconstruction

Denoising autoencoders (Vincent et al., 2008) are trained to reconstruct clean data from corrupted (noisy) input. They learn to remove noise and recover the original signal.

Training Process

Corrupt input with noise (Gaussian, mask, or salt-and-pepper). The model learns to denoise, forcing it to capture meaningful structure rather than memorizing.

🧹 Noise Removal🔍 Robust Features📊 Image Denoising

# Add Gaussian noise to input
def add_noise(x, noise_factor=0.3):
    noise = torch.randn_like(x) * noise_factor
    return x + noise

# During training: x_noisy = add_noise(x); x_hat = model(x_noisy)
# Loss: F.mse_loss(x_hat, x)  # Compare to clean input

🌿 Sparse Autoencoders

🎯 Activation Sparsity

Sparse autoencoders add a sparsity constraint on the latent representation (e.g., L1 penalty or KL divergence on average activations). This encourages the model to activate only a few neurons per sample.

Feature Discovery

Sparsity promotes the learning of interpretable, localized features. It is often used in neuroscience and for feature visualization.

🌿 L1 Penalty🧠 Sparse Features🔍 Interpretable

def sparse_loss(x, x_hat, z, rho=0.05, beta=3.0):
    recon_loss = F.mse_loss(x_hat, x)
    rho_hat = torch.mean(z, dim=0)
    kl_div = torch.sum(rho * torch.log(rho / (rho_hat + 1e-8)) +
                       (1 - rho) * torch.log((1 - rho) / (1 - rho_hat + 1e-8)))
    return recon_loss + beta * kl_div

🔒 Contractive Autoencoders

🎯 Robustness to Input Perturbations

Contractive autoencoders (CAEs) add a penalty on the Frobenius norm of the Jacobian of the encoder. This makes the representation insensitive to small input changes.

Mathematical Formulation

Loss = reconstruction + λ * ||J_f(x)||²_F, where J_f is the Jacobian of the encoder. This encourages the latent representation to be smooth and robust.

🔒 Jacobian Penalty📊 Smooth Representation🧹 Robust

def contractive_loss(x, x_hat, z, lambda_=0.01):
    recon_loss = F.mse_loss(x_hat, x)
    # Compute Jacobian (approximate via finite differences or automatic differentiation)
    # Simplified: we use L2 regularization of the latent space
    contractive_penalty = lambda_ * torch.mean(torch.norm(z, dim=1) ** 2)
    return recon_loss + contractive_penalty

🔍 Anomaly Detection

🎯 Identifying Outliers

Autoencoders are excellent for anomaly detection. They learn to reconstruct normal data well. Anomalies have high reconstruction error, serving as a scoring mechanism.

Use Cases

Fraud detection, network intrusion, medical imaging, manufacturing quality control. The reconstruction error threshold is set on the training data.

📊 Reconstruction Error🎯 Thresholding🔍 Outlier Detection

def anomaly_score(x, model):
    x_hat, _ = model(x)
    # MSE per sample
    error = torch.mean((x - x_hat) ** 2, dim=1)
    return error

# Set threshold from training data
threshold = torch.quantile(anomaly_score(train_data, model), 0.95)
# Classify: anomaly if score > threshold

🎨 Generative Applications

🎯 Creating New Data

VAEs and their variants (β-VAE, VQ-VAE) are used for image generation, interpolation between samples, and latent space manipulation. They enable controlled generation of data.

Applications

Face generation, style transfer, text-to-image, drug discovery, music generation. The smooth latent space allows meaningful interpolation and manipulation.

🎨 Image Generation🔀 Interpolation🧪 Drug Discovery

# Generate new samples from VAE
with torch.no_grad():
    z = torch.randn(16, latent_dim)  # Sample from prior
    generated = model.decoder(z)
    # generated contains new images

🚀 Deployment & Production

🎯 From Model to Service

Export trained autoencoder (PyTorch JIT, ONNX). Deploy for inference: anomaly scoring API, feature extraction service, or generative model API.

MLOps Considerations

Monitor reconstruction error distribution over time (drift). Version models and data. Use feature stores for extracted representations.

📦 ONNX🌐 FastAPI📊 Drift Monitoring

# Export to ONNX
dummy_input = torch.randn(1, input_dim)
torch.onnx.export(model, dummy_input, "autoencoder.onnx",
                  input_names=['input'], output_names=['output', 'latent'])

# FastAPI anomaly detection endpoint
@app.post("/anomaly")
def detect_anomaly(data: List[float]):
    x = torch.tensor(data).unsqueeze(0)
    with torch.no_grad():
        error = anomaly_score(x, model)
        is_anomaly = error > threshold
    return {"anomaly": bool(is_anomaly), "score": error.item()}