๐ค Transformers: Foundations
๐ฏ The Attention Revolution
Transformers (Vaswani et al., 2017, "Attention Is All You Need") revolutionized NLP by replacing recurrence with self-attention. The architecture uses a sequence-to-sequence model with an encoder and decoder, both built from attention mechanisms. It is parallelizable and captures long-range dependencies.
Deep Explanation
Key innovation: Self-attention computes weighted sums of all positions, capturing relationships regardless of distance. This eliminates the need for RNNs/LSTMs, enabling parallel training and better handling of long sequences.
Why it matters: Transformers power most state-of-the-art models today: BERT, GPT, T5, and all major multimodal models. They are the foundation of modern AI.
import torch
import torch.nn as nn
import torch.nn.functional as F
# Minimal Transformer block concept
class SelfAttention(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.mha = nn.MultiheadAttention(d_model, n_heads)
self.norm = nn.LayerNorm(d_model)
def forward(self, x):
attn_out, _ = self.mha(x, x, x)
return self.norm(x + attn_out)๐๏ธ Scaled Dot-Product Attention
๐ฏ The Core Formula
Attention(Q, K, V) = softmax(QKแต / โdโ) V. Q (queries), K (keys), V (values). Scaling by โdโ prevents softmax from entering regions with extremely small gradients.
Intuition
Each query compares to all keys, producing attention weights. These weights are used to aggregate values. The result is a context-aware representation for each position.
def scaled_dot_product_attention(Q, K, V, mask=None):
d_k = Q.size(-1)
scores = torch.matmul(Q, K.transpose(-2, -1)) / math.sqrt(d_k)
if mask is not None:
scores = scores.masked_fill(mask == 0, -1e9)
attn_weights = F.softmax(scores, dim=-1)
return torch.matmul(attn_weights, V)๐งฉ Multi-Head Attention
๐ฏ Multiple Perspectives
Run scaled dot-product attention in parallel with multiple projection matrices. Each head learns different relationship patterns.
Why Multiple Heads?
Each head can attend to different positions and represent different types of relationships (syntactic, semantic). The outputs are concatenated and projected.
class MultiHeadAttention(nn.Module):
def __init__(self, d_model, n_heads):
super().__init__()
self.n_heads = n_heads
self.d_k = d_model // n_heads
self.W_q = nn.Linear(d_model, d_model)
self.W_k = nn.Linear(d_model, d_model)
self.W_v = nn.Linear(d_model, d_model)
self.W_o = nn.Linear(d_model, d_model)
def forward(self, x):
# Implementation details: split into heads, apply attention, concat
pass๐ Positional Encoding
๐ฏ Injecting Order Information
Since attention is permutation-invariant, positional encodings add information about the position of each token. Uses sine/cosine functions of different frequencies.
Mathematical Formulation
PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d)). This allows the model to attend to relative positions.
class PositionalEncoding(nn.Module):
def __init__(self, d_model, max_len=5000):
super().__init__()
pe = torch.zeros(max_len, d_model)
pos = torch.arange(0, max_len).unsqueeze(1).float()
div_term = torch.exp(torch.arange(0, d_model, 2).float() * (-math.log(10000.0) / d_model))
pe[:, 0::2] = torch.sin(pos * div_term)
pe[:, 1::2] = torch.cos(pos * div_term)
self.register_buffer('pe', pe)
def forward(self, x):
return x + self.pe[:x.size(1)]๐๏ธ Encoder Architecture
๐ฏ Encoding the Input
The encoder consists of N identical layers, each with multi-head self-attention and a position-wise feed-forward network, with residual connections and layer normalization.
Processing Flow
Input โ Positional Encoding โ [Self-Attention โ Add & Norm โ FFN โ Add & Norm] ร N. The encoder produces contextualized representations.
class EncoderLayer(nn.Module):
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.self_attn = MultiHeadAttention(d_model, n_heads)
self.ffn = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.ReLU(),
nn.Linear(d_ff, d_model)
)
self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)
def forward(self, x):
x = self.norm1(x + self.self_attn(x))
x = self.norm2(x + self.ffn(x))
return x๐๏ธ Decoder Architecture
๐ฏ Generating the Output
The decoder has two attention layers: masked self-attention (to prevent looking ahead) and cross-attention (to attend to encoder outputs).
Autoregressive Generation
Masked attention ensures each position only attends to previous positions. Cross-attention uses encoder outputs as keys and values.
class DecoderLayer(nn.Module):
def __init__(self, d_model, n_heads, d_ff):
super().__init__()
self.masked_attn = MultiHeadAttention(d_model, n_heads)
self.cross_attn = MultiHeadAttention(d_model, n_heads)
self.ffn = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.ReLU(),
nn.Linear(d_ff, d_model)
)
self.norm1 = nn.LayerNorm(d_model)
self.norm2 = nn.LayerNorm(d_model)
self.norm3 = nn.LayerNorm(d_model)
def forward(self, x, encoder_out):
x = self.norm1(x + self.masked_attn(x, mask))
x = self.norm2(x + self.cross_attn(x, encoder_out, encoder_out))
x = self.norm3(x + self.ffn(x))
return xโ๏ธ Feed-Forward Networks
๐ฏ Position-Wise Transformation
FFN(x) = max(0, xWโ + bโ)Wโ + bโ. Applied independently to each position. Typically expands the dimension (d_model โ d_ff โ d_model).
Role in the Model
The FFN adds non-linearity and transforms the attention outputs into a richer representation. It is essentially a two-layer MLP with ReLU.
class PositionwiseFFN(nn.Module):
def __init__(self, d_model, d_ff, dropout=0.1):
super().__init__()
self.linear1 = nn.Linear(d_model, d_ff)
self.linear2 = nn.Linear(d_ff, d_model)
self.dropout = nn.Dropout(dropout)
def forward(self, x):
return self.linear2(self.dropout(F.relu(self.linear1(x))))๐งฎ Layer Normalization & Residuals
๐ฏ Stabilizing Training
Layer normalization normalizes the activations across features. Residual connections (Add) allow gradients to flow directly, enabling deep networks.
Pre-Norm vs Post-Norm
Modern transformers typically use pre-norm (normalize before attention/FFN) for better training stability.
# Pre-norm transformer block (modern)
class PreNormBlock(nn.Module):
def __init__(self, attn, ff, norm):
super().__init__()
self.attn = attn
self.ff = ff
self.norm = norm
def forward(self, x):
x = x + self.attn(self.norm(x))
x = x + self.ff(self.norm(x))
return x๐ง BERT: Bidirectional Context
๐ฏ Pre-training + Fine-tuning
BERT (Devlin et al., 2018) uses a transformer encoder with masked language modeling and next-sentence prediction. It learns deep bidirectional representations.
MLM & NSP
Masked LM: 15% of tokens are masked, the model predicts them. NSP: binary classification of whether two sentences are consecutive.
from transformers import BertModel, BertTokenizer
model = BertModel.from_pretrained('bert-base-uncased')
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
inputs = tokenizer("Hello world", return_tensors="pt")
outputs = model(**inputs)
print(outputs.last_hidden_state.shape) # (1, seq_len, 768)๐ฌ GPT: Autoregressive Generation
๐ฏ Left-to-Right Language Modeling
GPT (Radford et al.) uses a decoder-only architecture with causal (masked) self-attention. It is pre-trained on next-word prediction and can generate coherent text.
Zero-Shot Capabilities
GPT models exhibit impressive few-shot and zero-shot learning abilities, especially at larger scales (GPT-3, GPT-4).
from transformers import GPT2LMHeadModel, GPT2Tokenizer
model = GPT2LMHeadModel.from_pretrained('gpt2')
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0]))๐ T5: Text-to-Text Framework
๐ฏ Unified Text Processing
T5 (Raffel et al., 2019) frames all NLP tasks as text-to-text problems. It uses a standard encoder-decoder architecture with a unified training objective.
Prefix-LM
Each task is converted to a text prefix (e.g., "translate English to German: ..."), and the model generates the output text.
from transformers import T5ForConditionalGeneration, T5Tokenizer
model = T5ForConditionalGeneration.from_pretrained('t5-small')
tokenizer = T5Tokenizer.from_pretrained('t5-small')
inputs = tokenizer("translate English to German: Hello world", return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0]))๐๏ธ Vision Transformers (ViT)
๐ฏ Transformers for Images
ViT (Dosovitskiy et al., 2020) splits images into patches, linearly embeds them, and applies a standard transformer encoder. It achieves competitive results with CNNs.
Patch Embedding
An image of size HรWรC is divided into patches (e.g., 16ร16). Each patch is flattened and projected to a vector. Positional encodings are added.
from transformers import ViTForImageClassification, ViTImageProcessor
model = ViTForImageClassification.from_pretrained('google/vit-base-patch16-224')
processor = ViTImageProcessor.from_pretrained('google/vit-base-patch16-224')
# Preprocess image and predictโก Efficiency: FlashAttention, Sparse
๐ฏ Faster & Memory-Efficient Attention
FlashAttention (Dao et al., 2022) dramatically speeds up attention by leveraging GPU memory hierarchy. Sparse attention reduces complexity from O(nยฒ) to O(n log n).
Key Techniques
FlashAttention uses tiling to avoid materializing the full attention matrix. Sparse transformers use local, strided, or block-sparse attention patterns.
# Using FlashAttention (requires specific GPU)
# from flash_attn import flash_attn_func
# attn_out = flash_attn_func(q, k, v, dropout_p=0.0, softmax_scale=None, causal=False)
# Longformer (sparse attention)
from transformers import LongformerModel
model = LongformerModel.from_pretrained('allenai/longformer-base-4096')๐ Deployment & Production
๐ฏ From Model to Service
Export to ONNX, TensorRT, or use Hugging Face's Inference API. Deploy with FastAPI, Triton, or Hugging Face TGI (Text Generation Inference).
Optimization Techniques
Quantization (INT8, FP16), model distillation, and caching of key-value pairs for autoregressive generation.
# Hugging Face TGI server
# docker run --gpus all -p 8080:80 ghcr.io/huggingface/text-generation-inference \
# --model-id meta-llama/Llama-2-7b-chat-hf
# FastAPI example
from fastapi import FastAPI
app = FastAPI()
@app.post("/generate")
def generate(prompt: str):
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=100)
return {"text": tokenizer.decode(outputs[0])}