🧠 Segmentation Fundamentals
🎯 What is Segmentation?
Segmentation is the task of partitioning an image into meaningful regions at the pixel level. Unlike classification (one label per image) or detection (bounding boxes), segmentation assigns a label to every single pixel. It's essential for medical imaging, autonomous driving, AR/VR, and photo editing.
Why Segmentation Matters
Pixel-precision: Exact boundaries of objects. Dense prediction: Every pixel gets a label. Foundation: Basis for medical AI, self-driving. Versatility: Semantic, instance, panoptic variants. Modern AI: Powers SAM, DINOv2, and more. Real-world: From MRI scans to AR overlays.
📌 Pixel-level📌 Dense prediction📌 Boundaries📌 Foundation AI
# Segmentation — the big picture
import torch
import torch.nn as nn
import torch.nn.functional as F
class SegmentationModel(nn.Module):
"""Base segmentation model."""
def __init__(self, num_classes):
super().__init__()
# Encoder (feature extraction)
self.encoder = self.build_encoder()
# Decoder (upsampling to full resolution)
self.decoder = self.build_decoder()
# Classification head (per-pixel)
self.classifier = nn.Conv2d(64, num_classes, 1)
def build_encoder(self):
# Typically a pretrained CNN (ResNet, EfficientNet)
return nn.Sequential(
nn.Conv2d(3, 64, 3, padding=1),
nn.ReLU(),
nn.MaxPool2d(2)
)
def build_decoder(self):
# Upsampling + skip connections
return nn.Sequential(
nn.Upsample(scale_factor=2, mode='bilinear'),
nn.Conv2d(64, 64, 3, padding=1),
nn.ReLU()
)
def forward(self, x):
# Encoder
features = self.encoder(x)
# Decoder
upsampled = self.decoder(features)
# Per-pixel classification
logits = self.classifier(upsampled)
return logits # (B, num_classes, H, W)
# COMPARISON WITH OTHER TASKS
print("=== Task Comparison ===")
print("""
Classification:
Input: Image (H, W, 3)
Output: Class label (scalar)
Example: "This is a cat"
Detection:
Input: Image (H, W, 3)
Output: Bounding boxes + labels
Example: "Cat at (10, 20, 100, 150)"
Semantic Segmentation:
Input: Image (H, W, 3)
Output: Per-pixel label (H, W)
Example: "Pixel (i,j) is 'cat'"
Instance Segmentation:
Input: Image (H, W, 3)
Output: Per-pixel label + instance ID
Example: "Pixel (i,j) is cat #1"
Panoptic Segmentation:
Input: Image (H, W, 3)
Output: Semantic + instance labels
Example: "Pixel (i,j) is cat #1" and "Pixel (k,l) is 'road'"
""")
# REAL-WORLD IMPORTANCE
print("\n=== Real-World Importance ===")
print("""
Medical Imaging:
- Tumor detection in MRI/CT
- Organ segmentation
- Cell segmentation in microscopy
- Retinal vessel extraction
- Cancer diagnosis
Autonomous Driving:
- Road segmentation
- Pedestrian segmentation
- Vehicle segmentation
- Traffic sign segmentation
- Lane detection
AR/VR:
- Background removal
- Human segmentation (portrait mode)
- Real-time occlusion
- Hand/face segmentation
- Scene understanding
Satellite Imagery:
- Land cover classification
- Building detection
- Crop monitoring
- Forest mapping
- Disaster assessment
Robotics:
- Object manipulation
- Scene understanding
- Grasp planning
- Navigation
Photo Editing:
- Background removal
- Portrait mode
- Object selection
- Sky replacement
- Creative effects
""")📊 Types of Segmentation
🎯 Three Main Variants
Segmentation comes in three main flavors: semantic (class per pixel), instance (object per pixel), and panoptic (both). Each serves different applications.
Comparing Segmentation Types
Semantic: "This pixel is a car." Instance: "This pixel belongs to car #3." Panoptic: Both — "This pixel is part of car #3 (a thing), this pixel is sky (stuff)." Use cases: Semantic for scenes, instance for counting, panoptic for complete understanding.
🎨 Semantic🔢 Instance🌐 Panoptic📊 Compare
# Comparing segmentation types
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
# 1. SEMANTIC SEGMENTATION
# Each pixel → class label
# All cars are same class, no distinction between them
class SemanticSegmentation(nn.Module):
def __init__(self, num_classes):
super().__init__()
self.num_classes = num_classes
# Output: (B, num_classes, H, W)
self.classifier = nn.Conv2d(256, num_classes, 1)
def forward(self, features):
return self.classifier(features)
print("=== Semantic Segmentation ===")
print("""
Output: (H, W) label map
Each pixel gets ONE class label
All instances of same class are indistinguishable
Example output for a scene with 2 cars:
0 0 0 0 0 0 0 0 (0 = background)
0 1 1 0 1 1 0 0 (1 = car — both cars same label!)
0 1 1 0 1 1 0 0
0 0 0 0 0 0 0 0
Classes: {0: background, 1: car, 2: person, 3: road, ...}
Use cases:
- Scene understanding
- Road/pedestrian detection
- Medical imaging (organs, tumors)
- Satellite imagery (land cover)
Loss: Cross-entropy per pixel
Architectures: FCN, U-Net, DeepLab, SegFormer
""")
# 2. INSTANCE SEGMENTATION
# Each pixel → class label + instance ID
# Two cars are distinguished
print("\n=== Instance Segmentation ===")
print("""
Output: (H, W) label map with instance IDs
Each pixel gets class + instance ID
Different instances of same class are distinguished
Example output for a scene with 2 cars:
0 0 0 0 0 0 0 0 (0 = background)
0 1 1 0 2 2 0 0 (1 = car #1, 2 = car #2)
0 1 1 0 2 2 0 0
0 0 0 0 0 0 0 0
Classes: {0: bg, 1: car_1, 2: car_2, 3: person_1, ...}
Use cases:
- Object counting
- Robotics (individual objects)
- Medical (individual cells)
- Retail (product instances)
Loss: Mask R-CNN loss (classification + box + mask)
Architectures: Mask R-CNN, YOLACT, SOLOv2
""")
# 3. PANOPTIC SEGMENTATION
# Combines semantic + instance
print("\n=== Panoptic Segmentation ===")
print("""
Output: (H, W) with both stuff and things
- "Things" = countable objects (cars, people) → instance
- "Stuff" = amorphous regions (sky, road) → semantic
Example output:
0 0 0 0 0 0 0 0 (0 = sky)
0 1 1 0 2 2 0 0 (1 = car #1, 2 = car #2)
0 1 1 0 2 2 0 0
3 3 3 3 3 3 3 3 (3 = road)
Segment IDs:
0: sky (stuff)
1: car #1 (thing)
2: car #2 (thing)
3: road (stuff)
...
Use cases:
- Complete scene understanding
- Autonomous driving (full scene)
- Robotics
- City planning
Loss: Combined semantic + instance losses
Architectures: Panoptic FPN, UPSNet, Mask2Former
""")
# VISUALIZATION
def visualize_comparison():
"""Show difference between types."""
print("\n=== Visual Comparison ===")
print("""
Same image, different outputs:
Input image: Street scene with 2 cars, 1 person, road, buildings
Semantic Segmentation:
Label map: [0=bg, 1=car, 2=person, 3=road, 4=building]
All cars labeled "1", all persons labeled "2"
Instance Segmentation:
Label map: [car_1, car_2, person_1, road, building]
Each car/person has unique ID
Panoptic Segmentation:
Complete: cars (instances), person (instance),
road (stuff), building (stuff), sky (stuff)
Every pixel has semantic + instance (for things)
Comparison table:
Task Per-Pixel Instance ID Things Stuff
Semantic ✓ ✗ Partial ✓
Instance ✓ ✓ ✓ ✗
Panoptic ✓ ✓ ✓ ✓
""")
visualize_comparison()
# 4. COMPARISON TABLE
print("\n=== Comparison Table ===")
print(f"{'Aspect':<25} {'Semantic':<15} {'Instance':<15} {'Panoptic':<15}")
print("-" * 70)
print(f"{'Per-pixel labels':<25} {'✓':<15} {'✓':<15} {'✓':<15}")
print(f"{'Instance separation':<25} {'✗':<15} {'✓':<15} {'✓':<15}")
print(f"{'Stuff classes':<25} {'✓':<15} {'✗':<15} {'✓':<15}")
print(f"{'Thing classes':<25} {'Partial':<15} {'✓':<15} {'✓':<15}")
print(f"{'Output format':<25} {'(H,W)':<15} {'(N,H,W)':<15} {'(H,W)':<15}")
print(f"{'Training complexity':<25} {'Medium':<15} {'High':<15} {'Very High':<15}")
print(f"{'Inference speed':<25} {'Fast':<15} {'Medium':<15} {'Slower':<15}")
# 5. IMPLEMENTATION
class UnifiedSegmentationModel(nn.Module):
"""Base model for all segmentation types."""
def __init__(self, backbone='resnet50', num_classes=20, task='semantic'):
super().__init__()
self.task = task
self.num_classes = num_classes
# Shared backbone
from torchvision import models
self.backbone = models.resnet50(weights='IMAGENET1K_V2')
self.backbone = nn.Sequential(*list(self.backbone.children())[:-2])
# Task-specific heads
if task == 'semantic':
self.head = nn.Sequential(
nn.Conv2d(2048, 512, 3, padding=1),
nn.ReLU(),
nn.Conv2d(512, num_classes, 1)
)
elif task == 'instance':
# Similar to Mask R-CNN head
self.head = nn.Sequential(
nn.Conv2d(2048, 512, 3, padding=1),
nn.ReLU(),
nn.Conv2d(512, num_classes, 1)
)
elif task == 'panoptic':
# Semantic + instance branches
self.semantic_head = nn.Conv2d(2048, num_classes, 1)
self.instance_head = nn.Conv2d(2048, num_classes, 1)
def forward(self, x):
features = self.backbone(x)
if self.task == 'panoptic':
sem = self.semantic_head(features)
inst = self.instance_head(features)
return sem, inst
else:
return self.head(features)
# 6. CHOOSING THE RIGHT TYPE
print("\n=== Choosing the Right Type ===")
print("""
Use SEMANTIC when:
✓ Need region-level understanding
✓ Don't care about individual instances
✓ Examples: land cover, organs, road/pedestrian
✓ Simpler to train
✓ Faster inference
Use INSTANCE when:
✓ Need to count objects
✓ Need to track individual instances
✓ Examples: cell counting, product detection
✓ More complex
✓ Handles overlapping objects
Use PANOPTIC when:
✓ Need complete scene understanding
✓ Both stuff and things matter
✓ Examples: autonomous driving, robotics
✓ Most complex
✓ Richest output
Real-world choices:
- Medical imaging: Usually semantic
- Self-driving: Panoptic (full understanding)
- Retail: Instance (count products)
- AR/VR: Often semantic or instance
- Satellite: Semantic (land cover)
""")🎨 Semantic Segmentation
🎯 Classify Every Pixel
Semantic segmentation assigns a class label to every pixel in an image. It's the simplest form of dense prediction and forms the foundation for more complex segmentation tasks.
Core Concepts
Input: Image (H, W, 3). Output: Label map (H, W). Loss: Cross-entropy per pixel. Metric: mIoU (mean Intersection over Union). Challenge: Preserve spatial resolution. Solution: Encoder-decoder with skip connections.
🎯 Per-pixel📉 Cross-entropy📊 mIoU🔗 Skip connections
# Semantic segmentation — complete implementation
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import models
# 1. SIMPLE SEMANTIC SEGMENTATION MODEL
class SimpleSegNet(nn.Module):
"""Simple encoder-decoder for semantic segmentation."""
def __init__(self, num_classes=21):
super().__init__()
# Encoder: pretrained ResNet
resnet = models.resnet34(weights='IMAGENET1K_V1')
self.encoder = nn.Sequential(
resnet.conv1, # (B, 64, H/2, W/2)
resnet.bn1,
resnet.relu,
resnet.maxpool, # (B, 64, H/4, W/4)
resnet.layer1, # (B, 64, H/4, W/4)
resnet.layer2, # (B, 128, H/8, W/8)
resnet.layer3, # (B, 256, H/16, W/16)
resnet.layer4, # (B, 512, H/32, W/32)
)
# Decoder: upsampling to full resolution
self.decoder = nn.Sequential(
nn.ConvTranspose2d(512, 256, 4, 2, 1), # H/16
nn.ReLU(),
nn.ConvTranspose2d(256, 128, 4, 2, 1), # H/8
nn.ReLU(),
nn.ConvTranspose2d(128, 64, 4, 2, 1), # H/4
nn.ReLU(),
nn.ConvTranspose2d(64, 32, 4, 2, 1), # H/2
nn.ReLU(),
nn.ConvTranspose2d(32, 16, 4, 2, 1), # H
nn.ReLU(),
)
# Classification head
self.classifier = nn.Conv2d(16, num_classes, 1)
def forward(self, x):
# Encoder
features = self.encoder(x)
# Decoder
upsampled = self.decoder(features)
# Classification
logits = self.classifier(upsampled)
return logits
# 2. TRAINING
def train_semantic_seg(model, train_loader, val_loader,
epochs=20, lr=1e-3, num_classes=21):
"""Train semantic segmentation model."""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
criterion = nn.CrossEntropyLoss(ignore_index=255) # Ignore unlabeled
for epoch in range(epochs):
# Training
model.train()
train_loss = 0
for images, masks in train_loader:
images = images.to(device)
masks = masks.to(device).long()
optimizer.zero_grad()
logits = model(images) # (B, num_classes, H, W)
loss = criterion(logits, masks)
loss.backward()
optimizer.step()
train_loss += loss.item()
# Validation
model.eval()
val_loss = 0
with torch.no_grad():
for images, masks in val_loader:
images = images.to(device)
masks = masks.to(device).long()
logits = model(images)
val_loss += criterion(logits, masks).item()
print(f"Epoch {epoch+1}: "
f"Train Loss={train_loss/len(train_loader):.4f}, "
f"Val Loss={val_loss/len(val_loader):.4f}")
# 3. INFERENCE
def predict_segmentation(model, image, device='cuda'):
"""Run inference on a single image."""
model.eval()
with torch.no_grad():
# Add batch dimension
if image.dim() == 3:
image = image.unsqueeze(0)
image = image.to(device)
# Forward pass
logits = model(image) # (1, num_classes, H, W)
# Get predicted class per pixel
prediction = logits.argmax(dim=1) # (1, H, W)
# Convert to numpy
return prediction.squeeze(0).cpu().numpy()
# 4. POST-PROCESSING
def visualize_segmentation(mask, colormap):
"""Convert label map to RGB image."""
h, w = mask.shape
rgb = np.zeros((h, w, 3), dtype=np.uint8)
for class_id, color in enumerate(colormap):
rgb[mask == class_id] = color
return rgb
# 5. COMMON ARCHITECTURES
print("=== Semantic Segmentation Architectures ===")
print("""
FCN (Fully Convolutional Network):
- First end-to-end segmentation network
- Replaces FC layers with conv layers
- Skip connections for finer details
- Year: 2015
U-Net:
- Encoder-decoder with skip connections
- Symmetric architecture
- Great for medical imaging
- Year: 2015
SegNet:
- Encoder-decoder with pooling indices
- Memory efficient
- Year: 2017
DeepLab (v1-v3+):
- Atrous (dilated) convolutions
- ASPP (Atrous Spatial Pyramid Pooling)
- State-of-the-art accuracy
- Years: 2016-2018
PSPNet:
- Pyramid Pooling Module
- Multi-scale context
- Year: 2017
SegFormer:
- Transformer-based
- Hierarchical encoder
- Simple MLP decoder
- Year: 2021
Mask2Former:
- Universal segmentation
- Masked attention
- SOTA on all tasks
- Year: 2022
SAM (Segment Anything):
- Foundation model
- Promptable segmentation
- Zero-shot
- Year: 2023
""")
# 6. DATASETS
print("\n=== Common Datasets ===")
print("""
PASCAL VOC:
- 21 classes (20 + background)
- 10,582 images
- Standard benchmark
- mIoU metric
Cityscapes:
- 30 classes (19 used for eval)
- 5,000 fine + 20,000 coarse
- Urban street scenes
- Autonomous driving
ADE20K:
- 150 classes
- 25,000 images
- Diverse scenes
- Indoor + outdoor
COCO-Stuff:
- 171 classes (80 things + 91 stuff)
- 164,000 images
- Complex scenes
- Panoptic annotations
Medical:
- ISIC (skin lesions)
- KiTS (kidney)
- BraTS (brain tumors)
- LiTS (liver)
Satellite:
- SpaceNet
- DeepGlobe
- LoveDA
""")
# 7. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. START WITH PRETRAINED ENCODER
- ImageNet weights
- Faster convergence
- Better accuracy
2. USE SKIP CONNECTIONS
- Preserve spatial details
- Combine low + high level features
3. HANDLE CLASS IMBALANCE
- Background dominates
- Use weighted loss
- Or Dice loss
4. DATA AUGMENTATION
- Random crop, flip, rotate
- Color jitter
- Random scale
5. USE DILATED CONVOLUTIONS
- Larger receptive field
- No resolution loss
- DeepLab's trick
6. BATCH SIZE
- Small if memory limited
- Use gradient accumulation
- Or crop patches
7. POST-PROCESSING
- CRF (Conditional Random Field)
- Refine boundaries
- Optional but helps
8. EVALUATION
- mIoU per class
- Pixel accuracy
- Confusion matrix
""")🔢 Instance Segmentation
🎯 Separate Every Object
Instance segmentation goes beyond semantic segmentation by distinguishing between individual objects. Each car, person, or animal gets its own unique mask.
Core Concepts
Detection + Segmentation: Two-stage approach. Output: N masks + N classes. Top-down: Detect then segment. Bottom-up: Segment then group. Metrics: Mask AP, mAP. Challenge: Overlapping objects.
🎭 Mask R-CNN📊 Mask AP🔢 Counting🎯 Instances
# Instance segmentation — Mask R-CNN style
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.models.detection import maskrcnn_resnet50_fpn
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
# 1. LOAD PRETRAINED MASK R-CNN
def get_instance_segmentation_model(num_classes):
"""Load pretrained Mask R-CNN and customize for num_classes."""
# Load pretrained
model = maskrcnn_resnet50_fpn(weights='COCO_V1')
# Replace classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
# Replace mask predictor
in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
hidden_layer = 256
model.roi_heads.mask_predictor = MaskRCNNPredictor(
in_features_mask, hidden_layer, num_classes
)
return model
class MaskRCNNPredictor(nn.Module):
"""Mask prediction head."""
def __init__(self, in_channels, dim_reduced, num_classes):
super().__init__()
self.conv5_mask = nn.ConvTranspose2d(in_channels, dim_reduced, 2, 2, 0)
self.mask_fcn_logits = nn.Conv2d(dim_reduced, num_classes, 1)
for name, param in self.named_parameters():
if "bias" in name:
nn.init.constant_(param, 0)
elif "weight" in name:
nn.init.kaiming_normal_(param, mode="fan_out",
nonlinearity="relu")
def forward(self, x):
x = F.relu(self.conv5_mask(x))
return self.mask_fcn_logits(x)
# 2. COMPLETE MASK R-CNN ARCHITECTURE
class MaskRCNN(nn.Module):
"""
Mask R-CNN Architecture:
1. Backbone (ResNet + FPN): Extract features
2. RPN (Region Proposal Network): Generate proposals
3. RoI Align: Extract features for each proposal
4. Box head: Classify + refine boxes
5. Mask head: Predict per-instance masks
"""
def __init__(self, num_classes=81):
super().__init__()
self.num_classes = num_classes
# Load pretrained backbone
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
self.backbone = resnet_fpn_backbone('resnet50', pretrained=True)
# We'll use torchvision's implementation directly
self.model = maskrcnn_resnet50_fpn(weights='COCO_V1')
# Customize heads
in_features = self.model.roi_heads.box_predictor.cls_score.in_features
self.model.roi_heads.box_predictor = FastRCNNPredictor(
in_features, num_classes
)
def forward(self, images, targets=None):
return self.model(images, targets)
# 3. TRAINING MASK R-CNN
def train_mask_rcnn(model, train_loader, num_epochs=10, lr=1e-4):
"""Train Mask R-CNN."""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
# Separate parameters for different learning rates
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=lr, momentum=0.9,
weight_decay=1e-4)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
step_size=3,
gamma=0.1)
for epoch in range(num_epochs):
model.train()
total_loss = 0
for images, targets in train_loader:
images = [img.to(device) for img in images]
targets = [{k: v.to(device) for k, v in t.items()}
for t in targets]
# Forward pass (returns dict of losses)
loss_dict = model(images, targets)
losses = sum(loss for loss in loss_dict.values())
# Backward
optimizer.zero_grad()
losses.backward()
optimizer.step()
total_loss += losses.item()
lr_scheduler.step()
print(f"Epoch {epoch+1}: Loss = {total_loss/len(train_loader):.4f}")
# 4. INFERENCE
def predict_instance_segmentation(model, image, threshold=0.5):
"""Run instance segmentation on an image."""
model.eval()
with torch.no_grad():
if image.dim() == 3:
image = image.unsqueeze(0)
image = image.to(device='cuda')
predictions = model(image)
# Get predictions
pred = predictions[0]
boxes = pred['boxes'].cpu().numpy()
labels = pred['labels'].cpu().numpy()
scores = pred['scores'].cpu().numpy()
masks = pred['masks'].cpu().numpy()
# Filter by threshold
keep = scores > threshold
return {
'boxes': boxes[keep],
'labels': labels[keep],
'scores': scores[keep],
'masks': masks[keep] # (N, 1, H, W)
}
# 5. POPULAR ARCHITECTURES
print("=== Instance Segmentation Architectures ===")
print("""
TWO-STAGE (detect then segment):
Mask R-CNN:
- Most popular
- Extends Faster R-CNN with mask head
- RoI Align (better than RoI Pool)
- Year: 2017
Cascade Mask R-CNN:
- Cascade of detectors
- Progressive refinement
- Higher accuracy
HTC (Hybrid Task Cascade):
- Interweaves detection and segmentation
- State-of-the-art on COCO
ONE-STAGE (direct prediction):
YOLACT:
- Real-time (30+ FPS)
- Prototype masks + coefficients
- Simple but effective
SOLOv2:
- Location-based
- No detection
- Fast and accurate
TensorMask:
- Dense sliding windows
- Alternative paradigm
QUERY-BASED:
DETR:
- Transformer-based
- Set prediction
- No NMS needed
Mask2Former:
- Universal architecture
- Works for all segmentation tasks
- State-of-the-art
""")
# 6. METRICS
print("\n=== Instance Segmentation Metrics ===")
print("""
Mask AP (Average Precision):
- Primary metric
- IoU threshold: 0.5, 0.75, 0.5:0.95
- Per-class AP averaged
AP@0.5:
- Lenient threshold
- Common for comparison
AP@0.75:
- Strict threshold
- Tests precision
AP@0.5:0.95:
- COCO standard
- Average over 10 thresholds
- Most comprehensive
Mask AP vs Box AP:
- Box AP: detection quality
- Mask AP: segmentation quality
- Mask AP < Box AP usually
Per-class analysis:
- Small objects: hardest
- Large objects: easier
- Crowded scenes: challenging
""")
# 7. DATASETS
print("\n=== Instance Segmentation Datasets ===")
print("""
COCO (Common Objects in Context):
- 80 object classes
- 118,000 training images
- 5,000 test images
- Standard benchmark
Cityscapes:
- 8 instance classes
- 5,000 images
- Urban driving
LVIS:
- 1,200+ classes
- 164,000 images
- Long-tail distribution
- Rare classes
Open Images:
- 600+ classes
- 1.9M images
- Google's dataset
Medical:
- Cell tracking
- Organ instances
- Tumor instances
Retail:
- Product detection
- Shelf monitoring
- Counting
""")
# 8. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. START WITH PRETRAINED
- COCO weights available
- Fine-tune for your classes
- Much faster convergence
2. USE RoI ALIGN (NOT RoIPool)
- No quantization error
- Better mask quality
- Standard in modern methods
3. HANDLE OVERLAPPING OBJECTS
- NMS (Non-Maximum Suppression)
- Soft-NMS (better for overlaps)
- Mask-based NMS
4. DATA AUGMENTATION
- Random crop, flip
- Color jitter
- Mosaic (YOLO)
- Copy-paste
5. BATCH SIZE
- Smaller than semantic seg
- Use gradient accumulation
- Or distribute training
6. INFERENCE OPTIMIZATION
- FP16/INT8 quantization
- TensorRT
- ONNX Runtime
- Batching
7. COMMON ISSUES
- Small objects missed
- Crowded scenes
- Class imbalance
- Slow inference
""")🌐 Panoptic Segmentation
🎯 Complete Scene Understanding
Panoptic segmentation unifies semantic and instance segmentation. It segments everything in the image — countable objects ("things") get instance IDs, while amorphous regions ("stuff") get semantic labels.
Things vs Stuff
Things: Countable objects (cars, people, animals). Stuff: Amorphous regions (sky, road, grass). Output: Every pixel labeled. Things: Get instance IDs. Stuff: Only semantic labels. Format: Single label map with segment IDs.
🚗 Things🌫️ Stuff📊 PQ metric🌐 Complete
# Panoptic segmentation
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.models.detection import maskrcnn_resnet50_fpn
# 1. PANOPTIC ARCHITECTURE
class PanopticFPN(nn.Module):
"""
Panoptic FPN: Semantic + Instance branches on shared FPN.
Architecture:
1. Backbone + FPN
2. Semantic branch: per-pixel classification (all classes)
3. Instance branch: Mask R-CNN
4. Fusion: merge semantic and instance predictions
"""
def __init__(self, num_classes_things, num_classes_stuff):
super().__init__()
self.num_things = num_classes_things
self.num_stuff = num_classes_stuff
self.num_total = num_classes_things + num_classes_stuff
# Shared backbone (ResNet + FPN)
backbone = maskrcnn_resnet50_fpn(weights='COCO_V1')
self.backbone = backbone.backbone
self.fpn = backbone.backbone.fpn if hasattr(backbone.backbone, 'fpn') else None
# Semantic branch (stuff + things, per-pixel)
self.semantic_head = nn.Sequential(
nn.Conv2d(256, 128, 3, padding=1),
nn.ReLU(),
nn.Conv2d(128, self.num_total, 1)
)
# Instance branch
self.instance_head = maskrcnn_resnet50_fpn(weights='COCO_V1')
def forward(self, images):
# 1. Extract features
features = self.backbone(images)
# 2. Semantic segmentation
if isinstance(features, dict):
# FPN output
fpn_features = list(features.values())[0]
else:
fpn_features = features
semantic_logits = self.semantic_head(fpn_features)
semantic_logits = F.interpolate(
semantic_logits,
size=images.shape[-2:],
mode='bilinear',
align_corners=False
)
# 3. Instance segmentation
instance_predictions = self.instance_head(images)
return {
'semantic': semantic_logits,
'instances': instance_predictions
}
# 2. PANOPTIC FUSION
def fuse_panoptic(semantic_logits, instance_predictions,
thing_classes, stuff_classes,
overlap_threshold=0.5):
"""
Fuse semantic and instance predictions into panoptic output.
Steps:
1. Start with semantic segmentation
2. Override thing regions with instance masks
3. Resolve overlaps
"""
# Get semantic predictions
semantic_pred = semantic_logits.argmax(dim=1) # (B, H, W)
# Instance predictions
pred = instance_predictions[0]
masks = pred['masks'] # (N, 1, H, W)
labels = pred['labels']
scores = pred['scores']
# Create panoptic map
H, W = semantic_pred.shape[1:]
panoptic = semantic_pred.clone()
# Assign unique IDs to instances
next_id = len(stuff_classes) + len(thing_classes)
for i, (mask, label, score) in enumerate(zip(masks, labels, scores)):
# Threshold mask
mask_binary = (mask[0] > 0.5)
# Override semantic prediction with instance
panoptic[0][mask_binary] = next_id
next_id += 1
return panoptic
# 3. PANOPTIC QUALITY (PQ) METRIC
def compute_pq(pred_segments, gt_segments, num_classes):
"""
Panoptic Quality = SQ × RQ
SQ: Segmentation Quality (IoU of matched segments)
RQ: Recognition Quality (F1 of matched segments)
"""
# Match segments by IoU > 0.5
tp, fp, fn = 0, 0, 0
iou_sum = 0
for cls in range(num_classes):
pred_ids = np.unique(pred_segments[pred_segments // 1000 == cls])
gt_ids = np.unique(gt_segments[gt_segments // 1000 == cls])
matched_gt = set()
matched_pred = set()
for pred_id in pred_ids:
pred_mask = (pred_segments == pred_id)
best_iou = 0
best_gt = None
for gt_id in gt_ids:
if gt_id in matched_gt:
continue
gt_mask = (gt_segments == gt_id)
iou = compute_iou(pred_mask, gt_mask)
if iou > best_iou:
best_iou = iou
best_gt = gt_id
if best_iou > 0.5:
tp += 1
iou_sum += best_iou
matched_gt.add(best_gt)
matched_pred.add(pred_id)
else:
fp += 1
fn += len(gt_ids) - len(matched_gt)
# Compute metrics
sq = iou_sum / tp if tp > 0 else 0
rq = tp / (tp + 0.5 * fp + 0.5 * fn) if (tp + fp + fn) > 0 else 0
pq = sq * rq
return pq, sq, rq
def compute_iou(mask1, mask2):
"""Intersection over Union."""
intersection = (mask1 & mask2).sum()
union = (mask1 | mask2).sum()
return intersection / union if union > 0 else 0
import numpy as np
# 4. POPULAR ARCHITECTURES
print("=== Panoptic Segmentation Architectures ===")
print("""
Panoptic FPN:
- Baseline method
- Shared FPN backbone
- Separate semantic + instance heads
- Simple fusion
- Year: 2019
UPSNet:
- Unified panoptic
- Deformable convolutions
- Better fusion
- Year: 2019
Panoptic-DeepLab:
- Bottom-up approach
- No instance proposals
- Semantic + instance center + offset
- Year: 2020
MaX-DeepLab:
- Mask transformer
- End-to-end
- Dual-path transformer
- Year: 2021
MaskFormer:
- Universal segmentation
- Mask classification
- SOTA on all tasks
- Year: 2021
Mask2Former:
- Improved MaskFormer
- Masked attention
- Multi-scale features
- Year: 2022
OneFormer:
- One model for all tasks
- Task-conditioned training
- Year: 2023
""")
# 5. DATASETS
print("\n=== Panoptic Datasets ===")
print("""
COCO Panoptic:
- 133 classes (80 things + 53 stuff)
- 118,000 train images
- Standard benchmark
- PQ metric
Cityscapes Panoptic:
- 19 classes (11 things + 8 stuff)
- 5,000 images
- Urban driving
- High quality
ADE20K Panoptic:
- 150 classes
- 25,000 images
- Diverse scenes
- Challenging
Mapillary Vistas:
- 124 classes
- 25,000 images
- Street scenes
- Global coverage
BDD100K Panoptic:
- 10,000 images
- Autonomous driving
- Diverse weather
""")
# 6. APPLICATIONS
print("\n=== Panoptic Applications ===")
print("""
Autonomous Driving:
- Complete scene understanding
- All objects + regions
- Safety critical
- Example: Tesla, Waymo
Robotics:
- Grasp planning
- Scene manipulation
- Human-robot interaction
- Navigation
AR/VR:
- Complete scene parsing
- Realistic occlusion
- Interactive experiences
- Mixed reality
Medical Imaging:
- Organ + lesion segmentation
- Cell instances + tissue
- Complete diagnosis
Satellite Imagery:
- Land + object understanding
- Urban planning
- Disaster response
Video Analysis:
- Tracking + segmentation
- Video understanding
- Content creation
""")
# 7. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. USE PRETRAINED MODELS
- COCO panoptic weights
- Fine-tune for your data
2. HANDLE OVERLAPS
- Instance masks override semantic
- Order matters
- Overlap threshold tuning
3. CLASS BALANCE
- Stuff usually dominates
- Weight losses appropriately
- Per-class metrics
4. EVALUATION
- PQ, SQ, RQ
- Per-class analysis
- Confusion matrix
5. INFERENCE SPEED
- Two-stage is slower
- Consider one-stage methods
- Optimize post-processing
6. TRAINING DATA
- Need both types of labels
- Or generate from separate datasets
- Careful annotation
7. FUSION STRATEGY
- Simple override works
- Or learned fusion
- Dataset-specific
""")🔗 FCN & Encoder-Decoder
🎯 End-to-End Segmentation Networks
FCN (Fully Convolutional Network) was the first end-to-end segmentation architecture. It replaced fully connected layers with convolutions, enabling arbitrary input sizes and pixel-level output.
The Encoder-Decoder Pattern
Encoder: Extract features, reduce resolution. Decoder: Upsample, restore resolution. Skip connections: Combine features from encoder. Result: Full-resolution prediction. Pattern: Universal in segmentation. Examples: U-Net, SegNet, FCN.
🔽 Encoder🔼 Decoder🔗 Skip📐 Full-res
# FCN and Encoder-Decoder architectures
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision import models
# 1. BASIC FCN
class FCN(nn.Module):
"""
Fully Convolutional Network.
Key idea: Replace FC layers with convolutions.
Input can be any size.
Output is dense (per-pixel).
"""
def __init__(self, num_classes=21):
super().__init__()
# Encoder: VGG16
vgg = models.vgg16(weights='IMAGENET1K_V1')
features = list(vgg.features.children())
# Split VGG into blocks
self.block1 = nn.Sequential(*features[0:5]) # H/2
self.block2 = nn.Sequential(*features[5:10]) # H/4
self.block3 = nn.Sequential(*features[10:17]) # H/8
self.block4 = nn.Sequential(*features[17:24]) # H/16
self.block5 = nn.Sequential(*features[24:31]) # H/32
# Classifier (1x1 conv) at each scale
self.fc6 = nn.Conv2d(512, 4096, 7, padding=3)
self.fc7 = nn.Conv2d(4096, 4096, 1)
self.score_fr = nn.Conv2d(4096, num_classes, 1)
# Score at each scale
self.score_pool3 = nn.Conv2d(256, num_classes, 1)
self.score_pool4 = nn.Conv2d(512, num_classes, 1)
# Upsampling
self.upscore2 = nn.ConvTranspose2d(num_classes, num_classes,
4, stride=2, bias=False)
self.upscore8 = nn.ConvTranspose2d(num_classes, num_classes,
16, stride=8, bias=False)
self.upscore16 = nn.ConvTranspose2d(num_classes, num_classes,
32, stride=16, bias=False)
def forward(self, x):
# Encoder
pool1 = self.block1(x) # H/2
pool2 = self.block2(pool1) # H/4
pool3 = self.block3(pool2) # H/8
pool4 = self.block4(pool3) # H/16
pool5 = self.block5(pool4) # H/32
# FCN head
fc6 = F.relu(self.fc6(pool5))
fc7 = F.relu(self.fc7(fc6))
score_fr = self.score_fr(fc7)
# Score from pool4
score_pool4 = self.score_pool4(pool4)
upscore2 = self.upscore2(score_fr)
score_pool4c = score_pool4[:, :, 5:5+upscore2.size(2),
5:5+upscore2.size(3)]
fuse_pool4 = upscore2 + score_pool4c
# Score from pool3
score_pool3 = self.score_pool3(pool3)
upscore_pool4 = self.upscore2(fuse_pool4)
score_pool3c = score_pool3[:, :, 9:9+upscore_pool4.size(2),
9:9+upscore_pool4.size(3)]
fuse_pool3 = upscore_pool4 + score_pool3c
# Final upsampling
upscore8 = self.upscore8(fuse_pool3)
return upscore8
# 2. ENCODER-DECODER PATTERN
class EncoderDecoder(nn.Module):
"""
Generic encoder-decoder for segmentation.
Encoder: Downsample and extract features
Decoder: Upsample and combine with skip connections
"""
def __init__(self, num_classes):
super().__init__()
# Encoder
self.enc1 = self.conv_block(3, 64)
self.enc2 = self.conv_block(64, 128)
self.enc3 = self.conv_block(128, 256)
self.enc4 = self.conv_block(256, 512)
self.pool = nn.MaxPool2d(2)
# Bottleneck
self.bottleneck = self.conv_block(512, 1024)
# Decoder with skip connections
self.up4 = nn.ConvTranspose2d(1024, 512, 2, stride=2)
self.dec4 = self.conv_block(1024, 512) # 512 + 512
self.up3 = nn.ConvTranspose2d(512, 256, 2, stride=2)
self.dec3 = self.conv_block(512, 256) # 256 + 256
self.up2 = nn.ConvTranspose2d(256, 128, 2, stride=2)
self.dec2 = self.conv_block(256, 128) # 128 + 128
self.up1 = nn.ConvTranspose2d(128, 64, 2, stride=2)
self.dec1 = self.conv_block(128, 64) # 64 + 64
# Classifier
self.classifier = nn.Conv2d(64, num_classes, 1)
def conv_block(self, in_channels, out_channels):
return nn.Sequential(
nn.Conv2d(in_channels, out_channels, 3, padding=1),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),
nn.Conv2d(out_channels, out_channels, 3, padding=1),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),
)
def forward(self, x):
# Encoder
e1 = self.enc1(x) # H
e2 = self.enc2(self.pool(e1)) # H/2
e3 = self.enc3(self.pool(e2)) # H/4
e4 = self.enc4(self.pool(e3)) # H/8
# Bottleneck
b = self.bottleneck(self.pool(e4)) # H/16
# Decoder with skip connections
d4 = self.up4(b)
d4 = torch.cat([d4, e4], dim=1)
d4 = self.dec4(d4)
d3 = self.up3(d4)
d3 = torch.cat([d3, e3], dim=1)
d3 = self.dec3(d3)
d2 = self.up2(d3)
d2 = torch.cat([d2, e2], dim=1)
d2 = self.dec2(d2)
d1 = self.up1(d2)
d1 = torch.cat([d1, e1], dim=1)
d1 = self.dec1(d1)
# Classify
return self.classifier(d1)
# 3. UPSAMPLING METHODS
print("=== Upsampling Methods ===")
print("""
1. Transposed Convolution (Deconvolution):
- Learnable upsampling
- nn.ConvTranspose2d
- Can produce checkerboard artifacts
2. Bilinear Interpolation:
- No parameters
- Smooth
- Fixed (not learned)
3. Nearest Neighbor:
- Simplest
- Blocky output
- Used in some architectures
4. Pixel Shuffle:
- Rearranges channels to space
- Efficient
- No checkerboard artifacts
5. Unpooling:
- Reverses max pooling
- Uses pooling indices
- SegNet approach
Comparison:
Method Learnable Artifacts Speed
Transposed ✓ Sometimes Fast
Bilinear ✗ No Fast
Nearest ✗ Blocky Fastest
Pixel Shuffle ✓ No Fast
Unpooling ✗ No Fast
""")
# 4. SKIP CONNECTIONS
print("\n=== Skip Connections ===")
print("""
Why skip connections?
Problem:
- Encoder reduces resolution
- Decoder must recover fine details
- Upsampling loses spatial information
Solution: Skip connections
- Direct connection from encoder to decoder
- Concatenate (U-Net) or add (FCN)
- Preserves spatial details
- Critical for sharp boundaries
Types:
- Concatenation (U-Net, SegNet)
- Addition (FCN, ResNet-style)
- Attention (attention U-Net)
Number of skips:
- More skips = better boundaries
- But more parameters
- Typical: 4-5 skip connections
""")
# 5. ARCHITECTURES COMPARISON
print("\n=== FCN vs U-Net vs SegNet ===")
print("""
FCN:
- First end-to-end segmentation
- Skip connections via addition
- Uses VGG backbone
- Parameters: ~134M
- Speed: Medium
U-Net:
- Symmetric encoder-decoder
- Skip connections via concatenation
- Designed for medical imaging
- Parameters: ~31M
- Speed: Fast
SegNet:
- Stores pooling indices
- Uses unpooling for upsampling
- Memory efficient
- Parameters: ~29M
- Speed: Fast
Modern alternatives:
- DeepLab: Atrous convolution
- PSPNet: Pyramid pooling
- SegFormer: Transformers
- Mask2Former: Universal
""")
# 6. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. USE PRETRAINED ENCODER
- ImageNet weights
- Faster convergence
2. ADD BATCH NORM
- After each conv
- Before ReLU
- Stabilizes training
3. SKIP CONNECTIONS EVERYWHERE
- Concatenation preferred
- More skips = better
4. USE DILATED CONV IN BOTTLENECK
- Larger receptive field
- No extra parameters
5. DEEP SUPERVISION
- Loss at multiple scales
- Better gradient flow
- Common in modern methods
6. CLASS IMBALANCE
- Weight the loss
- Or use Dice loss
- Ignore background (255)
7. DATA AUGMENTATION
- Random crop, flip, rotate
- Color jitter
- Scale augmentation
8. INFERENCE
- Sliding window for large images
- Multi-scale TTA
- Ensemble models
""")🧬 U-Net Architecture
🎯 The Medical Imaging Standard
U-Net (Ronneberger et al., 2015) is a symmetric encoder-decoder network with skip connections. It's the most popular architecture for medical image segmentation and remains a strong baseline today.
Why U-Net Works
Symmetric: Encoder and decoder are mirror images. Skip connections: Concatenate encoder features to decoder. Preserves details: High-resolution information flows. Efficient: Works with small datasets. Universal: Medical, satellite, industrial. Foundation: Basis for many variants.
🔄 Symmetric🔗 Skip concat🏥 Medical📊 Small data
# U-Net implementation
import torch
import torch.nn as nn
import torch.nn.functional as F
class DoubleConv(nn.Module):
"""(Conv → BN → ReLU) × 2"""
def __init__(self, in_channels, out_channels):
super().__init__()
self.double_conv = nn.Sequential(
nn.Conv2d(in_channels, out_channels, 3, padding=1),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True),
nn.Conv2d(out_channels, out_channels, 3, padding=1),
nn.BatchNorm2d(out_channels),
nn.ReLU(inplace=True)
)
def forward(self, x):
return self.double_conv(x)
class Down(nn.Module):
"""Downscaling with maxpool then double conv."""
def __init__(self, in_channels, out_channels):
super().__init__()
self.maxpool_conv = nn.Sequential(
nn.MaxPool2d(2),
DoubleConv(in_channels, out_channels)
)
def forward(self, x):
return self.maxpool_conv(x)
class Up(nn.Module):
"""Upscaling then double conv with skip connection."""
def __init__(self, in_channels, out_channels, bilinear=True):
super().__init__()
if bilinear:
self.up = nn.Upsample(scale_factor=2, mode='bilinear',
align_corners=True)
self.conv = DoubleConv(in_channels, out_channels)
else:
self.up = nn.ConvTranspose2d(in_channels, in_channels // 2,
kernel_size=2, stride=2)
self.conv = DoubleConv(in_channels, out_channels)
def forward(self, x1, x2):
"""x1: from decoder, x2: from encoder (skip)."""
x1 = self.up(x1)
# Handle size mismatch (odd dimensions)
diffY = x2.size()[2] - x1.size()[2]
diffX = x2.size()[3] - x1.size()[3]
x1 = F.pad(x1, [diffX // 2, diffX - diffX // 2,
diffY // 2, diffY - diffY // 2])
# Concatenate skip connection
x = torch.cat([x2, x1], dim=1)
return self.conv(x)
class OutConv(nn.Module):
"""Final 1x1 convolution."""
def __init__(self, in_channels, out_channels):
super().__init__()
self.conv = nn.Conv2d(in_channels, out_channels, 1)
def forward(self, x):
return self.conv(x)
class UNet(nn.Module):
"""
U-Net: Symmetric encoder-decoder with skip connections.
Architecture:
Encoder: 64 → 128 → 256 → 512 (each with downsampling)
Bottleneck: 1024
Decoder: 512 → 256 → 128 → 64 (each with upsampling)
Output: num_classes
"""
def __init__(self, n_channels=3, n_classes=2, bilinear=False):
super().__init__()
self.n_channels = n_channels
self.n_classes = n_classes
self.bilinear = bilinear
# Encoder
self.inc = DoubleConv(n_channels, 64)
self.down1 = Down(64, 128)
self.down2 = Down(128, 256)
self.down3 = Down(256, 512)
factor = 2 if bilinear else 1
self.down4 = Down(512, 1024 // factor)
# Decoder
self.up1 = Up(1024, 512 // factor, bilinear)
self.up2 = Up(512, 256 // factor, bilinear)
self.up3 = Up(256, 128 // factor, bilinear)
self.up4 = Up(128, 64, bilinear)
# Output
self.outc = OutConv(64, n_classes)
def forward(self, x):
# Encoder
x1 = self.inc(x) # (B, 64, H, W)
x2 = self.down1(x1) # (B, 128, H/2, W/2)
x3 = self.down2(x2) # (B, 256, H/4, W/4)
x4 = self.down3(x3) # (B, 512, H/8, W/8)
x5 = self.down4(x4) # (B, 1024, H/16, W/16)
# Decoder with skip connections
x = self.up1(x5, x4) # (B, 512, H/8, W/8)
x = self.up2(x, x3) # (B, 256, H/4, W/4)
x = self.up3(x, x2) # (B, 128, H/2, W/2)
x = self.up4(x, x1) # (B, 64, H, W)
# Output
logits = self.outc(x) # (B, n_classes, H, W)
return logits
# TRAINING U-NET
def train_unet(model, train_loader, val_loader, epochs=50):
"""Train U-Net for segmentation."""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(
optimizer, 'min', patience=5
)
# Combined loss: CrossEntropy + Dice
ce_loss = nn.CrossEntropyLoss()
def dice_loss(pred, target, smooth=1e-6):
pred = F.softmax(pred, dim=1)
target_one_hot = F.one_hot(target, num_classes=pred.shape[1])
target_one_hot = target_one_hot.permute(0, 3, 1, 2).float()
intersection = (pred * target_one_hot).sum(dim=(2, 3))
union = pred.sum(dim=(2, 3)) + target_one_hot.sum(dim=(2, 3))
dice = (2. * intersection + smooth) / (union + smooth)
return 1 - dice.mean()
for epoch in range(epochs):
model.train()
train_loss = 0
for images, masks in train_loader:
images = images.to(device)
masks = masks.to(device).long()
optimizer.zero_grad()
logits = model(images)
# Combined loss
loss = ce_loss(logits, masks) + dice_loss(logits, masks)
loss.backward()
optimizer.step()
train_loss += loss.item()
# Validation
model.eval()
val_loss = 0
with torch.no_grad():
for images, masks in val_loader:
images = images.to(device)
masks = masks.to(device).long()
logits = model(images)
val_loss += ce_loss(logits, masks).item()
avg_train = train_loss / len(train_loader)
avg_val = val_loss / len(val_loader)
scheduler.step(avg_val)
print(f"Epoch {epoch+1}: "
f"Train={avg_train:.4f}, Val={avg_val:.4f}")
# U-NET VARIANTS
print("=== U-Net Variants ===")
print("""
3D U-Net:
- For volumetric data (CT, MRI)
- 3D convolutions
- Preserves depth dimension
U-Net++ (Nested U-Net):
- Dense skip connections
- Better feature fusion
- Improved accuracy
Attention U-Net:
- Attention gates on skips
- Focus on relevant regions
- Better for small structures
ResUNet:
- Residual blocks in encoder
- Easier training
- Deeper networks
U-Net with EfficientNet:
- Efficient encoder
- Better accuracy/efficiency
- Common in practice
nnU-Net:
- Automated U-Net configuration
- Self-configuring
- State-of-the-art in medical
- Won many challenges
TransUNet:
- Transformer encoder
- U-Net decoder
- Better long-range context
SwinUNet:
- Swin Transformer encoder
- U-Net decoder
- Hierarchical features
""")
# U-NET FOR MEDICAL IMAGING
print("\n=== U-Net for Medical Imaging ===")
print("""
Why U-Net dominates medical imaging:
1. WORKS WITH SMALL DATA
- Skip connections help
- Data augmentation effective
- Pretraining not always needed
2. PRECISE BOUNDARIES
- Full resolution output
- Skip connections preserve details
- Critical for medical (organ boundaries)
3. FLEXIBLE
- Any number of classes
- Any input size
- 2D or 3D
4. FAST
- Efficient architecture
- Real-time inference
- Can run on modest hardware
Applications:
- Organ segmentation
- Tumor detection
- Cell segmentation
- Vessel extraction
- Lesion quantification
Datasets:
- KiTS (kidney)
- BraTS (brain tumor)
- LiTS (liver)
- ISIC (skin)
- MoNuSeg (nuclei)
""")
# IMPLEMENTATION TIPS
print("\n=== Practical Tips ===")
print("""
1. USE BATCH NORMALIZATION
- After each conv
- Stabilizes training
- Faster convergence
2. PADDING MATTERS
- 'same' padding preserves size
- 'valid' shrinks (original U-Net)
- 'same' is standard now
3. HANDLE ODD DIMENSIONS
- Pad to multiple of 16
- Or use padding in Up block
- Common implementation detail
4. DEEP SUPERVISION
- Add loss at multiple decoder levels
- Better gradient flow
- Especially for deep U-Nets
5. DATA AUGMENTATION
- Essential for small medical datasets
- Random crop, flip, rotate
- Elastic deformations
- Intensity shifts
6. LOSS FUNCTION
- CrossEntropy alone can be bad (class imbalance)
- Combine with Dice or IoU
- Or use Focal loss
7. WEIGHT INITIALIZATION
- He/Kaiming for ReLU
- Xavier for tanh
8. INFERENCE
- Sliding window for large images
- TTA (Test Time Augmentation)
- Ensemble multiple models
""")🏔️ DeepLab & Atrous Convolution
🎯 Dilated Convolutions for Dense Prediction
DeepLab (Google, 2016-2018) introduced atrous (dilated) convolutions to segmentation. These expand the receptive field without reducing resolution, avoiding the need for aggressive downsampling.
Atrous Convolution Explained
Standard conv: 3×3 kernel looks at 9 pixels. Dilated conv (rate=2): 3×3 kernel looks at 25 pixels (skips every other). Rate=4: Even larger receptive field. Benefit: Large context without downsampling. ASPP: Multiple rates in parallel. Result: Multi-scale context.
🔍 Atrous📐 Receptive field🎯 ASPP🏔️ Multi-scale
# DeepLab and atrous (dilated) convolutions
import torch
import torch.nn as nn
import torch.nn.functional as F
# 1. ATROUS (DILATED) CONVOLUTION
class AtrousConv(nn.Module):
"""
Atrous (dilated) convolution.
Standard 3x3 conv (rate=1): Atrous 3x3 conv (rate=2):
X X X X . X . X
X X X . . . . .
X X X X . X . X
. . . . .
X . X . X
Receptive field: 3x3 → 5x5 (with same parameters!)
"""
def __init__(self, in_channels, out_channels, rate=2):
super().__init__()
# Effective kernel size: k + (k-1)*(rate-1)
self.conv = nn.Conv2d(
in_channels, out_channels,
kernel_size=3,
padding=rate, # Same size output
dilation=rate # The dilation rate
)
self.bn = nn.BatchNorm2d(out_channels)
self.relu = nn.ReLU(inplace=True)
def forward(self, x):
return self.relu(self.bn(self.conv(x)))
# Demonstrate receptive field growth
print("=== Receptive Field Growth ===")
print("""
Convolution rate → Effective kernel → Receptive field
rate=1: 3×3 kernel → 3×3 receptive field
rate=2: 3×3 kernel → 5×5 receptive field
rate=4: 3×3 kernel → 9×9 receptive field
rate=8: 3×3 kernel → 17×17 receptive field
rate=16: 3×3 kernel → 33×33 receptive field
Key insight: Same number of parameters, larger context!
""")
# 2. ASPP (ATROUS SPATIAL PYRAMID POOLING)
class ASPP(nn.Module):
"""
Atrous Spatial Pyramid Pooling.
Applies multiple atrous convolutions with different rates
in parallel, then combines them.
Captures multi-scale context!
"""
def __init__(self, in_channels, out_channels=256, rates=[6, 12, 18]):
super().__init__()
# 1x1 conv (rate=1)
self.conv1x1 = nn.Sequential(
nn.Conv2d(in_channels, out_channels, 1),
nn.BatchNorm2d(out_channels),
nn.ReLU()
)
# Atrous convolutions at multiple rates
self.atrous_convs = nn.ModuleList()
for rate in rates:
self.atrous_convs.append(
nn.Sequential(
nn.Conv2d(in_channels, out_channels, 3,
padding=rate, dilation=rate),
nn.BatchNorm2d(out_channels),
nn.ReLU()
)
)
# Global average pooling
self.global_pool = nn.Sequential(
nn.AdaptiveAvgPool2d(1),
nn.Conv2d(in_channels, out_channels, 1),
nn.BatchNorm2d(out_channels),
nn.ReLU()
)
# Project combined features
self.project = nn.Sequential(
nn.Conv2d(out_channels * (len(rates) + 2), out_channels, 1),
nn.BatchNorm2d(out_channels),
nn.ReLU(),
nn.Dropout(0.5)
)
def forward(self, x):
size = x.shape[2:]
# 1x1 branch
conv1x1 = self.conv1x1(x)
# Multi-rate atrous branches
atrous_outs = [conv(x) for conv in self.atrous_convs]
# Global pooling branch
global_out = self.global_pool(x)
global_out = F.interpolate(global_out, size=size,
mode='bilinear', align_corners=False)
# Concatenate all branches
combined = torch.cat([conv1x1] + atrous_outs + [global_out], dim=1)
return self.project(combined)
# 3. DEEPLABV3+ ARCHITECTURE
class DeepLabV3Plus(nn.Module):
"""
DeepLabV3+ architecture.
1. Backbone: ResNet with atrous convolutions
2. ASPP: Multi-scale context
3. Decoder: Combine low-level features + ASPP output
"""
def __init__(self, num_classes=21, backbone='resnet50'):
super().__init__()
# Encoder: ResNet with atrous convolutions
from torchvision import models
resnet = models.resnet50(weights='IMAGENET1K_V2')
# Modify ResNet for atrous convolutions
self.layer0 = nn.Sequential(
resnet.conv1, resnet.bn1, resnet.relu, resnet.maxpool
)
self.layer1 = resnet.layer1
self.layer2 = resnet.layer2
self.layer3 = resnet.layer3
self.layer4 = resnet.layer4
# ASPP
self.aspp = ASPP(2048, 256)
# Decoder
self.low_level_conv = nn.Sequential(
nn.Conv2d(256, 48, 1),
nn.BatchNorm2d(48),
nn.ReLU()
)
self.decoder = nn.Sequential(
nn.Conv2d(256 + 48, 256, 3, padding=1),
nn.BatchNorm2d(256),
nn.ReLU(),
nn.Conv2d(256, 256, 3, padding=1),
nn.BatchNorm2d(256),
nn.ReLU(),
nn.Conv2d(256, num_classes, 1)
)
def forward(self, x):
input_size = x.shape[2:]
# Encoder
x = self.layer0(x) # H/4
low_level = self.layer1(x) # H/4 (for decoder)
x = self.layer2(low_level) # H/8
x = self.layer3(x) # H/16
x = self.layer4(x) # H/32
# ASPP
x = self.aspp(x)
# Upsample ASPP output
x = F.interpolate(x, size=low_level.shape[2:],
mode='bilinear', align_corners=False)
# Process low-level features
low_level = self.low_level_conv(low_level)
# Concatenate
x = torch.cat([x, low_level], dim=1)
# Decoder
x = self.decoder(x)
# Final upsample to input size
x = F.interpolate(x, size=input_size,
mode='bilinear', align_corners=False)
return x
# 4. DEEPLAB EVOLUTION
print("\n=== DeepLab Evolution ===")
print("""
DeepLab V1 (2015):
- Atrous convolutions
- CRF post-processing
- VGG backbone
DeepLab V2 (2016):
- ASPP introduced
- Multi-scale context
- ResNet backbone
DeepLab V3 (2017):
- Improved ASPP
- Image-level features
- No CRF needed
- SOTA on PASCAL VOC
DeepLab V3+ (2018):
- Encoder-decoder
- Skip connections
- Xception backbone
- SOTA on multiple datasets
Key innovations:
- Atrous convolution: context without resolution loss
- ASPP: multi-scale features
- Encoder-decoder: fine details
- Backbone: any pretrained CNN
""")
# 5. OTHER ATROUS-BASED METHODS
print("\n=== Other Atrous-Based Methods ===")
print("""
PSPNet:
- Pyramid Pooling Module
- Multiple scales
- Similar to ASPP
- Year: 2017
DilatedNet:
- Earlier atrous work
- Progressive dilation
- Year: 2016
ENet:
- Efficient network
- Atrous convolutions
- Real-time
- Year: 2016
Dilated Residual Networks (DRN):
- Atrous for classification
- Preserves resolution
- Year: 2017
Modern:
- DeepLab concepts in almost all segmentation
- ASPP-like modules standard
- Atrous convolutions common
""")
# 6. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. CHOOSE RATES CAREFULLY
- Depends on feature map size
- Common: [6, 12, 18]
- Or [1, 3, 6, 12]
2. USE OUTPUT STRIDE
- DeepLab uses output_stride=16
- Balances resolution and context
- Adjust last layers
3. BATCH NORM
- Usually in atrous blocks
- Or use GroupNorm for small batches
4. COMPUTE CONSIDERATIONS
- Atrous convs are slower
- But no downsampling
- Trade-off between speed and quality
5. COMBINE WITH ASPP
- ASPP is the key ingredient
- Multi-scale is essential
6. USE PRETRAINED BACKBONE
- ImageNet weights
- Faster convergence
- Better accuracy
7. HANDLE BOUNDARIES
- Atrous can cause boundary effects
- Padding matters
- Reflect padding can help
8. MEMORY
- Atrous uses more memory than stride
- Reduce batch size if needed
- Gradient checkpointing
""")
# 7. IMPLEMENTATION DETAILS
print("\n=== Implementation Details ===")
print("""
Atrous rate scheduling:
- Higher rates → larger context
- But can cause gridding artifacts
- Balance needed
DeepLab's approach:
- os=16 for V3+
- rates=[6,12,18] for ASPP
- Plus 1x1 and global pooling
Gridding artifacts:
- When using high dilation rates
- Solution: use varied rates
- Or use HDC (Hybrid Dilated Convolution)
Memory:
- Atrous convs preserve resolution
- More memory than strided
- DeepLab needs 12GB+ for training
Speed:
- Slower than plain CNN
- But no upsampling needed
- FPS: 5-20 depending on model
""")🎭 Mask R-CNN
🎯 Instance Segmentation Milestone
Mask R-CNN (He et al., 2017) extended Faster R-CNN with a mask prediction branch. It was a landmark achievement, achieving state-of-the-art instance segmentation with minimal changes to the detection framework.
Mask R-CNN Key Ideas
Two-stage: Detect then segment. RoI Align: Better than RoI Pool. Mask branch: Parallel to classification. Per-class masks: One mask per class per RoI. Simple: Only adds a small head. Powerful: SOTA at the time.
🎯 Two-stage🔗 RoI Align🎭 Mask head🚀 SOTA
# Mask R-CNN implementation
import torch
import torch.nn as nn
import torch.nn.functional as F
from torchvision.models.detection import maskrcnn_resnet50_fpn
from torchvision.models.detection.faster_rcnn import FastRCNNPredictor
from torchvision.models.detection.mask_rcnn import MaskRCNNPredictor
# 1. MASK R-CNN ARCHITECTURE (using torchvision)
def get_mask_rcnn(num_classes=81):
"""
Get pretrained Mask R-CNN.
Architecture:
1. Backbone: ResNet + FPN
2. RPN: Region proposals
3. RoI Align: Feature extraction
4. Box head: Classification + regression
5. Mask head: Per-instance segmentation
"""
# Load pretrained
model = maskrcnn_resnet50_fpn(weights='COCO_V1')
# Replace box classifier
in_features = model.roi_heads.box_predictor.cls_score.in_features
model.roi_heads.box_predictor = FastRCNNPredictor(in_features, num_classes)
# Replace mask predictor
in_features_mask = model.roi_heads.mask_predictor.conv5_mask.in_channels
hidden_layer = 256
model.roi_heads.mask_predictor = MaskRCNNPredictor(
in_features_mask, hidden_layer, num_classes
)
return model
# 2. UNDERSTANDING THE ARCHITECTURE
print("=== Mask R-CNN Architecture ===")
print("""
INPUT: Image (H, W, 3)
STAGE 1: BACKBONE + FPN
- ResNet extracts features
- FPN combines multi-scale features
- Output: Feature pyramid (P2, P3, P4, P5, P6)
STAGE 2: RPN (Region Proposal Network)
- Anchors at each location
- Objectness score + box refinement
- Output: ~1000-2000 proposals
STAGE 3: RoI ALIGN
- Extract fixed-size features from each proposal
- No quantization (unlike RoI Pool)
- Output: 7x7 feature map per RoI
STAGE 4: BOX HEAD
- Classification: which class?
- Regression: refined box
- Output: class + box per proposal
STAGE 5: MASK HEAD
- Small FCN per RoI
- Output: 28x28 mask per class
- Only for positive proposals
OUTPUT: Instance masks + classes + scores
""")
# 3. RoI ALIGN (Key Improvement)
print("\n=== RoI Align vs RoI Pool ===")
print("""
RoI Pool (Faster R-CNN):
- Quantizes RoI boundaries to integers
- Quantizes bin boundaries
- Introduces misalignment
- OK for detection, bad for masks
RoI Align (Mask R-CNN):
- No quantization of boundaries
- Bilinear interpolation for sampling
- Exact alignment
- Critical for masks
Comparison:
RoI Pool: RoI Align:
[1.7, 2.3] [1.7, 2.3]
→ [1, 2] → No rounding
→ Integer coords → Float coords
→ Misalignment → Precise
Impact:
- Mask AP improves by 3-5%
- Better boundaries
- Smoother masks
- Essential for high-quality output
""")
# 4. IMPLEMENTATION DETAILS
class MaskRCNNDetailed(nn.Module):
"""Detailed Mask R-CNN components."""
def __init__(self, num_classes=81):
super().__init__()
# Simplified components
self.backbone = self.build_backbone()
self.rpn = self.build_rpn()
self.roi_heads = self.build_roi_heads(num_classes)
def build_backbone(self):
"""Backbone + FPN."""
from torchvision.models.detection.backbone_utils import resnet_fpn_backbone
return resnet_fpn_backbone('resnet50', pretrained=True)
def build_rpn(self):
"""Region Proposal Network."""
from torchvision.models.detection.rpn import AnchorGenerator, RPNHead, RegionProposalNetwork
anchor_generator = AnchorGenerator(
sizes=((32, 64, 128, 256, 512),),
aspect_ratios=((0.5, 1.0, 2.0),)
)
rpn_head = RPNHead(
in_channels=256,
num_anchors=3 * 3
)
return RegionProposalNetwork(
anchor_generator, rpn_head,
fg_iou_thresh=0.7, bg_iou_thresh=0.3,
batch_size_per_image=256, positive_fraction=0.5
)
def build_roi_heads(self, num_classes):
"""RoI heads: box + mask."""
from torchvision.models.detection.roi_heads import RoIHeads
from torchvision.ops import MultiScaleRoIAlign
from torchvision.models.detection.faster_rcnn import TwoMLPHead, FastRCNNPredictor
# RoI Align
box_roi_pool = MultiScaleRoIAlign(
featmap_names=['0', '1', '2', '3'],
output_size=7,
sampling_ratio=2
)
# Box head
resolution = 7
representation_size = 1024
box_head = TwoMLPHead(
in_channels=256 * resolution ** 2,
representation_size=representation_size
)
box_predictor = FastRCNNPredictor(representation_size, num_classes)
# Mask head
from torchvision.models.detection.mask_rcnn import MaskRCNNHeads, MaskRCNNPredictor
mask_roi_pool = MultiScaleRoIAlign(
featmap_names=['0', '1', '2', '3'],
output_size=14,
sampling_ratio=2
)
mask_layers = (256, 256, 256, 256)
mask_dilation = 1
mask_head = MaskRCNNHeads(
256, mask_layers, mask_dilation
)
mask_predictor = MaskRCNNPredictor(
256, 256, num_classes
)
return RoIHeads(
box_roi_pool, box_head, box_predictor,
mask_roi_pool, mask_head, mask_predictor
)
# 5. TRAINING
def train_mask_rcnn(model, train_loader, num_epochs=10, lr=1e-4):
"""Train Mask R-CNN."""
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = model.to(device)
params = [p for p in model.parameters() if p.requires_grad]
optimizer = torch.optim.SGD(params, lr=lr, momentum=0.9,
weight_decay=1e-4)
lr_scheduler = torch.optim.lr_scheduler.StepLR(optimizer,
step_size=3, gamma=0.1)
for epoch in range(num_epochs):
model.train()
epoch_loss = 0
for images, targets in train_loader:
images = list(img.to(device) for img in images)
targets = [{k: v.to(device) for k, v in t.items()}
for t in targets]
loss_dict = model(images, targets)
losses = sum(loss for loss in loss_dict.values())
optimizer.zero_grad()
losses.backward()
optimizer.step()
epoch_loss += losses.item()
lr_scheduler.step()
print(f"Epoch {epoch+1}: Loss = {epoch_loss/len(train_loader):.4f}")
# 6. LOSSES
print("\n=== Mask R-CNN Losses ===")
print("""
Total loss = L_class + L_box + L_mask
1. Classification Loss (RPN + RoI):
- Cross-entropy for class prediction
- Or binary cross-entropy for objectness
2. Box Regression Loss:
- Smooth L1 loss
- Only for positive RoIs
- Refines box coordinates
3. Mask Loss:
- Binary cross-entropy
- Per-pixel, per-class
- Only for positive RoIs
Weighting:
- All losses weighted equally (default)
- Can tune weights
Training balance:
- RPN: 256 anchors per image
- RoI: 512 RoIs per image (25% positive)
""")
# 7. INFERENCE
def inference_mask_rcnn(model, image, threshold=0.5):
"""Run inference."""
model.eval()
with torch.no_grad():
if image.dim() == 3:
image = image.unsqueeze(0)
predictions = model(image.to(device='cuda'))
pred = predictions[0]
# Filter by threshold
keep = pred['scores'] > threshold
return {
'boxes': pred['boxes'][keep],
'labels': pred['labels'][keep],
'scores': pred['scores'][keep],
'masks': pred['masks'][keep]
}
# 8. ADVANTAGES AND LIMITATIONS
print("\n=== Mask R-CNN Pros and Cons ===")
print("""
ADVANTAGES:
✓ SOTA accuracy on instance segmentation
✓ Simple extension of Faster R-CNN
✓ Handles overlapping objects
✓ Works for many classes
✓ Well-understood and documented
LIMITATIONS:
✗ Two-stage: slower than one-stage
✗ Not real-time (5-10 FPS typical)
✗ Memory intensive during training
✗ Complex implementation
✗ Hard to train from scratch
Variants for speed:
- YOLACT (real-time)
- SOLOv2 (real-time)
- BlendMask (fast)
- CondInst (fast)
Modern alternatives:
- Mask2Former (transformer)
- SAM (foundation model)
- DETR-based methods
""")
# 9. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. USE PRETRAINED COCO WEIGHTS
- Faster convergence
- Better accuracy
- Standard practice
2. FINE-TUNE CAREFULLY
- Small learning rate (1e-4)
- Fewer epochs needed
- Watch for overfitting
3. DATA AUGMENTATION
- Random crop, flip
- Color jitter
- Essential for small data
4. BATCH SIZE
- Usually 2-4 for training
- Memory intensive
- Gradient accumulation if needed
5. EVALUATION
- Use COCO eval metrics
- Mask AP@0.5, @0.75
- Per-class analysis
6. INFERENCE
- Threshold tuning
- NMS (already in model)
- Post-processing masks
7. COMMON ISSUES
- Small objects missed
- Crowded scenes
- Slow inference
- High memory
8. WHEN TO USE
- Need individual instances
- High accuracy required
- Speed not critical
- Have pretrained model
""")🤖 Transformer-based (SegFormer, SAM)
🎯 Attention is All You Need for Segmentation
Transformers have revolutionized segmentation. Vision Transformers (ViT) and their variants now dominate benchmarks. SegFormer, Mask2Former, and SAM are state-of-the-art.
Why Transformers for Segmentation?
Global context: Attention sees entire image. Long-range: No locality bias. Flexible: Handles variable sizes. Scalable: Scales with data. SOTA: Best accuracy. Universal: One model for all tasks.
🎯 Attention🌐 Global📈 Scalable🚀 SOTA
# Transformer-based segmentation
import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import SegformerForSemanticSegmentation, SegformerConfig
# 1. SEGFORMER (Efficient Transformer)
def get_segformer(num_classes=21, model_name='nvidia/segformer-b0-finetuned-ade-512-512'):
"""
SegFormer: Simple and Efficient Design for Semantic Segmentation.
Architecture:
1. Hierarchical Transformer Encoder (MiT)
2. Lightweight MLP Decoder
"""
model = SegformerForSemanticSegmentation.from_pretrained(
model_name,
num_labels=num_classes,
ignore_mismatched_sizes=True
)
return model
print("=== SegFormer Architecture ===")
print("""
SegFormer:
1. HIERARCHICAL ENCODER (MiT - Mix Transformer)
- Multi-scale features (like CNN)
- Efficient self-attention
- Overlapping patch embeddings
- Stages: 1/4, 1/8, 1/16, 1/32
2. LIGHTWEIGHT DECODER (MLP)
- Upsample all scales
- Concatenate
- MLP fusion
- Final prediction
Benefits:
- No positional encoding
- Efficient attention
- Fast inference
- SOTA accuracy
Variants:
B0: 3.7M params (mobile)
B1: 13.7M params
B2: 27.4M params
B3: 44.7M params
B4: 61.4M params
B5: 84.7M params
""")
# 2. MASK2FORMER (Universal Segmentation)
print("\n=== Mask2Former ===")
print("""
Mask2Former: Universal Architecture for All Segmentation.
Key idea: Predict masks directly, not per-pixel classification.
Components:
1. Backbone: ResNet or Swin
2. Pixel Decoder: Multi-scale features
3. Transformer Decoder: Learnable queries → mask embeddings
4. Mask Prediction: Query × features → mask
Benefits:
- Unified for all tasks (semantic, instance, panoptic)
- Better accuracy
- No per-pixel classification needed
- Handles varying numbers of objects
How it works:
1. Learnable queries (N of them)
2. Each query attends to image features
3. Each query produces a mask
4. Classification of each mask
Code (simplified):
from transformers import Mask2FormerForUniversalSegmentation
model = Mask2FormerForUniversalSegmentation.from_pretrained(
"facebook/mask2former-swin-base-coco-panoptic"
)
# One model, three tasks!
outputs = model(images)
# - Semantic segmentation
# - Instance segmentation
# - Panoptic segmentation
""")
# 3. SAM (Segment Anything Model)
print("\n=== SAM (Segment Anything) ===")
print("""
SAM (Meta AI, 2023): Foundation Model for Segmentation.
Key innovations:
1. Trained on 1.1 BILLION masks
2. Promptable: point, box, text
3. Zero-shot: works on any image
4. General: no fine-tuning needed
Architecture:
1. Image Encoder: ViT-H (heavy, runs once)
2. Prompt Encoder: points, boxes, text
3. Mask Decoder: lightweight, real-time
Types of prompts:
- Point: click on object
- Box: bounding box
- Text: "the cat"
- Mask: rough mask
- Multiple: any combination
Capabilities:
✓ Segment anything
✓ Zero-shot on new images
✓ Promptable interface
✓ Works for all domains
Limitations:
✗ Slow image encoder
✗ Needs prompt (not fully automatic)
✗ Large model (2.4GB)
✗ Sometimes wrong
Code:
from segment_anything import sam_model_registry, SamPredictor
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
predictor = SamPredictor(sam)
# Set image
predictor.set_image(image)
# Segment with point prompt
masks, scores, logits = predictor.predict(
point_coords=np.array([[500, 375]]),
point_labels=np.array([1]),
multimask_output=True
)
""")
# 4. OTHER TRANSFORMER METHODS
print("\n=== Other Transformer Methods ===")
print("""
DETR (2020):
- Transformer for detection
- Set prediction
- No NMS
- Extended to segmentation
Swin Transformer (2021):
- Hierarchical ViT
- Shifted windows
- Efficient
- Backbone for many methods
SegFormer (2021):
- Efficient transformer
- MLP decoder
- Mobile-friendly
MaskFormer (2021):
- Unified segmentation
- Mask classification
Mask2Former (2022):
- Improved MaskFormer
- Masked attention
OneFormer (2023):
- One model for all tasks
- Task-conditioned
DINOv2 (2023):
- Self-supervised vision
- Foundation features
- Segment with linear probe
""")
# 5. IMPLEMENTATION WITH HUGGING FACE
def practical_segformer_training():
"""Fine-tune SegFormer with Hugging Face."""
from transformers import (
SegformerForSemanticSegmentation,
SegformerImageProcessor,
TrainingArguments,
Trainer
)
from datasets import load_dataset
# Load dataset
dataset = load_dataset("scene_parse_150")
# Load model
model = SegformerForSemanticSegmentation.from_pretrained(
"nvidia/segformer-b0-finetuned-ade-512-512",
num_labels=150,
ignore_mismatched_sizes=True
)
# Preprocessor
processor = SegformerImageProcessor(
do_resize=True,
size={"height": 512, "width": 512},
do_normalize=True
)
# Training arguments
training_args = TrainingArguments(
output_dir="./segformer-finetuned",
learning_rate=6e-5,
num_train_epochs=50,
per_device_train_batch_size=8,
per_device_eval_batch_size=8,
evaluation_strategy="epoch",
save_strategy="epoch",
load_best_model_at_end=True,
)
# Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=dataset["train"],
eval_dataset=dataset["validation"],
)
# Train
trainer.train()
return model
# 6. COMPARISON
print("\n=== Architecture Comparison ===")
print(f"{'Method':<20} {'Params':<12} {'Speed':<12} {'Accuracy':<12}")
print("-" * 60)
print(f"{'U-Net':<20} {'31M':<12} {'Fast':<12} {'Good':<12}")
print(f"{'DeepLabV3+':<20} {'59M':<12} {'Medium':<12} {'Very Good':<12}")
print(f"{'Mask R-CNN':<20} {'44M':<12} {'Slow':<12} {'SOTA':<12}")
print(f"{'SegFormer-B0':<20} {'3.7M':<12} {'Fast':<12} {'SOTA':<12}")
print(f"{'SegFormer-B5':<20} {'85M':<12} {'Medium':<12} {'SOTA':<12}")
print(f"{'Mask2Former':<20} {'216M':<12} {'Slow':<12} {'Best':<12}")
print(f"{'SAM (ViT-H)':<20} {'2.4B':<12} {'Very Slow':<12} {'Zero-shot':<12}")
# 7. WHEN TO USE TRANSFORMERS
print("\n=== When to Use Transformers ===")
print("""
USE TRANSFORMERS WHEN:
✓ High accuracy needed
✓ Global context important
✓ Large dataset available
✓ Compute budget available
✓ Pretrained models available
USE CNNs WHEN:
✓ Real-time inference needed
✓ Small dataset
✓ Limited compute
✓ Local features sufficient
✓ Efficiency matters
HYBRID APPROACHES:
- CNN encoder + Transformer decoder
- Transformer encoder + CNN decoder
- CNN features + Transformer attention
Modern practice:
- SegFormer for efficient SOTA
- Mask2Former for best accuracy
- SAM for zero-shot
- U-Net for medical (still competitive)
- DINOv2 for features
""")
# 8. PRACTICAL TIPS
print("\n=== Practical Tips ===")
print("""
1. START WITH PRETRAINED
- Hugging Face has many models
- Fine-tune for your task
2. USE MIXED PRECISION
- Transformers are memory-heavy
- FP16 or BF16 training
- Gradient checkpointing
3. BATCH SIZE
- Usually smaller than CNNs
- 4-8 for training
- Adjust for memory
4. LEARNING RATE
- Lower than CNNs
- 1e-5 to 6e-5
- Warmup helps
5. DATA AUGMENTATION
- Random crop
- Color jitter
- Mixup/CutMix
6. INFERENCE
- Fixed input size (multiple of 32)
- Batch for efficiency
- ONNX/TensorRT for speed
7. SAM SPECIFIC
- Use ViT-B for speed
- ViT-H for quality
- MobileSAM for edge devices
""")📉 Loss Functions (Dice, IoU)
🎯 Training Segmentation Models
Segmentation requires specialized loss functions. Cross-entropy is standard, but Dice, IoU, and Focal losses handle class imbalance better.
Why Special Losses?
Class imbalance: Background dominates pixels. Small objects: Rarely represented. Boundaries: Need precise edges. CE alone: Biased toward majority. Dice: Overlap-based, handles imbalance. Combined: Best results.
📉 CE🎲 Dice📊 IoU🔥 Focal
# Loss functions for segmentation
import torch
import torch.nn as nn
import torch.nn.functional as F
# 1. CROSS-ENTROPY LOSS
class CrossEntropyLoss(nn.Module):
"""Standard cross-entropy for per-pixel classification."""
def __init__(self, weight=None, ignore_index=255):
super().__init__()
self.loss = nn.CrossEntropyLoss(
weight=weight,
ignore_index=ignore_index
)
def forward(self, logits, targets):
"""
logits: (B, C, H, W)
targets: (B, H, W) with class indices
"""
return self.loss(logits, targets)
# 2. DICE LOSS
class DiceLoss(nn.Module):
"""
Dice Loss: 1 - (2 * |A ∩ B|) / (|A| + |B|)
Handles class imbalance well.
Works well for binary and multi-class.
"""
def __init__(self, smooth=1e-6):
super().__init__()
self.smooth = smooth
def forward(self, logits, targets):
"""
logits: (B, C, H, W)
targets: (B, H, W) with class indices
"""
# Softmax to get probabilities
probs = F.softmax(logits, dim=1)
# One-hot encode targets
num_classes = logits.shape[1]
targets_one_hot = F.one_hot(targets, num_classes)
targets_one_hot = targets_one_hot.permute(0, 3, 1, 2).float()
# Flatten spatial dimensions
probs = probs.flatten(2) # (B, C, H*W)
targets_one_hot = targets_one_hot.flatten(2)
# Compute Dice per class
intersection = (probs * targets_one_hot).sum(dim=2)
union = probs.sum(dim=2) + targets_one_hot.sum(dim=2)
dice = (2. * intersection + self.smooth) / (union + self.smooth)
# Average over classes
return 1 - dice.mean()
# 3. IoU (JACCARD) LOSS
class IoULoss(nn.Module):
"""
IoU Loss: 1 - |A ∩ B| / |A ∪ B|
Directly optimizes IoU metric.
"""
def __init__(self, smooth=1e-6):
super().__init__()
self.smooth = smooth
def forward(self, logits, targets):
probs = F.softmax(logits, dim=1)
num_classes = logits.shape[1]
targets_one_hot = F.one_hot(targets, num_classes)
targets_one_hot = targets_one_hot.permute(0, 3, 1, 2).float()
probs = probs.flatten(2)
targets_one_hot = targets_one_hot.flatten(2)
intersection = (probs * targets_one_hot).sum(dim=2)
union = probs.sum(dim=2) + targets_one_hot.sum(dim=2) - intersection
iou = (intersection + self.smooth) / (union + self.smooth)
return 1 - iou.mean()
# 4. FOCAL LOSS
class FocalLoss(nn.Module):
"""
Focal Loss: -α(1-p)^γ log(p)
Down-weights easy examples.
Focuses on hard, misclassified examples.
"""
def __init__(self, alpha=0.25, gamma=2.0, ignore_index=255):
super().__init__()
self.alpha = alpha
self.gamma = gamma
self.ignore_index = ignore_index
def forward(self, logits, targets):
# Standard CE
ce = F.cross_entropy(
logits, targets,
reduction='none',
ignore_index=self.ignore_index
)
# Get probabilities
probs = F.softmax(logits, dim=1)
# Focal term
pt = probs.gather(1, targets.unsqueeze(1)).squeeze(1)
focal_weight = (1 - pt) ** self.gamma
# Alpha weighting
if self.alpha is not None:
alpha_weight = torch.ones_like(targets).float()
alpha_weight[targets == 1] = self.alpha
focal_weight = focal_weight * alpha_weight
# Apply focal weight
loss = ce * focal_weight
# Ignore invalid pixels
valid_mask = (targets != self.ignore_index)
loss = loss[valid_mask].mean()
return loss
# 5. Tversky LOSS
class TverskyLoss(nn.Module):
"""
Tversky Loss: generalization of Dice.
Tversky = |A ∩ B| / (|A ∩ B| + α|A \ B| + β|B \ A|)
α, β control FP vs FN trade-off.
"""
def __init__(self, alpha=0.5, beta=0.5, smooth=1e-6):
super().__init__()
self.alpha = alpha
self.beta = beta
self.smooth = smooth
def forward(self, logits, targets):
probs = F.softmax(logits, dim=1)
num_classes = logits.shape[1]
targets_one_hot = F.one_hot(targets, num_classes)
targets_one_hot = targets_one_hot.permute(0, 3, 1, 2).float()
probs = probs.flatten(2)
targets_one_hot = targets_one_hot.flatten(2)
tp = (probs * targets_one_hot).sum(dim=2)
fp = probs.sum(dim=2) - tp
fn = targets_one_hot.sum(dim=2) - tp
tversky = (tp + self.smooth) / (tp + self.alpha * fp + self.beta * fn + self.smooth)
return 1 - tversky.mean()
# 6. COMBINED LOSS (BEST RESULTS)
class CombinedLoss(nn.Module):
"""
Combined CE + Dice loss.
Best of both worlds:
- CE: pixel-wise accuracy
- Dice: overlap quality
"""
def __init__(self, ce_weight=1.0, dice_weight=1.0):
super().__init__()
self.ce_weight = ce_weight
self.dice_weight = dice_weight
self.ce = nn.CrossEntropyLoss()
self.dice = DiceLoss()
def forward(self, logits, targets):
ce_loss = self.ce(logits, targets)
dice_loss = self.dice(logits, targets)
return self.ce_weight * ce_loss + self.dice_weight * dice_loss
# COMPARISON
print("=== Loss Function Comparison ===")
print("""
Cross-Entropy:
+ Standard, well-understood
+ Works for multi-class
- Biased toward majority class
- Bad for imbalanced
Dice:
+ Handles imbalance
+ Directly optimizes overlap
+ Works for binary
- Can be unstable early
- Multi-class needs care
IoU:
+ Matches evaluation metric
+ Handles imbalance
- Less smooth than Dice
- Slower convergence
Focal:
+ Handles extreme imbalance
+ Focus on hard examples
+ Works for detection too
- More hyperparameters
Tversky:
+ Tunable FP/FN trade-off
+ Generalization of Dice
- Extra hyperparameters
Combined (CE + Dice):
+ Best of both worlds
+ Stable training
+ Good final accuracy
+ Most common in practice
""")
# 7. PRACTICAL USAGE
print("\n=== Practical Usage ===")
print("""
For most tasks:
- Use CE + Dice combined
- Weights: 1:1 initially
- Tune based on validation
For class imbalance:
- Weight CE by class frequency
- Or use Focal loss
- Dice helps too
For medical imaging:
- CE + Dice standard
- Or Dice + Boundary loss
- Focus on boundaries
For autonomous driving:
- Weighted CE
- Or CE + Lovasz-Softmax
- Optimize mIoU directly
Loss weight tuning:
- If classes imbalanced: increase Dice weight
- If boundaries matter: add boundary loss
- If easy samples dominate: use Focal
Training tips:
- Start with CE alone
- Add Dice after a few epochs
- Or use combined from start
- Monitor both components
""")
# 8. DEMONSTRATION
print("\n=== Loss Behavior Demo ===")
# Simulate predictions
batch_size = 2
num_classes = 3
H, W = 64, 64
logits = torch.randn(batch_size, num_classes, H, W)
targets = torch.randint(0, num_classes, (batch_size, H, W))
# Compute losses
ce_loss = CrossEntropyLoss()(logits, targets)
dice_loss = DiceLoss()(logits, targets)
iou_loss = IoULoss()(logits, targets)
combined = CombinedLoss()(logits, targets)
print(f"CE Loss: {ce_loss.item():.4f}")
print(f"Dice Loss: {dice_loss.item():.4f}")
print(f"IoU Loss: {iou_loss.item():.4f}")
print(f"Combined Loss: {combined.item():.4f}")
# 9. BEST PRACTICES
print("\n=== Best Practices ===")
print("""
1. START SIMPLE
- Cross-entropy first
- Add Dice if needed
- Or use combined
2. HANDLE IGNORE INDEX
- Set ignore_index=255 for unlabeled
- Essential for datasets with unknown pixels
3. WEIGHT CLASSES
- Inverse frequency
- Or median frequency
- Especially for rare classes
4. MONITOR COMPONENTS
- Track CE and Dice separately
- Watch for divergence
- Adjust weights
5. GRADIENT BALANCE
- Combined loss can have scaling issues
- Normalize or use weights
- Watch for instability
6. VALIDATION
- Use mIoU as primary metric
- Compare with task-specific metric
- Evaluate per-class
7. LOSS SCHEDULING
- Start with CE
- Add Dice after warmup
- Or use combined from start
8. DON'T OVER-OPTIMIZE LOSS
- Sometimes CE is enough
- Focus on data quality
- Architecture matters more
""")📊 Evaluation Metrics
🎯 Measuring Segmentation Quality
Segmentation evaluation requires specialized metrics. IoU and Dice are standard, but mIoU, Pixel Accuracy, and Boundary F1 capture different aspects.
Key Metrics
IoU: Intersection over Union. Dice: F1-like overlap. mIoU: Mean across classes. Pixel Acc: Fraction correct. Boundary F1: Boundary quality. PQ: Panoptic Quality.
📊 IoU🎲 Dice📈 mIoU🎯 Boundary
# Segmentation evaluation metrics
import numpy as np
import torch
import torch.nn.functional as F
# 1. IoU (INTERSECTION OVER UNION)
def compute_iou(pred, target, num_classes, ignore_index=255):
"""
Compute IoU per class.
IoU = |A ∩ B| / |A ∪ B|
Range: [0, 1], higher is better
"""
ious = []
# Flatten
pred = pred.flatten()
target = target.flatten()
# Ignore invalid pixels
valid = target != ignore_index
pred = pred[valid]
target = target[valid]
for cls in range(num_classes):
pred_mask = (pred == cls)
target_mask = (target == cls)
intersection = (pred_mask & target_mask).sum()
union = (pred_mask | target_mask).sum()
if union == 0:
continue # Class not present
iou = intersection / union
ious.append(iou)
return np.mean(ious), ious
# 2. DICE COEFFICIENT
def compute_dice(pred, target, num_classes, ignore_index=255):
"""
Dice = 2|A ∩ B| / (|A| + |B|)
Range: [0, 1], higher is better
Related to F1 score
"""
dice_scores = []
pred = pred.flatten()
target = target.flatten()
valid = target != ignore_index
pred = pred[valid]
target = target[valid]
for cls in range(num_classes):
pred_mask = (pred == cls)
target_mask = (target == cls)
intersection = (pred_mask & target_mask).sum()
total = pred_mask.sum() + target_mask.sum()
if total == 0:
continue
dice = 2 * intersection / total
dice_scores.append(dice)
return np.mean(dice_scores)
# 3. PIXEL ACCURACY
def pixel_accuracy(pred, target, ignore_index=255):
"""Fraction of correctly classified pixels."""
valid = target != ignore_index
correct = (pred[valid] == target[valid]).sum()
total = valid.sum()
return correct / total
# 4. BOUNDARY F1
def boundary_f1(pred, target, tolerance=2):
"""
Boundary F1 score: F1 over boundary pixels.
More sensitive to boundary quality than IoU.
"""
from scipy.ndimage import binary_erosion
# Extract boundaries
pred_boundary = pred - binary_erosion(pred, iterations=1)
target_boundary = target - binary_erosion(target, iterations=1)
# Dilate for tolerance
from scipy.ndimage import binary_dilation
pred_dilated = binary_dilation(pred_boundary, iterations=tolerance)
target_dilated = binary_dilation(target_boundary, iterations=tolerance)
# Compute precision and recall
precision = (pred_boundary & target_dilated).sum() / max(pred_boundary.sum(), 1)
recall = (target_boundary & pred_dilated).sum() / max(target_boundary.sum(), 1)
f1 = 2 * precision * recall / max(precision + recall, 1e-6)
return f1
# 5. COMPLETE EVALUATION
class SegmentationMetrics:
"""Complete evaluation suite."""
def __init__(self, num_classes, ignore_index=255):
self.num_classes = num_classes
self.ignore_index = ignore_index
self.reset()
def reset(self):
# Confusion matrix
self.confusion_matrix = np.zeros(
(self.num_classes, self.num_classes), dtype=np.int64
)
def update(self, pred, target):
"""Update confusion matrix."""
# Flatten
pred = pred.flatten()
target = target.flatten()
# Ignore invalid
valid = target != self.ignore_index
pred = pred[valid]
target = target[valid]
# Update confusion matrix
for t, p in zip(target, pred):
self.confusion_matrix[t, p] += 1
def compute_iou(self):
"""Compute IoU per class from confusion matrix."""
intersection = np.diag(self.confusion_matrix)
union = (self.confusion_matrix.sum(axis=1) +
self.confusion_matrix.sum(axis=0) -
intersection)
# Avoid division by zero
iou = intersection / np.maximum(union, 1)
# Only consider present classes
present = self.confusion_matrix.sum(axis=1) > 0
return iou, iou[present].mean()
def compute_dice(self):
"""Compute Dice per class."""
intersection = np.diag(self.confusion_matrix)
total = (self.confusion_matrix.sum(axis=1) +
self.confusion_matrix.sum(axis=0))
dice = 2 * intersection / np.maximum(total, 1)
present = self.confusion_matrix.sum(axis=1) > 0
return dice, dice[present].mean()
def compute_pixel_accuracy(self):
"""Overall pixel accuracy."""
correct = np.diag(self.confusion_matrix).sum()
total = self.confusion_matrix.sum()
return correct / total
def compute_mean_accuracy(self):
"""Mean per-class accuracy."""
accuracies = np.diag(self.confusion_matrix) / \
np.maximum(self.confusion_matrix.sum(axis=1), 1)
present = self.confusion_matrix.sum(axis=1) > 0
return accuracies[present].mean()
def summary(self):
"""Print full evaluation summary."""
iou, miou = self.compute_iou()
dice, mdice = self.compute_dice()
pixel_acc = self.compute_pixel_accuracy()
mean_acc = self.compute_mean_accuracy()
print("=" * 50)
print("SEGMENTATION EVALUATION")
print("=" * 50)
print(f"Pixel Accuracy: {pixel_acc:.4f}")
print(f"Mean Accuracy: {mean_acc:.4f}")
print(f"mIoU: {miou:.4f}")
print(f"Mean Dice: {mdice:.4f}")
print("\nPer-class IoU:")
for cls, i in enumerate(iou):
if self.confusion_matrix[cls].sum() > 0:
print(f" Class {cls}: {i:.4f}")
print("=" * 50)
return {
'pixel_accuracy': pixel_acc,
'mean_accuracy': mean_acc,
'miou': miou,
'mean_dice': mdice,
'per_class_iou': iou
}
# 6. PANOPTIC QUALITY (PQ)
def panoptic_quality(pred_segments, gt_segments, num_classes):
"""
PQ = SQ × RQ
SQ (Segmentation Quality): average IoU of matched segments
RQ (Recognition Quality): F1 score of matched segments
"""
tp, fp, fn = 0, 0, 0
iou_sum = 0
for cls in range(num_classes):
pred_ids = np.unique(pred_segments[pred_segments // 1000 == cls])
gt_ids = np.unique(gt_segments[gt_segments // 1000 == cls])
matched_gt = set()
for pred_id in pred_ids:
pred_mask = (pred_segments == pred_id)
best_iou = 0
best_gt = None
for gt_id in gt_ids:
if gt_id in matched_gt:
continue
gt_mask = (gt_segments == gt_id)
intersection = (pred_mask & gt_mask).sum()
union = (pred_mask | gt_mask).sum()
iou = intersection / max(union, 1)
if iou > best_iou:
best_iou = iou
best_gt = gt_id
if best_iou > 0.5:
tp += 1
iou_sum += best_iou
matched_gt.add(best_gt)
else:
fp += 1
fn += len(gt_ids) - len(matched_gt)
sq = iou_sum / max(tp, 1)
rq = tp / max(tp + 0.5 * fp + 0.5 * fn, 1)
pq = sq * rq
return pq, sq, rq
# 7. METRIC COMPARISON
print("=== Metric Comparison ===")
print("""
IoU:
- Intersection over Union
- Most common metric
- Sensitive to boundaries
- Range: [0, 1]
Dice:
- F1-like overlap
- Similar to IoU
- Dice = 2*IoU / (1 + IoU)
- Range: [0, 1]
Pixel Accuracy:
- Simple fraction correct
- Misleading for imbalanced
- Not recommended alone
Mean IoU (mIoU):
- Average IoU across classes
- Standard benchmark metric
- Used in PASCAL VOC, Cityscapes
Boundary F1:
- Measures boundary quality
- More sensitive than IoU
- Good for medical imaging
Panoptic Quality (PQ):
- For panoptic segmentation
- Combines SQ and RQ
- Single number summary
""")
# 8. PRACTICAL EVALUATION
def evaluate_model(model, test_loader, num_classes=21, device='cuda'):
"""Complete model evaluation."""
model.eval()
model = model.to(device)
metrics = SegmentationMetrics(num_classes)
with torch.no_grad():
for images, masks in test_loader:
images = images.to(device)
masks = masks.to(device).long()
# Predict
logits = model(images)
preds = logits.argmax(dim=1) # (B, H, W)
# Update metrics
for pred, mask in zip(preds.cpu().numpy(), masks.cpu().numpy()):
metrics.update(pred, mask)
return metrics.summary()
# 9. PRACTICAL TIPS
print("=== Practical Tips ===")
print("""
1. REPORT MULTIPLE METRICS
- mIoU + pixel accuracy
- Per-class breakdown
- Boundary metrics
2. HANDLE EDGE CASES
- Classes with no ground truth
- Classes with no predictions
- Ignore index (255)
3. CROSS-VALIDATION
- Multiple folds
- Report mean ± std
- Statistical significance
4. PER-CLASS ANALYSIS
- Which classes are hard?
- Confusion between classes
- Error patterns
5. VISUAL EVALUATION
- Overlay predictions
- Compare with ground truth
- Qualitive assessment
6. STANDARD DATASETS
- Use official eval scripts
- Compare fairly
- Report on test set
7. INFERENCE SPEED
- FPS (frames per second)
- Latency (ms)
- Memory usage
8. EFFICIENCY
- FLOPs
- Parameters
- Model size
""")
# 10. BENCHMARK COMPARISON
print("\n=== Benchmark Scores ===")
print(f"{'Dataset':<20} {'Metric':<12} {'SOTA':<12} {'Typical':<12}")
print("-" * 60)
print(f"{'PASCAL VOC':<20} {'mIoU':<12} {'89.0%':<12} {'75.0%':<12}")
print(f"{'Cityscapes':<20} {'mIoU':<12} {'85.0%':<12} {'75.0%':<12}")
print(f"{'ADE20K':<20} {'mIoU':<12} {'62.0%':<12} {'45.0%':<12}")
print(f"{'COCO (inst)':<20} {'Mask AP':<12} {'52.0%':<12} {'35.0%':<12}")
print(f"{'COCO (pan)':<20} {'PQ':<12} {'58.0%':<12} {'42.0%':<12}")
print(f"{'BraTS (med)':<20} {'Dice':<12} {'92.0%':<12} {'80.0%':<12}")💼 Applications (Medical, Self-Driving)
🎯 Segmentation in the Real World
Segmentation powers critical applications in healthcare, autonomous driving, AR/VR, and more. Each domain has unique challenges and requirements.
Application Domains
Medical: Organ, tumor, cell segmentation. Self-driving: Road, pedestrians, vehicles. AR/VR: Background removal, occlusion. Satellite: Land cover, buildings. Retail: Product, shelf. Robotics: Objects, scenes.
🏥 Medical🚗 Self-driving🥽 AR/VR🛰️ Satellite
# Applications of segmentation
import torch
import numpy as np
# 1. MEDICAL IMAGING
print("=== Medical Imaging ===")
print("""
Applications:
- Organ segmentation
* Kidney, liver, heart
* Surgical planning
* Radiation therapy
- Tumor segmentation
* Brain tumors (BraTS)
* Lung nodules
* Skin lesions (ISIC)
- Cell segmentation
* Nuclei (MoNuSeg)
* Cell tracking
* Pathology
Special considerations:
- Small datasets (100s of images)
- High resolution (512-2048 px)
- 3D volumes (CT, MRI)
- Class imbalance (small lesions)
- High accuracy required
Approaches:
- U-Net (still SOTA for many tasks)
- nnU-Net (auto-configured)
- DeepLab variants
- Attention U-Net
- 3D U-Net for volumes
Evaluation:
- Dice coefficient (main metric)
- IoU/Jaccard
- Hausdorff distance
- Clinical validation
Datasets:
- BraTS: Brain tumors
- KiTS: Kidney tumors
- LiTS: Liver tumors
- ISIC: Skin lesions
- MoNuSeg: Nuclei
""")
# 2. AUTONOMOUS DRIVING
print("\n=== Autonomous Driving ===")
print("""
Applications:
- Road segmentation
- Lane detection
- Pedestrian segmentation
- Vehicle segmentation
- Traffic sign segmentation
- Drivable area
Requirements:
- Real-time (30+ FPS)
- High accuracy (safety critical)
- All weather conditions
- Long-range perception
- Robust to sensor noise
Architectures:
- Efficient: ENet, ERFNet
- Accurate: DeepLab, PSPNet
- Transformers: SegFormer
- Real-time: BiSeNet, DDRNet
Datasets:
- Cityscapes: 5,000 images, 19 classes
- BDD100K: 10,000 images
- Mapillary Vistas: 25,000 images
- ApolloScape
- A2D2
Special challenges:
- Panoramic (multiple cameras)
- Temporal consistency
- Sensor fusion (LiDAR + camera)
- Edge deployment (embedded GPU)
- Fog, rain, night
""")
# 3. AR/VR
print("\n=== AR/VR ===")
print("""
Applications:
- Background removal
- Portrait mode
- Human segmentation
- Hand/face segmentation
- Real-time occlusion
- Mixed reality scenes
Requirements:
- Real-time (60+ FPS)
- Low latency (<20ms)
- Mobile-friendly
- Low power
- High quality edges
Architectures:
- MobileNetV3 + DeepLab
- PortraitNet
- MODNet (real-time matting)
- SAM (for high quality)
- BiRefNet (background removal)
Special challenges:
- Hair/fur matting
- Semi-transparent objects
- Motion blur
- Real-time constraints
- Limited compute
Real examples:
- iPhone Portrait Mode
- Zoom background blur
- Snapchat filters
- OBS virtual background
- AR occlusion
""")
# 4. SATELLITE IMAGERY
print("\n=== Satellite Imagery ===")
print("""
Applications:
- Land cover classification
- Building detection
- Crop monitoring
- Forest mapping
- Water body detection
- Change detection
- Disaster assessment
Requirements:
- Large images (10000+ px)
- Multi-spectral input
- Temporal (multiple dates)
- Global scale
- Robust to clouds
Datasets:
- SpaceNet (buildings)
- DeepGlobe (roads, buildings)
- LoveDA (land cover)
- BigEarthNet
- Open Cities AI
Approaches:
- U-Net for buildings
- Semantic segmentation for land cover
- Multi-temporal for change
- Self-supervised for features
Special challenges:
- Very large images (tiling)
- Multi-spectral (beyond RGB)
- Class imbalance
- Different resolutions
- Geographic domain shift
""")
# 5. RETAIL
print("\n=== Retail ===")
print("""
Applications:
- Product segmentation
- Shelf monitoring
- Inventory management
- Cashier-less stores
- Planogram compliance
- Customer analytics
Requirements:
- High accuracy (product ID)
- Crowded scenes (many objects)
- Similar products
- Real-time (shelf cameras)
Approaches:
- Instance segmentation
- Mask R-CNN
- YOLACT (real-time)
- SAM for zero-shot
Special challenges:
- Similar objects (same product)
- Occlusion (stacked items)
- Various lighting
- High object density
Real examples:
- Amazon Go
- Trigo (automated checkout)
- Trax (shelf analytics)
- Focal Systems
""")
# 6. INDUSTRIAL
print("\n=== Industrial ===")
print("""
Applications:
- Quality control
- Defect detection
- Surface inspection
- Assembly verification
- Sorting
- Robotic grasping
Requirements:
- Very high accuracy
- Fast inference
- Robust to variations
- 24/7 operation
Approaches:
- U-Net for defects
- DeepLab for surfaces
- Anomaly detection
- Few-shot methods
Special challenges:
- Small defects
- Class imbalance
- High variability
- Real-time (production line)
""")
# 7. AGRICULTURE
print("\n=== Agriculture ===")
print("""
Applications:
- Crop segmentation
- Weed detection
- Disease detection
- Yield estimation
- Precision spraying
Approaches:
- Semantic segmentation (crop vs weed)
- Instance segmentation (plants)
- Multi-spectral (NDVI)
Platforms:
- Drones (aerial)
- Tractors (ground)
- Satellites (large scale)
Special challenges:
- Outdoor lighting
- Similar visual appearance
- Small objects (seedlings)
- Real-time on tractors
""")
# 8. CASE STUDY: MEDICAL SEGMENTATION
print("\n=== Case Study: Medical Segmentation ===")
print("""
Task: Brain tumor segmentation (BraTS)
Dataset:
- 4 MRI modalities (T1, T1ce, T2, FLAIR)
- 3D volumes (240x240x155)
- 3 tumor classes (ET, ED, NCR)
- 285 training cases
Approach:
1. Preprocessing:
- Skull stripping
- Normalization
- Patch extraction (128³)
2. Architecture:
- 3D U-Net
- Residual blocks
- Deep supervision
3. Training:
- Dice + CE loss
- Data augmentation (flip, rotate, noise)
- 5-fold cross-validation
4. Inference:
- Sliding window
- Test-time augmentation
- Ensemble of folds
Results:
- Dice: ~90% for whole tumor
- Dice: ~80% for tumor core
- Dice: ~75% for enhancing tumor
Key insights:
- 3D context essential
- Multi-modal input crucial
- Data augmentation critical
- Ensemble improves results
""")
# 9. CASE STUDY: SELF-DRIVING
print("\n=== Case Study: Self-Driving ===")
print("""
Task: Urban scene segmentation (Cityscapes)
Dataset:
- 5,000 finely annotated images
- 20,000 coarsely annotated
- 19 classes (road, car, person, etc.)
- 1024x2048 resolution
Approach:
1. Backbone: DeepLabV3+ or SegFormer
2. Training:
- Cross-entropy + Dice
- Random crop (768x768)
- Flip, scale augmentation
- Class weighting
3. Inference:
- Sliding window
- Multi-scale
- Ensemble
Results:
- mIoU: ~85% (SOTA)
- FPS: 5-30 depending on model
Key insights:
- High resolution helps
- Class balance matters
- Boundary quality critical
- Real-time constraints
- Safety requires accuracy
""")
# 10. PRODUCTION CONSIDERATIONS
print("\n=== Production Considerations ===")
print("""
1. LATENCY
- Real-time: 30+ FPS
- Interactive: 10+ FPS
- Batch: any speed
2. ACCURACY
- Safety critical: 99%+
- Medical: 95%+
- General: 90%+
3. ROBUSTNESS
- Different conditions
- Edge cases
- Adversarial inputs
4. RESOURCES
- Edge device (Jetson, mobile)
- Server (data center)
- Cloud (API)
5. PRIVACY
- On-device processing
- Federated learning
- No data leaves device
6. MONITORING
- Performance tracking
- Drift detection
- Error analysis
7. UPDATES
- Model versioning
- A/B testing
- Continuous improvement
""")🚀 Future: Foundation Segmentation
🎯 The Next Generation
Segmentation is being transformed by foundation models. SAM, DINOv2, Grounding DINO, and Grounded-SAM enable zero-shot segmentation of anything.
Future Directions
Foundation: One model for all segmentation. Zero-shot: No training needed. Promptable: Any input modality. Video: Consistent across frames. 3D: Native 3D segmentation. Real-time: Efficient on edge.
🎯 SAM🦖 DINOv2📝 Grounded-SAM🎬 Video
# Future of segmentation — foundation models
import torch
# 1. SAM (SEGMENT ANYTHING)
print("=== SAM: Segment Anything ===")
print("""
SAM (Meta AI, 2023):
- Trained on 1.1B masks
- 11M images
- Zero-shot on any image
- Promptable (point, box, text)
Architecture:
- Image encoder: ViT-H (heavy)
- Prompt encoder: points, boxes, text
- Mask decoder: lightweight
Capabilities:
✓ Segment any object
✓ Zero-shot
✓ Works for any domain
✓ Promptable interface
Limitations:
✗ Slow image encoder (once per image)
✗ Needs prompt
✗ Large model (2.4GB)
✗ Not semantic (no classes)
SAM 2 (2024):
- Video support
- Temporal consistency
- Better masks
- Streaming
Code:
from segment_anything import SamPredictor, sam_model_registry
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
predictor = SamPredictor(sam)
predictor.set_image(image)
masks, scores, _ = predictor.predict(
point_coords=np.array([[500, 375]]),
point_labels=np.array([1]),
multimask_output=True
)
""")
# 2. DINOv2 (Self-Supervised Features)
print("\n=== DINOv2 ===")
print("""
DINOv2 (Meta AI, 2023):
- Self-supervised vision transformer
- No labels needed
- Universal features
- State-of-the-art
Capabilities:
- Dense feature extraction
- Semantic segmentation (linear probe)
- Depth estimation
- Instance retrieval
- Zero-shot
Why it matters:
- No fine-tuning needed
- Transfer to any task
- Best features available
- Open source
For segmentation:
- Extract features
- Train simple head
- Often SOTA with minimal training
Code:
import torch
from transformers import AutoImageProcessor, AutoModel
processor = AutoImageProcessor.from_pretrained("facebook/dinov2-base")
model = AutoModel.from_pretrained("facebook/dinov2-base")
inputs = processor(images=image, return_tensors="pt")
outputs = model(**inputs)
features = outputs.last_hidden_state # (B, N, 768)
""")
# 3. GROUNDING DINO + SAM
print("\n=== Grounded-SAM ===")
print("""
Grounded-SAM combines:
- Grounding DINO: text-to-box
- SAM: box-to-mask
Capabilities:
- Text-prompted segmentation
- "Segment the cat"
- "Segment all cars"
- Zero-shot, any class
Workflow:
1. Text prompt: "car. person. tree."
2. Grounding DINO: detect boxes
3. SAM: segment each box
4. Result: labeled masks
Code:
from groundingdino.util.inference import load_model, predict
from segment_anything import SamPredictor
# Detect with text
boxes, logits, phrases = predict(
model=grounding_dino,
image=image,
caption="car. person. tree."
)
# Segment with SAM
predictor.set_image(image)
for box in boxes:
mask, _, _ = predictor.predict(box=box)
""")
# 4. OTHER FOUNDATION MODELS
print("\n=== Other Foundation Models ===")
print("""
CLIP (2021):
- Image-text alignment
- Zero-shot classification
- Guidance for segmentation
DINO (2021):
- Self-supervised ViT
- Emergent segmentation
- No labels needed
MAE (2021):
- Masked autoencoders
- Reconstruction-based
- Good features
BEiT (2021):
- BERT for images
- Masked image modeling
Swin (2021):
- Hierarchical ViT
- Efficient attention
- Backbone for many methods
ViT-Adapter (2022):
- ViT + adapter
- Dense prediction
- Better for segmentation
Mask2Former (2022):
- Universal segmentation
- SOTA on all tasks
OneFormer (2023):
- One model for all tasks
- Task-conditioned
SAM 2 (2024):
- Video segmentation
- Streaming
- Temporal consistency
""")
# 5. VIDEO SEGMENTATION
print("\n=== Video Segmentation ===")
print("""
Task: Segment objects in video (temporally consistent).
Approaches:
- Tracking + segmentation
- Mask propagation
- Video object segmentation (VOS)
- SAM 2 (video)
Challenges:
- Temporal consistency
- Occlusions
- Fast motion
- Real-time
Methods:
- XMem (2023)
- DEVA (2023)
- SAM 2 (2024)
- Cutie (2024)
Applications:
- Video editing
- Autonomous driving
- Surveillance
- Sports analysis
- AR/VR
SAM 2 for video:
- Streaming memory
- Real-time
- Zero-shot
- Interactive
Code (conceptual):
from sam2 import SAM2VideoPredictor
predictor = SAM2VideoPredictor()
predictor.set_video(video)
masks = predictor.track(prompt_point, frame_idx=0)
""")
# 6. 3D SEGMENTATION
print("\n=== 3D Segmentation ===")
print("""
Tasks:
- Point cloud segmentation
- Volumetric segmentation (CT, MRI)
- 3D scene understanding
- Mesh segmentation
Approaches:
- 3D U-Net (medical)
- PointNet++ (point clouds)
- MinkowskiNet (sparse)
- SAM3D (2D to 3D)
- Segment3D
Foundation models:
- SAM (2D projections)
- DINOv2 (features)
- Point-E / Shap-E
Applications:
- Medical imaging
- Autonomous driving (LiDAR)
- Robotics
- AR/VR
Challenges:
- Data sparsity
- Computational cost
- Rotation invariance
- Annotation cost
""")
# 7. INTERACTIVE SEGMENTATION
print("\n=== Interactive Segmentation ===")
print("""
Task: Segment with human-in-the-loop.
Approaches:
- Point prompts (SAM)
- Box prompts (SAM)
- Scribble prompts
- Text prompts
Tools:
- SAM (click to segment)
- SAM 2 (video)
- Grounded-SAM (text)
- SEEM (multi-modal)
- ClickSEG
Applications:
- Photo editing
- Medical annotation
- Robotics
- AR/VR
- Content creation
Future:
- Real-time interactive
- Multi-modal prompts
- Zero-shot
- On-device
""")
# 8. EFFICIENT SEGMENTATION
print("\n=== Efficient Segmentation ===")
print("""
Trends:
1. SMALL MODELS
- MobileSAM (40MB)
- FastSAM (68MB)
- EfficientSAM
- Edge deployment
2. QUANTIZATION
- INT8
- INT4
- ONNX Runtime
- TensorRT
3. DISTILLATION
- Teacher → Student
- Lightweight models
- Preserved accuracy
4. PRUNING
- Remove weights
- Sparse models
- Hardware friendly
5. REAL-TIME
- BiSeNet
- DDRNet
- PIDNet
- 100+ FPS
Goal:
- SAM quality
- Real-time speed
- On-device
- Low power
""")
# 9. FOUNDATION MODELS IMPACT
print("\n=== Impact of Foundation Models ===")
print("""
Transformation:
Before: Train per task, per domain
After: One model, any task
Benefits:
✓ No training needed for many tasks
✓ Zero-shot capabilities
✓ Better generalization
✓ Reduced annotation cost
New workflows:
- Prompt-based segmentation
- Interactive annotation
- Auto-labeling pipelines
- Few-shot adaptation
Challenges:
- Model size (2.4GB for SAM)
- Inference speed
- Domain-specific accuracy
- Fine-tuning cost
Future:
- Smaller foundation models
- Better efficiency
- More modalities
- Real-time everything
""")
# 10. OPEN PROBLEMS
print("\n=== Open Problems ===")
print("""
1. UNIFIED SEGMENTATION
- One model for all tasks
- Semantic + instance + panoptic
- Video + 3D
- Any modality
2. EFFICIENCY
- Real-time foundation models
- On-device
- Low power
3. ACCURACY
- Boundary precision
- Small objects
- Ambiguous cases
- Domain-specific
4. INTERPRETABILITY
- Understanding predictions
- Explainability
- Trust
5. INTERACTION
- Natural interfaces
- Multi-modal prompts
- Conversational
6. GENERALIZATION
- Cross-domain
- Cross-task
- Zero-shot
7. ANNOTATION
- Auto-labeling
- Active learning
- Semi-supervised
""")
# 11. RECOMMENDATIONS
print("\n=== Recommendations ===")
print("""
For practitioners:
1. Start with SAM for zero-shot
2. Fine-tune SegFormer for domain
3. Use Grounded-SAM for text
4. Consider DINOv2 features
5. Profile before deploying
For researchers:
1. Study foundation models
2. Efficient architectures
3. Video + 3D
4. Interactive methods
5. Evaluation benchmarks
For industry:
1. Auto-labeling pipelines
2. Active learning
3. On-device deployment
4. Continuous improvement
5. Privacy-preserving
For learners:
1. Understand U-Net first
2. Master one framework (PyTorch)
3. Experiment with SAM
4. Learn PEFT methods
5. Build projects
""")
# 12. FINAL THOUGHTS
print("\n=== Final Thoughts ===")
print("""
Segmentation has evolved:
1980s-2000s: Classical methods
- Thresholding
- Region growing
- Graph cuts
2010s: Deep learning era
- FCN (2015)
- U-Net (2015)
- DeepLab (2016)
- Mask R-CNN (2017)
2020s: Transformers + Foundation
- SegFormer (2021)
- Mask2Former (2022)
- SAM (2023)
- Grounded-SAM (2023)
- SAM 2 (2024)
Key trends:
1. Foundation models dominate
2. Zero-shot is now reality
3. Interactive is powerful
4. Video + 3D
5. Efficiency matters
What's next:
- Better foundation models
- Real-time everything
- Multi-modal
- On-device
- Universal segmentation
You now understand:
- Fundamentals of segmentation
- All major architectures
- Loss functions
- Evaluation metrics
- Real-world applications
- Future directions
"Segmentation is the key to
understanding the visual world,
pixel by pixel."
— Anonymous
Go build the future of visual AI!
""")