🎯 Object Detection Fundamentals

🎯 What is Object Detection?

Object detection is a computer vision task that identifies and locates objects in images or videos. It provides both classification (what) and localization (where) of objects.

Key concept: Object detection outputs bounding boxes with class labels and confidence scores for each detected object.

Deep Explanation

Object detection is a fundamental problem in computer vision. It combines classification and regression — classifying objects and regressing their bounding box coordinates.

Why it matters: Object detection enables autonomous driving, surveillance, medical imaging, and augmented reality applications.

🎯 Classification📊 Localization📋 Bounding Boxes🔄 Confidence

# Object detection inference example
import torch
import torchvision

def detect_objects(image, model, threshold=0.5):
    # Preprocess image
    image_tensor = preprocess(image)
    
    # Run inference
    with torch.no_grad():
        predictions = model(image_tensor)
    
    # Extract predictions
    boxes = predictions[0]['boxes']
    scores = predictions[0]['scores']
    labels = predictions[0]['labels']
    
    # Filter by threshold
    mask = scores > threshold
    boxes = boxes[mask]
    scores = scores[mask]
    labels = labels[mask]
    
    # Draw bounding boxes
    for box, score, label in zip(boxes, scores, labels):
        draw_box(image, box, label, score)
    
    return boxes, scores, labels

📊 Image Classification

🎯 Classification Foundation

Image classification assigns a single label to an entire image. It's the foundation for object detection, which adds localization.

Classification Details

Architectures: CNNs like ResNet, VGG, EfficientNet. Output: Probability distribution over classes. Training: Cross-entropy loss on labeled images.

Relation to detection: Object detection uses classification to classify each object candidate.

📊 CNN📋 Probabilities🔄 ResNet📈 Cross-Entropy

# Image classification (PyTorch)
import torch.nn as nn

class ImageClassifier(nn.Module):
    def __init__(self, num_classes=1000):
        super().__init__()
        self.features = nn.Sequential(
            nn.Conv2d(3, 64, 7, stride=2),
            nn.ReLU(),
            nn.MaxPool2d(3, stride=2),
            # ... more layers
        )
        self.classifier = nn.Sequential(
            nn.Linear(2048, 512),
            nn.ReLU(),
            nn.Linear(512, num_classes)
        )
    
    def forward(self, x):
        x = self.features(x)
        x = x.mean([2, 3])  # Global average pooling
        return self.classifier(x)

# Usage
model = ImageClassifier(num_classes=1000)
output = model(image)  # Shape: (batch_size, 1000)
probs = torch.softmax(output, dim=1)

📍 Localization

🎯 Locating Objects

Localization predicts the bounding box coordinates of an object. It outputs (x, y, w, h) or (x1, y1, x2, y2).

Localization Methods

Regression: Directly predict box coordinates. Anchor boxes: Predict offsets from predefined boxes. Heatmaps: Predict center points and box dimensions.

Loss: L1/L2 loss for coordinates, IoU loss for box overlap.

📍 Regression📊 Anchors📋 Heatmaps🔄 IoU

# Bounding box operations
class BoundingBox:
    def __init__(self, x1, y1, x2, y2):
        self.x1 = x1
        self.y1 = y1
        self.x2 = x2
        self.y2 = y2
    
    def area(self):
        return (self.x2 - self.x1) * (self.y2 - self.y1)
    
    def iou(self, other):
        # Intersection over Union
        x1 = max(self.x1, other.x1)
        y1 = max(self.y1, other.y1)
        x2 = min(self.x2, other.x2)
        y2 = min(self.y2, other.y2)
        
        if x2 <= x1 or y2 <= y1:
            return 0.0
        
        intersection = (x2 - x1) * (y2 - y1)
        union = self.area() + other.area() - intersection
        return intersection / union
    
    def to_xywh(self):
        # Convert to center x, center y, width, height
        cx = (self.x1 + self.x2) / 2
        cy = (self.y1 + self.y2) / 2
        w = self.x2 - self.x1
        h = self.y2 - self.y1
        return cx, cy, w, h

🔄 Sliding Window

🎯 Traditional Approach

Sliding window is a traditional approach where a classifier is applied to every possible window at multiple scales and locations.

Sliding Window Details

Process: Slide a window across the image, classify each window. Multi-scale: Use different window sizes. Computational cost: Very expensive, especially with many classes.

Improvements: Use image pyramids, efficient classifiers, and region proposals.

🔄 Window📊 Multi-Scale📋 Expensive🔄 Pyramids

# Sliding window implementation
def sliding_window(image, window_size, step_size):
    # Generate windows at multiple scales
    windows = []
    scales = [1.0, 0.75, 0.5, 0.25]
    
    for scale in scales:
        scaled_image = resize(image, scale)
        h, w = scaled_image.shape[:2]
        window_h, window_w = window_size
        
        for y in range(0, h - window_h + 1, step_size):
            for x in range(0, w - window_w + 1, step_size):
                window = scaled_image[y:y+window_h, x:x+window_w]
                windows.append((window, x, y, scale))
    
    return windows

# Apply classifier to each window
def detect_with_sliding(image, classifier):
    windows = sliding_window(image, (64, 64), 16)
    detections = []
    
    for window, x, y, scale in windows:
        score = classifier.predict(window)
        if score > threshold:
            detections.append({
                'box': (x, y, x+64, y+64),
                'score': score,
                'scale': scale
            })
    
    return detections

📋 Region-Based CNN

🎯 R-CNN Family

R-CNN uses region proposals to reduce the number of windows. It evolved through Fast R-CNN, Faster R-CNN, and Mask R-CNN.

R-CNN Evolution

R-CNN: Extract region proposals, warp them, classify each. Fast R-CNN: Single forward pass for feature extraction. Faster R-CNN: Region Proposal Network (RPN) integrates proposal generation. Mask R-CNN: Adds instance segmentation.

Performance: High accuracy, but slower than one-stage detectors.

📋 Proposals📊 RPN📋 Fast R-CNN🔄 Mask R-CNN

# Faster R-CNN architecture (simplified)
class FasterRCNN(nn.Module):
    def __init__(self, backbone, num_classes):
        super().__init__()
        self.backbone = backbone
        self.rpn = RPN(backbone.out_channels)
        self.roi_head = RoIHead(backbone.out_channels, num_classes)
    
    def forward(self, x):
        # Feature extraction
        features = self.backbone(x)
        
        # Region Proposal Network
        proposals, rpn_losses = self.rpn(features)
        
        # ROI Pooling and classification
        detections, roi_losses = self.roi_head(features, proposals)
        
        return detections, rpn_losses, roi_losses

class RPN(nn.Module):
    def __init__(self, in_channels):
        super().__init__()
        self.conv = nn.Conv2d(in_channels, 512, 3, padding=1)
        self.classifier = nn.Conv2d(512, 2 * 9, 1)  # 2 classes × 9 anchors
        self.regressor = nn.Conv2d(512, 4 * 9, 1)    # 4 coords × 9 anchors
        self.anchors = generate_anchors()
    
    def forward(self, features):
        # Generate proposals
        scores = self.classifier(self.conv(features))
        offsets = self.regressor(self.conv(features))
        proposals = self.decode_anchors(self.anchors, offsets)
        return proposals

⚡ YOLO

🎯 You Only Look Once

YOLO (You Only Look Once) is a one-stage detector that predicts bounding boxes and class probabilities directly from the full image in a single forward pass.

YOLO Details

YOLOv1: Divides image into grid, each cell predicts boxes. YOLOv2/v3: Added anchor boxes, multi-scale predictions. YOLOv4/v5/v8: Continual improvements in speed and accuracy.

Advantages: Very fast (real-time), simple architecture.

⚡ One-Stage📊 Grid📋 Real-Time🔄 Versions

# YOLO-style detection (simplified)
class YOLODetector:
    def __init__(self, grid_size=7, num_boxes=2, num_classes=20):
        self.grid_size = grid_size
        self.num_boxes = num_boxes
        self.num_classes = num_classes
    
    def decode_predictions(self, predictions):
        # predictions shape: (batch, grid, grid, num_boxes * 5 + num_classes)
        batch_size = predictions.shape[0]
        detections = []
        
        for b in range(batch_size):
            for i in range(self.grid_size):
                for j in range(self.grid_size):
                    cell_pred = predictions[b, i, j]
                    
                    for box_idx in range(self.num_boxes):
                        # Extract box coordinates
                        cx = (j + cell_pred[box_idx*5]) / self.grid_size
                        cy = (i + cell_pred[box_idx*5+1]) / self.grid_size
                        w = cell_pred[box_idx*5+2]
                        h = cell_pred[box_idx*5+3]
                        conf = cell_pred[box_idx*5+4]
                        
                        if conf > 0.5:
                            # Class scores
                            class_scores = cell_pred[self.num_boxes*5:]
                            class_id = class_scores.argmax()
                            class_prob = class_scores[class_id]
                            
                            detections.append({
                                'box': (cx-w/2, cy-h/2, cx+w/2, cy+h/2),
                                'class': class_id,
                                'confidence': conf * class_prob
                            })
        
        return detections

🚀 SSD

🎯 Single Shot MultiBox Detector

SSD (Single Shot MultiBox Detector) is another one-stage detector that uses multi-scale feature maps and anchor boxes.

SSD Features

Multi-scale: Predictions from different feature map layers. Anchors: Different aspect ratios and scales. Speed: Faster than YOLO (original). Accuracy: Comparable to two-stage detectors.

Architecture: Base network (e.g., VGG) with extra convolutional layers.

🚀 Multi-Scale📊 Anchors📋 Fast🔄 Accurate

# SSD architecture (simplified)
class SSD(nn.Module):
    def __init__(self, base_network, num_classes, aspect_ratios):
        super().__init__()
        self.base = base_network
        self.extra_layers = self.create_extra_layers()
        self.classifiers = nn.ModuleList()
        self.regressors = nn.ModuleList()
        self.aspect_ratios = aspect_ratios
        
        # Create prediction heads for each feature map
        for feature_map in self.get_feature_maps():
            n_channels = feature_map.shape[1]
            self.classifiers.append(
                nn.Conv2d(n_channels, len(aspect_ratios) * num_classes, 1)
            )
            self.regressors.append(
                nn.Conv2d(n_channels, len(aspect_ratios) * 4, 1)
            )
    
    def forward(self, x):
        features = []
        for layer in [self.base, self.extra_layers]:
            x = layer(x)
            features.append(x)
        
        predictions = []
        for feat, cls_head, reg_head in zip(features, self.classifiers, self.regressors):
            # Apply prediction heads
            class_pred = cls_head(feat)
            box_pred = reg_head(feat)
            predictions.append((class_pred, box_pred))
        
        return predictions

🏔️ FPN

🎯 Feature Pyramid Network

FPN (Feature Pyramid Network) builds a feature pyramid that improves detection of objects at different scales.

FPN Details

Bottom-up: Standard CNN backbone (ResNet). Top-down: Upsample higher-level features. Lateral connections: Combine bottom-up and top-down features. Multi-scale: Each level handles different object sizes.

Integration: Used in RetinaNet, Mask R-CNN, and many modern detectors.

🏔️ Pyramid📊 Top-Down📋 Lateral🔄 Multi-Scale

# FPN implementation (simplified)
class FPN(nn.Module):
    def __init__(self, backbone):
        super().__init__()
        self.backbone = backbone
        
        # Lateral connections
        self.lateral = nn.ModuleList([
            nn.Conv2d(c, 256, 1) for c in [2048, 1024, 512, 256]
        ])
        
        # Top-down outputs
        self.output = nn.ModuleList([
            nn.Conv2d(256, 256, 3, padding=1) for _ in range(4)
        ])
    
    def forward(self, x):
        # Bottom-up features
        c5, c4, c3, c2 = self.backbone(x)  # from deepest to shallowest
        
        # Top-down path
        p5 = self.lateral[0](c5)
        p4 = self.lateral[1](c4) + nn.Upsample(scale_factor=2)(p5)
        p3 = self.lateral[2](c3) + nn.Upsample(scale_factor=2)(p4)
        p2 = self.lateral[3](c2) + nn.Upsample(scale_factor=2)(p3)
        
        # Output features
        p5 = self.output[0](p5)
        p4 = self.output[1](p4)
        p3 = self.output[2](p3)
        p2 = self.output[3](p2)
        
        return p2, p3, p4, p5

📐 Anchor Boxes

🎯 Anchor-Based Detection

Anchor boxes are predefined bounding boxes used as references. The model predicts offsets from these anchors.

Anchor Design

Scales: Different sizes (e.g., 32, 64, 128, 256). Aspect ratios: 1:1, 1:2, 2:1. Placement: Centered at feature map locations. K-means clustering: Used in YOLO to determine anchor sizes from training data.

Positive anchors: Those with high IoU with ground truth boxes.

📐 Scales📊 Ratios📋 K-Means🔄 Offsets

# Anchor generation
def generate_anchors(scales=[32, 64, 128], aspect_ratios=[0.5, 1, 2]):
    anchors = []
    for scale in scales:
        for ratio in aspect_ratios:
            w = scale * math.sqrt(ratio)
            h = scale / math.sqrt(ratio)
            anchors.append((-w/2, -h/2, w/2, h/2))
    return anchors

# K-means for anchor clustering (YOLO)
def kmeans_anchors(boxes, k=9):
    # boxes: list of (w, h) from dataset
    # Use k-means to find optimal anchor sizes
    
    # Initialize centroids randomly
    centroids = random.sample(boxes, k)
    
    for _ in range(100):
        # Assign to clusters
        clusters = [[] for _ in range(k)]
        for box in boxes:
            distances = [iou_distance(box, c) for c in centroids]
            closest = distances.index(min(distances))
            clusters[closest].append(box)
        
        # Update centroids
        for i in range(k):
            if clusters[i]:
                centroids[i] = np.mean(clusters[i], axis=0)
    
    return centroids

🚫 Non-Max Suppression

🎯 Removing Duplicate Detections

Non-Max Suppression (NMS) removes duplicate detections by keeping only the highest-scoring box among overlapping ones.

NMS Process

Step 1: Sort detections by confidence score. Step 2: Select the highest-scoring box. Step 3: Remove boxes with IoU > threshold. Step 4: Repeat until all boxes processed.

Variants: Soft-NMS (weights scores instead of removing), NMS with adaptive thresholds.

🚫 Sort📊 Select📋 Remove🔄 Soft-NMS

# Non-Max Suppression implementation
def non_max_suppression(boxes, scores, iou_threshold=0.5):
    # boxes: list of (x1, y1, x2, y2)
    # scores: list of confidence scores
    
    if len(boxes) == 0:
        return []
    
    # Sort by score descending
    indices = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)
    
    keep = []
    while indices:
        # Keep the best
        current = indices.pop(0)
        keep.append(current)
        
        # Remove boxes with high IoU
        to_remove = []
        for i, idx in enumerate(indices):
            iou = compute_iou(boxes[current], boxes[idx])
            if iou > iou_threshold:
                to_remove.append(i)
        
        # Remove in reverse order
        for i in reversed(to_remove):
            indices.pop(i)
    
    return keep

# Soft-NMS (differentiable)
def soft_nms(boxes, scores, iou_threshold=0.5, sigma=0.5):
    # Weight scores instead of removing
    keep = []
    while boxes:
        # Best box
        best_idx = scores.argmax()
        keep.append(best_idx)
        
        # Update scores of remaining boxes
        for i in range(len(boxes)):
            if i != best_idx:
                iou = compute_iou(boxes[best_idx], boxes[i])
                if iou > iou_threshold:
                    scores[i] *= np.exp(-(iou**2) / sigma)
        
        # Remove the best box
        boxes = np.delete(boxes, best_idx, axis=0)
        scores = np.delete(scores, best_idx)
    
    return keep

📊 Evaluation Metrics

🎯 Measuring Performance

Common metrics include mAP (mean Average Precision), IoU, precision, recall, and F1-score.

Metric Details

mAP: Area under the precision-recall curve, averaged over classes. IoU threshold: Typically 0.5 (mAP@0.5) or multiple thresholds (mAP@[0.5:0.95]). Precision: True positives / (True positives + False positives). Recall: True positives / (True positives + False negatives).

F1-score: Harmonic mean of precision and recall.

📊 mAP📋 IoU🔄 Precision📈 Recall

# mAP calculation
def compute_map(predictions, ground_truth, iou_threshold=0.5):
    # predictions: list of (boxes, scores, labels)
    # ground_truth: list of (boxes, labels)
    
    # Group by image
    image_results = {}
    for pred, gt in zip(predictions, ground_truth):
        # Compute true/false positives for each class
        pass
    
    # Compute AP per class
    aps = []
    for class_id in classes:
        # Get predictions and ground truth for this class
        class_preds = []
        class_gts = []
        
        # Sort by score
        class_preds.sort(key=lambda x: x['score'], reverse=True)
        
        # Compute precision-recall curve
        tp = np.zeros(len(class_preds))
        fp = np.zeros(len(class_preds))
        for i, pred in enumerate(class_preds):
            # Check if any ground truth matches
            matched = False
            for gt in class_gts:
                if compute_iou(pred['box'], gt['box']) > iou_threshold:
                    if not gt['matched']:
                        gt['matched'] = True
                        matched = True
                        break
            if matched:
                tp[i] = 1
            else:
                fp[i] = 1
        
        # Compute precision and recall
        tp_cumsum = np.cumsum(tp)
        fp_cumsum = np.cumsum(fp)
        recall = tp_cumsum / len(class_gts)
        precision = tp_cumsum / (tp_cumsum + fp_cumsum)
        
        # Compute AP (area under PR curve)
        ap = np.trapz(precision, recall)
        aps.append(ap)
    
    return np.mean(aps)

📚 Datasets

🎯 Object Detection Datasets

Popular datasets include COCO, Pascal VOC, Open Images, and KITTI.

Dataset Details

COCO: Common Objects in Context - 80 classes, 330K images, 1.5M objects. Pascal VOC: 20 classes, 11K images. Open Images: 600 classes, 9M images. KITTI: Autonomous driving dataset.

Format: COCO JSON format or Pascal VOC XML format.

📚 COCO📊 VOC📋 Open Images🔄 KITTI

# COCO dataset loader (simplified)
class COCODataset:
    def __init__(self, annotation_path, image_dir):
        self.annotation_path = annotation_path
        self.image_dir = image_dir
        self.load_annotations()
    
    def load_annotations(self):
        import json
        with open(self.annotation_path, 'r') as f:
            self.data = json.load(f)
        
        self.images = {img['id']: img for img in self.data['images']}
        self.categories = {cat['id']: cat for cat in self.data['categories']}
        
        # Group annotations by image
        self.annotations = {}
        for ann in self.data['annotations']:
            img_id = ann['image_id']
            if img_id not in self.annotations:
                self.annotations[img_id] = []
            self.annotations[img_id].append(ann)
    
    def get_item(self, idx):
        img_info = self.images[idx]
        img_path = f"{self.image_dir}/{img_info['file_name']}"
        anns = self.annotations.get(idx, [])
        
        boxes = []
        labels = []
        for ann in anns:
            x, y, w, h = ann['bbox']
            boxes.append([x, y, x + w, y + h])
            labels.append(ann['category_id'])
        
        return img_path, boxes, labels

💼 Applications

🎯 Real-World Uses

Object detection powers autonomous driving, surveillance, medical imaging, retail, and augmented reality.

Application Domains

Autonomous driving: Pedestrian, vehicle, and traffic sign detection. Surveillance: Person detection and tracking. Medical: Tumor detection in radiology. Retail: Object counting and inventory management.

AR: Object detection for AR interaction.

🚗 Autonomous📊 Surveillance📋 Medical🔄 Retail

# Application: Pedestrian detection
class PedestrianDetector:
    def __init__(self, model_path):
        self.model = torch.load(model_path)
        self.classes = ['person', 'car', 'bicycle', 'motorcycle']
    
    def detect(self, image):
        # Run detection
        detections = self.model(image)
        
        # Filter for pedestrians
        pedestrians = []
        for det in detections:
            if det['class'] == 'person' and det['confidence'] > 0.7:
                pedestrians.append(det)
        
        return pedestrians
    
    def track_pedestrians(self, video_frames):
        # Tracking over time
        tracks = []
        for frame in video_frames:
            people = self.detect(frame)
            # Apply tracking (e.g., Kalman filter)
            tracks = update_tracks(tracks, people)
        return tracks

🚀 Future Directions

🎯 Emerging Trends

Future directions include real-time on-device detection, transformer-based detectors, and few-shot learning.

Future Trends

On-device: Efficient models for mobile and edge. Transformers: DETR and its variants. Few-shot: Detecting novel classes with few examples. Video: Spatio-temporal detection. 3D: Point cloud object detection.

Foundation models: Large vision-language models for detection.

🚀 On-Device📊 Transformers📋 Few-Shot🔄 3D

# Future: DETR (Detection Transformer)
class DETR(nn.Module):
    def __init__(self, backbone, num_classes, num_queries=100):
        super().__init__()
        self.backbone = backbone
        self.transformer = nn.Transformer(
            d_model=512,
            nhead=8,
            num_encoder_layers=6,
            num_decoder_layers=6
        )
        self.query_pos = nn.Parameter(torch.randn(num_queries, 512))
        self.classifier = nn.Linear(512, num_classes + 1)
        self.bbox_predictor = nn.Linear(512, 4)
    
    def forward(self, x):
        # Extract features
        features = self.backbone(x)
        features = features.flatten(2).permute(2, 0, 1)
        
        # Transformer decoder
        hs = self.transformer(features, self.query_pos.unsqueeze(1).repeat(1, 1, 1))
        
        # Predict classes and boxes
        classes = self.classifier(hs)
        boxes = self.bbox_predictor(hs)
        
        return classes, boxes

# Future: EfficientViT for on-device detection
class EfficientDetector:
    def __init__(self):
        self.model = self.build_efficient_model()
    
    def build_efficient_model(self):
        # Lightweight model for edge devices
        # Using MobileNet/EfficientNet backbone
        pass