🧠 Object Detection AR Fundamentals
🎯 What is Object Detection AR?
Object Detection AR combines computer vision (object detection/recognition) with augmented reality to recognize real-world objects and overlay relevant digital content on them. Unlike marker-based AR (QR codes) or markerless AR (planes), object detection AR understands what it sees — a chair, a car, a product — and can attach contextual AR experiences to specific objects.
Why Object Detection AR Matters
Context-aware: AR content adapts to what you see. No markers: Works with real objects (products, furniture, machines). Interactive: Tap, rotate, or manipulate recognized objects. Information overlay: Show specs, instructions, reviews. Applications: Retail (try before you buy), maintenance (repair guides), education (3D anatomy), gaming (real-world objects as game pieces).
# Object Detection AR pipeline (conceptual)
class ObjectDetectionAR:
def __init__(self):
self.detector = YOLOv8() # or SSD, EfficientDet
self.tracker = ByteTrack()
self.pose_estimator = PoseEstimator()
self.ar_renderer = ARRenderer()
def process_frame(self, camera_frame, camera_pose):
# 1. Detect objects
detections = self.detector.detect(camera_frame)
# detections: [(class_id, confidence, bbox), ...]
# 2. Track objects across frames
tracked = self.tracker.update(detections)
# 3. Estimate 3D pose of each object
for obj in tracked:
obj.pose_3d = self.pose_estimator.estimate(
obj.bbox, camera_pose, camera_frame
)
# 4. Anchor AR content to objects
for obj in tracked:
ar_content = self.get_ar_content(obj.class_id)
self.ar_renderer.place(ar_content, obj.pose_3d)
# 5. Render AR overlay
return self.ar_renderer.render(camera_frame)📷 Computer Vision Basics
🎯 How machines see
Computer vision enables machines to interpret images. Core tasks: image classification (what is it?), object detection (where is it?), semantic segmentation (pixel-level labels), instance segmentation (individual objects).
CV Tasks Hierarchy
Classification: One label per image. Detection: Bounding boxes + labels. Semantic segmentation: Per-pixel class. Instance segmentation: Per-pixel + object identity. Keypoint detection: Landmarks (pose, face). Depth estimation: Per-pixel distance. AR uses all of these.
# CV tasks with PyTorch
import torch
import torchvision.transforms as T
from PIL import Image
# 1. Image Classification
from torchvision.models import resnet50
model = resnet50(pretrained=True).eval()
img = Image.open("image.jpg")
transform = T.Compose([T.Resize(256), T.CenterCrop(224), T.ToTensor()])
input_tensor = transform(img).unsqueeze(0)
with torch.no_grad():
output = model(input_tensor)
probabilities = torch.nn.functional.softmax(output[0], dim=0)
top5_prob, top5_catid = torch.topk(probabilities, 5)
for i in range(top5_prob.size(0)):
print(f"{top5_catid[i]}: {top5_prob[i]:.3f}")
# 2. Object Detection (torchvision)
from torchvision.models.detection import fasterrcnn_resnet50_fpn
detection_model = fasterrcnn_resnet50_fpn(pretrained=True).eval()
with torch.no_grad():
predictions = detection_model([input_tensor[0]])
# predictions[0]['boxes'], ['labels'], ['scores']
# 3. Semantic Segmentation (DeepLab)
from torchvision.models.segmentation import deeplabv3_resnet50
seg_model = deeplabv3_resnet50(pretrained=True).eval()
with torch.no_grad():
seg_output = seg_model(input_tensor)['out']
seg_map = seg_output.argmax(1).squeeze().cpu().numpy()
# 4. Instance Segmentation (Mask R-CNN)
from torchvision.models.detection import maskrcnn_resnet50_fpn
mask_model = maskrcnn_resnet50_fpn(pretrained=True).eval()
with torch.no_grad():
mask_predictions = mask_model([input_tensor[0]])
# mask_predictions[0]['masks'] — per-instance masks
# 5. Keypoint Detection (Keypoint R-CNN)
from torchvision.models.detection import keypointrcnn_resnet50_fpn
kp_model = keypointrcnn_resnet50_fpn(pretrained=True).eval()
with torch.no_grad():
kp_predictions = kp_model([input_tensor[0]])
# kp_predictions[0]['keypoints'] — 17 human keypoints🤖 Detection Models (YOLO, SSD)
🎯 Real-time object detection
Detection models locate and classify objects in images. State-of-the-art: YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), EfficientDet, DETR, and RT-DETR.
Model Comparison
YOLOv8/v11: Fastest, most accurate, easy to use. SSD: Lightweight, mobile-friendly. EfficientDet: Good accuracy/efficiency tradeoff. DETR: Transformer-based, no anchors. RT-DETR: Real-time DETR. MobileNet-SSD: For mobile AR. Nano models: YOLOv8n, YOLOv11n for edge devices.
# YOLOv8 object detection (Ultralytics)
from ultralytics import YOLO
# Load pretrained model
model = YOLO('yolov8n.pt') # nano, small, medium, large, xlarge
# Detect on image
results = model('image.jpg')
# Process results
for result in results:
boxes = result.boxes
for box in boxes:
# Bounding box
x1, y1, x2, y2 = box.xyxy[0].tolist()
confidence = box.conf[0].item()
class_id = int(box.cls[0].item())
class_name = model.names[class_id]
print(f"{class_name}: {confidence:.2f} at ({x1:.0f}, {y1:.0f}, {x2:.0f}, {y2:.0f})")
# Real-time detection on webcam
import cv2
cap = cv2.VideoCapture(0)
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
results = model(frame, stream=True)
for result in results:
annotated = result.plot()
cv2.imshow('YOLOv8', annotated)
if cv2.waitKey(1) & 0xFF == 27:
break
cap.release()
cv2.destroyAllWindows()
# Export for mobile (TFLite, ONNX, CoreML)
model.export(format='tflite') # Android
model.export(format='coreml') # iOS
model.export(format='onnx') # Cross-platform🏷️ Classification & Recognition
🎯 What is this object?
Object classification identifies the category of an object (e.g., "chair", "laptop", "coffee cup"). Recognition goes further — identifying specific instances (e.g., "my laptop", "Coca-Cola can").
Classification Approaches
CNN classifiers: ResNet, EfficientNet, MobileNet. Fine-grained: Distinguish similar classes (bird species, car models). Few-shot: Classify with few examples (CLIP, Prototypical Networks). Instance recognition: Identify specific objects (product recognition). Multi-label: Multiple tags per image.
# Classification with CLIP (zero-shot)
import torch
import clip
from PIL import Image
# Load CLIP model
device = "cuda" if torch.cuda.is_available() else "cpu"
model, preprocess = clip.load("ViT-B/32", device=device)
# Prepare image and text
image = preprocess(Image.open("image.jpg")).unsqueeze(0).to(device)
text_labels = ["a photo of a chair", "a photo of a table", "a photo of a lamp"]
text = clip.tokenize(text_labels).to(device)
# Classify
with torch.no_grad():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
# Compute similarity
logits_per_image, _ = model(image, text)
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
for label, prob in zip(text_labels, probs[0]):
print(f"{label}: {prob:.3f}")
# Few-shot classification (Prototypical Networks)
class PrototypicalNetwork:
def __init__(self, encoder):
self.encoder = encoder # pretrained CNN
def compute_prototypes(self, support_set):
"""Compute class prototypes from support examples."""
prototypes = {}
for class_name, images in support_set.items():
embeddings = [self.encoder(img) for img in images]
prototypes[class_name] = torch.mean(torch.stack(embeddings), dim=0)
return prototypes
def classify(self, query_image, prototypes):
"""Classify query image by nearest prototype."""
query_embedding = self.encoder(query_image)
distances = {}
for class_name, prototype in prototypes.items():
distances[class_name] = torch.norm(query_embedding - prototype)
return min(distances, key=distances.get)🔄 Object Tracking
🎯 Following objects over time
Object tracking maintains the identity of detected objects across frames. Essential for AR — AR content should stay attached to the same object as it moves. Algorithms: SORT, ByteTrack, DeepSORT, OC-SORT.
Tracking Methods
SORT: Kalman filter + Hungarian algorithm. ByteTrack: Uses low-confidence detections. DeepSORT: Adds appearance features (ReID). OC-SORT: Handles occlusions. Single object: KCF, CSRT, MOSSE. Multi-object: Track multiple objects simultaneously.
# ByteTrack object tracking
from ultralytics import YOLO
import cv2
# Load model with tracking
model = YOLO('yolov8n.pt')
# Track on video
cap = cv2.VideoCapture('video.mp4')
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
# Track with ByteTrack
results = model.track(frame, persist=True, tracker='bytetrack.yaml')
if results[0].boxes.id is not None:
boxes = results[0].boxes.xyxy.cpu().numpy()
ids = results[0].boxes.id.cpu().numpy()
classes = results[0].boxes.cls.cpu().numpy()
for box, track_id, cls in zip(boxes, ids, classes):
x1, y1, x2, y2 = box
cv2.rectangle(frame, (int(x1), int(y1)), (int(x2), int(y2)), (0, 255, 0), 2)
cv2.putText(frame, f"ID {int(track_id)}", (int(x1), int(y1) - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
cv2.imshow('Tracking', frame)
if cv2.waitKey(1) & 0xFF == 27:
break
# Custom Kalman filter tracker
import numpy as np
class KalmanTracker:
def __init__(self):
# State: [x, y, vx, vy]
self.kf = cv2.KalmanFilter(4, 2)
self.kf.measurementMatrix = np.array([[1, 0, 0, 0], [0, 1, 0, 0]], np.float32)
self.kf.transitionMatrix = np.array([[1, 0, 1, 0], [0, 1, 0, 1], [0, 0, 1, 0], [0, 0, 0, 1]], np.float32)
self.kf.processNoiseCov = np.eye(4, dtype=np.float32) * 0.03
def predict(self):
return self.kf.predict()
def update(self, measurement):
self.kf.correct(np.array(measurement, np.float32))📐 6-DOF Pose Estimation
🎯 Where and how is it oriented?
6-DOF pose estimation determines an object's 3D position (x, y, z) and orientation (roll, pitch, yaw) relative to the camera. Essential for anchoring AR content to real objects.
Pose Estimation Methods
PnP: Perspective-n-Point (known 3D model). Deep learning: PoseCNN, PVNet, FoundationPose. Keypoints: Detect 2D keypoints, solve PnP. Template matching: Match object templates. Depth-based: Use depth maps for 3D pose. 6D pose: Full 3D position + orientation.
# 6-DOF pose estimation with PnP
import cv2
import numpy as np
def estimate_pose_6dof(image_points, object_points, camera_matrix, dist_coeffs):
"""
Estimate 6-DOF pose using PnP.
image_points: 2D points in image (N, 2)
object_points: 3D points in object frame (N, 3)
"""
success, rvec, tvec, inliers = cv2.solvePnPRansac(
object_points,
image_points,
camera_matrix,
dist_coeffs,
flags=cv2.SOLVEPNP_ITERATIVE,
iterationsCount=100,
reprojectionError=8.0,
confidence=0.99
)
if not success:
return None
# Convert rotation vector to matrix
R, _ = cv2.Rodrigues(rvec)
# Build 4x4 pose matrix
pose = np.eye(4)
pose[:3, :3] = R
pose[:3, 3] = tvec.flatten()
return pose, inliers
# Deep learning 6D pose (FoundationPose)
# FoundationPose is a unified foundation model for 6D object pose
# It works with or without CAD models
# Keypoint-based pose estimation
def estimate_pose_from_keypoints(keypoints_2d, keypoints_3d, camera_matrix):
"""
Estimate pose from 2D-3D keypoint correspondences.
keypoints_2d: (N, 2) detected 2D keypoints
keypoints_3d: (N, 3) known 3D keypoints in object frame
"""
# Use EPnP for better accuracy with few points
success, rvec, tvec = cv2.solvePnP(
keypoints_3d,
keypoints_2d,
camera_matrix,
None,
flags=cv2.SOLVEPNP_EPNP
)
if success:
R, _ = cv2.Rodrigues(rvec)
pose = np.eye(4)
pose[:3, :3] = R
pose[:3, 3] = tvec.flatten()
return pose
return None
# Pose refinement with ICP
def refine_pose_icp(initial_pose, depth_image, camera_matrix, object_mesh):
"""
Refine pose using Iterative Closest Point.
"""
import open3d as o3d
# Create point cloud from depth
points_3d = depth_to_pointcloud(depth_image, camera_matrix)
source = o3d.geometry.PointCloud()
source.points = o3d.utility.Vector3dVector(points_3d)
# Target: object mesh
target = object_mesh
# ICP
result = o3d.pipelines.registration.registration_icp(
source, target, 0.02, initial_pose,
o3d.pipelines.registration.TransformationEstimationPointToPlane()
)
return result.transformation✂️ Instance Segmentation
🎯 Pixel-perfect object boundaries
Instance segmentation provides per-pixel masks for each object, not just bounding boxes. This enables precise AR occlusion, object manipulation, and realistic integration. Models: Mask R-CNN, YOLOv8-seg, Segment Anything (SAM).
Segmentation Models
Mask R-CNN: Two-stage, accurate. YOLOv8-seg: Fast, single-stage. SAM (Segment Anything): Zero-shot, class-agnostic. SAM 2: Video segmentation. MobileSAM: Lightweight for mobile. Use in AR: Occlusion masks, object manipulation, realistic shadows.
# YOLOv8 instance segmentation
from ultralytics import YOLO
import cv2
import numpy as np
model = YOLO('yolov8n-seg.pt')
# Segment on image
results = model('image.jpg')
for result in results:
# Get masks
if result.masks is not None:
masks = result.masks.data.cpu().numpy() # (N, H, W)
boxes = result.boxes.xyxy.cpu().numpy()
classes = result.boxes.cls.cpu().numpy()
for mask, box, cls in zip(masks, boxes, classes):
# Create binary mask
binary_mask = (mask > 0.5).astype(np.uint8) * 255
# Apply mask for AR occlusion
# AR content will be occluded by this object
apply_occlusion_mask(binary_mask)
# Or use mask for object manipulation
object_pixels = image * (binary_mask[..., None] / 255)
# Segment Anything (SAM) — zero-shot segmentation
from segment_anything import sam_model_registry, SamPredictor
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
predictor = SamPredictor(sam)
# Set image
predictor.set_image(image)
# Segment with point prompt
input_point = np.array([[500, 375]])
input_label = np.array([1])
masks, scores, logits = predictor.predict(
point_coords=input_point,
point_labels=input_label,
multimask_output=True
)
# Segment with box prompt
input_box = np.array([100, 100, 500, 500])
masks, scores, logits = predictor.predict(
box=input_box,
multimask_output=False
)
# Use mask for AR occlusion
def apply_occlusion_mask(mask):
"""Apply segmentation mask as depth for AR occlusion."""
# Convert mask to depth (near = 0, far = 1)
depth = np.where(mask > 0, 0.5, 1.0) # object at 0.5m, background at 1m
return depth📱 On-Device Inference
🎯 AI on mobile AR
On-device inference is critical for AR — no cloud latency, works offline, privacy-preserving. Frameworks: TensorFlow Lite, Core ML, ONNX Runtime, NCNN, MediaPipe, ExecuTorch.
Mobile Inference Options
TensorFlow Lite: Android + iOS, GPU/NPU delegate. Core ML: iOS, Neural Engine. ONNX Runtime: Cross-platform. MediaPipe: Google's ML pipeline. NCNN: Tencent's mobile inference. ExecuTorch: PyTorch mobile. Quantization: INT8 for 4x speedup. Pruning: Remove unnecessary weights.
// TensorFlow Lite on Android (Kotlin)
import org.tensorflow.lite.Interpreter
import org.tensorflow.lite.gpu.GpuDelegate
class ObjectDetector(context: Context, modelPath: String) {
private val interpreter: Interpreter
private val gpuDelegate: GpuDelegate
init {
// Enable GPU acceleration
gpuDelegate = GpuDelegate()
val options = Interpreter.Options().apply {
addDelegate(gpuDelegate)
setNumThreads(4)
}
// Load model
val model = FileUtil.loadMappedFile(context, modelPath)
interpreter = Interpreter(model, options)
}
fun detect(bitmap: Bitmap): List {
// Preprocess
val input = preprocess(bitmap) // 640x640x3
// Run inference
val output = Array(1) { Array(84) { FloatArray(8400) } }
interpreter.run(input, output)
// Postprocess (NMS, etc.)
return postprocess(output)
}
}
// Core ML on iOS (Swift)
import CoreML
import Vision
class ObjectDetector {
private var model: VNCoreMLModel
init() {
let config = MLModelConfiguration()
config.computeUnits = .all // CPU + GPU + Neural Engine
let mlModel = try! YOLOv8(configuration: config).model
model = try! VNCoreMLModel(for: mlModel)
}
func detect(image: CVPixelBuffer, completion: @escaping ([Detection]) -> Void) {
let request = VNCoreMLRequest(model: model) { request, error in
guard let results = request.results as? [VNRecognizedObjectObservation] else {
completion([])
return
}
let detections = results.map { observation in
Detection(
boundingBox: observation.boundingBox,
label: observation.labels.first?.identifier ?? "unknown",
confidence: observation.confidence
)
}
completion(detections)
}
let handler = VNImageRequestHandler(cvPixelBuffer: image)
try? handler.perform([request])
}
}
// Model quantization (Python)
import tensorflow as tf
# Convert to TFLite with quantization
converter = tf.lite.TFLiteConverter.from_saved_model('model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_types = [tf.float16] # or tf.int8
tflite_model = converter.convert()
with open('model_quantized.tflite', 'wb') as f:
f.write(tflite_model) ⚓ Anchoring AR Content
🎯 Attaching content to objects
Once an object is detected and tracked, AR content must be anchored to it. This requires converting from 2D detection coordinates to 3D world coordinates, then attaching content with the correct pose.
Anchoring Pipeline
1. Detect: 2D bounding box. 2. Estimate depth: From depth sensor or monocular depth. 3. Unproject: 2D + depth → 3D point. 4. Estimate pose: 6-DOF from PnP or network. 5. Create anchor: ARAnchor at 3D position. 6. Attach content: Place AR object at anchor. 7. Track: Update anchor as object moves.
// ARKit object anchoring (Swift)
import ARKit
import Vision
class ObjectDetectionAR: NSObject, ARSessionDelegate {
var session: ARSession!
var sceneView: ARSCNView!
var visionModel: VNCoreMLModel!
func session(_ session: ARSession, didUpdate frame: ARFrame) {
// 1. Get camera image
let pixelBuffer = frame.capturedImage
// 2. Run object detection
let request = VNCoreMLRequest(model: visionModel) { [weak self] request, error in
guard let results = request.results as? [VNRecognizedObjectObservation] else { return }
for observation in results {
// 3. Get bounding box
let boundingBox = observation.boundingBox
// 4. Estimate depth (raycast from camera through box center)
let boxCenter = CGPoint(x: boundingBox.midX, y: boundingBox.midY)
let raycastQuery = frame.raycastQuery(
from: boxCenter,
allowing: .estimatedPlane,
alignment: .any
)
if let result = session.raycast(raycastQuery).first {
// 5. Create anchor at 3D position
let anchor = ARAnchor(transform: result.worldTransform)
session.add(anchor: anchor)
// 6. Attach AR content
self?.attachContent(to: anchor, for: observation.labels.first?.identifier)
}
}
}
// Run Vision request
let handler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: .right)
try? handler.perform([request])
}
func attachContent(to anchor: ARAnchor, for label: String?) {
// Load AR content based on label
let scene = SCNScene(named: "\(label ?? "default").scn")!
let node = scene.rootNode.clone()
// Create anchor node
let anchorNode = SCNNode()
anchorNode.simdTransform = anchor.transform
anchorNode.addChildNode(node)
sceneView.scene.rootNode.addChildNode(anchorNode)
}
}
// ARCore object anchoring (Kotlin)
fun onUpdate(frame: Frame) {
// 1. Get camera image
val image = frame.acquireCameraImage()
// 2. Run object detection (TFLite)
val detections = detector.detect(image)
for (detection in detections) {
// 3. Get hit pose from depth
val depthImage = frame.acquireDepthImage16Bits()
val depth = getDepthAt(depthImage, detection.center)
// 4. Unproject to 3D
val cameraIntrinsics = frame.camera.imageIntrinsics
val point3d = unproject(detection.center, depth, cameraIntrinsics)
// 5. Create anchor
val cameraPose = frame.camera.pose
val worldPoint = cameraPose.transformPoint(point3d)
val anchor = session.createAnchor(Pose(worldPoint, cameraPose.rotationQuaternion))
// 6. Attach content
attachContent(anchor, detection.label)
}
}🖱️ Object-Based Interaction
🎯 Interacting with recognized objects
Once objects are detected and anchored, users can interact with them: tap for info, grab to manipulate, point to select, or use gestures. Interaction is object-specific — a chair behaves differently from a phone.
Interaction Patterns
Tap: Select object, show info. Grab: Pick up, rotate, scale. Point: Highlight, select. Gesture: Swipe, pinch, rotate. Context menu: Actions specific to object. Physics: Objects fall, collide, stack. Haptic feedback: Vibration on touch.
// Object interaction in Unity AR
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
public class ObjectInteraction : MonoBehaviour {
public ARRaycastManager raycastManager;
public Camera arCamera;
public ObjectDetector detector;
void Update() {
// Detect objects
var detections = detector.Detect(arCamera);
foreach (var detection in detections) {
// Check for tap on detected object
if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Began) {
Vector2 touchPos = Input.GetTouch(0).position;
if (detection.boundingBox.Contains(touchPos)) {
// Tapped on this object!
OnObjectTapped(detection);
}
}
}
}
void OnObjectTapped(Detection detection) {
// Show contextual menu based on object class
switch (detection.label) {
case "chair":
ShowChairMenu(detection);
break;
case "laptop":
ShowLaptopMenu(detection);
break;
case "cup":
ShowCupMenu(detection);
break;
}
// Haptic feedback
Handheld.Vibrate();
}
void ShowChairMenu(Detection detection) {
// Options: "Try in room", "Change color", "View specs"
var menu = Instantiate(menuPrefab);
menu.transform.position = detection.worldPosition;
menu.AddButton("Try in room", () => TryInRoom(detection));
menu.AddButton("Change color", () => ChangeColor(detection));
menu.AddButton("View specs", () => ViewSpecs(detection));
}
}
// Physics-based interaction
public class ObjectPhysics : MonoBehaviour {
void OnCollisionEnter(Collision collision) {
// Check if collided with detected object
var detectedObj = collision.gameObject.GetComponent();
if (detectedObj != null) {
// Apply physics based on object properties
float mass = detectedObj.estimatedMass;
float restitution = detectedObj.estimatedBounciness;
var rb = GetComponent();
rb.mass = mass;
rb.drag = 1.0f - restitution;
}
}
} 🧱 Object Occlusion
🎯 Realistic depth integration
Object occlusion ensures AR content is properly hidden behind real objects (and vice versa). It uses depth maps or segmentation masks to determine what's in front.
Occlusion Techniques
Depth-based: Compare AR depth with real depth. Segmentation-based: Use object masks as occlusion. Mesh-based: Use spatial mesh for occlusion. Soft occlusion: Fade edges for realism. People occlusion: Segment people. Object occlusion: Segment specific objects.
// Object occlusion with segmentation mask (Unity)
using UnityEngine;
using UnityEngine.XR.ARFoundation;
public class ObjectOcclusion : MonoBehaviour {
public AROcclusionManager occlusionManager;
public Material occlusionMaterial;
public ObjectDetector detector;
void Update() {
// Get environment depth
if (occlusionManager.TryAcquireEnvironmentDepthCpuImage(out var depthImage)) {
// Update occlusion material with depth
occlusionMaterial.SetTexture("_EnvironmentDepth", depthImage.texture);
}
// Get object segmentation mask
var detections = detector.Detect(Camera.main);
foreach (var detection in detections) {
// Create occlusion mask from bounding box
var mask = CreateMask(detection);
// Apply as occlusion
occlusionMaterial.SetTexture("_ObjectMask", mask);
}
}
Texture2D CreateMask(Detection detection) {
// Create a texture with 1 where object is, 0 elsewhere
var mask = new Texture2D(Screen.width, Screen.height);
var pixels = new Color32[Screen.width * Screen.height];
for (int y = 0; y < Screen.height; y++) {
for (int x = 0; x < Screen.width; x++) {
bool inBox = detection.boundingBox.Contains(new Vector2(x, y));
pixels[y * Screen.width + x] = inBox ? new Color32(255, 255, 255, 255) : new Color32(0, 0, 0, 0);
}
}
mask.SetPixels32(pixels);
mask.Apply();
return mask;
}
}
// Occlusion shader
Shader "AR/ObjectOcclusion" {
Properties {
_MainTex ("Texture", 2D) = "white" {}
_EnvironmentDepth ("Environment Depth", 2D) = "white" {}
_ObjectMask ("Object Mask", 2D) = "white" {}
}
SubShader {
Tags { "Queue"="Transparent" "RenderType"="Transparent" }
Pass {
ZWrite On
ZTest LEqual
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#include "UnityCG.cginc"
sampler2D _MainTex;
sampler2D _EnvironmentDepth;
sampler2D _ObjectMask;
struct v2f {
float4 pos : SV_POSITION;
float2 uv : TEXCOORD0;
float4 screenPos : TEXCOORD1;
};
v2f vert(appdata_base v) {
v2f o;
o.pos = UnityObjectToClipPos(v.vertex);
o.uv = v.texCoord;
o.screenPos = ComputeScreenPos(o.pos);
return o;
}
fixed4 frag(v2f i) : SV_Target {
float2 screenUV = i.screenPos.xy / i.screenPos.w;
// Sample environment depth
float envDepth = tex2D(_EnvironmentDepth, screenUV).r;
// Sample object mask
float objMask = tex2D(_ObjectMask, screenUV).r;
// Compute fragment depth
float fragDepth = i.screenPos.z / i.screenPos.w;
// Discard if behind environment or object
if (fragDepth > envDepth || objMask > 0.5) {
discard;
}
return tex2D(_MainTex, i.uv);
}
ENDCG
}
}
}🧪 Training Custom Models
🎯 Teaching AI your objects
Custom object detection models are trained on your specific objects: products, equipment, tools. Process: data collection, annotation, training, export, deploy.
Training Pipeline
1. Collect data: Photos of objects from many angles, lighting. 2. Annotate: Draw bounding boxes (LabelImg, CVAT, Roboflow). 3. Train: YOLOv8, Detectron2, or custom. 4. Evaluate: mAP, precision, recall. 5. Export: TFLite, CoreML, ONNX. 6. Deploy: Integrate into AR app. 7. Iterate: Add hard examples, retrain.
# Training YOLOv8 on custom dataset
from ultralytics import YOLO
# 1. Prepare dataset (YOLO format)
# dataset/
# images/
# train/
# val/
# labels/
# train/
# val/
# data.yaml
# data.yaml:
# train: dataset/images/train
# val: dataset/images/val
# nc: 3 # number of classes
# names: ['chair', 'table', 'lamp']
# 2. Train
model = YOLO('yolov8n.pt') # start from pretrained
results = model.train(
data='data.yaml',
epochs=100,
imgsz=640,
batch=16,
patience=20,
device=0, # GPU
augment=True, # data augmentation
mosaic=1.0, # mosaic augmentation
mixup=0.1, # mixup augmentation
degrees=10, # rotation
translate=0.1, # translation
scale=0.5, # scaling
fliplr=0.5, # horizontal flip
hsv_h=0.015, # hue
hsv_s=0.7, # saturation
hsv_v=0.4 # value
)
# 3. Evaluate
metrics = model.val()
print(f"mAP@0.5: {metrics.box.map50}")
print(f"mAP@0.5:0.95: {metrics.box.map}")
# 4. Export for mobile
model.export(format='tflite', int8=True) # quantized for mobile
model.export(format='coreml', nms=True) # iOS
model.export(format='onnx') # cross-platform
# 5. Inference with custom model
custom_model = YOLO('runs/detect/train/weights/best.pt')
results = custom_model('test_image.jpg')
# Data augmentation for AR-specific challenges
import albumentations as A
transform = A.Compose([
A.RandomBrightnessContrast(p=0.5), # lighting changes
A.MotionBlur(blur_limit=7, p=0.3), # motion blur
A.GaussNoise(p=0.3), # sensor noise
A.Perspective(scale=(0.05, 0.1), p=0.5), # viewpoint changes
A.Rotate(limit=30, p=0.5), # rotation
A.RandomShadow(p=0.3), # shadows
A.CoarseDropout(p=0.3), # occlusion
A.HorizontalFlip(p=0.5),
], bbox_params=A.BboxParams(format='yolo'))
# Active learning: add hard examples
# 1. Run model on unlabeled data
# 2. Find low-confidence detections
# 3. Annotate those
# 4. Retrain
# 5. Repeat⚠️ Challenges & Failure Modes
🎯 When detection fails
Object detection AR faces unique challenges: varying lighting, occlusion, motion blur, unusual angles, and the sim-to-real gap. Robust systems must handle these gracefully.
Common Issues
Lighting: Model trained on daylight fails at night. Occlusion: Partial objects hard to detect. Motion blur: Fast movement causes blur. Scale: Very small/large objects missed. Viewpoint: Unusual angles fail. Domain shift: Real world differs from training. Latency: Detection takes too long. Power: Continuous inference drains battery.
# Handling detection failures gracefully
class RobustObjectDetectionAR:
def __init__(self):
self.detector = YOLOv8()
self.tracker = ByteTrack()
self.detection_history = {}
self.confidence_threshold = 0.5
def process_frame(self, frame):
# 1. Run detection
detections = self.detector.detect(frame)
# 2. Filter by confidence
detections = [d for d in detections if d.confidence > self.confidence_threshold]
# 3. If detection fails, use tracking prediction
if len(detections) == 0:
# Use tracker to predict where objects are
tracked = self.tracker.predict()
if tracked:
# Show "lost" state, but keep AR content
for obj in tracked:
obj.state = "lost"
obj.confidence *= 0.9 # decay confidence
# 4. Smooth detections over time
smoothed = self.smooth_detections(detections)
# 5. Handle edge cases
for det in smoothed:
# Check if detection is stable
if self.is_stable(det):
det.state = "stable"
else:
det.state = "unstable"
return smoothed
def smooth_detections(self, detections):
"""Smooth detection results over time (EMA)."""
alpha = 0.3
smoothed = []
for det in detections:
key = (det.class_id, det.track_id) if det.track_id else det.class_id
if key in self.detection_history:
prev = self.detection_history[key]
# Exponential moving average
det.bbox = (
alpha * det.bbox[0] + (1 - alpha) * prev.bbox[0],
alpha * det.bbox[1] + (1 - alpha) * prev.bbox[1],
alpha * det.bbox[2] + (1 - alpha) * prev.bbox[2],
alpha * det.bbox[3] + (1 - alpha) * prev.bbox[3]
)
det.confidence = alpha * det.confidence + (1 - alpha) * prev.confidence
self.detection_history[key] = det
smoothed.append(det)
return smoothed
def is_stable(self, detection, min_frames=5):
"""Check if detection has been stable for N frames."""
key = (detection.class_id, detection.track_id) if detection.track_id else detection.class_id
if key not in self.detection_history:
return False
# Count consecutive frames
return self.detection_history[key].frame_count >= min_frames
# Handling low-light conditions
def enhance_for_detection(frame):
"""Enhance image for better detection in low light."""
import cv2
# Convert to LAB
lab = cv2.cvtColor(frame, cv2.COLOR_BGR2LAB)
# Apply CLAHE to L channel
clahe = cv2.createCLAHE(clipLimit=3.0, tileGridSize=(8, 8))
lab[:, :, 0] = clahe.apply(lab[:, :, 0])
# Convert back
enhanced = cv2.cvtColor(lab, cv2.COLOR_LAB2BGR)
return enhanced
# Handling motion blur
def deblur_for_detection(frame, blur_kernel=15):
"""Deblur image for better detection."""
import cv2
import numpy as np
# Wiener filter (simplified)
kernel = np.ones((blur_kernel, blur_kernel)) / (blur_kernel ** 2)
deblurred = cv2.filter2D(frame, -1, kernel)
return deblurred🚀 Future: Foundation Models + AR
🎯 The next generation of object detection AR
Foundation models (SAM, DINOv2, CLIP) and generative AI are transforming object detection AR: zero-shot detection, open-vocabulary recognition, 3D-aware understanding, and text-to-3D content creation.
Foundation Models in AR
SAM (Segment Anything): Zero-shot segmentation of any object. DINOv2: Self-supervised visual features. CLIP: Open-vocabulary recognition. Grounding DINO: Text-to-object detection. GPT-4V: Multimodal understanding. 3D generation: Text-to-3D for AR content. NeRF/Gaussian Splatting: Photorealistic AR objects.
# Grounding DINO + SAM: open-vocabulary object detection
import torch
from groundingdino.util.inference import load_model, predict
from segment_anything import sam_model_registry, SamPredictor
# Load models
grounding_dino = load_model("groundingdino.pth")
sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth")
sam_predictor = SamPredictor(sam)
def detect_and_segment(image, text_prompt):
"""
Detect objects using text prompt (open-vocabulary),
then segment them precisely with SAM.
"""
# 1. Grounding DINO: text -> bounding boxes
boxes, logits, phrases = predict(
model=grounding_dino,
image=image,
caption=text_prompt, # e.g., "chair. table. lamp."
box_threshold=0.3,
text_threshold=0.25
)
# 2. SAM: boxes -> precise masks
sam_predictor.set_image(image)
masks = []
for box in boxes:
# Convert box to SAM format
box_np = box.cpu().numpy() * torch.tensor([image.width, image.height, image.width, image.height])
mask, _, _ = sam_predictor.predict(
box=box_np,
multimask_output=False
)
masks.append(mask[0])
return boxes, masks, phrases
# Usage: detect any object with text
image = load_image("scene.jpg")
boxes, masks, phrases = detect_and_segment(image, "chair. table. laptop.")
for box, mask, phrase in zip(boxes, masks, phrases):
print(f"Detected: {phrase}")
# Use mask for AR occlusion, box for anchoring
# Text-to-3D for AR content
# Use a generative model to create 3D content from text
# Then place it on the detected object
def generate_ar_content(text_prompt):
"""
Generate 3D AR content from text prompt.
Uses a text-to-3D model (e.g., Shap-E, DreamFusion, or Gaussian Splatting).
"""
# 1. Text -> 3D mesh
# (using a hypothetical text-to-3D API)
mesh = text_to_3d(text_prompt)
# 2. Optimize for mobile AR
mesh = simplify_mesh(mesh, target_faces=5000)
mesh = bake_textures(mesh)
# 3. Export as AR-ready asset
return mesh
# Example: "a wooden chair"
chair_mesh = generate_ar_content("a wooden chair")
# Place on detected chair in AR
# 3D-aware object detection
# Detect objects in 3D space, not just 2D boxes
def detect_3d_objects(point_cloud, text_prompt):
"""
Detect objects in 3D point cloud using language.
Uses a 3D foundation model (e.g., OpenScene, LERF).
"""
# 1. Extract 3D features (e.g., using a 3D backbone)
features = extract_3d_features(point_cloud)
# 2. Compute text embedding
text_embedding = clip.encode_text(text_prompt)
# 3. Find regions matching text
similarity = compute_similarity(features, text_embedding)
# 4. Extract 3D bounding boxes
boxes_3d = extract_3d_boxes(similarity)
return boxes_3d