🧠 Markerless Fundamentals
🎯 What is Markerless Tracking?
Markerless tracking (also called Natural Feature Tracking or NFT) detects and tracks features naturally present in the environment — edges, corners, textures — without requiring physical markers like QR codes or ArUco tags. It's the foundation of modern AR experiences like ARKit, ARCore, and HoloLens.
How It Works
1. Detect → Find distinctive natural features (corners, blobs) in the camera image. 2. Describe → Compute a descriptor for each feature (e.g., ORB, SIFT). 3. Match → Find correspondences between current frame and map/keyframe. 4. Estimate Pose → Solve PnP to get camera position and orientation. 5. Track → Continuously update pose as the camera moves.
# Markerless tracking pipeline (conceptual)
def markerless_track(frame, map):
# 1. Detect features
keypoints, descriptors = orb.detectAndCompute(frame)
# 2. Match to map
matches = matcher.match(descriptors, map.descriptors)
# 3. Estimate pose with PnP
pose = solvePnP(map.points_3d, keypoints[matches], camera_matrix)
# 4. Update map if needed
if is_keyframe(frame, pose):
map.add_keyframe(frame, pose, keypoints, descriptors)
return pose📌 Marker vs Markerless
🎯 Two approaches to AR tracking
Marker-based: Uses fiducial markers (QR, ArUco, image targets). Markerless: Uses natural features from the environment. Each has trade-offs.
Comparison
Marker-based: Robust, fast, easy to implement. Requires printed markers. Markerless: No preparation needed, works anywhere, more immersive. Requires more computation and can fail in textureless environments.
# Marker vs Markerless decision
if environment_has_markers:
# Use ArUco or QR tracking
corners, ids = aruco.detectMarkers(frame)
pose = aruco.estimatePoseSingleMarkers(corners, marker_length)
else:
# Use natural feature tracking
keypoints = orb.detect(frame)
pose = track_natural_features(keypoints, map)🔍 Natural Feature Detection
🎯 Finding distinctive points
Natural features are corners, edges, blobs, and textured regions that are stable across viewpoints and lighting changes.
Feature Detectors
FAST: Fast corner detector. Harris: Classic corner detector. ORB: FAST + BRIEF, rotation-invariant. SIFT/SURF: Scale-invariant, patented. SuperPoint: Deep learning-based.
// FAST corner detection (OpenCV)
#include
Ptr fast = FastFeatureDetector::create(50);
vector keypoints;
fast->detect(image, keypoints);
// Draw keypoints
Mat output;
drawKeypoints(image, keypoints, output, Scalar::all(-1), DrawMatchesFlags::DEFAULT); 🧬 Feature Descriptors
🎯 Encoding feature identity
Descriptors are compact representations of the local image patch around a keypoint. They enable matching across frames and viewpoints.
Descriptor Types
BRIEF: Binary, fast, not rotation-invariant. ORB: Oriented BRIEF, rotation-invariant. SIFT: 128-dim float, scale-invariant. SuperPoint: 256-dim, learned. Binary descriptors use Hamming distance; float descriptors use L2.
// ORB descriptor extraction
Ptr orb = ORB::create(1000);
vector keypoints;
Mat descriptors;
orb->detectAndCompute(image, noArray(), keypoints, descriptors);
// descriptors is a Mat of N x 32 bytes (256 bits)
// Matching uses Hamming distance
BFMatcher matcher(NORM_HAMMING);
vector matches;
matcher.match(descriptors1, descriptors2, matches); 🔗 Feature Matching
🎯 Finding correspondences
Matching compares descriptors to find the same feature in different frames. Robust matching filters outliers with RANSAC.
Matching Strategies
Brute-force: Compare every descriptor. FLANN: Approximate nearest neighbor, faster. Ratio test: Lowe's ratio (0.7-0.8) filters ambiguous matches. Cross-check: Mutual best matches only. RANSAC: Geometric verification.
// FLANN matching with ratio test
Ptr matcher = FlannBasedMatcher::create();
vector> knn_matches;
matcher->knnMatch(desc1, desc2, knn_matches, 2);
vector good_matches;
for (auto& m : knn_matches) {
if (m[0].distance < 0.75 * m[1].distance) {
good_matches.push_back(m[0]);
}
}
// RANSAC geometric verification
Mat H = findHomography(pts1, pts2, RANSAC); 📐 Pose Estimation (PnP)
🎯 3D-2D correspondences → pose
Perspective-n-Point (PnP) estimates camera pose given 3D points and their 2D projections. It's the core of markerless tracking.
PnP Algorithms
P3P: Minimal 3 points. EPnP: Efficient, O(n). Iterative: Levenberg-Marquardt refinement. RANSAC: Robust to outliers. OpenCV's solvePnPRansac handles all.
// PnP pose estimation with RANSAC
Mat rvec, tvec;
vector inliers;
solvePnPRansac(
object_points, // 3D points
image_points, // 2D projections
camera_matrix,
dist_coeffs,
rvec, tvec,
false,
100, // iterations
8.0, // reprojection error
0.99, // confidence
inliers
);
// Convert to 4x4 pose matrix
Mat R;
Rodrigues(rvec, R);
Mat pose = Mat::eye(4, 4, CV_64F);
R.copyTo(pose(Rect(0, 0, 3, 3)));
tvec.copyTo(pose(Rect(3, 0, 1, 3))); 🎯 Frame-to-Frame Tracking
🎯 Continuous pose updates
Tracking maintains pose across frames using optical flow or feature matching. It's faster than full relocalization.
Optical Flow (KLT)
Lucas-Kanade tracks features between consecutive frames by assuming small motion. It's fast and works well for continuous tracking. When tracking fails, relocalization is triggered.
// Lucas-Kanade optical flow
vector prev_pts, next_pts;
vector status;
vector err;
calcOpticalFlowPyrLK(
prev_gray, next_gray,
prev_pts, next_pts,
status, err,
Size(21, 21), // window size
3 // pyramid levels
);
// Filter good points
vector good_new, good_old;
for (size_t i = 0; i < status.size(); i++) {
if (status[i]) {
good_new.push_back(next_pts[i]);
good_old.push_back(prev_pts[i]);
}
} 🗝️ Keyframes & Mapping
🎯 Building a persistent map
Keyframes are selected frames that contribute to the map. They store pose, features, and 3D points. The map enables relocalization.
Keyframe Selection
A frame becomes a keyframe if: significant translation/rotation from last keyframe, few tracked features (tracking degrading), or new area detected. Map points are triangulated from keyframe pairs.
// Keyframe selection heuristic
bool shouldBeKeyframe(Pose current_pose, Pose last_keyframe_pose, int tracked_features) {
float translation = (current_pose.position - last_keyframe_pose.position).norm();
float rotation = current_pose.rotation.angleTo(last_keyframe_pose.rotation);
if (translation > 0.1f || // 10cm
rotation > 0.1f || // ~5.7 degrees
tracked_features < 50) { // tracking degraded
return true;
}
return false;
}📍 Relocalization
🎯 Recovering from tracking loss
When tracking is lost (rapid motion, occlusion), relocalization finds the camera pose in the existing map using feature matching + PnP.
Relocalization Pipeline
1. Extract features from current frame. 2. Match against map keyframes (using BoW for fast retrieval). 3. PnP + RANSAC for pose estimation. 4. Verify with geometric checks. 5. Resume tracking.
# Relocalization with DBoW and PnP
def relocalize(frame, map, db):
# 1. Extract features
kps, desc = orb.detectAndCompute(frame)
# 2. Query database for candidate keyframes
results = db.query(desc, max_results=5)
for result in results:
keyframe = map.get_keyframe(result.id)
# 3. Match features
matches = matcher.match(desc, keyframe.descriptors)
if len(matches) < 20:
continue
# 4. PnP with RANSAC
pose, inliers = solvePnPRansac(
keyframe.points_3d, kps[matches], camera_matrix
)
# 5. Verify
if len(inliers) > 30:
return pose # relocalized!
return None # still lost🧠 Deep Learning Features
🎯 Learned features for robust tracking
Deep learning models like SuperPoint, SuperGlue, and D2-Net outperform handcrafted features in challenging conditions.
SuperPoint + SuperGlue
SuperPoint: Self-supervised detector + descriptor. SuperGlue: Graph neural network for matching. Together they achieve state-of-the-art performance in extreme viewpoint and lighting changes. DINOv2 provides general-purpose features for tracking.
# SuperPoint + SuperGlue (PyTorch)
import torch
from superpoint import SuperPoint
from superglue import SuperGlue
# Load models
sp = SuperPoint().eval().cuda()
sg = SuperGlue().eval().cuda()
# Extract features
with torch.no_grad():
kps1, desc1 = sp(img1)
kps2, desc2 = sp(img2)
# Match
matches = sg(kps1, desc1, kps2, desc2)📏 Plane & Surface Detection
🎯 Finding flat surfaces
Plane detection identifies horizontal and vertical surfaces (floors, tables, walls) for placing AR objects. Essential for ARKit/ARCore.
How Planes Are Detected
Feature points are clustered and RANSAC fits a plane model. Planes are tracked and merged over time. ARPlaneManager in AR Foundation exposes detected planes. Bounded vs infinite planes.
// AR Foundation plane detection
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;
public class PlaneDetector : MonoBehaviour {
ARPlaneManager planeManager;
void Start() {
planeManager = GetComponent();
planeManager.requestedDetectionMode = PlaneDetectionMode.Horizontal | PlaneDetectionMode.Vertical;
}
void OnEnable() {
planeManager.planesChanged += OnPlanesChanged;
}
void OnPlanesChanged(ARPlanesChangedEventArgs args) {
foreach (ARPlane plane in args.added) {
Debug.Log($"Plane detected: {plane.size} at {plane.center}");
}
}
} 🖼️ Image & Object Tracking
🎯 Tracking known images and 3D objects
Image tracking recognizes and tracks 2D images (posters, cards). Object tracking recognizes 3D objects (toys, products).
Reference Images & Objects
ARCore Augmented Images: Provide a library of reference images. ARKit Reference Images: Similar, with ARReferenceImage. Object Tracking: Uses 3D scans or CAD models. Both provide pose and tracking quality.
// ARCore Augmented Images
using Google.XR.ARCoreExtensions;
using UnityEngine.XR.ARFoundation;
public class ImageTracker : MonoBehaviour {
ARTrackedImageManager imageManager;
void Start() {
imageManager = GetComponent();
imageManager.trackedImagesChanged += OnImagesChanged;
}
void OnImagesChanged(ARTrackedImagesChangedEventArgs args) {
foreach (ARTrackedImage image in args.added) {
Debug.Log($"Image detected: {image.referenceImage.name}");
// Place content on image
Instantiate(prefab, image.transform.position, image.transform.rotation);
}
}
} ⚠️ Challenges & Failure Modes
🎯 When markerless tracking fails
Markerless tracking struggles with textureless surfaces, repetitive patterns, rapid motion, dynamic scenes, and lighting changes.
Common Issues
Textureless walls: No features to detect. Repetitive patterns: Ambiguous matching (e.g., tiled floors). Rapid motion: Motion blur, tracking loss. Dynamic objects: Moving people confuse tracking. Lighting changes: Features change appearance. Solutions: IMU fusion, deep learning, multi-sensor.
# Tracking quality assessment
def assess_tracking_quality(matches, inliers, reprojection_error):
quality = "GOOD"
if len(matches) < 20:
quality = "POOR" # few features
elif len(inliers) / len(matches) < 0.5:
quality = "POOR" # many outliers
elif reprojection_error > 5.0:
quality = "POOR" # high error
if quality == "POOR":
trigger_relocalization()
return quality🚀 Future: Neural Tracking
🎯 AI-powered markerless tracking
Neural networks are replacing traditional feature-based pipelines, enabling robust tracking in previously impossible conditions.
Neural Tracking Systems
DROID-SLAM: Recurrent optical flow for dense tracking. SuperGlue: Graph neural network matching. NeRF-based tracking: iMAP, NICE-SLAM. Gaussian Splatting: Real-time photorealistic tracking. These systems handle textureless, dynamic, and changing environments.
# Neural tracking with DROID-SLAM
import torch
from droid_slam import Droid
model = Droid().cuda()
model.eval()
poses = []
for image in video_stream:
image = preprocess(image).cuda()
pose = model.track(image)
poses.append(pose)
# Global optimization
if len(poses) % 10 == 0:
poses = model.global_ba(poses)