🧠 Spatial Mapping Fundamentals

🎯 Building a Digital Twin of the Real World

Spatial mapping (also called scene reconstruction or spatial understanding) is the process of creating a 3D representation of the physical environment. It uses depth sensors, cameras, and SLAM to build a mesh, point cloud, or voxel grid that AR/VR systems can understand and interact with.

Why Spatial Mapping Matters

Occlusion: Virtual objects hide behind real ones. Physics: Objects collide and rest on real surfaces. Navigation: Avatars and NPCs walk on real floors. Placement: Content snaps to walls, floors, tables. Interaction: Rays bounce off real geometry. It's the foundation of believable mixed reality.

📌 3D Reconstruction📌 Occlusion📌 Physics📌 Semantic Understanding

# Spatial mapping pipeline (conceptual)
class SpatialMappingSystem:
    def __init__(self):
        self.volume = TSDFVolume(resolution=0.05)  # 5cm voxels
        self.mesh = None
        self.planes = []
        self.semantic_labels = {}
    
    def process_frame(self, depth_map, color_image, camera_pose):
        # 1. Integrate depth into volumetric grid
        self.volume.integrate(depth_map, camera_pose)
        
        # 2. Extract mesh (marching cubes)
        self.mesh = self.volume.extract_mesh()
        
        # 3. Extract planes (RANSAC)
        self.planes = extract_planes(self.mesh)
        
        # 4. Semantic labeling (optional)
        self.semantic_labels = classify_surfaces(self.mesh, color_image)
        
        return self.mesh, self.planes

📡 Depth Sensors & Data

🎯 Capturing the third dimension

Depth sensors provide per-pixel distance to surfaces. Types: structured light (Kinect v1, Face ID), time-of-flight (Kinect v2, LiDAR), stereo (Quest, HoloLens), LiDAR (iPhone Pro, iPad Pro).

Sensor Comparison

Structured light: High accuracy, short range. ToF: Medium accuracy, longer range. Stereo IR: Low power, medium accuracy. LiDAR: High accuracy, long range, expensive. Software depth: ARCore Depth API, monocular depth estimation (MiDaS, Depth Anything).

💡 Structured Light⏱️ ToF👓 Stereo IR📡 LiDAR

// Depth map processing (OpenCV + NumPy)
import numpy as np
import cv2

def process_depth(depth_image, camera_matrix):
    # Convert to meters (depends on sensor)
    depth_meters = depth_image.astype(np.float32) / 1000.0  # mm to m
    
    # Filter invalid depths
    valid_mask = (depth_meters > 0.1) & (depth_meters < 5.0)
    
    # Bilateral filter for smoothing
    depth_filtered = cv2.bilateralFilter(depth_meters, 5, 0.5, 0.5)
    
    # Back-project to 3D point cloud
    height, width = depth_filtered.shape
    fx = camera_matrix[0, 0]
    fy = camera_matrix[1, 1]
    cx = camera_matrix[0, 2]
    cy = camera_matrix[1, 2]
    
    points_3d = []
    for v in range(height):
        for u in range(width):
            if valid_mask[v, u]:
                z = depth_filtered[v, u]
                x = (u - cx) * z / fx
                y = (v - cy) * z / fy
                points_3d.append([x, y, z])
    
    return np.array(points_3d), depth_filtered

🧊 TSDF & Volumetric Fusion

🎯 Fusing depth into a volumetric model

TSDF (Truncated Signed Distance Function) is a volumetric representation that stores the signed distance to the nearest surface in each voxel. It fuses multiple depth frames into a consistent 3D model.

How TSDF Works

Voxel grid: 3D grid covering the scene. Signed distance: Positive = in front of surface, negative = behind. Truncation: Values clamped to [-1, 1] around surface. Weighted average: New depth measurements update voxels. Marching cubes: Extract mesh at zero-crossing.

🧊 Voxel Grid📏 Signed Distance🔗 Weighted Fusion🕸️ Marching Cubes

# TSDF integration (simplified)
class TSDFVolume:
    def __init__(self, resolution, size):
        self.resolution = resolution
        self.size = size
        self.voxels = np.zeros((size, size, size), dtype=np.float32)
        self.weights = np.zeros((size, size, size), dtype=np.float32)
    
    def integrate(self, depth_map, camera_pose, camera_matrix, truncation=0.1):
        # For each voxel, project to camera and compare depth
        for x in range(self.size):
            for y in range(self.size):
                for z in range(self.size):
                    # Voxel center in world coordinates
                    world_pos = self.voxel_to_world(x, y, z)
                    
                    # Project to camera
                    camera_pos = camera_pose.inverse() @ world_pos
                    pixel = camera_matrix @ camera_pos[:3]
                    u, v = int(pixel[0] / pixel[2]), int(pixel[1] / pixel[2])
                    
                    if 0 <= u < depth_map.shape[1] and 0 <= v < depth_map.shape[0]:
                        depth = depth_map[v, u]
                        sdf = depth - camera_pos[2]
                        
                        if sdf > -truncation:
                            # Truncate and update
                            tsdf = np.clip(sdf / truncation, -1, 1)
                            old_weight = self.weights[x, y, z]
                            new_weight = old_weight + 1
                            self.voxels[x, y, z] = (self.voxels[x, y, z] * old_weight + tsdf) / new_weight
                            self.weights[x, y, z] = new_weight
    
    def extract_mesh(self):
        # Marching cubes at zero-crossing
        from skimage import measure
        verts, faces, normals, values = measure.marching_cubes(self.voxels, level=0)
        return verts, faces, normals

🕸️ Mesh Generation

🎯 From voxels to triangles

Mesh generation converts the volumetric representation (TSDF, occupancy grid) into a triangle mesh that can be rendered, physics-simulated, and used for occlusion.

Mesh Extraction Methods

Marching Cubes: Classic algorithm — extracts isosurface from voxel grid. Marching Tetrahedra: Avoids ambiguity. Dual Contouring: Better topology. Poisson Reconstruction: Smooth surfaces from point clouds. Delaunay: Triangulation from points.

🧊 Marching Cubes🔺 Dual Contouring🌊 Poisson

// Marching cubes mesh extraction (Unity)
using System.Collections.Generic;
using UnityEngine;

public class MarchingCubes {
    public static Mesh ExtractMesh(float[,,] voxels, float isoLevel, float scale) {
        List vertices = new List();
        List triangles = new List();
        
        int width = voxels.GetLength(0);
        int height = voxels.GetLength(1);
        int depth = voxels.GetLength(2);
        
        for (int x = 0; x < width - 1; x++) {
            for (int y = 0; y < height - 1; y++) {
                for (int z = 0; z < depth - 1; z++) {
                    // Get cube corner values
                    float[] cube = new float[8];
                    cube[0] = voxels[x, y, z];
                    cube[1] = voxels[x + 1, y, z];
                    cube[2] = voxels[x + 1, y, z + 1];
                    cube[3] = voxels[x, y, z + 1];
                    cube[4] = voxels[x, y + 1, z];
                    cube[5] = voxels[x + 1, y + 1, z];
                    cube[6] = voxels[x + 1, y + 1, z + 1];
                    cube[7] = voxels[x, y + 1, z + 1];
                    
                    // Determine cube configuration
                    int config = 0;
                    for (int i = 0; i < 8; i++) {
                        if (cube[i] < isoLevel) config |= (1 << i);
                    }
                    
                    // Look up triangle edges (table omitted for brevity)
                    // Add vertices and triangles...
                }
            }
        }
        
        Mesh mesh = new Mesh();
        mesh.vertices = vertices.ToArray();
        mesh.triangles = triangles.ToArray();
        mesh.RecalculateNormals();
        return mesh;
    }
}

🏷️ Semantic Mapping

🎯 Understanding what surfaces are

Semantic mapping adds labels to geometry: floor, wall, ceiling, table, chair, person. This enables intelligent AR interactions — placing content only on tables, avoiding walls, etc.

Semantic Segmentation

2D segmentation: Label pixels in camera images (DeepLab, Mask R-CNN). 3D fusion: Project labels onto mesh. ARKit: Scene Geometry with classification (floor, wall, ceiling, table, seat, window, door). Use cases: Intelligent placement, navigation, context-aware UI.

🪑 Furniture🧱 Walls📏 Floor🚪 Doors

// ARKit scene geometry classification (Swift)
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
    for anchor in anchors {
        if let meshAnchor = anchor as? ARMeshAnchor {
            let geometry = meshAnchor.geometry
            let classification = geometry.classification
            
            // Iterate over faces
            for i in 0..



  
  
    📏 Plane Extraction
    🎯 Finding flat surfaces
Plane extraction identifies large flat surfaces (floors, walls, tables) from the mesh or point cloud. It's essential for object placement and room layout understanding.
     Plane Extraction Methods
RANSAC: Iteratively fit planes to point cloud. Region growing: Group coplanar points. Normal clustering: Group similar normals. Mesh segmentation: Use mesh connectivity. Output: Plane equation (normal + distance), boundary polygon, center, extent.
📐 RANSAC🌱 Region Growing🔺 Mesh Segmentation
    # RANSAC plane extraction (Open3D)
import open3d as o3d
import numpy as np

def extract_planes(point_cloud, max_planes=5, distance_threshold=0.01):
    planes = []
    remaining = point_cloud
    
    for _ in range(max_planes):
        if len(remaining.points) < 100:
            break
        
        # Fit plane with RANSAC
        plane_model, inliers = remaining.segment_plane(
            distance_threshold=distance_threshold,
            ransac_n=3,
            num_iterations=1000
        )
        
        # Extract inlier points
        inlier_cloud = remaining.select_by_index(inliers)
        outlier_cloud = remaining.select_by_index(inliers, invert=True)
        
        # Compute plane properties
        a, b, c, d = plane_model
        normal = np.array([a, b, c])
        normal /= np.linalg.norm(normal)
        
        planes.append({
            'normal': normal,
            'distance': d,
            'center': np.mean(np.asarray(inlier_cloud.points), axis=0),
            'extent': np.ptp(np.asarray(inlier_cloud.points), axis=0),
            'points': inlier_cloud
        })
        
        remaining = outlier_cloud
    
    return planes
  

  
  
    ⚙️ Physics & Collision
    🎯 Making virtual objects interact with real geometry
Spatial mapping enables physics simulation in AR/VR: virtual objects collide with real walls, fall on real floors, and rest on real tables.
     Physics with Spatial Maps
Mesh colliders: Use spatial mesh as collision geometry. Rigidbodies: Virtual objects with mass, velocity. Gravity: Objects fall to real floor. Raycasts: Interaction rays hit real surfaces. Performance: Simplify mesh for physics (convex hulls, simplified colliders).
🧱 Mesh Collider⚙️ Rigidbody🌍 Gravity🎯 Raycast
    // Unity: Using spatial mesh for physics
using UnityEngine;
using UnityEngine.XR.ARFoundation;

public class SpatialPhysics : MonoBehaviour {
    public ARMeshManager meshManager;
    public GameObject virtualBall;
    
    void Start() {
        meshManager.meshesChanged += OnMeshesChanged;
    }
    
    void OnMeshesChanged(ARMeshesChangedEventArgs args) {
        foreach (var meshFilter in args.added) {
            // Add MeshCollider to spatial mesh
            var collider = meshFilter.gameObject.AddComponent();
            collider.sharedMesh = meshFilter.mesh;
            collider.convex = false;  // static environment
        }
    }
    
    void SpawnBall() {
        // Instantiate ball with Rigidbody
        var ball = Instantiate(virtualBall, Camera.main.transform.position + Camera.main.transform.forward * 2, Quaternion.identity);
        var rb = ball.GetComponent();
        rb.useGravity = true;
        // Ball will collide with real surfaces via MeshCollider
    }
}
  

  
  
    🧱 Occlusion & Depth
    🎯 Hiding virtual objects behind real ones
Occlusion is the #1 realism booster in AR. Spatial mapping provides the geometry needed to occlude virtual objects with real surfaces — and vice versa.
     Occlusion Techniques
Depth-based: Compare virtual depth with real depth per pixel. Mesh-based: Render spatial mesh as depth-only. Soft occlusion: Fade edges for smoother transitions. People occlusion: Segment people and occlude. ARKit/ARCore: Provide depth and mesh for occlusion.
📡 Depth Map🕸️ Mesh🧑 People🌫️ Soft Edges
    // Occlusion shader (Unity URP)
Shader "AR/Occlusion" {
    SubShader {
        Tags { "RenderType"="Opaque" "Queue"="Geometry-1" }
        
        Pass {
            ZWrite On
            ZTest LEqual
            ColorMask 0  // Don't write color
            
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            
            struct appdata {
                float4 vertex : POSITION;
            };
            
            struct v2f {
                float4 pos : SV_POSITION;
            };
            
            v2f vert(appdata v) {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                return o;
            }
            
            fixed4 frag(v2f i) : SV_Target {
                return fixed4(0, 0, 0, 0);
            }
            ENDCG
        }
    }
}

// Apply to spatial mesh to occlude virtual objects
// Virtual objects render after, with ZTest LEqual
  

  
  
    🚶 NavMesh & Pathfinding
    🎯 Navigating the real world
A NavMesh (Navigation Mesh) is generated from the spatial map to enable pathfinding for virtual characters, robots, or guided tours in AR/VR.
     NavMesh Generation
Walkable areas: Extract floor surfaces. Obstacles: Mark walls, furniture. NavMesh baking: Generate navigation graph. Pathfinding: A* or NavMesh path. Dynamic updates: Re-bake as map updates. Use cases: Virtual pets, guided tours, robot navigation.
🚶 Walkable🧱 Obstacles🔍 A*🔄 Dynamic
    // Unity NavMesh from spatial mesh
using UnityEngine;
using UnityEngine.AI;
using UnityEngine.XR.ARFoundation;

public class SpatialNavMesh : MonoBehaviour {
    public ARMeshManager meshManager;
    public NavMeshSurface navMeshSurface;
    
    void Start() {
        meshManager.meshesChanged += OnMeshesChanged;
    }
    
    void OnMeshesChanged(ARMeshesChangedEventArgs args) {
        // Collect all spatial mesh objects
        var sources = new System.Collections.Generic.List();
        var settings = NavMesh.GetSettingsByID(0);
        var bounds = new Bounds(Vector3.zero, Vector3.one * 100);
        
        foreach (var meshFilter in meshManager.meshes) {
            var source = new NavMeshBuildSource();
            source.shape = NavMeshBuildSourceShape.Mesh;
            source.mesh = meshFilter.mesh;
            source.transform = meshFilter.transform.localToWorldMatrix;
            source.area = 0;  // walkable
            sources.Add(source);
        }
        
        // Build NavMesh
        NavMeshBuilder.BuildNavMeshData(settings, sources, bounds, Vector3.zero, Quaternion.identity);
    }
}
  

  
  
    💾 Map Persistence
    🎯 Saving and loading spatial maps
Persistence allows spatial maps to be saved and reloaded across sessions, enabling persistent AR experiences where content stays in place.
     Persistence Methods
ARKit ARWorldMap: Save/load complete world map. ARCore: Cloud Anchors + persistent anchors. Mesh serialization: Save mesh vertices/faces to disk. TSDF serialization: Save volumetric grid. Use cases: Persistent AR content, multi-session experiences.
🗺️ ARWorldMap☁️ Cloud Anchors💾 Mesh Serialization
    // ARKit ARWorldMap persistence (Swift)
func saveWorldMap() {
    session.getCurrentWorldMap { worldMap, error in
        guard let map = worldMap else { return }
        
        do {
            let data = try NSKeyedArchiver.archivedData(withRootObject: map, requiringSecureCoding: true)
            try data.write(to: self.worldMapURL)
            print("World map saved")
        } catch {
            print("Failed to save: \(error)")
        }
    }
}

func loadWorldMap() {
    do {
        let data = try Data(contentsOf: worldMapURL)
        let worldMap = try NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: data)
        
        let config = ARWorldTrackingConfiguration()
        config.initialWorldMap = worldMap
        config.planeDetection = [.horizontal, .vertical]
        session.run(config)
    } catch {
        print("Failed to load: \(error)")
    }
}

// ARCore: Save mesh to disk (Kotlin)
fun saveMesh(mesh: Mesh) {
    val file = File(context.filesDir, "spatial_mesh.obj")
    file.outputStream().use { output ->
        // Write vertices
        for (i in 0 until mesh.vertexCount) {
            val vertex = mesh.vertices[i]
            output.write("v ${vertex.x} ${vertex.y} ${vertex.z}\n".toByteArray())
        }
        // Write faces...
    }
}
  

  
  
    
    
☁️ AR Cloud & Shared Maps
    🎯 Maps that live in the cloud
AR Cloud stores spatial maps in the cloud, enabling multiple devices to share the same map and persistent AR experiences across sessions and users.
     AR Cloud Platforms
ARCore Cloud Anchors: Google's shared anchors. Azure Spatial Anchors: Microsoft's cross-platform solution. Niantic Lightship: VPS for large-scale AR. Immersal: Visual positioning. Use cases: Multiplayer AR, persistent content, location-based experiences.
☁️ Cloud Anchors🌍 VPS👥 Shared📍 Persistent
    // Azure Spatial Anchors (Unity)
using Microsoft.Azure.SpatialAnchors;
using Microsoft.Azure.SpatialAnchors.Unity;

public class SpatialAnchorManager : MonoBehaviour {
    private CloudSpatialAnchorSession cloudSession;
    
    async void Start() {
        cloudSession = new CloudSpatialAnchorSession();
        cloudSession.Configuration.AccountDomain = "your-account.azure.com";
        cloudSession.Configuration.AccountId = "your-account-id";
        cloudSession.Configuration.AccountKey = "your-key";
        cloudSession.Start();
    }
    
    async Task CreateAnchor(Vector3 position) {
        var anchor = new GameObject("Anchor").AddComponent();
        anchor.transform.position = position;
        
        var cloudAnchor = new CloudSpatialAnchor();
        cloudAnchor.LocalAnchor = anchor.NativeAnchor;
        cloudAnchor.Expiration = DateTimeOffset.Now.AddDays(7);
        
        await cloudSession.CreateAnchorAsync(cloudAnchor);
        string anchorId = cloudAnchor.Identifier;
        // Share anchorId with other users
    }
    
    async Task LocateAnchor(string anchorId) {
        var criteria = new AnchorLocateCriteria();
        criteria.Identifiers = new[] { anchorId };
        var watcher = cloudSession.CreateWatcher(criteria);
        // Wait for located event...
    }
}
  

  
  
    ⚡ Optimization & LOD
    🎯 Keeping spatial mapping performant
Spatial maps can be huge — millions of triangles. Optimization is critical for mobile AR/VR: LOD, mesh simplification, culling, and compression.
     Optimization Techniques
Mesh simplification: Reduce triangle count (quadric decimation). LOD: Multiple detail levels based on distance. Frustum culling: Render only visible parts. Occlusion culling: Skip hidden geometry. Compression: Draco, meshopt. Async updates: Update map in background.
📉 Simplification🔍 LOD✂️ Culling🗜️ Compression
    // Mesh simplification (Unity + MeshSimplify)
using UnityEngine;

public class MeshOptimizer : MonoBehaviour {
    public MeshFilter meshFilter;
    public float quality = 0.5f;  // 0 = max simplification, 1 = original
    
    void Start() {
        Mesh originalMesh = meshFilter.mesh;
        
        // Simplify mesh
        Mesh simplified = MeshSimplifier.Simplify(originalMesh, quality);
        meshFilter.mesh = simplified;
        
        Debug.Log($"Original: {originalMesh.triangles.Length / 3} tris, " +
                  $"Simplified: {simplified.triangles.Length / 3} tris");
    }
    
    // LOD Group setup
    void SetupLOD() {
        var lodGroup = gameObject.AddComponent();
        var lods = new LOD[3];
        
        // LOD 0: Full detail (0-10m)
        lods[0] = new LOD(0.5f, new Renderer[] { GetComponent() });
        
        // LOD 1: Medium detail (10-20m)
        var mediumMesh = MeshSimplifier.Simplify(meshFilter.mesh, 0.5f);
        var mediumObj = CreateLODObject(mediumMesh);
        lods[1] = new LOD(0.2f, new Renderer[] { mediumObj.GetComponent() });
        
        // LOD 2: Low detail (20m+)
        var lowMesh = MeshSimplifier.Simplify(meshFilter.mesh, 0.1f);
        var lowObj = CreateLODObject(lowMesh);
        lods[2] = new LOD(0.01f, new Renderer[] { lowObj.GetComponent() });
        
        lodGroup.SetLODs(lods);
    }
}
  

  
  
    ⚠️ Challenges & Failure Modes
    🎯 When spatial mapping breaks
Spatial mapping faces challenges: noisy depth, dynamic scenes, textureless surfaces, and memory constraints. Understanding these helps build robust systems.
     Common Issues
Depth noise: Sensor noise creates artifacts. Dynamic objects: Moving people/objects corrupt the map. Textureless surfaces: White walls are hard to map. Memory: Large scenes exceed device RAM. Latency: Map updates lag behind camera. Drift: Map misalignment over time.
📡 Noise🚶 Dynamic🧱 Textureless💾 Memory
    # Robust spatial mapping: outlier removal
import numpy as np
from scipy.spatial import KDTree

def remove_outliers(point_cloud, k=10, std_ratio=2.0):
    """Remove statistical outliers from point cloud."""
    tree = KDTree(point_cloud)
    distances, _ = tree.query(point_cloud, k=k)
    
    # Mean distance to k nearest neighbors
    mean_distances = np.mean(distances[:, 1:], axis=1)
    
    # Global mean and std
    global_mean = np.mean(mean_distances)
    global_std = np.std(mean_distances)
    
    # Keep points within std_ratio * std
    threshold = global_mean + std_ratio * global_std
    inliers = mean_distances < threshold
    
    return point_cloud[inliers]

# Temporal consistency check
def temporal_filter(current_points, previous_points, threshold=0.05):
    """Keep only points consistent with previous frame."""
    if previous_points is None:
        return current_points
    
    tree = KDTree(previous_points)
    distances, _ = tree.query(current_points)
    
    # Keep points near previous frame's points
    consistent = distances < threshold
    return current_points[consistent]
  

  
  
    🚀 Future: Neural Mapping
    🎯 AI-powered spatial understanding
Neural networks are revolutionizing spatial mapping: NeRF, Gaussian Splatting, semantic understanding, and generative scene completion.
     Neural Mapping Techniques
NeRF: Photorealistic volumetric reconstruction. Gaussian Splatting: Real-time photorealistic rendering. Semantic NeRF: Label-aware reconstruction. Generative: Complete missing parts of the scene. Neural TSDF: Learned implicit surfaces. These enable photorealistic AR/VR with full scene understanding.
🧠 NeRF✨ Gaussian Splat🏷️ Semantic🎨 Generative
    # Neural implicit mapping (simplified)
import torch
import torch.nn as nn

class NeuralSDF(nn.Module):
    """Neural network representing a signed distance function."""
    def __init__(self, hidden_dim=256):
        super().__init__()
        self.mlp = nn.Sequential(
            nn.Linear(3, hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),
            nn.Linear(hidden_dim, 1)  # SDF value
        )
    
    def forward(self, points):
        return self.mlp(points)

def train_neural_sdf(model, depth_maps, poses, camera_matrix, iterations=1000):
    """Train neural SDF from depth data."""
    optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
    
    for i in range(iterations):
        # Sample points along rays
        rays = sample_rays(camera_matrix, poses, depth_maps)
        points = sample_points_along_rays(rays)
        
        # Predict SDF
        sdf_values = model(points)
        
        # Depth supervision
        depth_loss = compute_depth_loss(sdf_values, rays.depths)
        
        # Eikonal regularization (|∇SDF| = 1)
        eikonal_loss = compute_eikonal_loss(model, points)
        
        loss = depth_loss + 0.1 * eikonal_loss
        
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()