🧠 Point Cloud Fundamentals

🎯 What is a Point Cloud?

A point cloud is a set of data points in 3D space, typically produced by 3D scanners (LiDAR, depth cameras, photogrammetry). Each point has (x, y, z) coordinates and may include color (RGB), normals, intensity, and classification labels. Point clouds are the raw output of most 3D sensing — the "pixels of the 3D world."

Why Point Clouds Matter

Raw 3D data: Direct representation of geometry without topology. Versatile: Can be converted to meshes, voxels, or implicit surfaces. Rich: Color, intensity, normals, semantic labels. Applications: Autonomous driving, AR/VR, robotics, surveying, cultural heritage, medical imaging.

📌 3D Points📌 XYZ + RGB📌 No Topology📌 Raw Sensor Data
# Point cloud basics with Open3D import open3d as o3d import numpy as np # Create a point cloud from NumPy array points = np.random.rand(1000, 3) # 1000 random 3D points colors = np.random.rand(1000, 3) # random RGB colors pcd = o3d.geometry.PointCloud() pcd.points = o3d.utility.Vector3dVector(points) pcd.colors = o3d.utility.Vector3dVector(colors) # Basic properties print(f"Number of points: {len(pcd.points)}") print(f"Bounding box: {pcd.get_axis_aligned_bounding_box()}") print(f"Center: {pcd.get_center()}") # Visualize o3d.visualization.draw_geometries([pcd])

📡 Acquisition & Sensors

🎯 How point clouds are captured

Point clouds come from: LiDAR (laser scanning), depth cameras (structured light, ToF, stereo), photogrammetry (multi-view stereo), and SLAM systems. Each has different density, accuracy, and cost.

Acquisition Methods

Terrestrial LiDAR: High accuracy, static scans. Mobile LiDAR: Vehicle-mounted, large areas. Aerial LiDAR: Drone-based, terrain mapping. Depth cameras: Real-time, short range (Kinect, RealSense, iPhone LiDAR). Photogrammetry: From photos, dense but noisy. SLAM: Real-time, incremental mapping.

🔦 LiDAR📷 Depth Camera📸 Photogrammetry📍 SLAM
# Reading a LiDAR point cloud (LAS/LAZ format) import laspy import numpy as np # Read LAS file las = laspy.read("scan.las") # Extract coordinates x = las.x y = las.y z = las.z # Extract attributes intensity = las.intensity classification = las.classification return_number = las.return_number # Convert to NumPy array points = np.vstack((x, y, z)).T print(f"Loaded {len(points)} points") # Filter ground points (classification 2) ground_mask = classification == 2 ground_points = points[ground_mask] print(f"Ground points: {len(ground_points)}") # Create Open3D point cloud import open3d as o3d pcd = o3d.geometry.PointCloud() pcd.points = o3d.utility.Vector3dVector(points) pcd.colors = o3d.utility.Vector3dVector( np.vstack((intensity/65535, intensity/65535, intensity/65535)).T ) # Save as PLY o3d.io.write_point_cloud("output.ply", pcd)

📁 Formats & Storage

🎯 Storing 3D point data

Point cloud formats: PLY, PCD, LAS/LAZ, E57, OBJ, XYZ, and PTS. Each has different features (color, normals, classification, compression).

Format Comparison

PLY: Flexible, supports color/normals. PCD: PCL native, efficient. LAS/LAZ: LiDAR standard, supports classification. E57: ASTM standard for 3D imaging. OBJ: Text-based, simple. Draco: Google's compression for 3D. Potree: Web-based visualization format.

📄 PLY📦 PCD🛰️ LAS/LAZ🌐 Potree
# Format conversion with Open3D import open3d as o3d # Read various formats pcd_ply = o3d.io.read_point_cloud("input.ply") pcd_pcd = o3d.io.read_point_cloud("input.pcd") pcd_xyz = o3d.io.read_point_cloud("input.xyz") # Write to different formats o3d.io.write_point_cloud("output.ply", pcd_ply, write_ascii=False, compressed=True) o3d.io.write_point_cloud("output.pcd", pcd_ply, write_ascii=False) o3d.io.write_point_cloud("output.xyz", pcd_ply, write_ascii=True) # LAS conversion (laspy) import laspy header = laspy.LasHeader(point_format=3, version="1.2") header.offsets = np.min(points, axis=0) header.scales = np.array([0.001, 0.001, 0.001]) las = laspy.LasData(header) las.x = points[:, 0] las.y = points[:, 1] las.z = points[:, 2] las.red = (colors[:, 0] * 65535).astype(np.uint16) las.green = (colors[:, 1] * 65535).astype(np.uint16) las.blue = (colors[:, 2] * 65535).astype(np.uint16) las.write("output.las") # Compressed LAZ las.write("output.laz") # automatically compresses

🔗 Registration & Alignment

🎯 Aligning multiple scans

Point cloud registration aligns multiple scans into a common coordinate frame. Methods: ICP (Iterative Closest Point), feature-based (FPFH, RANSAC), and global registration.

Registration Pipeline

1. Feature extraction: Compute FPFH descriptors. 2. Feature matching: Find correspondences. 3. RANSAC: Robust initial alignment. 4. ICP refinement: Fine-tune with point-to-point or point-to-plane ICP. 5. Global optimization: Pose graph for multiple scans.

🔄 ICP🔍 FPFH🎯 RANSAC🌍 Global
# Point cloud registration with Open3D import open3d as o3d import numpy as np # Load two point clouds source = o3d.io.read_point_cloud("scan1.ply") target = o3d.io.read_point_cloud("scan2.ply") # Downsample for speed voxel_size = 0.05 source_down = source.voxel_down_sample(voxel_size) target_down = target.voxel_down_sample(voxel_size) # Compute normals source_down.estimate_normals( o3d.geometry.KDTreeSearchParamHybrid(radius=0.1, max_nn=30) ) target_down.estimate_normals( o3d.geometry.KDTreeSearchParamHybrid(radius=0.1, max_nn=30) ) # Compute FPFH features source_fpfh = o3d.pipelines.registration.compute_fpfh_feature( source_down, o3d.geometry.KDTreeSearchParamHybrid(radius=0.25, max_nn=100) ) target_fpfh = o3d.pipelines.registration.compute_fpfh_feature( target_down, o3d.geometry.KDTreeSearchParamHybrid(radius=0.25, max_nn=100) ) # Global registration (RANSAC) result_global = o3d.pipelines.registration.registration_ransac_based_on_feature_matching( source_down, target_down, source_fpfh, target_fpfh, mutual_filter=True, max_correspondence_distance=0.075, estimation_method=o3d.pipelines.registration.TransformationEstimationPointToPoint(False), ransac_n=4, checkers=[ o3d.pipelines.registration.CorrespondenceCheckerBasedOnEdgeLength(0.9), o3d.pipelines.registration.CorrespondenceCheckerBasedOnDistance(0.075) ], criteria=o3d.pipelines.registration.RANSACConvergenceCriteria(100000, 0.999) ) # ICP refinement result_icp = o3d.pipelines.registration.registration_icp( source, target, 0.02, result_global.transformation, o3d.pipelines.registration.TransformationEstimationPointToPlane() ) print(f"ICP fitness: {result_icp.fitness:.4f}") print(f"ICP inlier RMSE: {result_icp.inlier_rmse:.4f}") print(f"Transformation:\n{result_icp.transformation}") # Apply transformation source.transform(result_icp.transformation) o3d.visualization.draw_geometries([source, target])

🧹 Filtering & Denoising

🎯 Cleaning up point clouds

Raw point clouds contain noise, outliers, and redundant points. Filtering improves quality: voxel downsampling, statistical outlier removal, bilateral filtering, and radius outlier removal.

Filtering Techniques

Voxel downsampling: Reduce density uniformly. Statistical outlier removal: Remove points far from neighbors. Radius outlier removal: Remove points with few neighbors. Bilateral filter: Smooth while preserving edges. Pass-through filter: Crop to bounding box. ROR: Radius-based outlier removal.

📦 Voxel📊 Statistical🔘 Radius✂️ Crop
# Point cloud filtering with Open3D import open3d as o3d import numpy as np pcd = o3d.io.read_point_cloud("noisy.ply") print(f"Original: {len(pcd.points)} points") # 1. Voxel downsampling voxel_size = 0.02 # 2cm voxels pcd_down = pcd.voxel_down_sample(voxel_size) print(f"After voxel downsampling: {len(pcd_down.points)} points") # 2. Statistical outlier removal cl, ind = pcd_down.remove_statistical_outlier(nb_neighbors=20, std_ratio=2.0) pcd_stat = pcd_down.select_by_index(ind) print(f"After statistical removal: {len(pcd_stat.points)} points") # 3. Radius outlier removal cl, ind = pcd_stat.remove_radius_outlier(nb_points=16, radius=0.05) pcd_radius = pcd_stat.select_by_index(ind) print(f"After radius removal: {len(pcd_radius.points)} points") # 4. Pass-through filter (crop) points = np.asarray(pcd_radius.points) mask = (points[:, 2] > 0.0) & (points[:, 2] < 2.0) # Z between 0 and 2m pcd_cropped = pcd_radius.select_by_index(np.where(mask)[0]) print(f"After cropping: {len(pcd_cropped.points)} points") # 5. Bilateral filter (smooth) pcd_smooth = pcd_cropped.filter_smooth_bilateral( number_of_iterations=5, sigma_color=0.1, sigma_space=0.1 ) # Visualize o3d.visualization.draw_geometries([pcd_smooth])

✂️ Segmentation

🎯 Dividing point clouds into parts

Segmentation partitions a point cloud into meaningful regions: plane segmentation (RANSAC), cluster segmentation (DBSCAN), region growing, and semantic segmentation (deep learning).

Segmentation Methods

RANSAC plane: Extract dominant planes (floor, walls, tables). DBSCAN: Density-based clustering. Euclidean clustering: Group nearby points. Region growing: Grow regions based on normals/curvature. Semantic: PointNet, PointNet++, KPConv for labeled segmentation.

📐 RANSAC🔵 DBSCAN🌱 Region Growing🧠 Semantic
# Segmentation with Open3D import open3d as o3d import numpy as np pcd = o3d.io.read_point_cloud("scene.ply") # 1. Plane segmentation (RANSAC) plane_model, inliers = pcd.segment_plane( distance_threshold=0.01, ransac_n=3, num_iterations=1000 ) [a, b, c, d] = plane_model print(f"Plane equation: {a:.2f}x + {b:.2f}y + {c:.2f}z + {d:.2f} = 0") # Extract plane and non-plane points plane_cloud = pcd.select_by_index(inliers) non_plane_cloud = pcd.select_by_index(inliers, invert=True) # Color the plane red plane_cloud.paint_uniform_color([1, 0, 0]) # 2. Euclidean clustering on non-plane points with o3d.utility.VerbosityContextManager(o3d.utility.VerbosityLevel.Debug): labels = np.array(non_plane_cloud.cluster_dbscan( eps=0.02, min_points=10, print_progress=True )) max_label = labels.max() print(f"Found {max_label + 1} clusters") # Color each cluster differently colors = plt.get_cmap("tab20")(labels / (max_label if max_label > 0 else 1)) colors[labels < 0] = 0 # noise = black non_plane_cloud.colors = o3d.utility.Vector3dVector(colors[:, :3]) # Visualize o3d.visualization.draw_geometries([plane_cloud, non_plane_cloud])

🔍 Feature Extraction

🎯 Describing local geometry

Point cloud features capture local geometry: normals, curvature, FPFH (Fast Point Feature Histograms), SHOT, and 3D SIFT. Used for registration, recognition, and classification.

Feature Types

Normals: Surface orientation. Curvature: How curved the surface is. FPFH: 33-dimensional descriptor for registration. SHOT: Signature of Histograms of Orientations. Spin images: 2D representation of local geometry. Learned: PointNet features.

🧭 Normals📐 Curvature🔗 FPFH🎯 SHOT
# Feature extraction with Open3D import open3d as o3d import numpy as np pcd = o3d.io.read_point_cloud("object.ply") pcd.estimate_normals( o3d.geometry.KDTreeSearchParamHybrid(radius=0.1, max_nn=30) ) # Compute FPFH features fpfh = o3d.pipelines.registration.compute_fpfh_feature( pcd, o3d.geometry.KDTreeSearchParamHybrid(radius=0.25, max_nn=100) ) print(f"FPFH shape: {fpfh.data.shape}") # (33, num_points) # Compute normals and curvature points = np.asarray(pcd.points) normals = np.asarray(pcd.normals) # Estimate curvature from eigenvalues of local covariance def compute_curvature(pcd, radius=0.1): tree = o3d.geometry.KDTreeFlann(pcd) curvatures = [] for i in range(len(pcd.points)): k, idx, _ = tree.search_radius_vector_3d(pcd.points[i], radius) if k < 3: curvatures.append(0) continue neighbors = np.asarray(pcd.points)[idx, :] neighbors_centered = neighbors - np.mean(neighbors, axis=0) cov = np.dot(neighbors_centered.T, neighbors_centered) / k eigenvalues, _ = np.linalg.eig(cov) eigenvalues = np.sort(eigenvalues) curvature = eigenvalues[0] / (eigenvalues[0] + eigenvalues[1] + eigenvalues[2]) curvatures.append(curvature) return np.array(curvatures) curvatures = compute_curvature(pcd) print(f"Mean curvature: {np.mean(curvatures):.4f}") # Color by curvature colors = plt.get_cmap("jet")(curvatures / np.max(curvatures))[:, :3] pcd.colors = o3d.utility.Vector3dVector(colors)

🕸️ Meshing & Reconstruction

🎯 From points to surfaces

Meshing converts point clouds into triangle meshes for rendering, physics, and analysis. Methods: Poisson reconstruction, Ball Pivoting, Alpha Shapes, and Marching Cubes on TSDF.

Meshing Methods

Poisson: Smooth, watertight surfaces — best for closed objects. Ball Pivoting: Fast, preserves detail — good for clean data. Alpha Shapes: Generalization of convex hull. TSDF + Marching Cubes: Volumetric fusion for noisy data. Delaunay: 2.5D triangulation for terrain.

🌊 Poisson⚽ Ball Pivoting🔺 Alpha Shapes🧊 TSDF
# Meshing with Open3D import open3d as o3d import numpy as np pcd = o3d.io.read_point_cloud("object.ply") pcd.estimate_normals( o3d.geometry.KDTreeSearchParamHybrid(radius=0.1, max_nn=30) ) # Orient normals consistently pcd.orient_normals_consistent_tangent_plane(k=30) # 1. Poisson reconstruction mesh_poisson, densities = o3d.geometry.TriangleMesh.create_from_point_cloud_poisson( pcd, depth=9, width=0, scale=1.1, linear_fit=False ) print(f"Poisson mesh: {len(mesh_poisson.vertices)} vertices, {len(mesh_poisson.triangles)} triangles") # Remove low-density vertices (noise) densities = np.asarray(densities) density_threshold = np.quantile(densities, 0.05) vertices_to_remove = densities < density_threshold mesh_poisson.remove_vertices_by_mask(vertices_to_remove) # 2. Ball Pivoting radii = [0.005, 0.01, 0.02, 0.04] mesh_ball = o3d.geometry.TriangleMesh.create_from_point_cloud_ball_pivoting( pcd, o3d.utility.DoubleVector(radii) ) print(f"Ball Pivoting mesh: {len(mesh_ball.vertices)} vertices") # 3. Alpha Shapes alpha = 0.03 mesh_alpha = o3d.geometry.TriangleMesh.create_from_point_cloud_alpha_shape( pcd, alpha ) mesh_alpha.compute_vertex_normals() print(f"Alpha Shape mesh: {len(mesh_alpha.vertices)} vertices") # Save meshes o3d.io.write_triangle_mesh("mesh_poisson.ply", mesh_poisson) o3d.io.write_triangle_mesh("mesh_ball.ply", mesh_ball) o3d.io.write_triangle_mesh("mesh_alpha.ply", mesh_alpha) # Visualize o3d.visualization.draw_geometries([mesh_poisson])

🎨 Visualization

🎯 Seeing the data

Point cloud visualization ranges from simple (Open3D, PCL Viewer) to advanced (Potree for web, CloudCompare, MeshLab). Key considerations: performance, color mapping, and interaction.

Visualization Tools

Open3D: Python/C++ library, easy to use. CloudCompare: Desktop tool for inspection and editing. MeshLab: Open-source mesh/point cloud processor. Potree: Web-based, handles billions of points. Three.js: WebGL for custom viewers. Unity/Unreal: Game engines with point cloud plugins.

🐍 Open3D☁️ CloudCompare🌐 Potree🎮 Unity
# Advanced visualization with Open3D import open3d as o3d import numpy as np pcd = o3d.io.read_point_cloud("scene.ply") # Color by height points = np.asarray(pcd.points) z_min, z_max = points[:, 2].min(), points[:, 2].max() colors = plt.get_cmap("viridis")((points[:, 2] - z_min) / (z_max - z_min))[:, :3] pcd.colors = o3d.utility.Vector3dVector(colors) # Create visualizer vis = o3d.visualization.Visualizer() vis.create_window(window_name="Point Cloud Viewer", width=1024, height=768) # Add point cloud vis.add_geometry(pcd) # Add coordinate frame coord_frame = o3d.geometry.TriangleMesh.create_coordinate_frame(size=1.0) vis.add_geometry(coord_frame) # Set render options opt = vis.get_render_option() opt.background_color = np.asarray([0.1, 0.1, 0.1]) # dark gray opt.point_size = 2.0 opt.show_coordinate_frame = True # Set view ctr = vis.get_view_control() ctr.set_zoom(0.8) ctr.set_front([0, 0, -1]) ctr.set_up([0, -1, 0]) # Run vis.run() vis.destroy_window() # Web-based visualization with Three.js (JavaScript) # # # const loader = new THREE.PLYLoader(); # loader.load('scene.ply', (geometry) => { # const material = new THREE.PointsMaterial({ size: 0.01, vertexColors: true }); # const points = new THREE.Points(geometry, material); # scene.add(points); # });

🧠 Deep Learning on Point Clouds

🎯 Neural networks for 3D data

Deep learning on point clouds has revolutionized 3D understanding: PointNet, PointNet++, Point Transformer, KPConv, and PointNeXt. Used for classification, segmentation, and detection.

Key Architectures

PointNet: First deep network for raw point clouds — permutation invariant. PointNet++: Hierarchical feature learning. Point Transformer: Attention-based. KPConv: Kernel point convolution. PointNeXt: Improved training and scaling. Applications: Classification, part segmentation, semantic segmentation, object detection.

🧠 PointNet🔺 PointNet++🤖 Transformer🎯 KPConv
# PointNet classification with PyTorch import torch import torch.nn as nn import torch.nn.functional as F class PointNet(nn.Module): def __init__(self, num_classes=40): super().__init__() # Input transform (3x3) self.input_transform = nn.Sequential( nn.Conv1d(3, 64, 1), nn.BatchNorm1d(64), nn.ReLU(), nn.Conv1d(64, 128, 1), nn.BatchNorm1d(128), nn.ReLU(), nn.Conv1d(128, 1024, 1), nn.BatchNorm1d(1024), nn.ReLU(), nn.AdaptiveMaxPool1d(1) ) # Feature transform (64x64) self.feature_transform = nn.Sequential( nn.Conv1d(64, 64, 1), nn.BatchNorm1d(64), nn.ReLU() ) # MLP for classification self.fc = nn.Sequential( nn.Linear(1024, 512), nn.BatchNorm1d(512), nn.ReLU(), nn.Dropout(0.3), nn.Linear(512, 256), nn.BatchNorm1d(256), nn.ReLU(), nn.Dropout(0.3), nn.Linear(256, num_classes) ) def forward(self, x): # x: (B, 3, N) point cloud B, _, N = x.shape # Input transform x = self.input_transform(x) # (B, 1024, 1) x = x.view(B, 1024) # Classification x = self.fc(x) return F.log_softmax(x, dim=1) # Usage model = PointNet(num_classes=40) points = torch.randn(8, 3, 1024) # batch of 8 point clouds output = model(points) print(f"Output shape: {output.shape}") # (8, 40) # Training optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) criterion = nn.NLLLoss() for epoch in range(100): for points, labels in dataloader: optimizer.zero_grad() output = model(points) loss = criterion(output, labels) loss.backward() optimizer.step()

🏷️ Classification & Segmentation

🎯 Labeling points and objects

Point cloud classification assigns a label to the entire cloud (e.g., chair, car). Segmentation assigns a label to each point (e.g., part of a chair, road vs building).

Tasks & Datasets

Classification: ModelNet40 (40 object categories). Part segmentation: ShapeNet (50 parts). Semantic segmentation: S3DIS, SemanticKITTI. Instance segmentation: Separate objects. Metrics: Accuracy, IoU, mIoU.

📦 ModelNet🚗 KITTI🏠 S3DIS📊 IoU
# Semantic segmentation with PointNet++ import torch import torch.nn as nn class PointNetSeg(nn.Module): def __init__(self, num_classes=13): super().__init__() # Encoder (shared MLP) self.encoder = nn.Sequential( nn.Conv1d(3, 64, 1), nn.BatchNorm1d(64), nn.ReLU(), nn.Conv1d(64, 128, 1), nn.BatchNorm1d(128), nn.ReLU(), nn.Conv1d(128, 1024, 1), nn.BatchNorm1d(1024), nn.ReLU() ) # Decoder (per-point) self.decoder = nn.Sequential( nn.Conv1d(1024 + 64, 512, 1), nn.BatchNorm1d(512), nn.ReLU(), nn.Conv1d(512, 256, 1), nn.BatchNorm1d(256), nn.ReLU(), nn.Conv1d(256, 128, 1), nn.BatchNorm1d(128), nn.ReLU(), nn.Conv1d(128, num_classes, 1) ) def forward(self, x): # x: (B, 3, N) B, _, N = x.shape # Local features local_feat = self.encoder[:6](x) # (B, 64, N) # Global features global_feat = self.encoder(x) # (B, 1024, N) global_feat = torch.max(global_feat, dim=2, keepdim=True)[0] # (B, 1024, 1) global_feat = global_feat.repeat(1, 1, N) # (B, 1024, N) # Concatenate local and global combined = torch.cat([local_feat, global_feat], dim=1) # (B, 1088, N) # Per-point classification output = self.decoder(combined) # (B, num_classes, N) return output # Usage model = PointNetSeg(num_classes=13) points = torch.randn(4, 3, 4096) # batch of 4 scenes output = model(points) print(f"Output shape: {output.shape}") # (4, 13, 4096) # Each point gets a class prediction predictions = output.argmax(dim=1) # (4, 4096) print(f"Predictions shape: {predictions.shape}")

📍 Point Clouds in SLAM

🎯 Mapping with SLAM

SLAM systems (ORB-SLAM, LIO-SAM, LOAM) produce point clouds as maps. These maps are used for localization, navigation, and AR/VR anchoring.

SLAM Point Cloud Types

Sparse: Feature points (ORB-SLAM). Semi-dense: Edge/depth points. Dense: Full point cloud (LIO-SAM, LOAM). Surfel-based: ElasticFusion. TSDF: Volumetric fusion (KinectFusion). Loop closure: Corrects drift in the map.

📍 Sparse🧱 Dense🌊 Surfel🔄 Loop Closure
# LIO-SAM: LiDAR-inertial SLAM (conceptual) # Point cloud registration for SLAM import numpy as np from scipy.spatial import KDTree class PointCloudSLAM: def __init__(self): self.map_points = [] self.keyframes = [] self.current_pose = np.eye(4) def process_scan(self, points, imu_data): # 1. Deskew (motion compensation using IMU) points_deskewed = self.deskew(points, imu_data) # 2. Feature extraction (edge and planar features) edge_features, planar_features = self.extract_features(points_deskewed) # 3. Scan-to-map matching (ICP) if len(self.map_points) > 0: transformation = self.scan_to_map_matching(edge_features, planar_features) self.current_pose = self.current_pose @ transformation # 4. Add to map self.map_points.extend(points_deskewed) # 5. Keyframe selection if self.is_keyframe(): self.keyframes.append({ 'pose': self.current_pose.copy(), 'points': points_deskewed }) return self.current_pose def scan_to_map_matching(self, edge_features, planar_features): # Build KD-tree of map map_array = np.array(self.map_points) tree = KDTree(map_array) # Find correspondences distances, indices = tree.query(edge_features) # Solve for transformation (point-to-line, point-to-plane) # ... (optimization) return np.eye(4) # placeholder

🥽 Point Clouds in AR/VR

🎯 3D data in immersive experiences

Point clouds are used in AR/VR for: spatial mapping (occlusion, physics), visualization (scientific data), telepresence (3D video), and content creation (scanning real objects).

AR/VR Point Cloud Applications

Spatial mapping: Real-time mesh from depth (ARKit, ARCore). Point cloud rendering: Efficient visualization of large scans. Telepresence: 3D point cloud video of people. Scanning: Capture real objects for AR. VPS: Visual positioning using point cloud maps. Performance: LOD, culling, compression (Draco, Potree).

🗺️ Spatial Mapping🎥 Telepresence📸 Scanning📍 VPS
// Rendering point clouds in Unity (C#) using UnityEngine; using System.Collections.Generic; public class PointCloudRenderer : MonoBehaviour { public TextAsset pointCloudFile; // PLY or XYZ public Material pointMaterial; public float pointSize = 0.01f; void Start() { LoadPointCloud(); } void LoadPointCloud() { string[] lines = pointCloudFile.text.Split('\n'); List points = new List(); List colors = new List(); foreach (string line in lines) { if (line.StartsWith("#") || line.StartsWith("ply") || line.StartsWith("end_header") || string.IsNullOrWhiteSpace(line)) continue; string[] parts = line.Split(' '); if (parts.Length >= 3) { float x = float.Parse(parts[0]); float y = float.Parse(parts[1]); float z = float.Parse(parts[2]); points.Add(new Vector3(x, y, z)); if (parts.Length >= 6) { float r = float.Parse(parts[3]) / 255f; float g = float.Parse(parts[4]) / 255f; float b = float.Parse(parts[5]) / 255f; colors.Add(new Color(r, g, b)); } else { colors.Add(Color.white); } } } // Create mesh Mesh mesh = new Mesh(); mesh.indexFormat = UnityEngine.Rendering.IndexFormat.UInt32; mesh.vertices = points.ToArray(); mesh.colors = colors.ToArray(); // Create indices (each point as a vertex) int[] indices = new int[points.Count]; for (int i = 0; i < points.Count; i++) indices[i] = i; mesh.SetIndices(indices, MeshTopology.Points, 0); // Assign to MeshFilter MeshFilter meshFilter = gameObject.AddComponent(); meshFilter.mesh = mesh; // Assign material MeshRenderer renderer = gameObject.AddComponent(); renderer.material = pointMaterial; Debug.Log($"Loaded {points.Count} points"); } }

🚀 Future: Neural Point Clouds

🎯 AI and point clouds

Neural networks are transforming point cloud processing: Neural Radiance Fields (NeRF), Gaussian Splatting, Point Transformers, and generative models for 3D content creation.

Neural Point Cloud Advances

NeRF: Neural radiance fields — photorealistic novel view synthesis. Gaussian Splatting: Real-time photorealistic rendering from point clouds. Point Transformers: Attention-based point cloud processing. Generative: Point-E, Diffusion for 3D generation. Neural compression: Learned point cloud compression. Foundation models: General-purpose 3D understanding.

🧠 NeRF✨ Gaussian Splat🤖 Transformer🎨 Generative
# Gaussian Splatting for point cloud rendering (simplified) import torch import torch.nn as nn class GaussianSplatting(nn.Module): def __init__(self, num_points=10000): super().__init__() # Learnable parameters per point self.positions = nn.Parameter(torch.randn(num_points, 3)) self.scales = nn.Parameter(torch.ones(num_points, 3) * 0.01) self.rotations = nn.Parameter(torch.zeros(num_points, 4)) # quaternions self.opacities = nn.Parameter(torch.ones(num_points, 1)) self.colors = nn.Parameter(torch.rand(num_points, 3)) def forward(self, camera_pose, camera_intrinsics): # Project 3D Gaussians to 2D # Compute covariance matrices # Sort by depth # Alpha compositing # Return rendered image pass # Training (from multi-view images) model = GaussianSplatting(num_points=50000) optimizer = torch.optim.Adam(model.parameters(), lr=1e-3) for iteration in range(30000): # Random camera camera_pose, camera_intrinsics = sample_camera() # Render rendered = model(camera_pose, camera_intrinsics) # Loss against ground truth gt_image = get_ground_truth(camera_pose) loss = photometric_loss(rendered, gt_image) optimizer.zero_grad() loss.backward() optimizer.step() # Densify/prune points periodically if iteration % 1000 == 0: model.densify_and_prune()