๐ง NeRF Fundamentals
๐ฏ What is NeRF?
NeRF (Neural Radiance Field) is a deep learning technique that represents a 3D scene as a continuous volumetric function. Given a set of 2D images with known camera poses, NeRF learns to synthesize novel views of the scene with photorealistic quality. It was introduced by Mildenhall et al. (2020) and has revolutionized 3D reconstruction and neural rendering.
How NeRF Works
1. Input: 2D images + camera poses. 2. NeRF MLP: A neural network that maps (x, y, z, ฮธ, ฯ) โ (RGB, density). 3. Volume rendering: Integrate color and density along camera rays. 4. Loss: Compare rendered pixel with ground truth. 5. Optimization: Backpropagate to learn the radiance field. 6. Novel views: Query the network from any camera pose.
๐ Radiance Field๐ Volume Rendering๐ MLP๐ Novel View
# NeRF pipeline (conceptual)
class NeRF:
def __init__(self):
self.mlp = MLP() # neural network
self.optimizer = Adam(self.mlp.parameters(), lr=5e-4)
def render_ray(self, ray_origin, ray_direction):
# 1. Sample points along the ray
t_vals = torch.linspace(near, far, num_samples)
points = ray_origin + ray_direction * t_vals
# 2. Query network at each point
rgb, density = self.mlp(points, ray_direction)
# 3. Volume rendering (integrate)
color = self.volume_render(rgb, density, t_vals)
return color
def train_step(self, images, poses):
# 1. Sample rays from images
rays = sample_rays(images, poses)
# 2. Render rays
rendered = [self.render_ray(o, d) for o, d in rays]
# 3. Compare with ground truth
loss = mse_loss(rendered, ground_truth_colors)
# 4. Optimize
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
return loss๐ Volume Rendering
๐ฏ Integrating light along rays
Volume rendering integrates the color and density along each camera ray to produce a 2D image. It's the mathematical foundation of NeRF. The rendering equation: C = โซ T(t) ฯ(t) c(t) dt, where T is transmittance, ฯ is density, and c is color.
Volume Rendering Equation
Transmittance: T(t) = exp(-โซ ฯ(s) ds) โ probability of light reaching t. Density: ฯ(t) โ how opaque the point is. Color: c(t) โ view-dependent RGB. Discrete approximation: Sum over samples. Alpha compositing: Front-to-back or back-to-front. Importance sampling: Sample more where density is high.
๐ Transmittance๐ Density๐จ Color๐ Alpha Blending
# Volume rendering (PyTorch)
import torch
def volume_rendering(rgb, density, t_vals, ray_directions):
"""
Volume rendering along a ray.
rgb: (N_samples, 3) color at each sample
density: (N_samples, 1) density at each sample
t_vals: (N_samples,) distance along ray
"""
# Compute distances between samples
dists = t_vals[..., 1:] - t_vals[..., :-1]
dists = torch.cat([dists, torch.tensor([1e10])], dim=-1)
# Scale by ray direction magnitude
dists = dists * torch.norm(ray_directions, dim=-1, keepdim=True)
# Alpha = 1 - exp(-density * distance)
alpha = 1.0 - torch.exp(-density * dists)
# Transmittance
# T_i = prod_{j๐ Positional Encoding
๐ฏ Helping networks learn high frequencies
Neural networks struggle to learn high-frequency functions. Positional encoding maps low-dimensional inputs (x, y, z) to higher-dimensional space using sinusoidal functions, enabling NeRF to represent fine details.
Fourier Features
Formula: ฮณ(p) = (sin(2โฐฯp), cos(2โฐฯp), sin(2ยนฯp), cos(2ยนฯp), ..., sin(2^(L-1)ฯp), cos(2^(L-1)ฯp)). L: Number of frequency octaves (10 for position, 4 for direction). Effect: Network can represent high-frequency details. Without encoding: Blurry, smooth results. NTK theory: Explains why this works.
๐ Sinusoidal๐ Fourier Features๐ข Octavesโจ High Frequency
# Positional encoding (PyTorch)
import torch
import numpy as np
class PositionalEncoding(torch.nn.Module):
def __init__(self, num_frequencies=10, include_input=True):
super().__init__()
self.num_frequencies = num_frequencies
self.include_input = include_input
# Frequency bands: 2^0, 2^1, ..., 2^(L-1)
self.freq_bands = 2.0 ** torch.linspace(0, num_frequencies - 1, num_frequencies)
def forward(self, x):
"""
x: (..., input_dim) input coordinates
Returns: (..., input_dim * 2 * num_frequencies + input_dim) encoded features
"""
encoded = []
if self.include_input:
encoded.append(x)
for freq in self.freq_bands:
encoded.append(torch.sin(x * freq * np.pi))
encoded.append(torch.cos(x * freq * np.pi))
return torch.cat(encoded, dim=-1)
# Usage in NeRF
class NeRFMLP(torch.nn.Module):
def __init__(self, pos_encoding=10, dir_encoding=4):
super().__init__()
self.pos_encoder = PositionalEncoding(pos_encoding)
self.dir_encoder = PositionalEncoding(dir_encoding)
# Input: 3D position + 3D direction
# Position: 3 + 3*2*10 = 63 features
# Direction: 3 + 3*2*4 = 27 features
self.pos_dim = 3 + 3 * 2 * pos_encoding
self.dir_dim = 3 + 3 * 2 * dir_encoding
# MLP layers
self.layers = torch.nn.Sequential(
torch.nn.Linear(self.pos_dim, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
torch.nn.Linear(256, 256), torch.nn.ReLU(),
)
# Density head (view-independent)
self.density_head = torch.nn.Sequential(
torch.nn.Linear(256, 1),
torch.nn.ReLU() # density must be positive
)
# Color head (view-dependent)
self.color_head = torch.nn.Sequential(
torch.nn.Linear(256 + self.dir_dim, 128),
torch.nn.ReLU(),
torch.nn.Linear(128, 3),
torch.nn.Sigmoid() # RGB in [0, 1]
)
def forward(self, positions, directions):
# Encode positions
pos_encoded = self.pos_encoder(positions)
# MLP
features = self.layers(pos_encoded)
# Density (view-independent)
density = self.density_head(features)
# Encode directions
dir_encoded = self.dir_encoder(directions)
# Color (view-dependent)
color_input = torch.cat([features, dir_encoded], dim=-1)
color = self.color_head(color_input)
return color, density
# Example usage
nerf = NeRFMLP(pos_encoding=10, dir_encoding=4)
positions = torch.randn(1024, 3) # 1024 points
directions = torch.randn(1024, 3) # 1024 directions
directions = directions / torch.norm(directions, dim=-1, keepdim=True)
color, density = nerf(positions, directions)
print(f"Color: {color.shape}, Density: {density.shape}")๐งฌ MLP Architecture
๐ฏ The neural network behind NeRF
NeRF uses a multi-layer perceptron (MLP) with skip connections. The network takes encoded position and direction, and outputs color and density. Typical: 8 layers, 256 neurons, ReLU activations, with a skip connection at layer 5.
Architecture Details
Input: Position (63 dim) + Direction (27 dim). Hidden: 8 layers ร 256 neurons. Skip connection: At layer 5, concatenate input. Density head: 1 neuron, ReLU (density โฅ 0). Color head: 3 neurons, Sigmoid (RGB โ [0,1]). View dependence: Color depends on direction; density does not.
๐งฌ 8 Layers๐ Skip Connection๐ 256 Neurons๐๏ธ View-Dependent
# Full NeRF MLP with skip connections
import torch
import torch.nn as nn
class NeRFMLP(nn.Module):
def __init__(self, pos_encoding=10, dir_encoding=4, hidden_dim=256, num_layers=8):
super().__init__()
self.pos_encoding = pos_encoding
self.dir_encoding = dir_encoding
self.hidden_dim = hidden_dim
# Input dimensions
pos_dim = 3 + 3 * 2 * pos_encoding # 63
dir_dim = 3 + 3 * 2 * dir_encoding # 27
# Position encoding layers
self.pos_encoder = PositionalEncoding(pos_encoding)
self.dir_encoder = PositionalEncoding(dir_encoding)
# First layer
self.fc1 = nn.Linear(pos_dim, hidden_dim)
# Intermediate layers (with skip connection at layer 5)
self.fc2 = nn.Linear(hidden_dim, hidden_dim)
self.fc3 = nn.Linear(hidden_dim, hidden_dim)
self.fc4 = nn.Linear(hidden_dim, hidden_dim)
self.fc5 = nn.Linear(hidden_dim + pos_dim, hidden_dim) # skip
self.fc6 = nn.Linear(hidden_dim, hidden_dim)
self.fc7 = nn.Linear(hidden_dim, hidden_dim)
self.fc8 = nn.Linear(hidden_dim, hidden_dim)
# Density head
self.density_head = nn.Linear(hidden_dim, 1)
# Color head (takes features + direction encoding)
self.color_head = nn.Sequential(
nn.Linear(hidden_dim + dir_dim, 128),
nn.ReLU(),
nn.Linear(128, 3),
nn.Sigmoid()
)
def forward(self, positions, directions):
# Encode inputs
pos_enc = self.pos_encoder(positions) # (N, 63)
dir_enc = self.dir_encoder(directions) # (N, 27)
# Layer 1
x = F.relu(self.fc1(pos_enc))
# Layers 2-4
x = F.relu(self.fc2(x))
x = F.relu(self.fc3(x))
x = F.relu(self.fc4(x))
# Layer 5 with skip connection
x = torch.cat([x, pos_enc], dim=-1)
x = F.relu(self.fc5(x))
# Layers 6-8
x = F.relu(self.fc6(x))
x = F.relu(self.fc7(x))
x = F.relu(self.fc8(x))
# Density (view-independent)
density = F.relu(self.density_head(x)) # (N, 1)
# Color (view-dependent)
color_input = torch.cat([x, dir_enc], dim=-1)
color = self.color_head(color_input) # (N, 3)
return color, density
# Instantiate
nerf = NeRFMLP(pos_encoding=10, dir_encoding=4)
print(f"Parameters: {sum(p.numel() for p in nerf.parameters()):,}")
# Forward pass
positions = torch.randn(1024, 3)
directions = torch.randn(1024, 3)
directions = directions / torch.norm(directions, dim=-1, keepdim=True)
color, density = nerf(positions, directions)
print(f"Color: {color.shape}, Density: {density.shape}")๐ Training NeRF
๐ฏ Learning from images
Training NeRF requires: input images with known camera poses (from COLMAP), ray sampling, volume rendering, and photometric loss. Training takes hours on a single GPU, but recent variants (Instant-NGP) reduce this to minutes.
Training Pipeline
1. Data: 20-100+ images, camera poses from COLMAP. 2. Ray sampling: Random rays from images. 3. Coarse network: Sample N points, render. 4. Fine network: Importance sample, render. 5. Loss: MSE between rendered and GT pixels. 6. Optimization: Adam, lr=5e-4, 100k-300k iterations.
๐ธ COLMAP๐ฏ Ray Sampling๐ MSE Lossโ๏ธ Adam
# Full NeRF training loop
import torch
import torch.nn.functional as F
from torch.optim import Adam
class NeRFTrainer:
def __init__(self, model, images, poses, intrinsics):
self.model = model
self.images = images # (N_images, H, W, 3)
self.poses = poses # (N_images, 4, 4)
self.intrinsics = intrinsics # (3, 3)
self.optimizer = Adam(model.parameters(), lr=5e-4, betas=(0.9, 0.999))
self.num_iterations = 300000
def train(self):
for iteration in range(self.num_iterations):
# 1. Sample a random image
img_idx = torch.randint(0, len(self.images), (1,))
image = self.images[img_idx]
pose = self.poses[img_idx]
# 2. Sample random rays
rays_o, rays_d, gt_colors = self.sample_rays(image, pose)
# 3. Coarse sampling
t_vals_coarse = torch.linspace(2.0, 6.0, 64)
pts_coarse = rays_o + rays_d * t_vals_coarse[..., None]
# Query network
rgb_coarse, density_coarse = self.model(pts_coarse, rays_d)
# Volume render
color_coarse, _, _, weights_coarse = volume_rendering(
rgb_coarse, density_coarse, t_vals_coarse, rays_d
)
# 4. Fine sampling (importance sampling)
t_vals_fine = hierarchical_sampling(
density_coarse, t_vals_coarse, 128
)
pts_fine = rays_o + rays_d * t_vals_fine[..., None]
# Query network
rgb_fine, density_fine = self.model(pts_fine, rays_d)
# Volume render
color_fine, _, _, _ = volume_rendering(
rgb_fine, density_fine, t_vals_fine, rays_d
)
# 5. Loss
loss_coarse = F.mse_loss(color_coarse, gt_colors)
loss_fine = F.mse_loss(color_fine, gt_colors)
loss = loss_coarse + loss_fine
# 6. Optimize
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
# 7. Logging
if iteration % 1000 == 0:
print(f"Iteration {iteration}: Loss = {loss.item():.6f}")
# Save checkpoint
if iteration % 10000 == 0:
torch.save({
'iteration': iteration,
'model_state_dict': self.model.state_dict(),
'optimizer_state_dict': self.optimizer.state_dict(),
'loss': loss.item()
}, f'nerf_checkpoint_{iteration}.pt')
def sample_rays(self, image, pose, num_rays=1024):
"""Sample rays from an image."""
H, W = image.shape[:2]
# Random pixel coordinates
pixels_x = torch.randint(0, W, (num_rays,))
pixels_y = torch.randint(0, H, (num_rays,))
# Convert to camera coordinates
fx, fy = self.intrinsics[0, 0], self.intrinsics[1, 1]
cx, cy = self.intrinsics[0, 2], self.intrinsics[1, 2]
# Ray directions in camera space
dirs_x = (pixels_x - cx) / fx
dirs_y = (pixels_y - cy) / fy
dirs_z = torch.ones_like(dirs_x)
dirs_cam = torch.stack([dirs_x, dirs_y, dirs_z], dim=-1)
# Transform to world space
R = pose[:3, :3]
t = pose[:3, 3]
rays_d = (R @ dirs_cam.T).T
rays_d = rays_d / torch.norm(rays_d, dim=-1, keepdim=True)
# Ray origins (camera position)
rays_o = t.expand(num_rays, 3)
# Ground truth colors
gt_colors = image[pixels_y, pixels_x] # (num_rays, 3)
return rays_o, rays_d, gt_colors
# Training
model = NeRFMLP()
trainer = NeRFTrainer(model, images, poses, intrinsics)
trainer.train()
# COLMAP: camera pose estimation
# colmap automatic_reconstructor \
# --workspace_path ./scene \
# --image_path ./images \
# --quality high๐๏ธ Novel View Synthesis
๐ฏ Rendering from any viewpoint
Once trained, NeRF can render the scene from any camera pose โ even viewpoints not seen during training. This is called novel view synthesis and is the primary application of NeRF.
View Synthesis Applications
Virtual cinematography: Change camera angles in post. 3D photography: Explore a scene from any angle. AR/VR: Render 3D content from user's viewpoint. Telepresence: Immersive 3D video. Cultural heritage: Digitize artifacts. E-commerce: 3D product views.
๐ฅ Cinematography๐ธ 3D Photo๐ฅฝ AR/VR๐๏ธ Heritage
# Novel view synthesis with trained NeRF
import torch
def render_novel_view(model, camera_pose, intrinsics, H, W, chunk_size=1024):
"""Render an image from a novel camera pose."""
model.eval()
with torch.no_grad():
# Generate rays for all pixels
rays_o, rays_d = generate_rays(camera_pose, intrinsics, H, W)
rays_o = rays_o.reshape(-1, 3)
rays_d = rays_d.reshape(-1, 3)
# Render in chunks (to fit in memory)
colors = []
depths = []
for i in range(0, len(rays_o), chunk_size):
chunk_o = rays_o[i:i+chunk_size]
chunk_d = rays_d[i:i+chunk_size]
# Sample points
t_vals = torch.linspace(2.0, 6.0, 128)
pts = chunk_o[:, None, :] + chunk_d[:, None, :] * t_vals[None, :, None]
# Query network
rgb, density = model(pts, chunk_d[:, None, :].expand_as(pts))
# Volume render
color, depth, _, _ = volume_rendering(
rgb, density, t_vals, chunk_d
)
colors.append(color)
depths.append(depth)
# Reshape to image
colors = torch.cat(colors, dim=0).reshape(H, W, 3)
depths = torch.cat(depths, dim=0).reshape(H, W)
return colors, depths
def generate_rays(camera_pose, intrinsics, H, W):
"""Generate rays for all pixels in an image."""
# Pixel coordinates
i, j = torch.meshgrid(
torch.arange(W), torch.arange(H), indexing='xy'
)
# Camera intrinsics
fx, fy = intrinsics[0, 0], intrinsics[1, 1]
cx, cy = intrinsics[0, 2], intrinsics[1, 2]
# Ray directions in camera space
dirs = torch.stack([
(i - cx) / fx,
-(j - cy) / fy,
-torch.ones_like(i)
], dim=-1) # (H, W, 3)
# Transform to world space
R = camera_pose[:3, :3]
t = camera_pose[:3, 3]
rays_d = dirs @ R.T # (H, W, 3)
rays_o = t.expand(rays_d.shape)
return rays_o, rays_d
# Render novel views
novel_pose = torch.tensor([
[1, 0, 0, 0.5],
[0, 1, 0, 0.2],
[0, 0, 1, 2.0],
[0, 0, 0, 1]
], dtype=torch.float32)
colors, depths = render_novel_view(model, novel_pose, intrinsics, 800, 800)
print(f"Rendered image: {colors.shape}")
print(f"Depth map: {depths.shape}")
# Save image
import torchvision
torchvision.utils.save_image(colors.permute(2, 0, 1), 'novel_view.png')
# Create a video by moving the camera
def create_orbit_video(model, intrinsics, H, W, num_frames=120):
"""Create an orbiting camera video."""
frames = []
for i in range(num_frames):
angle = 2 * np.pi * i / num_frames
radius = 3.0
# Camera pose orbiting the scene
pose = torch.tensor([
[np.cos(angle), 0, np.sin(angle), radius * np.sin(angle)],
[0, 1, 0, 0.5],
[-np.sin(angle), 0, np.cos(angle), radius * np.cos(angle)],
[0, 0, 0, 1]
], dtype=torch.float32)
colors, _ = render_novel_view(model, pose, intrinsics, H, W)
frames.append(colors)
# Save as video
import imageio
frames_np = [(f.cpu().numpy() * 255).astype(np.uint8) for f in frames]
imageio.mimsave('orbit.mp4', frames_np, fps=30)๐ NeRF Variants (Instant-NGP, etc.)
๐ฏ Faster, better, smaller
Original NeRF is slow (hours to train, seconds to render). Variants: Instant-NGP (hash encoding, seconds to train), Plenoxels (sparse voxels), Mip-NeRF (anti-aliasing), NeRF-W (in-the-wild), Zip-NeRF (fast + anti-aliased).
Key Variants
Instant-NGP: Multi-resolution hash encoding โ 1000x faster. Plenoxels: Sparse voxel grid, no MLP. Mip-NeRF: Cone tracing for anti-aliasing. NeRF-W: Handles varying lighting. TensoRF: Tensor decomposition. Zip-NeRF: Fast + anti-aliased. 3D Gaussian Splatting: Real-time rendering.
โก Instant-NGP๐ง Plenoxels๐ Mip-NeRFโจ Gaussian Splatting
# Instant-NGP: Multi-resolution hash encoding
import torch
import torch.nn as nn
class HashEncoding(nn.Module):
"""Multi-resolution hash grid encoding (Instant-NGP)."""
def __init__(self, num_levels=16, min_res=16, max_res=512,
log2_hashmap_size=19, features_per_level=2):
super().__init__()
self.num_levels = num_levels
self.min_res = min_res
self.max_res = max_res
self.log2_hashmap_size = log2_hashmap_size
self.features_per_level = features_per_level
# Resolution per level (geometric progression)
resolutions = [min_res * (max_res / min_res) ** (i / (num_levels - 1))
for i in range(num_levels)]
self.resolutions = resolutions
# Hash tables (one per level)
self.hash_tables = nn.ParameterList([
nn.Parameter(torch.randn(2 ** log2_hashmap_size, features_per_level) * 0.01)
for _ in range(num_levels)
])
def forward(self, x):
"""
x: (N, 3) positions in [0, 1]
Returns: (N, num_levels * features_per_level) encoded features
"""
N = x.shape[0]
features = []
for level in range(self.num_levels):
resolution = self.resolutions[level]
# Scale to grid coordinates
scaled = x * resolution
floored = torch.floor(scaled).long()
frac = scaled - floored
# Get 8 corner indices
corners = []
for dx in [0, 1]:
for dy in [0, 1]:
for dz in [0, 1]:
corner = floored + torch.tensor([dx, dy, dz])
# Hash to table index
hash_idx = self.hash_corner(corner, resolution)
corners.append(hash_idx)
# Trilinear interpolation
corner_features = []
for i, hash_idx in enumerate(corners):
dx, dy, dz = i // 4, (i // 2) % 2, i % 2
weight = (frac[:, 0] if dx else 1 - frac[:, 0]) * \
(frac[:, 1] if dy else 1 - frac[:, 1]) * \
(frac[:, 2] if dz else 1 - frac[:, 2])
feat = self.hash_tables[level][hash_idx % (2 ** self.log2_hashmap_size)]
corner_features.append(feat * weight[:, None])
level_features = sum(corner_features)
features.append(level_features)
return torch.cat(features, dim=-1)
def hash_corner(self, corner, resolution):
"""Hash a 3D corner to a table index."""
# Simple spatial hash
primes = torch.tensor([1, 2654435761, 805459861], device=corner.device)
idx = (corner * primes).sum(dim=-1)
return idx % (2 ** self.log2_hashmap_size)
# Instant-NGP full model
class InstantNGP(nn.Module):
def __init__(self):
super().__init__()
self.encoder = HashEncoding()
# Small MLP (much smaller than original NeRF)
self.mlp = nn.Sequential(
nn.Linear(16 * 2, 64), nn.ReLU(),
nn.Linear(64, 64), nn.ReLU(),
nn.Linear(64, 64), nn.ReLU(),
nn.Linear(64, 4) # RGB + density
)
def forward(self, positions, directions):
# Encode positions
encoded = self.encoder(positions)
# MLP
out = self.mlp(encoded)
color = torch.sigmoid(out[..., :3])
density = torch.relu(out[..., 3:])
return color, density
# Training: 1000x faster than original NeRF
model = InstantNGP()
# Train in seconds instead of hours!โจ Gaussian Splatting
๐ฏ Real-time photorealistic rendering
3D Gaussian Splatting (Kerbl et al., 2023) represents scenes as millions of 3D Gaussians. Unlike NeRF (which uses ray marching), Gaussian Splatting uses rasterization, enabling real-time rendering at 100+ fps with photorealistic quality.
Gaussian Splatting Explained
Representation: 3D Gaussians (position, scale, rotation, opacity, SH coefficients). Rendering: Project Gaussians to 2D, sort by depth, alpha blend. Training: Optimize Gaussian parameters from images. Advantages: Real-time, photorealistic, editable. Disadvantages: Large file size, no view-dependent effects (basic).
โก Real-timeโจ Photorealistic๐จ Editable๐ฆ Large Files
# 3D Gaussian Splatting (simplified)
import torch
import torch.nn as nn
import numpy as np
class GaussianSplatting(nn.Module):
"""3D Gaussian Splatting scene representation."""
def __init__(self, num_gaussians=100000):
super().__init__()
self.num_gaussians = num_gaussians
# Learnable parameters
self.positions = nn.Parameter(torch.randn(num_gaussians, 3) * 0.5)
self.scales = nn.Parameter(torch.ones(num_gaussians, 3) * 0.01)
self.rotations = nn.Parameter(torch.zeros(num_gaussians, 4)) # quaternions
self.opacities = nn.Parameter(torch.ones(num_gaussians, 1) * 0.5)
# Spherical harmonics (view-dependent color)
self.sh_coeffs = nn.Parameter(torch.randn(num_gaussians, 3, 16) * 0.1)
def forward(self, camera_pose, intrinsics, H, W):
"""Render an image from a camera."""
# 1. Transform Gaussians to camera space
R = camera_pose[:3, :3]
t = camera_pose[:3, 3]
positions_cam = (R @ self.positions.T).T + t
# 2. Project to 2D
fx, fy = intrinsics[0, 0], intrinsics[1, 1]
cx, cy = intrinsics[0, 2], intrinsics[1, 2]
u = fx * positions_cam[:, 0] / positions_cam[:, 2] + cx
v = fy * positions_cam[:, 1] / positions_cam[:, 2] + cy
# 3. Compute 2D covariance (simplified)
# (in practice, use EWA splatting)
scales_2d = self.scales[:, :2] / positions_cam[:, 2:3]
# 4. Sort by depth (back-to-front)
depths = positions_cam[:, 2]
sorted_idx = torch.argsort(depths, descending=True)
# 5. Alpha blend
image = torch.zeros(H, W, 3, device=self.positions.device)
for i in sorted_idx:
if u[i] < 0 or u[i] >= W or v[i] < 0 or v[i] >= H:
continue
# Gaussian contribution
color = self.evaluate_sh(self.sh_coeffs[i], camera_pose)
# Alpha
alpha = self.opacities[i] * torch.exp(
-((u[i] - u[i]) ** 2 + (v[i] - v[i]) ** 2) / (2 * scales_2d[i, 0] ** 2)
)
# Blend
x, y = int(u[i]), int(v[i])
image[y, x] = image[y, x] * (1 - alpha) + color * alpha
return image
def evaluate_sh(self, sh_coeffs, camera_pose):
"""Evaluate spherical harmonics for view-dependent color."""
# View direction
view_dir = camera_pose[:3, 2] # camera looking direction
# Evaluate SH (simplified)
color = sh_coeffs[:, 0] # DC component
return torch.sigmoid(color)
# Training
model = GaussianSplatting(num_gaussians=100000)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
for iteration in range(30000):
# Random camera
camera_pose, intrinsics = sample_camera()
H, W = 800, 800
# Render
rendered = model(camera_pose, intrinsics, H, W)
# Ground truth
gt_image = get_ground_truth(camera_pose)
# Loss
loss = F.mse_loss(rendered, gt_image)
# Optimize
optimizer.zero_grad()
loss.backward()
optimizer.step()
# Densify and prune (adaptive)
if iteration % 1000 == 0:
model.densify_and_prune()
if iteration % 1000 == 0:
print(f"Iteration {iteration}: Loss = {loss.item():.6f}")
# Real-time rendering (100+ fps)
def render_real_time(model, camera_pose, intrinsics):
with torch.no_grad():
return model(camera_pose, intrinsics, 800, 800)
# Gaussian Splatting in Unity/Unreal
# Use plugins like:
# - Unity: aras-p/UnityGaussianSplatting
# - Unreal: Gaussian Splatting plugin
# - Web: gsplat.js, Three.js GaussianSplats3D๐ Dynamic NeRF (4D)
๐ฏ Adding time to NeRF
Dynamic NeRF extends NeRF to handle moving scenes (4D: x, y, z, t). Applications: video synthesis, free-viewpoint video, telepresence. Variants: D-NeRF, Nerfies, HyperNeRF, K-Planes.
Dynamic NeRF Approaches
D-NeRF: Deformable NeRF with time input. Nerfies: Handles non-rigid deformation. HyperNeRF: Higher-dimensional space for topology changes. K-Planes: Decomposes 4D into 2D planes. Applications: Video, telepresence, sports replay.
โฑ๏ธ 4D๐ฅ Video๐ Deformation๐น Free-viewpoint
# D-NeRF: Deformable NeRF with time
import torch
import torch.nn as nn
class DeformationNetwork(nn.Module):
"""Predicts deformation of each point at a given time."""
def __init__(self, hidden_dim=256):
super().__init__()
self.network = nn.Sequential(
nn.Linear(4, hidden_dim), # (x, y, z, t)
nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),
nn.Linear(hidden_dim, hidden_dim), nn.ReLU(),
nn.Linear(hidden_dim, 3) # (ฮx, ฮy, ฮz)
)
def forward(self, positions, time):
"""
positions: (N, 3)
time: (N, 1) or scalar
"""
if isinstance(time, (int, float)):
time = torch.full((positions.shape[0], 1), time, device=positions.device)
input = torch.cat([positions, time], dim=-1)
delta = self.network(input)
return positions + delta # deformed positions
class DNeRF(nn.Module):
def __init__(self):
super().__init__()
self.deformation = DeformationNetwork()
self.nerf = NeRFMLP() # standard NeRF
def forward(self, positions, directions, time):
# Deform positions based on time
deformed_positions = self.deformation(positions, time)
# Query NeRF at deformed positions
color, density = self.nerf(deformed_positions, directions)
return color, density
# Training on video
model = DNeRF()
optimizer = torch.optim.Adam(model.parameters(), lr=5e-4)
for iteration in range(300000):
# Sample random frame (time)
time = torch.rand(1)
# Sample rays from that frame
rays_o, rays_d, gt_colors = sample_rays_from_frame(time)
# Render
color = render_ray_with_time(model, rays_o, rays_d, time)
# Loss
loss = F.mse_loss(color, gt_colors)
optimizer.zero_grad()
loss.backward()
optimizer.step()
# HyperNeRF: Higher-dimensional space
class HyperNeRF(nn.Module):
def __init__(self):
super().__init__()
# Ambient space network
self.ambient = nn.Sequential(
nn.Linear(4, 256), nn.ReLU(), # (x, y, z, t)
nn.Linear(256, 256), nn.ReLU(),
nn.Linear(256, 256), nn.ReLU(),
)
# Deformation to canonical space
self.deform = nn.Linear(256, 3)
# Canonical NeRF
self.nerf = NeRFMLP()
def forward(self, positions, directions, time):
# Encode (position, time)
input = torch.cat([positions, time], dim=-1)
features = self.ambient(input)
# Deform to canonical space
canonical_positions = positions + self.deform(features)
# Query canonical NeRF
color, density = self.nerf(canonical_positions, directions)
return color, density๐ฅฝ NeRF in AR/VR
๐ฏ Immersive neural rendering
NeRF enables photorealistic AR/VR experiences: 3D video for telepresence, neural avatars, realistic environments, and dynamic scenes. Combined with eye tracking and foveated rendering, it's becoming real-time.
NeRF in XR
Telepresence: 3D video of people (Nerfies). Avatars: Neural avatar rendering. Environments: Capture real spaces in 3D. Foveated: Render high-res only where looking. Streaming: Compressed NeRF over network. Gaussian Splatting: Real-time for VR.
๐ฅ Telepresence๐ง Avatars๐ Environmentsโก Real-time
// NeRF rendering in Unity (via plugin)
using UnityEngine;
using UnityEngine.XR;
public class NeRFVRRenderer : MonoBehaviour {
public TextAsset nerfModel; // ONNX or custom format
public Shader nerfShader;
public Camera xrCamera;
private ComputeShader nerfCompute;
private RenderTexture nerfOutput;
void Start() {
// Load NeRF model
nerfCompute = Instantiate(Resources.Load("NeRFCompute"));
nerfOutput = new RenderTexture(2048, 2048, 24);
nerfOutput.enableRandomWrite = true;
nerfOutput.Create();
}
void Update() {
// Get camera pose from XR
Vector3 cameraPos = xrCamera.transform.position;
Quaternion cameraRot = xrCamera.transform.rotation;
// Render NeRF from this pose
RenderNeRF(cameraPos, cameraRot);
// Apply to display
GetComponent().material.mainTexture = nerfOutput;
}
void RenderNeRF(Vector3 position, Quaternion rotation) {
// Set camera parameters
nerfCompute.SetVector("_CameraPosition", position);
nerfCompute.SetVector("_CameraForward", rotation * Vector3.forward);
nerfCompute.SetVector("_CameraUp", rotation * Vector3.up);
nerfCompute.SetVector("_CameraRight", rotation * Vector3.right);
// Foveated rendering: higher resolution in center
Vector2 gazeCenter = GetGazeDirection();
nerfCompute.SetVector("_GazeCenter", gazeCenter);
nerfCompute.SetFloat("_FovealRadius", 0.2f);
// Dispatch
nerfCompute.SetTexture(0, "_Output", nerfOutput);
nerfCompute.Dispatch(0, 2048 / 8, 2048 / 8, 1);
}
Vector2 GetGazeDirection() {
// From eye tracking
InputDevice eyeDevice = InputDevices.GetDeviceAtXRNode(XRNode.Head);
if (eyeDevice.TryGetFeatureValue(CommonUsages.eyesData, out Eyes eyes)) {
if (eyes.TryGetLeftEyePosition(out Vector3 leftPos)) {
Vector3 local = xrCamera.transform.InverseTransformPoint(leftPos);
return new Vector2(local.x, local.y);
}
}
return Vector2.zero;
}
}
// NeRF telepresence (3D video)
public class NeRFTelepresence : MonoBehaviour {
public string streamUrl;
private NeRFStreamDecoder decoder;
void Start() {
decoder = new NeRFStreamDecoder(streamUrl);
}
void Update() {
// Receive compressed NeRF frames
NeRFFrame frame = decoder.GetLatestFrame();
// Render from user's viewpoint
RenderTexture output = RenderNeRF(frame, Camera.main);
// Display
GetComponent().material.mainTexture = output;
}
}
// WebXR NeRF (Three.js + gsplat)
/*
import * as THREE from 'three';
import { GaussianSplats3D } from 'gaussian-splats-3d';
const viewer = new GaussianSplats3D.Viewer({
cameraUp: [0, -1, 0],
initialCameraPosition: [0, 0, 5],
initialCameraLookAt: [0, 0, 0]
});
viewer.addSplatScene('scene.splat', {
splatAlphaRemovalThreshold: 5,
showLoadingUI: true
});
renderer.xr.enabled = true;
renderer.setAnimationLoop((time, frame) => {
viewer.update();
viewer.render();
});
*/ ๐ NeRF + SLAM
๐ฏ Neural SLAM
Neural SLAM combines NeRF with SLAM to build photorealistic maps in real-time. Systems: iMAP, NICE-SLAM, Co-SLAM, Point-SLAM. Enables dense, photorealistic mapping for AR/VR and robotics.
Neural SLAM Systems
iMAP: First NeRF-based SLAM (2021). NICE-SLAM: Hierarchical feature grids. Co-SLAM: Joint coordinate and parametric encoding. Point-SLAM: Neural point cloud. Applications: AR mapping, robot navigation, 3D reconstruction.
๐ง iMAP๐๏ธ NICE-SLAM๐ Co-SLAM๐ Point-SLAM
# Neural SLAM pipeline (conceptual)
class NeuralSLAM:
def __init__(self):
self.nerf = NeRFMLP() # or Instant-NGP
self.tracker = Tracker()
self.mapper = Mapper()
self.keyframes = []
def process_frame(self, rgb, depth, camera_pose_init):
"""Process a new RGB-D frame."""
# 1. Track camera pose
pose = self.tracker.track(rgb, depth, self.nerf, camera_pose_init)
# 2. Check if keyframe
if self.is_keyframe(pose):
self.keyframes.append(Keyframe(rgb, depth, pose))
# 3. Update NeRF map
self.mapper.update(self.nerf, self.keyframes[-1])
# 4. Optional: bundle adjustment
if len(self.keyframes) % 10 == 0:
self.optimize_poses()
return pose
def is_keyframe(self, pose):
"""Determine if current frame should be a keyframe."""
if not self.keyframes:
return True
last_pose = self.keyframes[-1].pose
translation = np.linalg.norm(pose[:3, 3] - last_pose[:3, 3])
rotation = np.arccos((np.trace(pose[:3, :3].T @ last_pose[:3, :3]) - 1) / 2)
return translation > 0.1 or rotation > 0.1
def render_from_pose(self, pose):
"""Render from a given pose using the NeRF map."""
rays_o, rays_d = generate_rays(pose, self.intrinsics, H, W)
return render_rays(self.nerf, rays_o, rays_d)
# iMAP: Implicit Mapping and Positioning
class iMAP:
def __init__(self):
self.nerf = NeRFMLP()
self.optimizer = Adam(self.nerf.parameters(), lr=1e-3)
def track(self, rgb, depth, pose_init):
"""Track camera pose by minimizing rendering loss."""
pose = pose_init.clone().requires_grad_(True)
pose_optimizer = Adam([pose], lr=1e-3)
for _ in range(50):
# Render from current pose
rendered_rgb, rendered_depth = self.render(pose)
# Loss
loss = F.mse_loss(rendered_rgb, rgb) + \
F.mse_loss(rendered_depth, depth)
pose_optimizer.zero_grad()
loss.backward()
pose_optimizer.step()
return pose.detach()
def map(self, keyframes):
"""Update NeRF map from keyframes."""
for _ in range(100):
# Sample a keyframe
kf = random.choice(keyframes)
# Render
rendered_rgb, rendered_depth = self.render(kf.pose)
# Loss
loss = F.mse_loss(rendered_rgb, kf.rgb) + \
F.mse_loss(rendered_depth, kf.depth)
# Optimize NeRF
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()๐จ Generative NeRF
๐ฏ Creating 3D from text/images
Generative NeRF combines NeRF with generative models (GANs, diffusion) to create 3D content from text or single images. Models: DreamFusion, Magic3D, Zero-1-to-3, LRM, TripoSR.
Generative 3D Approaches
DreamFusion: Text-to-3D via score distillation. Magic3D: Coarse-to-fine 3D generation. Zero-1-to-3: Single image to 3D. LRM: Large Reconstruction Model. TripoSR: Fast single-image to 3D. Applications: AR content, games, e-commerce.
๐ Text-to-3D๐ผ๏ธ Image-to-3D๐จ DreamFusionโก TripoSR
# DreamFusion: Text-to-3D via score distillation
import torch
from diffusers import StableDiffusionPipeline
class DreamFusion:
def __init__(self):
# 2D diffusion model (text-to-image)
self.sd = StableDiffusionPipeline.from_pretrained(
"runwayml/stable-diffusion-v1-5"
).to("cuda")
# 3D representation (NeRF or Gaussian Splatting)
self.nerf = NeRFMLP()
self.optimizer = Adam(self.nerf.parameters(), lr=1e-3)
def generate(self, text_prompt, num_iterations=10000):
"""Generate 3D content from text prompt."""
# Encode text
text_embedding = self.sd.encode_prompt(text_prompt)
for iteration in range(num_iterations):
# 1. Random camera pose
pose = sample_random_camera()
# 2. Render NeRF from pose
rendered_image = self.render(pose)
# 3. Score distillation: use 2D diffusion as critic
# Add noise
noise = torch.randn_like(rendered_image)
t = torch.randint(0, 1000, (1,))
noisy_image = self.sd.scheduler.add_noise(rendered_image, noise, t)
# Predict noise
noise_pred = self.sd.unet(noisy_image, t, text_embedding).sample
# SDS loss (Score Distillation Sampling)
loss = F.mse_loss(noise_pred, noise)
# 4. Optimize NeRF
self.optimizer.zero_grad()
loss.backward()
self.optimizer.step()
if iteration % 1000 == 0:
print(f"Iteration {iteration}: Loss = {loss.item():.4f}")
return self.nerf
def render(self, pose):
"""Render NeRF from camera pose."""
rays_o, rays_d = generate_rays(pose, self.intrinsics, 64, 64)
color = render_rays(self.nerf, rays_o, rays_d)
return color.permute(2, 0, 1).unsqueeze(0)
# Usage
dreamfusion = DreamFusion()
nerf = dreamfusion.generate("a delicious hamburger")
# Save as mesh
mesh = extract_mesh(nerf)
mesh.export("hamburger.obj")
# TripoSR: Fast single-image to 3D
from tsr.system import TSR
class TripoSR:
def __init__(self):
self.model = TSR.from_pretrained(
"stabilityai/TripoSR",
config_name="config.yaml",
weight_name="model.ckpt"
)
self.model.renderer.set_chunk_size(8192)
self.model.to("cuda")
def generate(self, image):
"""Generate 3D mesh from a single image."""
with torch.no_grad():
scene_codes = self.model([image], device="cuda")
# Extract mesh
meshes = self.model.extract_mesh(scene_codes, resolution=256)
return meshes[0]
def generate_with_texture(self, image):
"""Generate 3D mesh with texture."""
with torch.no_grad():
scene_codes = self.model([image], device="cuda")
# Extract mesh with texture
mesh = self.model.extract_mesh(scene_codes, True, resolution=256)[0]
return mesh
# Usage
triposr = TripoSR()
mesh = triposr.generate(image)
mesh.export("generated.obj")โ ๏ธ Challenges & Limitations
๐ฏ What NeRF can't do (yet)
NeRF faces challenges: slow training, slow rendering, large models, static scenes only, and difficulty with reflective/transparent surfaces. Recent variants address many of these.
Key Limitations
Speed: Original NeRF takes hours to train. Rendering: Slow (seconds per frame). Model size: 100s of MB. Static: Can't handle motion (without variants). Reflections: Struggles with mirrors, glass. Transparency: Hard to model. Lighting: Baked-in, can't relight. Data: Needs many images + poses.
๐ข Slow๐ฆ Large๐ช Reflections๐ก Lighting
# Addressing NeRF limitations
class FastNeRF:
"""Instant-NGP: 1000x faster training and rendering."""
def __init__(self):
self.encoder = HashEncoding()
self.mlp = SmallMLP() # tiny MLP
def train(self, images, poses, iterations=10000):
# Train in seconds/minutes instead of hours
for i in range(iterations):
# ...
pass
class RelightableNeRF:
"""NeRF that can be relit with new lighting."""
def __init__(self):
self.albedo_net = MLP() # base color
self.normal_net = MLP() # surface normal
self.roughness_net = MLP() # roughness
self.light_net = MLP() # lighting
def render(self, pose, light_direction):
# Render with new lighting
albedo = self.albedo_net(pose)
normal = self.normal_net(pose)
roughness = self.roughness_net(pose)
# Apply lighting
color = self.shade(albedo, normal, roughness, light_direction)
return color
class ReflectiveNeRF:
"""NeRF for reflective surfaces."""
def __init__(self):
self.nerf = NeRFMLP()
self.reflection_net = MLP()
def render(self, pose):
# Direct view
direct = self.nerf(pose)
# Reflection view (mirror the ray)
reflected_pose = reflect(pose)
reflection = self.reflection_net(reflected_pose)
# Combine
return direct + reflection
# Data requirements
def assess_data_requirements(scene_type):
"""Estimate how many images are needed."""
requirements = {
'small_object': 20,
'room': 50,
'building': 200,
'outdoor': 500,
'large_scene': 1000
}
return requirements.get(scene_type, 100)
# Typical NeRF training data:
# - 20-100 images for small scenes
# - 100-500 for rooms
# - 500-1000+ for large scenes
# - Camera poses from COLMAP (or known)
# COLMAP pose estimation
"""
colmap automatic_reconstructor \
--workspace_path ./scene \
--image_path ./images \
--quality high \
--camera_model OPENCV
"""๐ Future: Neural Rendering
๐ฏ The future of 3D
Neural rendering is evolving rapidly: real-time NeRF, relightable, generative, dynamic, and integrated into AR/VR. The line between captured and generated 3D is blurring.
Future Directions
Real-time: Gaussian Splatting, Instant-NGP for VR. Relightable: Separate albedo, normal, lighting. Generative: Text/image-to-3D in seconds. Dynamic: 4D NeRF for telepresence. Compression: Small models for mobile. Foundation: General-purpose 3D models. Neural graphics: Replacing traditional rendering.
โก Real-time๐ก Relightable๐จ Generative๐ Dynamic
# Future: Real-time relightable NeRF in VR
class RealTimeNeRF:
"""NeRF optimized for VR: 90fps, relightable, dynamic."""
def __init__(self):
# Multi-resolution hash encoding
self.encoder = HashEncoding(num_levels=16)
# Small MLP (real-time)
self.mlp = TinyMLP(hidden_dim=64)
# Separate heads
self.albedo_head = nn.Linear(64, 3)
self.normal_head = nn.Linear(64, 3)
self.roughness_head = nn.Linear(64, 1)
self.density_head = nn.Linear(64, 1)
def render(self, pose, light_dir, eye_tracking):
"""
Render in real-time with:
- Foveated rendering (high-res where looking)
- Relighting (dynamic light)
- 90fps performance
"""
# Foveated ray sampling
rays = sample_rays_foveated(pose, eye_tracking)
# Query network
features = self.mlp(self.encoder(rays.positions))
# Decode
albedo = self.albedo_head(features)
normal = self.normal_head(features)
roughness = self.roughness_head(features)
density = self.density_head(features)
# Relight
color = shade(albedo, normal, roughness, light_dir)
# Volume render
image = volume_render(color, density, rays.t_vals)
return image
# Neural graphics pipeline
"""
Traditional rendering:
Geometry โ Materials โ Lighting โ Rasterization โ Image
Neural rendering:
Images โ Neural Network โ Novel Views
(No explicit geometry, materials, or lighting)
"""
# Foundation models for 3D
class Zero123:
"""Zero-shot single image to 3D (Zero-1-to-3)."""
def __init__(self):
self.model = load_model("zero123")
def generate(self, image, angle):
"""Generate novel view from single image at given angle."""
return self.model(image, angle)
class LRM:
"""Large Reconstruction Model (LRM)."""
def __init__(self):
self.model = load_model("lrm")
def reconstruct(self, image):
"""Reconstruct 3D from single image in seconds."""
return self.model(image)
# Integration with AR/VR platforms
"""
Unity:
- aras-p/UnityGaussianSplatting
- NeRF plugin (emerging)
Unreal:
- Gaussian Splatting plugin
- NeRF plugin (emerging)
WebXR:
- gsplat.js
- Three.js GaussianSplats3D
- NeRF in WebGPU
Vision Pro:
- Neural rendering pipeline
- Gaussian Splatting support
"""