🕶️ AR Core Foundations

🎯 What is Augmented Reality?

AR overlays digital content onto the real world in real-time. Unlike VR, AR does not replace reality — it enhances it. Core components: sensors (camera, IMU, GPS), tracking (SLAM), rendering (3D graphics), and interaction (touch, voice, gesture). ARKit, ARCore, and WebXR are major platforms.

The AR Pipeline

Sensor fusion combines camera, gyroscope, and accelerometer data to understand motion. SLAM (Simultaneous Localization and Mapping) builds a map of the environment while tracking the device's position. Projection maps 3D virtual objects to 2D screen coordinates. The entire pipeline must run at 60+ fps for a comfortable experience.

📌 SLAM📌 Sensor Fusion📌 60fps📌 ARKit/ARCore

// AR session pseudocode (ARKit style)
ARConfiguration config = new ARWorldTrackingConfiguration();
config.planeDetection = PlaneDetection.Horizontal;
ARSession session = new ARSession();
session.run(config);

session.onFrame = (frame) => {
  Matrix4x4 cameraPose = frame.camera.transform;
  List anchors = frame.anchors;
  renderVirtualObjects(cameraPose, anchors);
};

📍 SLAM & Tracking

🎯 Simultaneous Localization and Mapping

SLAM is the core technology that enables AR devices to understand where they are in the world. It builds a sparse or dense map of feature points while tracking the device's 6-DOF (degrees of freedom) pose.

Visual-Inertial SLAM

Modern AR uses visual-inertial odometry — fusing camera images with IMU data for robust tracking even in textureless environments. Loop closure detection corrects drift over time. ORB-SLAM, VINS-Mono are popular algorithms.

🧭 6-DOF📷 Feature Points🔄 Loop Closure

// Feature extraction (ORB)
vector keypoints;
Mat descriptors;
ORB::create()->detectAndCompute(frame, noArray(), keypoints, descriptors);

// Pose estimation via PnP
solvePnP( 3D_points, 2D_points, cameraMatrix, distCoeffs, rvec, tvec );

📌 Marker & Markerless AR

🎯 Image targets & Natural Features

Marker-based AR uses fiducial markers (QR codes, ArUco) for easy tracking. Markerless AR detects natural features (edges, textures) from the environment — more flexible but computationally heavier.

When to use what?

Markers are robust and fast, ideal for educational kits or business cards. Markerless enables world-scale AR (e.g., Google Maps Live View) and requires SLAM. Vuforia, ARCore Augmented Images support both.

📸 Marker detection🌍 World tracking

// ArUco marker detection
cv::aruco::detectMarkers(image, dictionary, markerCorners, markerIds);
if (!markerIds.empty()) {
  cv::aruco::estimatePoseSingleMarkers(markerCorners, markerLength, camMatrix, distCoeffs, rvecs, tvecs);
}

📐 Projection & Calibration

🎯 3D → 2D projection

Projection matrix transforms 3D world points to 2D screen coordinates. Camera calibration (intrinsic/extrinsic parameters) is critical for correct alignment.

Intrinsic & Extrinsic

Intrinsic: focal length, principal point, distortion. Extrinsic: rotation & translation of camera relative to world. AR frameworks handle this automatically, but understanding projection is key for custom rendering.

🎯 Projection Matrix📏 Calibration

// OpenGL projection matrix (perspective)
float fov = 60.0f, aspect = 1.33f, near = 0.1f, far = 100.0f;
Matrix4x4 proj = Matrix4x4::Perspective(fov, aspect, near, far);
// Model-view matrix from SLAM pose
Matrix4x4 view = cameraPose.inverse();
Matrix4x4 mvp = proj * view * model;

🎨 Real-time Rendering

🎯 3D graphics for AR

Rendering virtual objects with correct lighting, shadows, and occlusion. OpenGL, Vulkan, or Unity/Unreal engines are commonly used.

Performance & Quality

AR rendering must balance visual quality and battery life. Deferred shading, LOD, and instancing are techniques to maintain 60fps. PBR (physically based rendering) helps objects blend with real-world lighting.

⚡ 60fps🎨 PBR📦 Instancing

// Simple shader (GLSL)
#version 330 core
uniform mat4 mvp;
layout(location=0) in vec3 position;
void main() {
  gl_Position = mvp * vec4(position, 1.0);
}

💡 Lighting & Reflection

🎯 Environment lighting

AR apps capture environment lighting to cast realistic shadows and reflections on virtual objects. Light estimation (intensity, direction, color temperature) is essential.

Light Estimation

ARKit/ARCore provide ambient light intensity and environment probe for reflections. This makes virtual objects look like they belong in the real scene.

☀️ Light Probe🔆 Ambient Intensity

// ARCore light estimation
LightEstimate lightEstimate = frame.getLightEstimate();
float intensity = lightEstimate.getPixelIntensity();
ColorCorrection colorCorrection = lightEstimate.getColorCorrection();
shader.setUniform("lightIntensity", intensity);

✋ Interaction & Gestures

🎯 Touch, voice, hand tracking

AR interaction goes beyond taps: gestures (pinch, swipe), voice commands, and hand tracking (via depth sensors or ML).

Natural UX

Hand tracking with MediaPipe or Ultraleap allows direct manipulation of virtual objects. Gesture recognition (e.g., "pinch to place") is intuitive for AR.

🖐️ Hand Tracking👌 Gesture Recognition

// Gesture detection (pinch)
onTouchEvent(event) {
  if (event.pointerCount == 2) {
    float distance = distance(event.getX(0), event.getY(0), event.getX(1), event.getY(1));
    if (distance < threshold) handlePinch();
  }
}

🧱 Occlusion & Depth

🎯 Realistic occlusion

Virtual objects must be occluded by real objects for realism. Depth sensors (LiDAR, ToF) provide pixel-depth maps.

Depth-based Occlusion

LiDAR on iPhone/iPad gives dense depth maps. The depth map is used as a stencil or z-buffer to hide virtual geometry behind real surfaces.

📡 LiDAR🧩 Depth Buffer

// Using depth map as occlusion mask
depthTexture = frame.getDepthImage();
occlusionShader.setUniform("depthTexture", depthTexture);
// In fragment shader: if (depth < virtualDepth) discard;

☁️ AR Cloud & Persistence

🎯 Persistent AR experiences

AR Cloud stores spatial maps in the cloud, enabling shared and persistent AR experiences across devices and sessions.

Shared Anchors

Platforms like ARCore Cloud Anchors and ARKit Location Anchors allow users to see the same AR content anchored to real-world locations.

🌐 Persistent Anchors👥 Multi-user

// ARCore Cloud Anchor
if (isHosting) {
  CloudAnchor cloudAnchor = session.hostAnchor(anchor);
  // share cloudAnchor ID
} else {
  CloudAnchor resolved = session.resolveCloudAnchor(cloudAnchorId);
}

🗺️ Scene Mapping & Meshing

🎯 3D scene reconstruction

Mesh generation from depth data enables physics, collision, and realistic occlusion. Used for room-scale AR.

Mesh Reconstruction

ARKit Scene Geometry and ARCore Depth API provide real-time meshes. Useful for object placement on floors/walls and for physics simulations.

📐 3D Mesh🧱 Physics

// ARKit scene mesh
let sceneGeometry = frame.anchors.compactMap { $0 as? ARMeshAnchor }
for mesh in sceneGeometry {
  let geometry = mesh.geometry
  // use vertices, faces for rendering or physics
}

👥 Multi-user AR

🎯 Collaborative AR

Multiple users see and interact with shared AR content in real-time. Requires synchronization of poses and anchors.

Synchronization

Each device runs its own SLAM, but anchors are shared via a server. Peer-to-peer or client-server architectures are used.

📡 Anchor Sync🔄 Pose Interpolation

// Multi-user anchor sharing (simplified)
onAnchorCreated(anchor) {
  sendToServer({ type: "anchor", id: anchor.id, pose: anchor.pose });
}
onReceive(serverMsg) {
  if (msg.type === "anchor") {
    createAnchor(msg.pose, msg.id);
  }
}

⌚ Wearable AR (HMD)

🎯 Head-Mounted Displays

HoloLens, Magic Leap, and glasses-like devices offer hands-free AR. Display technologies: waveguides, retinal projection.

Optical vs Video See-through

Optical see-through (HoloLens) projects onto transparent lens. Video see-through (VR-like) uses cameras and displays. Each has trade-offs in FOV, latency, and brightness.

🕶️ FOV⚡ Latency

// HoloLens spatial mapping
SpatialMappingManager spatialMapping = GetComponent();
spatialMapping.StartObserver();
List surfaces = spatialMapping.GetSurfaces();

👁️ CV & Object Recognition

🎯 Recognizing real objects

AR can recognize objects (e.g., furniture, faces) using ML models and overlay relevant information or animations.

Object Detection + AR

Models like YOLO, SSD detect objects in camera frames. The bounding box is used to place AR labels or interactive elements on the object.

🤖 YOLO🎯 Bounding Box

// Object detection with TensorFlow Lite
Interpreter tflite = new Interpreter(model);
tflite.run(inputImage, output);
List detections = output.decode();
foreach (Detection d in detections) {
  if (d.className == "chair") {
    showARLabel(d.boundingBox);
  }
}

🚀 Future: AR + AI

🎯 AI-powered AR

Generative AI, LLMs, and real-time scene understanding will make AR context-aware, interactive, and truly intelligent.

Generative AR

Imagine describing what you want to see in AR — and it appears. NeRF, Gaussian Splatting and LLM agents will enable dynamic, interactive AR worlds.

🧠 NeRF🤖 LLM Agents🌌 Generative

// Conceptual: AR + LLM
def handle_ar_query(query, context):
  response = llm.generate(query, context)
  if "place a tree" in query:
    position = get_ground_plane()
    render_tree(position, response["tree_type"])
  return response["text"]