🎮 Unity XR Foundations

🎯 Unity as an XR Powerhouse

Unity is the most popular engine for XR development, powering ~70% of AR/VR content. Its cross-platform support, extensive XR toolkit, and C# scripting make it ideal for rapid prototyping and production. Unity XR supports AR (AR Foundation), VR (OpenXR, Oculus, SteamVR), and MR (HoloLens, Quest Pro).

Unity XR Ecosystem

Core packages: XR Plugin Management, XR Interaction Toolkit, AR Foundation, Input System. Platforms: Meta Quest, HoloLens, PlayStation VR, SteamVR, ARKit/ARCore phones. Unity's component-based architecture lets you build once and deploy across devices with minimal changes.

📌 Cross-platform📌 XR Toolkit📌 C# Scripting📌 AR + VR + MR

// Basic Unity XR setup (C#)
using UnityEngine;
using UnityEngine.XR;

public class XRChecker : MonoBehaviour {
    void Start() {
        if (XRSettings.isDeviceActive) {
            Debug.Log($"XR Device: {XRSettings.loadedDeviceName}");
            Debug.Log($"Eye Texture Size: {XRSettings.eyeTextureWidth}x{XRSettings.eyeTextureHeight}");
        }
    }
}

🔌 XR Plugin Management

🎯 Configuring XR per platform

XR Plugin Management replaces the legacy XR SDK. It allows enabling/disabling XR providers per platform (Oculus, OpenXR, ARCore, ARKit, etc.) without rebuilding from scratch.

Loaders & Providers

XR Plug-in Management is installed via Package Manager. Under Project Settings → XR Plug-in Management, you choose providers. For Android, enable Oculus and ARCore. For iOS, ARKit. For PC, OpenXR.

🔧 Per-platform📦 Package Manager🔄 OpenXR

// Programmatic XR loader check
using UnityEngine.XR.Management;

void Start() {
    XRGeneralSettings settings = XRGeneralSettings.Instance;
    if (settings.Manager.activeLoader != null) {
        Debug.Log($"Active Loader: {settings.Manager.activeLoader.name}");
    }
}

📍 XR Origin & Camera Rig

🎯 The player's presence in XR

XR Origin (formerly XR Rig) is the root GameObject containing the Camera Offset and Main Camera. It translates and rotates based on headset tracking.

XR Origin Components

Camera Offset positions the camera at floor level. Main Camera has the Tracked Pose Driver. XR Origin handles room-scale movement — when the user walks, the camera moves but the origin stays fixed.

🎥 Tracked Pose🧍 Floor Level📐 Room Scale

// Accessing XR camera
using UnityEngine;
using UnityEngine.XR;

public class XRCameraInfo : MonoBehaviour {
    void Update() {
        InputDevice head = InputDevices.GetDeviceAtXRNode(XRNode.Head);
        if (head.TryGetFeatureValue(CommonUsages.devicePosition, out Vector3 pos)) {
            Debug.Log($"Head Position: {pos}");
        }
    }
}

🎮 XR Input System

🎯 Controllers, hands, and more

The new Input System in Unity supports XR controllers (triggers, grips, joysticks, buttons) and hand tracking. Actions map to inputs across devices.

Input Actions & Bindings

Define Input Actions (e.g., "Select", "Grab") and bind them to controller buttons. The same action works across Oculus, Vive, and WMR controllers. XR Input Device classes give direct access to controller state.

🎮 Action-based🔗 Bindings📱 Cross-device

// Direct controller input (XR Input System)
using UnityEngine.XR;

void Update() {
    InputDevice rightHand = InputDevices.GetDeviceAtXRNode(XRNode.RightHand);
    if (rightHand.TryGetFeatureValue(CommonUsages.trigger, out float trigger)) {
        if (trigger > 0.5f) Debug.Log("Trigger pressed");
    }
    if (rightHand.TryGetFeatureValue(CommonUsages.gripButton, out bool grip)) {
        if (grip) Debug.Log("Grip held");
    }
}

✋ XR Interaction Toolkit

🎯 High-level interaction framework

The XR Interaction Toolkit provides components for grabbing, pointing, teleporting, and UI interaction — essential for VR/AR apps.

Interactors & Interactables

Interactors (e.g., XR Ray Interactor, XR Direct Interactor) detect and interact with Interactables (e.g., XR Grab Interactable). Events like OnSelectEntered let you respond to grabs.

🖐️ Grab🎯 Raycast🚀 Teleport🖱️ UI

// Grab interactable event
using UnityEngine.XR.Interaction.Toolkit;

public class GrabHandler : MonoBehaviour {
    public XRGrabInteractable grabInteractable;
    
    void OnEnable() {
        grabInteractable.selectEntered.AddListener(OnGrabbed);
        grabInteractable.selectExited.AddListener(OnReleased);
    }
    void OnGrabbed(SelectEnterEventArgs args) {
        Debug.Log($"Grabbed by {args.interactorObject}");
    }
    void OnReleased(SelectExitEventArgs args) {
        Debug.Log("Released");
    }
}

🚶 Locomotion Systems

🎯 Moving in VR

Locomotion providers: Teleportation, Continuous Move, Snap Turn, Continuous Turn. Combine them in a Locomotion System.

Locomotion Providers

Teleportation Provider for comfort. Continuous Move Provider for smooth movement (use vignette). Snap Turn Provider for rotation in 45° increments. All providers share the Locomotion System to avoid conflicts.

🌀 Teleport🕹️ Continuous🔄 Snap Turn

// Teleportation setup
using UnityEngine.XR.Interaction.Toolkit;

public class TeleportActivator : MonoBehaviour {
    public XRRayInteractor rayInteractor;
    public TeleportationProvider teleportProvider;
    
    void Update() {
        if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit)) {
            TeleportRequest request = new TeleportRequest() {
                destinationPosition = hit.point,
                matchOrientation = MatchOrientation.WorldSpaceUp
            };
            teleportProvider.QueueTeleportRequest(request);
        }
    }
}

🖥️ World Space UI in XR

🎯 UI that lives in 3D

XR requires World Space canvases and interaction via rays. Unity UI (uGUI) works with XR Interaction Toolkit's XR UI Input Module.

UI Setup

Set Canvas to World Space, add Tracked Device Graphic Raycaster, and XR UI Input Module to the Event System. Buttons work with ray interactors. For hand tracking, use poke interaction.

🌍 World Space🎯 Raycast UI👉 Poke

// XR UI Input Module setup (via code)
using UnityEngine.XR.Interaction.Toolkit.UI;

void SetupXRUI() {
    var inputModule = gameObject.AddComponent();
    inputModule.enableXRInput = true;
    inputModule.enableMouseInput = false;
}

📱 AR Foundation

🎯 Cross-platform AR in Unity

AR Foundation abstracts ARCore (Android) and ARKit (iOS) into a unified API. Write once, deploy to both platforms.

Core AR Components

ARSession manages lifecycle. ARSessionOrigin (now XR Origin) parents AR content. ARPlaneManager detects surfaces. ARRaycastManager places objects. ARAnchorManager for persistent anchors.

📐 Plane Detection🎯 Raycast⚓ Anchors🖼️ Image Tracking

// AR Raycast placement
using UnityEngine.XR.ARFoundation;
using UnityEngine.XR.ARSubsystems;

public class ARPlacement : MonoBehaviour {
    public ARRaycastManager raycastManager;
    public GameObject prefab;
    
    void Update() {
        if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Began) {
            List hits = new List();
            if (raycastManager.Raycast(Input.GetTouch(0).position, hits, TrackableType.PlaneWithinPolygon)) {
                Instantiate(prefab, hits[0].pose.position, hits[0].pose.rotation);
            }
        }
    }
}

🤖 ARCore & ARKit

🎯 Platform-specific AR features

AR Foundation exposes platform-specific features: ARCore Depth API, ARKit LiDAR, face tracking, body tracking, and more.

Feature Access

ARCore Extensions for Cloud Anchors and Depth. ARKit XR Plugin for face tracking, body tracking, and scene reconstruction (LiDAR). Use #if UNITY_ANDROID / #if UNITY_IOS for platform-specific code.

📡 Depth API☁️ Cloud Anchors😊 Face Tracking

// Face tracking (ARKit)
#if UNITY_IOS
using UnityEngine.XR.ARKit;

void Update() {
    ARFace face = GetComponent();
    if (face != null) {
        // access blendshapes
        float jawOpen = face.blendShapes[ARKitBlendShapeLocation.JawOpen];
    }
}
#endif

🖐️ Hand Tracking

🎯 Natural hand interaction

Unity XR Hands package provides hand tracking for Quest, HoloLens 2, and other devices. Use hand joints for gesture recognition and direct interaction.

XR Hands API

XRHandSubsystem provides hand data. Access joint poses (wrist, knuckles, fingertips) and gestures (pinch, grab). Combine with XR Interaction Toolkit's XRHandInteractor for poke and grab.

✋ Joint Poses🤏 Pinch👉 Poke

// Hand tracking with XR Hands
using UnityEngine.XR.Hands;

public class HandTracker : MonoBehaviour {
    XRHandSubsystem handSubsystem;
    
    void Start() {
        var subsystems = new List();
        SubsystemManager.GetSubsystems(subsystems);
        if (subsystems.Count > 0) handSubsystem = subsystems[0];
    }
    
    void Update() {
        if (handSubsystem.running) {
            XRHand leftHand = handSubsystem.leftHand;
            if (leftHand.isTracked) {
                XRHandJoint joint = leftHand.GetJoint(XRHandJointID.IndexTip);
                if (joint.TryGetPose(out Pose pose)) {
                    Debug.Log($"Index tip: {pose.position}");
                }
            }
        }
    }
}

⚡ XR Performance Tuning

🎯 Hitting 72/90/120 fps

XR requires consistent frame rates. Optimize CPU/GPU, reduce draw calls, use foveated rendering, and enable single-pass instanced rendering.

Optimization Techniques

Single Pass Instanced rendering for stereo. Foveated Rendering (Quest Pro, PSVR2). LOD groups, occlusion culling, GPU instancing. Use the Unity Profiler and XR Stats to monitor performance.

⚡ Single Pass🎯 Foveated📉 LOD📊 Profiler

// Enable single pass instanced rendering
using UnityEngine.XR;

void Start() {
    XRSettings.stereoRenderingMode = XRSettings.StereoRenderingMode.SinglePassInstanced;
}

// Foveated rendering (Oculus)
using Unity.XR.Oculus;
void EnableFoveation() {
    Utils.SetFoveationLevel(FoveationLevel.High);
}

🎨 XR Graphics & Shaders

🎯 URP for XR

Universal Render Pipeline (URP) is recommended for XR. It supports single-pass instancing, foveated rendering, and optimized shaders.

Shader Tips for XR

Use URP shaders and avoid complex vertex/fragment operations. Stereo-aware shaders handle eye indices automatically. For AR, use light estimation to match real-world lighting.

🎨 URP👁️ Stereo Shaders💡 Light Estimation

// URP shader with stereo support
Shader "XR/StereoShader" {
    SubShader {
        Tags { "RenderPipeline"="UniversalPipeline" }
        Pass {
            HLSLPROGRAM
            #pragma multi_compile_instancing
            #include "Packages/com.unity.render-pipelines.universal/ShaderLibrary/Core.hlsl"
            ENDHLSL
        }
    }
}

🚀 Building & Deployment

🎯 From editor to device

Build settings for Android (Quest, ARCore), iOS (ARKit), PC (OpenXR), and UWP (HoloLens).

Deployment Checklist

Android: Switch to Android, set minimum API 24+, enable XR plugins. iOS: Build with Xcode, enable ARKit. Quest: Use Oculus Integration, enable Developer Mode. HoloLens: UWP build with IL2CPP and ARM64.

📱 Android🍎 iOS🥽 Quest🕶️ HoloLens

// Build script for Android XR
using UnityEditor;
using UnityEditor.Build.Reporting;

public class XRBuilder {
    [MenuItem("XR/Build Android")]
    static void BuildAndroid() {
        BuildPlayerOptions options = new BuildPlayerOptions();
        options.scenes = new[] { "Assets/Scenes/Main.unity" };
        options.locationPathName = "Builds/XR.apk";
        options.target = BuildTarget.Android;
        options.options = BuildOptions.None;
        BuildReport report = BuildPipeline.BuildPlayer(options);
        Debug.Log($"Build result: {report.summary.result}");
    }
}

🔮 Future: Unity XR + AI

🎯 AI-powered XR experiences

Unity Sentis (AI inference), generative AI for content creation, and intelligent NPCs will define next-gen XR.

Unity Sentis & AI

Unity Sentis runs neural networks in Unity (object detection, pose estimation). Combine with LLM agents for conversational NPCs. Procedural generation via AI for infinite worlds.

🧠 Sentis🤖 LLM NPCs🌌 Procedural

// Unity Sentis object detection (simplified)
using Unity.Sentis;

public class ARDetector : MonoBehaviour {
    Model runtimeModel;
    IWorker worker;
    
    void Start() {
        runtimeModel = ModelLoader.Load(modelAsset);
        worker = WorkerFactory.CreateWorker(BackendType.GPUCompute, runtimeModel);
    }
    
    void Update() {
        // run inference on camera texture
        Tensor input = new Tensor(texture);
        worker.Execute(input);
        Tensor output = worker.PeekOutput() as Tensor;
        // parse detections
    }
}