🥽 VR Core Foundations
🎯 What is Virtual Reality?
Virtual Reality immerses the user in a completely digital environment, replacing the real world with a simulated one. Key components: Head-Mounted Display (HMD), tracking (position + orientation), interaction (controllers, gestures), and rendering (real-time 3D graphics). Platforms: SteamVR, Oculus, PlayStation VR, WebXR.
The VR Pipeline
From sensor to pixels: IMU + optical tracking → pose estimation → 3D scene rendering (stereo) → lens distortion correction → display. Latency must be sub-20ms to avoid motion sickness. Presence is the ultimate goal — the feeling of "being there".
// VR session initialization (OpenVR/SteamVR)
vr::IVRSystem* system = vr::VR_Init(&error, vr::VRApplication_Scene);
if (system) {
vr::TrackedDevicePose_t poses[vr::k_unMaxTrackedDeviceCount];
system->GetDeviceToAbsoluteTrackingPose(vr::TrackingUniverseStanding, 0, poses, vr::k_unMaxTrackedDeviceCount);
// render each eye
}🕶️ HMD & Display Tech
🎯 Head-Mounted Displays
HMDs use OLED, LCD, or micro-OLED panels. Key specs: resolution (per eye), refresh rate (90Hz+), FOV (100°+), and pixel density (PPD).
Display Technologies
OLED offers true blacks and fast response (low persistence). LCD provides higher brightness and sub-pixel resolution. Micro-OLED enables compact form factors (Bigscreen Beyond).
// Query display properties (OpenXR)
XrViewConfigurationView viewConfig;
xrEnumerateViewConfigurationViews(instance, systemId, viewConfigType, 1, &viewCount, &viewConfig);
// viewConfig.recommendedImageRectWidth, recommendedImageRectHeight📍 6-DOF Tracking
🎯 Position + Orientation
6-DOF (degrees of freedom) tracks movement in 3D space: x, y, z (position) and roll, pitch, yaw (rotation). Inside-out vs outside-in tracking.
Inside-out vs Outside-in
Outside-in (base stations) — more accurate, requires external sensors. Inside-out (camera-based) — more portable, used in Quest, Vive XR Elite. IMU (gyro/accelerometer) fusion is essential for low-latency tracking.
// IMU + optical pose fusion (Kalman filter)
Pose estimate = kalmanFilter.predict(imuData);
Pose corrected = kalmanFilter.update(opticalPose);
// transform for rendering
Matrix4x4 viewMatrix = corrected.inverse();🎨 VR Rendering Pipeline
🎯 Real-time 3D rendering
VR rendering requires high frame rate (90+ fps) and low latency. Techniques: single-pass stereo, foveated rendering, ASW (Asynchronous SpaceWarp).
Optimization Techniques
Single-pass stereo renders both eyes in one draw call (Nvidia VRWorks). Foveated rendering reduces resolution in peripheral vision (eye tracking). Fixed foveated is commonly used on mobile VR.
// Single-pass stereo (shader)
// gl_ViewIndex = 0 for left, 1 for right
layout(location=0) in vec3 position;
void main() {
mat4 view = viewMatrices[gl_ViewIndex];
gl_Position = projection * view * model * vec4(position, 1.0);
}👁️ Stereoscopic 3D
🎯 Two eyes, two views
VR displays two slightly offset images (left/right) to create depth perception. Interpupillary distance (IPD) calibration is critical.
Depth Cues
Binocular disparity, convergence, and accommodation are key. Vergence-accommodation conflict causes discomfort — solved by light field displays or varifocal optics.
// Eye offsets
float ipd = 0.063f; // meters
vec3 leftEyePos = cameraPos - cameraRight * (ipd / 2.0f);
vec3 rightEyePos = cameraPos + cameraRight * (ipd / 2.0f);
// render for each eye🔍 Optics & Distortion
🎯 Lenses and chromatic aberration
VR lenses magnify the display, but introduce distortion (barrel/pincushion) and chromatic aberration. Pre-distortion shaders correct these.
Lens Correction
Barrel distortion is countered by rendering with opposite distortion. Chromatic aberration is corrected by scaling colors differently. Modern headsets use Fresnel lenses or pancake optics.
// Distortion shader (simplified)
vec2 distort(vec2 uv) {
vec2 r = uv - 0.5;
float k = 0.35; // distortion coefficient
float scale = 1.0 + k * dot(r, r);
return uv + r * (scale - 1.0);
}✋ Controllers & Hand Tracking
🎯 Input methods
VR interaction includes motion controllers (with buttons, triggers, joysticks), hand tracking (camera-based), and gaze/voice.
Natural Interaction
Motion controllers provide haptic feedback and precise input. Hand tracking (Quest, Leap Motion) enables natural gestures. Gaze-based selection is used in accessibility.
// Controller input (OpenXR)
XrActionSet actionSet;
xrCreateActionSet(session, &actionSetInfo, &actionSet);
XrAction gripAction;
xrCreateAction(actionSet, &gripActionInfo, &gripAction);
// in loop: xrGetActionStateFloat(session, gripAction, &state);🚶 Locomotion & Comfort
🎯 Moving in VR without motion sickness
Locomotion techniques: teleportation, smooth locomotion, room-scale, redirected walking.
Comfort-first design
Teleportation is the safest for most users. Smooth locomotion (joystick movement) can cause cybersickness. Vignetting (narrowing FOV) during movement reduces discomfort.
// Teleportation system (pseudocode)
if (controller.triggerPressed && raycastGround()) {
Vector3 target = hitPoint;
// fade out, move camera, fade in
StartCoroutine(Teleport(target));
}🎧 Spatial Audio
🎯 3D sound for presence
Spatial audio uses HRTF (head-related transfer function) to position sounds in 3D space, matching head orientation.
Sound Localization
HRTF simulates how ears perceive direction. Ambisonics for full-sphere audio. Occlusion and reverb add realism.
// Spatial audio (Unity)
AudioSource src = GetComponent();
src.spatialBlend = 1.0f;
src.spatialize = true;
src.SetAmbisonicDecoderFloat(0, 1.0f); // enable ambisonics ⚡ Performance & Optimization
🎯 Sustained 90+ fps
VR is unforgiving: missed frames cause stutter and sickness. Optimization is mandatory.
Optimization Checklist
Draw call batching, LOD, texture atlasing, GPU instancing, ASW (asynchronous spacewarp) for frame interpolation.
// GPU instancing in Unity
#pragma multi_compile_instancing
// C#: MaterialPropertyBlock for per-instance properties
Graphics.DrawMeshInstanced(mesh, 0, material, matrices, batchCount);🌀 Presence & Immersion
🎯 The "being there" feeling
Presence is the holy grail of VR. Achieved through low latency, high FOV, realistic graphics, and sensory coherence (visual + audio + haptic).
Factors of Presence
Sensorimotor contingencies — your actions must have predictable responses. Place illusion (feeling of location) and plausibility illusion (feeling that events are real).
// Haptic feedback on interaction
controller.TriggerHapticPulse(0.5f, 0.2f); // amplitude, duration
// Combined with audio and visual cues🌐 Social VR & Networking
🎯 Multi-user VR
Shared virtual spaces (VRChat, Rec Room) require avatar synchronization, voice chat, and low-latency networking.
Synchronization
Pose interpolation smooths network updates. Relevancy (only sync nearby objects) and authoritative server prevent cheating.
// Networked avatar (Photon/Unity)
void OnPhotonSerializeView(PhotonStream stream, PhotonMessageInfo info) {
if (stream.isWriting) {
stream.SendNext(transform.position);
stream.SendNext(transform.rotation);
} else {
transform.position = (Vector3)stream.ReceiveNext();
transform.rotation = (Quaternion)stream.ReceiveNext();
}
}🎯 UX for VR
🎯 Human-centered design
VR UX must consider comfort, accessibility, and intuitive interaction. Avoid "gorilla arm" (holding arms up for long), use affordances.
UX Guidelines
FOV, comfort, and fatigue are key. Use comfort modes (snap turn, vignette). Provide clear feedback for every action.
// Snap turn (comfort)
if (joystick.right) {
transform.Rotate(Vector3.up, 45f);
// instant or smooth
}🚀 Future: VR + AI
🎯 Generative VR
AI will generate virtual worlds on the fly, create realistic NPCs, and personalize experiences. NeRFs and generative models will revolutionize content creation.
AI-driven VR
NeRF (Neural Radiance Fields) for photorealistic scenes from images. LLM-powered NPCs that converse naturally. Procedural generation for infinite worlds.
// NeRF rendering (simplified)
def render_nerf(rays, model):
points = sample_along_rays(rays)
densities, colors = model(points)
# volumetric rendering
return integrate(densities, colors)