🧠 Eye Tracking Fundamentals
🎯 What is Eye Tracking?
Eye tracking is the process of measuring either the point of gaze (where someone is looking) or the motion of an eye relative to the head. It uses cameras, infrared light, and computer vision to detect pupil position, corneal reflections, and eye movements. It's a core technology in AR/VR/MR, neuroscience, UX research, and accessibility.
How Eye Tracking Works
1. Illuminate → Infrared LEDs illuminate the eye. 2. Capture → IR cameras capture eye images. 3. Detect → Find pupil center and corneal reflections (glints). 4. Compute → Calculate gaze vector from pupil-glint relationship. 5. Calibrate → Map gaze to screen/world coordinates. This runs at 30-120+ Hz for real-time tracking.
# Eye tracking pipeline (conceptual)
def eye_tracking_pipeline(eye_image, calibration_data):
# 1. Detect pupil
pupil_center, pupil_radius = detect_pupil(eye_image)
# 2. Detect corneal reflection (glint)
glint_center = detect_glint(eye_image)
# 3. Compute pupil-glint vector
pupil_glint_vector = pupil_center - glint_center
# 4. Estimate gaze direction
gaze_vector = estimate_gaze(pupil_glint_vector, calibration_data)
# 5. Map to screen/world coordinates
gaze_point = map_to_screen(gaze_vector, calibration_data)
return gaze_point, gaze_vector👁️ Eye Anatomy & Movements
🎯 The biological camera
The human eye has several key components for tracking: pupil (aperture), iris (colored ring), cornea (transparent front), retina (sensor), and fovea (high-acuity center). Eye movements include saccades, smooth pursuit, vergence, and fixations.
Eye Movement Types
Saccades: Rapid, ballistic eye movements (up to 900°/s). Smooth pursuit: Slow tracking of moving objects. Vergence: Both eyes move inward/outward for depth. Fixations: Brief pauses (200-300ms) where gaze is stable. Microsaccades: Tiny involuntary movements during fixation. Tremor: High-frequency jitter. Drift: Slow random movement.
// Eye movement classification (Python)
import numpy as np
from scipy.signal import savgol_filter
def classify_eye_movements(gaze_points, timestamps, velocity_threshold=30):
"""Classify eye movements into fixations and saccades."""
# Compute velocity
dt = np.diff(timestamps)
dx = np.diff(gaze_points[:, 0])
dy = np.diff(gaze_points[:, 1])
velocity = np.sqrt(dx**2 + dy**2) / dt
# Smooth velocity
velocity_smooth = savgol_filter(velocity, window_length=5, polyorder=2)
# Classify
movements = []
for i, v in enumerate(velocity_smooth):
if v < velocity_threshold:
movements.append('fixation')
else:
movements.append('saccade')
# Group consecutive same movements
grouped = []
current = movements[0]
start = 0
for i in range(1, len(movements)):
if movements[i] != current:
grouped.append({
'type': current,
'start': timestamps[start],
'end': timestamps[i],
'duration': timestamps[i] - timestamps[start]
})
current = movements[i]
start = i
return grouped📷 Sensors & Modalities
🎯 Capturing the eye
Eye tracking uses infrared (IR) cameras and IR LEDs to capture eye images. Modalities: video-oculography (VOG), electro-oculography (EOG), scleral search coils, and optical sensors.
Sensor Technologies
VOG: Camera-based, most common in AR/VR. EOG: Electrodes around eyes, measures corneal-retinal potential. Search coils: Contact lenses with coils, gold standard but invasive. IR LEDs: Illuminate eye for pupil detection. Depth cameras: For 3D eye tracking.
// IR camera setup for eye tracking (OpenCV)
#include
// Configure IR camera
cv::VideoCapture cap(0);
cap.set(cv::CAP_PROP_FRAME_WIDTH, 640);
cap.set(cv::CAP_PROP_FRAME_HEIGHT, 480);
cap.set(cv::CAP_PROP_FPS, 120);
// IR LEDs controlled via GPIO
void set_IR_LED(int brightness) {
// PWM control for IR LED brightness
// GPIO pin -> MOSFET -> IR LED
}
// Capture eye image
cv::Mat frame;
cap >> frame;
cv::Mat gray;
cv::cvtColor(frame, gray, cv::COLOR_BGR2GRAY);
// Threshold to isolate pupil (dark)
cv::Mat pupil_mask;
cv::threshold(gray, pupil_mask, 50, 255, cv::THRESH_BINARY_INV);
// Find contours (pupil candidates)
std::vector> contours;
cv::findContours(pupil_mask, contours, cv::RETR_EXTERNAL, cv::CHAIN_APPROX_SIMPLE);
// Select largest contour (pupil)
auto largest = std::max_element(contours.begin(), contours.end(),
[](const std::vector& a, const std::vector& b) {
return cv::contourArea(a) < cv::contourArea(b);
}); 🔵 Pupil Detection & Center
🎯 Finding the pupil
Pupil detection locates the dark circular aperture in the eye. Techniques: thresholding, contour detection, Hough circles, ellipse fitting, and deep learning (PupilNet, DeepVOG).
Pupil Detection Methods
Thresholding: Pupil is darker than iris — simple but sensitive to lighting. Hough circles: Detect circular shapes. Ellipse fitting: Pupil is often elliptical at angles. Deep learning: Robust to noise, glasses, and reflections. Center detection: Centroid, ellipse center, or model-based.
# Pupil detection with ellipse fitting (OpenCV + Python)
import cv2
import numpy as np
def detect_pupil(eye_image):
# Preprocess
gray = cv2.cvtColor(eye_image, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (7, 7), 0)
# Adaptive threshold
thresh = cv2.adaptiveThreshold(blurred, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C,
cv2.THRESH_BINARY_INV, 11, 2)
# Morphological operations
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5, 5))
thresh = cv2.morphologyEx(thresh, cv2.MORPH_OPEN, kernel)
thresh = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel)
# Find contours
contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
if not contours:
return None
# Select largest contour
pupil_contour = max(contours, key=cv2.contourArea)
# Fit ellipse
if len(pupil_contour) >= 5:
ellipse = cv2.fitEllipse(pupil_contour)
center = ellipse[0] # (x, y)
axes = ellipse[1] # (major, minor)
angle = ellipse[2]
return center, axes, angle
return None
# Deep learning approach (MediaPipe Iris)
import mediapipe as mp
mp_iris = mp.solutions.iris
with mp_iris.Iris() as iris:
results = iris.process(rgb_image)
if results.iris_landmarks:
# 5 landmarks per iris (center + 4 boundary)
left_iris = results.iris_landmarks[0]
pupil_center = left_iris.landmark[0]🎯 Gaze Estimation
🎯 Where are you looking?
Gaze estimation computes the 3D direction of gaze (or 2D point of regard) from eye features. Methods: model-based (geometric), appearance-based (ML), and hybrid.
Gaze Estimation Methods
Model-based: Use 3D eye model (pupil center + glint) to compute gaze vector. Appearance-based: Train CNN to map eye images to gaze direction (GazeNet, iTracker). Feature-based: Use pupil-glint vector + calibration. Hybrid: Combine geometric and learned features. Output: 3D gaze vector or 2D screen coordinates.
# Gaze estimation with CNN (PyTorch)
import torch
import torch.nn as nn
class GazeNet(nn.Module):
def __init__(self):
super().__init__()
# Eye image encoder
self.eye_encoder = nn.Sequential(
nn.Conv2d(3, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(32, 64, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Conv2d(64, 128, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
nn.Flatten()
)
# Face encoder (for head pose)
self.face_encoder = nn.Sequential(
nn.Linear(6, 64), nn.ReLU(), # head pose (pitch, yaw, roll, x, y, z)
nn.Linear(64, 128), nn.ReLU()
)
# Fusion
self.fc = nn.Sequential(
nn.Linear(128 * 4 * 4 + 128, 256), nn.ReLU(),
nn.Linear(256, 128), nn.ReLU(),
nn.Linear(128, 3) # gaze vector (x, y, z)
)
def forward(self, eye_image, head_pose):
eye_features = self.eye_encoder(eye_image)
face_features = self.face_encoder(head_pose)
combined = torch.cat([eye_features, face_features], dim=1)
gaze = self.fc(combined)
return gaze / torch.norm(gaze, dim=1, keepdim=True) # normalize
# Training
model = GazeNet()
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
criterion = nn.CosineEmbeddingLoss()
for eye_img, head_pose, gaze_gt in dataloader:
gaze_pred = model(eye_img, head_pose)
loss = criterion(gaze_pred, gaze_gt, torch.ones(1))
optimizer.zero_grad()
loss.backward()
optimizer.step()📐 Calibration
🎯 Personalizing the gaze model
Calibration maps raw eye features (pupil-glint vector) to screen/world coordinates for each user. It accounts for individual eye geometry and glasses.
Calibration Process
1. Display targets at known positions (e.g., 9-point grid). 2. User looks at each target. 3. Record features (pupil-glint vectors). 4. Fit mapping (polynomial regression, neural network). 5. Validate with additional points. Types: Explicit (user follows dots), implicit (during use), one-point, 9-point.
# 9-point calibration with polynomial regression
import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import Ridge
class GazeCalibrator:
def __init__(self, degree=2):
self.poly = PolynomialFeatures(degree=degree)
self.model_x = Ridge(alpha=1.0)
self.model_y = Ridge(alpha=1.0)
self.calibrated = False
def calibrate(self, features, targets):
"""features: (N, 2) pupil-glint vectors
targets: (N, 2) screen coordinates"""
X = self.poly.fit_transform(features)
self.model_x.fit(X, targets[:, 0])
self.model_y.fit(X, targets[:, 1])
self.calibrated = True
def predict(self, features):
if not self.calibrated:
raise ValueError("Not calibrated")
X = self.poly.transform(features.reshape(1, -1))
x = self.model_x.predict(X)[0]
y = self.model_y.predict(X)[0]
return x, y
def validate(self, features, targets):
"""Compute calibration error."""
predictions = [self.predict(f) for f in features]
errors = [np.linalg.norm(np.array(p) - np.array(t))
for p, t in zip(predictions, targets)]
return np.mean(errors), np.std(errors)
# Usage
calibrator = GazeCalibrator(degree=2)
# Collect 9 points
features = [] # pupil-glint vectors
targets = [] # screen coordinates
for point in calibration_points:
display_target(point)
user_looks_at(point)
features.append(get_eye_features())
targets.append(point)
calibrator.calibrate(np.array(features), np.array(targets))
error, std = calibrator.validate(np.array(features), np.array(targets))
print(f"Calibration error: {error:.2f}° ± {std:.2f}°")⚡ Foveated Rendering
🎯 Rendering where you look
Foveated rendering uses eye tracking to render high resolution only where the user is looking (fovea), and lower resolution in the periphery. This dramatically reduces GPU load — up to 50-70% savings.
Foveated Rendering Techniques
Fixed foveated: Fixed high-res center (no eye tracking). Dynamic foveated: High-res region follows gaze. Multi-resolution shading: Render at different resolutions. Variable rate shading (VRS): Hardware feature for per-region shading rates. Used in: PSVR2, Quest Pro, Varjo, Apple Vision Pro.
// Foveated rendering (Unity URP + eye tracking)
using UnityEngine;
using UnityEngine.XR;
public class FoveatedRenderer : MonoBehaviour {
public Camera xrCamera;
public ComputeShader foveationShader;
void Update() {
// Get gaze direction from eye tracking
Vector3 gazeDirection = GetGazeDirection();
// Compute foveal region center in screen space
Vector3 screenPoint = xrCamera.WorldToScreenPoint(
xrCamera.transform.position + gazeDirection * 10f
);
Vector2 fovealCenter = new Vector2(
screenPoint.x / Screen.width,
screenPoint.y / Screen.height
);
// Update foveation shader
foveationShader.SetVector("_FovealCenter", fovealCenter);
foveationShader.SetFloat("_FovealRadius", 0.15f); // 15% of screen
foveationShader.SetFloat("_PeripheralScale", 0.25f); // 1/4 resolution
// Apply to render texture
RenderTexture rt = RenderTexture.GetTemporary(
Screen.width, Screen.height, 24
);
foveationShader.SetTexture(0, "_Source", xrCamera.targetTexture);
foveationShader.SetTexture(0, "_Result", rt);
foveationShader.Dispatch(0, Screen.width / 8, Screen.height / 8, 1);
Graphics.Blit(rt, null as RenderTexture);
RenderTexture.ReleaseTemporary(rt);
}
Vector3 GetGazeDirection() {
// From eye tracking API (e.g., OpenXR XR_EXT_eye_gaze_interaction)
InputDevice eyeDevice = InputDevices.GetDeviceAtXRNode(XRNode.Head);
if (eyeDevice.TryGetFeatureValue(CommonUsages.eyesData, out Eyes eyes)) {
if (eyes.TryGetLeftEyePosition(out Vector3 leftPos) &&
eyes.TryGetLeftEyeRotation(out Quaternion leftRot)) {
return leftRot * Vector3.forward;
}
}
return xrCamera.transform.forward;
}
}🖱️ Gaze Interaction
🎯 Using your eyes as input
Gaze interaction uses eye tracking for hands-free control: gaze + dwell, gaze + pinch, gaze + voice, and gaze + controller. It's fast, intuitive, and accessible.
Gaze Interaction Patterns
Gaze + dwell: Look at target for 500ms-1s to select. Gaze + pinch: Look at target, pinch to confirm (Vision Pro). Gaze + voice: Look and say "select". Gaze + controller: Look and press button. Midas touch problem: Everything you look at triggers — need confirmation.
// Gaze + dwell interaction (Unity)
using UnityEngine;
using UnityEngine.UI;
public class GazeDwell : MonoBehaviour {
public Image dwellIndicator;
public float dwellTime = 1.0f; // seconds
public float gazeTolerance = 2.0f; // degrees
private float dwellTimer = 0f;
private GameObject currentTarget;
void Update() {
// Raycast from eye gaze
Ray gazeRay = new Ray(GetGazeOrigin(), GetGazeDirection());
if (Physics.Raycast(gazeRay, out RaycastHit hit, 100f)) {
GameObject target = hit.collider.gameObject;
if (target != currentTarget) {
// New target
currentTarget = target;
dwellTimer = 0f;
} else {
// Same target — accumulate dwell time
dwellTimer += Time.deltaTime;
// Update dwell indicator
dwellIndicator.fillAmount = dwellTimer / dwellTime;
dwellIndicator.transform.position = hit.point;
if (dwellTimer >= dwellTime) {
// Select!
target.SendMessage("OnGazeSelect", SendMessageOptions.DontRequireReceiver);
dwellTimer = 0f;
}
}
} else {
currentTarget = null;
dwellTimer = 0f;
dwellIndicator.fillAmount = 0f;
}
}
}🧠 Attention & Analytics
🎯 Understanding where people look
Eye tracking provides objective data on attention: fixations, saccades, heatmaps, scanpaths, and time to first fixation. Used in UX research, advertising, and training.
Attention Metrics
Fixation duration: How long gaze stays in one place (attention). Time to first fixation: How quickly something is noticed. Heatmap: Aggregate gaze density. Scanpath: Sequence of fixations. Pupil dilation: Cognitive load and arousal. Blink rate: Fatigue and engagement.
# Attention analytics with gaze data
import numpy as np
import matplotlib.pyplot as plt
from scipy.ndimage import gaussian_filter
class GazeAnalytics:
def __init__(self, screen_width, screen_height):
self.width = screen_width
self.height = screen_height
self.gaze_points = []
self.timestamps = []
def add_gaze(self, x, y, timestamp):
self.gaze_points.append((x, y))
self.timestamps.append(timestamp)
def compute_heatmap(self, sigma=50):
"""Generate attention heatmap."""
heatmap = np.zeros((self.height, self.width))
for x, y in self.gaze_points:
if 0 <= x < self.width and 0 <= y < self.height:
heatmap[int(y), int(x)] += 1
return gaussian_filter(heatmap, sigma=sigma)
def compute_fixations(self, velocity_threshold=30):
"""Detect fixations from gaze data."""
points = np.array(self.gaze_points)
times = np.array(self.timestamps)
dt = np.diff(times)
velocity = np.linalg.norm(np.diff(points, axis=0), axis=1) / dt
fixations = []
in_fixation = False
start = 0
for i, v in enumerate(velocity):
if v < velocity_threshold and not in_fixation:
in_fixation = True
start = i
elif v >= velocity_threshold and in_fixation:
in_fixation = False
fixations.append({
'start': times[start],
'end': times[i],
'duration': times[i] - times[start],
'x': np.mean(points[start:i, 0]),
'y': np.mean(points[start:i, 1])
})
return fixations
def time_to_first_fixation(self, roi):
"""Time until gaze enters region of interest."""
x1, y1, x2, y2 = roi
for (x, y), t in zip(self.gaze_points, self.timestamps):
if x1 <= x <= x2 and y1 <= y <= y2:
return t - self.timestamps[0]
return None❤️ Biometrics & Health
🎯 The eye as a window to health
Eye tracking provides biometric signals: pupil dilation (cognitive load, arousal), blink rate (fatigue), saccade velocity (neurological health), and gaze patterns (autism, ADHD, concussion).
Biometric Applications
Cognitive load: Pupil dilation increases with mental effort. Fatigue detection: Blink rate, eyelid closure (PERCLOS). Neurological: Saccade abnormalities in Parkinson's, Alzheimer's. Concussion: Smooth pursuit and saccade testing. Autism: Atypical gaze patterns. Emotion: Pupil size correlates with arousal.
# Cognitive load estimation from pupil diameter
import numpy as np
from scipy.signal import butter, filtfilt
class CognitiveLoadEstimator:
def __init__(self, sampling_rate=120):
self.fs = sampling_rate
self.pupil_history = []
# Bandpass filter for pupil (0.1 - 10 Hz)
nyq = 0.5 * self.fs
self.b, self.a = butter(4, [0.1/nyq, 10/nyq], btype='band')
def add_pupil_diameter(self, diameter_mm, timestamp):
self.pupil_history.append((diameter_mm, timestamp))
def estimate_cognitive_load(self, window_seconds=5):
"""Estimate cognitive load from pupil dilation."""
if len(self.pupil_history) < self.fs * window_seconds:
return None
# Get recent window
recent = self.pupil_history[-int(self.fs * window_seconds):]
diameters = np.array([d for d, _ in recent])
times = np.array([t for _, t in recent])
# Detrend
diameters_detrended = diameters - np.mean(diameters)
# Filter
filtered = filtfilt(self.b, self.a, diameters_detrended)
# Compute features
mean_diameter = np.mean(diameters)
std_diameter = np.std(filtered)
peak_dilation = np.max(diameters) - np.min(diameters)
# Simple cognitive load index (normalized)
# Higher = more cognitive load
load_index = (mean_diameter - 3.0) / 2.0 # baseline ~3mm, max ~5mm
load_index = np.clip(load_index, 0, 1)
return {
'load_index': load_index,
'mean_diameter': mean_diameter,
'std_diameter': std_diameter,
'peak_dilation': peak_dilation
}
def detect_fatigue(self, window_seconds=60):
"""Detect fatigue from blink rate and PERCLOS."""
recent = self.pupil_history[-int(self.fs * window_seconds):]
# Blink detection would require eyelid tracking
# PERCLOS = percentage of time eyes are 80% closed
pass🔒 Privacy & Ethics
🎯 The sensitive data problem
Eye tracking data is highly sensitive — it reveals attention, cognitive state, health, and even sexual orientation. Privacy and ethics are critical for responsible deployment.
Privacy Considerations
Biometric data: Eye movements are unique identifiers. Informed consent: Users must know what's tracked and why. Data minimization: Collect only what's needed. On-device processing: Avoid sending raw eye data to cloud. Transparency: Show when eye tracking is active. Regulations: GDPR, BIPA (Illinois), CCPA.
// Privacy-preserving eye tracking (on-device processing)
// Never send raw eye images to cloud
class PrivacyPreservingEyeTracker {
constructor() {
this.onDeviceModel = null;
this.gazeHistory = [];
this.consentGiven = false;
}
async initialize() {
// Load model on-device (WebNN, TensorFlow.js)
this.onDeviceModel = await tf.loadGraphModel('gaze_model/model.json');
}
async requestConsent() {
// Explicit consent dialog
const consent = await showConsentDialog({
title: "Eye Tracking",
description: "This app uses eye tracking to improve interaction. " +
"Raw eye images are processed on-device and never uploaded.",
dataTypes: ["gaze_direction", "pupil_diameter"],
retention: "session_only"
});
this.consentGiven = consent;
return consent;
}
async processEyeImage(imageTensor) {
if (!this.consentGiven) {
throw new Error("Consent not given");
}
// Process on-device
const gaze = await this.onDeviceModel.predict(imageTensor);
// Store only derived features (not raw images)
this.gazeHistory.push({
direction: gaze.direction,
timestamp: Date.now()
});
// Limit retention
if (this.gazeHistory.length > 1000) {
this.gazeHistory.shift();
}
return gaze;
}
anonymizeData() {
// Remove identifying patterns before any upload
// Only send aggregate statistics
return {
average_fixation_duration: this.computeAverageFixation(),
attention_heatmap: this.computeHeatmap(),
// No raw gaze points, no timestamps
};
}
}🥽 Eye Tracking in AR/VR
🎯 The killer feature for XR
Eye tracking is transforming AR/VR: foveated rendering, gaze interaction, avatar eye contact, attention analytics, and social presence. It's in PSVR2, Quest Pro, Varjo, Apple Vision Pro, and HoloLens 2.
XR Eye Tracking Applications
Foveated rendering: Render only where you look. Gaze interaction: Select with eyes. Avatar eyes: Realistic eye contact in social VR. Attention: Know what users look at. IPD: Auto-adjust lens distance. Vergence: Correct depth focus. Presence: Eye contact increases social presence.
// OpenXR eye gaze interaction (XR_EXT_eye_gaze_interaction)
#include
// Enable extension
const char* extensions[] = {XR_EXT_EYE_GAZE_INTERACTION_EXTENSION_NAME};
// Create eye gaze action
XrAction eyeGazeAction;
XrActionCreateInfo gazeInfo{XR_TYPE_ACTION_CREATE_INFO};
strcpy(gazeInfo.actionName, "eye_gaze");
gazeInfo.actionType = XR_ACTION_TYPE_POSE_INPUT;
xrCreateAction(actionSet, &gazeInfo, &eyeGazeAction);
// Suggest bindings
XrPath eyeGazePath;
xrStringToPath(instance, "/user/eyes_ext/input/gaze_ext/pose", &eyeGazePath);
XrActionSuggestedBinding binding{eyeGazeAction, eyeGazePath};
XrInteractionProfileSuggestedBinding suggested{XR_TYPE_INTERACTION_PROFILE_SUGGESTED_BINDING};
suggested.interactionProfile = eyeGazePath;
suggested.suggestedBindings = &binding;
suggested.countSuggestedBindings = 1;
xrSuggestInteractionProfileBindings(instance, &suggested);
// Create action space
XrActionSpaceCreateInfo spaceInfo{XR_TYPE_ACTION_SPACE_CREATE_INFO};
spaceInfo.action = eyeGazeAction;
spaceInfo.subactionPath = XR_NULL_PATH;
spaceInfo.poseInActionSpace = identityPose;
XrSpace eyeGazeSpace;
xrCreateActionSpace(session, &spaceInfo, &eyeGazeSpace);
// In frame loop: locate gaze
XrSpaceLocation gazeLocation{XR_TYPE_SPACE_LOCATION};
xrLocateSpace(eyeGazeSpace, stageSpace, predictedDisplayTime, &gazeLocation);
if (gazeLocation.locationFlags & XR_SPACE_LOCATION_ORIENTATION_VALID_BIT) {
XrQuaternionf gazeOrientation = gazeLocation.pose.orientation;
XrVector3f gazeDirection = rotateVector({0, 0, -1}, gazeOrientation);
// Use gaze direction for foveated rendering, interaction, etc.
} ⚠️ Challenges & Failure Modes
🎯 When eye tracking fails
Eye tracking faces challenges: glasses, contact lenses, lighting, eye makeup, droopy eyelids, and individual anatomical differences. Robust systems must handle these variations.
Common Issues
Glasses: Reflections, refraction, frame occlusion. Contact lenses: Can shift, affect pupil detection. Lighting: Sunlight, IR interference. Eye makeup: Eyeliner, mascara change pupil contrast. Eyelids: Droopy eyelids occlude pupil. Anatomy: Different eye shapes, pupil sizes. Motion: Fast head movements. Calibration drift: Over time.
# Robust eye tracking with glasses (deep learning)
# Use a model trained on diverse datasets including glasses
class RobustEyeTracker:
def __init__(self):
# Load model trained with glasses augmentation
self.model = load_model('eye_tracking_robust.h5')
# Glasses detection
self.glasses_detector = load_model('glasses_detector.h5')
def process(self, eye_image):
# Check if glasses are present
has_glasses = self.glasses_detector.predict(eye_image) > 0.5
if has_glasses:
# Use glasses-specific preprocessing
# Remove reflections
eye_image = self.remove_reflections(eye_image)
# Enhance pupil contrast
eye_image = self.enhance_pupil(eye_image)
# Use glasses-trained model
pupil = self.model.predict(eye_image, mode='glasses')
else:
pupil = self.model.predict(eye_image, mode='normal')
return pupil
def remove_reflections(self, image):
"""Remove specular reflections from glasses."""
# Detect bright spots (reflections)
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
_, bright = cv2.threshold(gray, 240, 255, cv2.THRESH_BINARY)
# Inpaint reflections
image = cv2.inpaint(image, bright, 3, cv2.INPAINT_TELEA)
return image
def enhance_pupil(self, image):
"""Enhance pupil contrast for glasses users."""
# CLAHE (Contrast Limited Adaptive Histogram Equalization)
clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
lab = cv2.cvtColor(image, cv2.COLOR_BGR2LAB)
lab[:, :, 0] = clahe.apply(lab[:, :, 0])
return cv2.cvtColor(lab, cv2.COLOR_LAB2BGR)🚀 Future: Neural Eye Tracking
🎯 AI-powered eye tracking
Deep learning is revolutionizing eye tracking: appearance-based gaze estimation, event cameras, neural rendering, and even passive eye tracking from ordinary webcams.
Neural Eye Tracking Advances
Appearance-based: GazeNet, iTracker — no calibration needed. Event cameras: Microsecond latency, no motion blur. Webcam gaze: MediaPipe, WebGazer — no special hardware. Neural rendering: Eye models for avatars. Foundation models: General-purpose gaze estimation. Predictive: Anticipate gaze before it happens.
# MediaPipe Iris: webcam-based eye tracking (no special hardware)
import cv2
import mediapipe as mp
mp_face_mesh = mp.solutions.face_mesh
mp_drawing = mp.solutions.drawing_utils
# Iris landmarks indices
LEFT_IRIS = [474, 475, 476, 477]
RIGHT_IRIS = [469, 470, 471, 472]
with mp_face_mesh.FaceMesh(
max_num_faces=1,
refine_landmarks=True,
min_detection_confidence=0.5,
min_tracking_confidence=0.5
) as face_mesh:
cap = cv2.VideoCapture(0)
while cap.isOpened():
ret, frame = cap.read()
if not ret:
break
rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
results = face_mesh.process(rgb)
if results.multi_face_landmarks:
mesh_points = np.array([
np.multiply([p.x, p.y], [frame.shape[1], frame.shape[0]]).astype(int)
for p in results.multi_face_landmarks[0].landmark
])
# Get iris centers
left_iris_center = mesh_points[LEFT_IRIS].mean(axis=0)
right_iris_center = mesh_points[RIGHT_IRIS].mean(axis=0)
# Get eye corners
left_eye_left = mesh_points[33]
left_eye_right = mesh_points[133]
right_eye_left = mesh_points[362]
right_eye_right = mesh_points[263]
# Compute gaze ratio (horizontal)
left_gaze_ratio = (left_iris_center[0] - left_eye_left[0]) / (left_eye_right[0] - left_eye_left[0])
right_gaze_ratio = (right_iris_center[0] - right_eye_left[0]) / (right_eye_right[0] - right_eye_left[0])
avg_gaze_ratio = (left_gaze_ratio + right_gaze_ratio) / 2
# Map to screen coordinates (after calibration)
screen_x = avg_gaze_ratio * screen_width
screen_y = (left_iris_center[1] / frame.shape[0]) * screen_height
print(f"Gaze: ({screen_x:.0f}, {screen_y:.0f})")
cv2.imshow('Eye Tracking', frame)
if cv2.waitKey(1) & 0xFF == 27:
break
cap.release()
cv2.destroyAllWindows()