🧠 ARKit & ARCore Foundations
🎯 The Two Giants of Mobile AR
ARKit (Apple, 2017) and ARCore (Google, 2018) are the dominant AR frameworks for iOS and Android respectively. Both use Visual-Inertial Odometry (VIO) for 6-DOF tracking, plane detection, light estimation, and anchors. They power millions of AR apps — from IKEA Place to Pokémon GO.
What They Share
Common capabilities: World tracking (6-DOF), plane detection (horizontal + vertical), light estimation, image tracking, face tracking, anchors, and cloud anchors. Both use camera + IMU fusion. Differences: ARKit has LiDAR (iPhone Pro/iPad Pro) for instant depth; ARCore has Depth API (software depth) and Geospatial API (Google Street View-based localization).
// ARKit session setup (Swift)
import ARKit
let configuration = ARWorldTrackingConfiguration()
configuration.planeDetection = [.horizontal, .vertical]
configuration.environmentTexturing = .automatic
configuration.frameSemantics = [.sceneDepth, .smoothedSceneDepth]
let session = ARSession()
session.run(configuration)
// ARCore session setup (Kotlin)
val config = Config(session).apply {
planeFindingMode = Config.PlaneFindingMode.HORIZONTAL_AND_VERTICAL
lightEstimationMode = Config.LightEstimationMode.ENVIRONMENTAL_HDR
depthMode = Config.DepthMode.AUTOMATIC
}
session.configure(config)🌍 World Tracking
🎯 6-DOF pose estimation
World tracking uses VIO (camera + IMU) to estimate the device's 3D position and orientation in real-time. It creates a persistent coordinate frame for placing AR content.
How It Works
Feature points are extracted from camera images and tracked across frames. IMU data (gyro + accelerometer) provides high-frequency motion estimates. Sensor fusion combines both for robust 6-DOF tracking. The result is a camera transform (position + rotation) updated 60 times per second.
// ARKit camera transform (Swift)
func session(_ session: ARSession, didUpdate frame: ARFrame) {
let transform = frame.camera.transform // 4x4 matrix
let position = SIMD3(transform.columns.3.x,
transform.columns.3.y,
transform.columns.3.z)
let rotation = simd_quatf(transform)
print("Camera position: \(position)")
}
// ARCore camera pose (Kotlin)
fun onUpdate(frame: Frame) {
val camera = frame.camera
val pose = camera.pose // Pose object
val translation = pose.translation // FloatArray[3]
val rotation = pose.rotationQuaternion // FloatArray[4]
} 📏 Plane Detection
🎯 Finding flat surfaces
Plane detection identifies horizontal and vertical surfaces (floors, tables, walls) in the environment. Essential for placing AR objects realistically.
How Planes Are Detected
Feature points are clustered and RANSAC fits a plane model. Planes are tracked and merged over time as the camera explores. ARKit/ARCore provide plane boundaries, center, extent, and alignment (horizontal/vertical).
// ARKit plane detection (Swift)
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
for anchor in anchors {
if let planeAnchor = anchor as? ARPlaneAnchor {
print("Plane detected: \(planeAnchor.alignment)")
print("Extent: \(planeAnchor.planeExtent)")
print("Center: \(planeAnchor.center)")
}
}
}
// ARCore plane detection (Kotlin)
override fun onUpdate(frame: Frame) {
for (plane in frame.getUpdatedTrackables(Plane::class.java)) {
when (plane.trackingState) {
TrackingState.TRACKING -> {
val planeType = plane.type // HORIZONTAL_UPWARD_FACING, VERTICAL
val centerPose = plane.centerPose
val extentX = plane.extentX
val extentZ = plane.extentZ
}
}
}
}⚓ Anchors & Persistence
🎯 Pinning content to the world
Anchors are fixed points in the real world that AR content can be attached to. They persist across frames and can be saved/loaded for persistence.
Anchor Types
Local anchors: Live in the current session only. Cloud anchors: Shared across devices and sessions. Geospatial anchors: Anchored to GPS coordinates (ARCore Geospatial API). Persistent anchors: Saved to disk and reloaded later (ARKit ARWorldMap).
// ARKit anchor creation and persistence (Swift)
let anchor = ARAnchor(transform: matrix_identity_float4x4)
session.add(anchor: anchor)
// Save world map
session.getCurrentWorldMap { worldMap, error in
guard let map = worldMap else { return }
let data = try! NSKeyedArchiver.archivedData(withRootObject: map)
try! data.write(to: worldMapURL)
}
// Load world map
let data = try! Data(contentsOf: worldMapURL)
let worldMap = try! NSKeyedUnarchiver.unarchivedObject(ofClass: ARWorldMap.self, from: data)
let config = ARWorldTrackingConfiguration()
config.initialWorldMap = worldMap
session.run(config)💡 Light Estimation
🎯 Matching real-world lighting
Light estimation captures ambient light intensity, color temperature, and direction to make virtual objects look like they belong in the real scene.
Light Estimation Modes
Ambient intensity: Overall brightness (lumens). Ambient color: Color temperature (Kelvin). Directional light: Direction and intensity of primary light source. HDR environment map: Full environment lighting (ARCore Environmental HDR, ARKit 4+).
// ARKit light estimation (Swift)
func session(_ session: ARSession, didUpdate frame: ARFrame) {
if let lightEstimate = frame.lightEstimate {
let ambientIntensity = lightEstimate.ambientIntensity // lumens
let ambientColorTemperature = lightEstimate.ambientColorTemperature // Kelvin
print("Intensity: \(ambientIntensity), Temp: \(ambientColorTemperature)K")
}
}
// ARCore light estimation (Kotlin)
override fun onUpdate(frame: Frame) {
val lightEstimate = frame.lightEstimate
if (lightEstimate.state == LightEstimate.State.VALID) {
val pixelIntensity = lightEstimate.pixelIntensity
val colorCorrection = lightEstimate.colorCorrection // FloatArray[4]
}
}📡 Depth API & Occlusion
🎯 Realistic occlusion and physics
Depth APIs provide per-pixel depth maps, enabling virtual objects to be occluded by real objects and vice versa. ARKit uses LiDAR (iPhone Pro/iPad Pro); ARCore uses software depth estimation.
Depth-Based Features
Occlusion: Hide virtual objects behind real ones. Physics: Collisions with real surfaces. Measurements: Real-world distance measurements. Scene reconstruction: Dense mesh of the environment (ARKit Scene Geometry).
// ARKit depth API (Swift)
func session(_ session: ARSession, didUpdate frame: ARFrame) {
guard let depthMap = frame.sceneDepth?.depthMap else { return }
let width = CVPixelBufferGetWidth(depthMap)
let height = CVPixelBufferGetHeight(depthMap)
// depthMap is a CVPixelBuffer with depth values in meters
// Use as occlusion mask in shader
}
// ARCore Depth API (Kotlin)
override fun onUpdate(frame: Frame) {
if (frame.hasDepth()) {
val depthImage = frame.acquireDepthImage16Bits()
// depthImage is an Image with depth values in millimeters
// Use as occlusion mask
}
}🖼️ Image Tracking
🎯 Recognizing and tracking 2D images
Image tracking detects known 2D images (posters, cards, product packaging) and tracks their pose in real-time. Enables marker-based AR without special markers.
Reference Images
You provide a library of reference images. ARKit/ARCore match camera frames against these images and return the pose when detected. Physical size must be specified for correct scale. Tracking quality indicates how well the image is tracked.
// ARKit image tracking (Swift)
let referenceImages = ARReferenceImage.referenceImages(inGroupNamed: "AR Resources", bundle: nil)
let config = ARWorldTrackingConfiguration()
config.detectionImages = referenceImages
config.maximumNumberOfTrackedImages = 2
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
for anchor in anchors {
if let imageAnchor = anchor as? ARImageAnchor {
print("Image detected: \(imageAnchor.referenceImage.name ?? "")")
// Place content on image
}
}
}
// ARCore image tracking (Kotlin)
val imageDatabase = AugmentedImageDatabase(session)
imageDatabase.addImage("poster", posterBitmap, 0.5f) // 0.5m width
val config = Config(session).apply {
augmentedImageDatabase = imageDatabase
}🧸 Object Tracking
🎯 Tracking 3D objects
Object tracking recognizes and tracks 3D objects (toys, products, equipment) using 3D scans or CAD models. Enables product visualization and interactive AR.
Object Tracking Setup
ARKit Object Tracking: Requires a 3D scan (.arobject file) created with the ARKit Scanner app. ARCore: No native 3D object tracking, but can be achieved with ML Kit or custom models. Use cases: Product demos, maintenance guides, interactive toys.
// ARKit object tracking (Swift)
let referenceObjects = ARReferenceObject.referenceObjects(inGroupNamed: "AR Objects", bundle: nil)
let config = ARWorldTrackingConfiguration()
config.detectionObjects = referenceObjects
func session(_ session: ARSession, didAdd anchors: [ARAnchor]) {
for anchor in anchors {
if let objectAnchor = anchor as? ARObjectAnchor {
print("Object detected: \(objectAnchor.referenceObject.name ?? "")")
// Place content on object
}
}
}😊 Face Tracking
🎯 Tracking facial expressions
Face tracking detects and tracks the user's face, including 3D mesh, blend shapes (expressions), and eye gaze. Powers Animoji, AR filters, and face-based AR.
Face Tracking Features
ARKit Face Tracking: TrueDepth camera (iPhone X+) provides 52 blend shapes, 3D mesh, and eye tracking. ARCore Face Tracking: Uses front camera, provides 6 blend shapes and 3D mesh (less detailed). Use cases: Animoji, AR masks, face filters, attention detection.
// ARKit face tracking (Swift)
let config = ARFaceTrackingConfiguration()
config.maximumNumberOfTrackedFaces = 1
session.run(config)
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
for anchor in anchors {
if let faceAnchor = anchor as? ARFaceAnchor {
let blendShapes = faceAnchor.blendShapes
let jawOpen = blendShapes[.jawOpen]?.floatValue ?? 0
let smileLeft = blendShapes[.mouthSmileLeft]?.floatValue ?? 0
print("Jaw open: \(jawOpen), Smile: \(smileLeft)")
}
}
}🏃 Body Tracking
🎯 Tracking human motion
Body tracking detects and tracks the human body, including 2D/3D joint positions. Enables fitness AR, motion capture, and interactive experiences.
Body Tracking Features
ARKit Body Tracking: 2D and 3D skeleton tracking (up to 2 people). ARCore: No native body tracking, but ML Kit Pose Detection provides 2D joints. Use cases: Fitness apps, dance games, motion capture, sports training.
// ARKit body tracking (Swift)
let config = ARBodyTrackingConfiguration()
session.run(config)
func session(_ session: ARSession, didUpdate anchors: [ARAnchor]) {
for anchor in anchors {
if let bodyAnchor = anchor as? ARBodyAnchor {
let skeleton = bodyAnchor.skeleton
for jointName in ARSkeletonDefinition.defaultBody3D.jointNames {
let jointTransform = skeleton.jointModelTransforms(for: jointName)
// Use joint transforms for motion capture
}
}
}
}☁️ Cloud Anchors & Geospatial
🎯 Persistent, shared AR
Cloud Anchors enable multiple users to see the same AR content at the same physical location, across devices and sessions. Geospatial API anchors content to GPS coordinates using Google Street View data.
Cloud Anchor & Geospatial
ARCore Cloud Anchors: Host an anchor on the cloud, share the ID, other users resolve it. ARKit: No native cloud anchors; use third-party (e.g., Azure Spatial Anchors) or ARWorldMap. Geospatial API: Localize using Google Street View, place content at lat/long/altitude.
// ARCore Cloud Anchor hosting (Kotlin)
val cloudAnchor = session.hostCloudAnchor(anchor)
// Wait for result
when (cloudAnchor.cloudAnchorState) {
CloudAnchorState.SUCCESS -> {
val cloudAnchorId = cloudAnchor.cloudAnchorId
// Share cloudAnchorId with other users
}
}
// ARCore Geospatial API (Kotlin)
val earth = session.earth
if (earth.trackingState == TrackingState.TRACKING) {
val cameraGeospatialPose = earth.cameraGeospatialPose
val latitude = cameraGeospatialPose.latitude
val longitude = cameraGeospatialPose.longitude
val altitude = cameraGeospatialPose.altitude
// Create anchor at specific lat/long
val anchor = earth.createAnchor(lat, lng, alt, quaternion)
}👥 People Occlusion
🎯 Virtual objects behind real people
People occlusion uses person segmentation to make virtual objects appear behind real people, enhancing realism in AR scenes.
People Occlusion
ARKit: People occlusion via frameSemantics = .personSegmentation (requires A12+). ARCore: Depth API provides depth-based occlusion, including people. Use cases: AR games, collaborative AR, realistic scene composition.
// ARKit people occlusion (Swift)
let config = ARWorldTrackingConfiguration()
config.frameSemantics = [.personSegmentationWithDepth]
session.run(config)
// In renderer, use segmentation buffer as occlusion mask
func renderer(_ renderer: SCNSceneRenderer, updateAtTime time: TimeInterval) {
guard let frame = session.currentFrame,
let segmentationBuffer = frame.segmentationBuffer else { return }
// Apply segmentation as occlusion mask in shader
}🌐 Multiplayer AR
🎯 Shared AR experiences
Multiplayer AR synchronizes AR content across multiple devices in real-time, enabling collaborative experiences like shared games, design reviews, and social AR.
Multiplayer AR Architecture
Shared anchors: All devices agree on a common coordinate frame. Cloud anchors: Host once, resolve on all devices. Networking: Sync object poses, interactions, and state. Frameworks: Photon, Mirror, Unity Netcode, ARCore Cloud Anchors + Firebase.
// Multiplayer AR with ARCore Cloud Anchors + Firebase
// Host: create cloud anchor, save to Firebase
val cloudAnchor = session.hostCloudAnchor(localAnchor)
firebaseRef.child("anchors").child(roomId).setValue(cloudAnchor.cloudAnchorId)
// Client: read from Firebase, resolve anchor
firebaseRef.child("anchors").child(roomId).get()
.addOnSuccessListener { snapshot ->
val cloudAnchorId = snapshot.value as String
val cloudAnchor = session.resolveCloudAnchor(cloudAnchorId)
// Place shared content at resolved anchor
}🚀 Future: ARKit + ARCore + AI
🎯 AI-powered mobile AR
AI is transforming ARKit and ARCore: semantic understanding, generative AR, intelligent agents, and real-time scene interpretation.
AI + AR Integration
Semantic segmentation: Understand what objects are in the scene (chair, table, person). Generative AR: Create 3D content from text prompts. LLM agents: Conversational AR assistants. NeRF/Gaussian Splatting: Photorealistic scene reconstruction on mobile.
// ARCore + ML Kit: semantic understanding
val segmenter = ImageSegmenter.createFromOptions(context, options)
val result = segmenter.segment(image)
// result.categoryMask contains pixel labels (person, sky, building, etc.)
// Use for semantic AR: place content only on tables, avoid people, etc.
// ARKit + Core ML: object detection
let model = try VNCoreMLModel(for: YOLOv8().model)
let request = VNCoreMLRequest(model: model) { request, error in
guard let results = request.results as? [VNRecognizedObjectObservation] else { return }
for observation in results {
let boundingBox = observation.boundingBox
let label = observation.labels.first?.identifier
// Place AR content based on detected objects
}
}