Mirage is a visionOS app that uses visionOS 2's room tracking capabilities to customize your place with captivating visualizations - all while staying connected to your surroundings. Change the scenery, relax in the sky, or meditate in a lava lamp with Mirage.

Mirage uses:

• visionOS 2
• Room Tracking
• ARKit
• Plane Detection
• RealityKit
• Reality Composer Pro
• Shader Graph Materials
• Geometry Modifiers

Project Type: 1984 product

This GitHub repo serves as a starting point for people interested in visionOS 2's new object tracking capabilities. Pairing ARKit's hand tracking + object tracking opens up fun & interesting interaction possibilities based on hand proximity.

Object Tracking Demo uses:

• visionOS 2
• ARKit
• Hand Tracking
• Object Tracking
• RealityKit
• Reality Composer Pro
• GitHub

Project Type: Open-source

This 1984 research project syncs a map of Middle Earth to the Lord of the Rings Extended Edition movies. Once the Vision Pro recognizes the paper map, an augmented reality version of the map tracks the location of each character of The Fellowship as the story unfolds.

This demo uses:

• A Paper Map Made by the Brilliant: Myths and Maps
• ARKit
• RealityKit
• Reality Composer Pro
• Image Tracking
• ParticleEmitterComponent
• External DVD Drive

Project Type: 1984 internal research

Twin is a visionOS and iOS app that uses photogrammetry to create custom 3D models of your stuff for use in Vision Pro's spatial computing world. Twin implements Shader Graph Materials that enable you to remix & filter the textures of your stuff in real-time, creating brand new colorways and new versions of your things that are impossible in the real world.

Twin uses:

• RealityKit
• SwiftData
• Reality Composer Pro
• Object Capture
• Shader Graph Materials
• RevenueCat
• RealityView

Project Type: 1984 product

Minimap Level 2 uses two iPhones to display a real-time spatial map. This second level of the prototype understands the structure of a room, segments a human from video frames, and updates the 3D position of that frame in real-time.

Minimap Level 2 uses:

• RoomPlan
• SwiftData
• RealityKit Postprocessing
• ARKit
• VideoToolbox
• TextureResource DrawableQueue

Project Type: 1984 internal research

Minimap is an ongoing internal research project exploring spatial mapping of a location. This initial demo understands the structure of a room using RoomPlan and paints object textures using ARKit & RealityKit.

Minimap uses:

• RoomPlan
• Metal Shaders
• RealityKit
• ARKit
• SwiftUI

Project Type: 1984 internal research

Reality Synthesizer is an open-source, augmented reality app that visualizes MIDI notes played on a paired synthesizer. The MIDI notes are interpreted by Metal shaders to product augmented reality visualizations. Reality Synthesizer utilizes LiDAR capabilities introduced in iOS 15.4.

Reality Synthesizer uses:

• LiDAR
• Metal Shaders
• AVFoundation
• AudioKit
• Linnstrument Helper

Project Type: Open-source

Rock Paper Checkmate is an augmented reality auto-chess game played against a computer by playing Rock Paper Scissors. Rock Paper Checkmate includes people occlusion with depth, a custom machine learning model, and recognizes hand gestures shown to the camera so anyone can play augmented reality auto-chess.

Rock Paper Checkmate uses:

• ARKit 5
• RealityKit 2 and Reality Composer
• Create ML for Hand Pose Classification
• SwiftUI
• Vision Framework for Human Hand Pose Estimations

Project Type: 1984 product

Mask Me is an ecommerce app which uses face landmarks, face anchors, and saliency analysis to enable anyone to visualize, create, and purchase custom face masks in real-time using augmented reality.

Mask Me uses:

• ARKit 4
• Vision Framework to Detect Face Landmarks
• Vision Framework for Objectness Saliency Analysis
• ARKit Face Anchors
• WooCommerce and Stripe for Ecommerce

Project Type: Client work for Mask Market

CyberWave uses the new LiDAR sensor to visualize your music in augmented reality. Play your music to change your surroundings in real-time.

CyberWave uses:

• ARKit 3.5
• LiDAR for ARMeshAnchors
• Auto-Renewable Subscriptions for In-App Purchases
• Person Segmentation with Depth
• GLSL Shaders

Project Type: 1984 product

Primer helps you visualize your home with new wallpaper, tiling, and paint. Primer utilizes Core Animation to display intuitive, animated, augmented reality tooltips to visually onboard users.

Primer uses:

• ARKit 3
• Core Animation
• Person Segmentation with Depth

Project Type: Client work for Primer

WebXR Viewer is an open-source, augmented reality, WebXR browser for iOS. WebXR Viewer has a fine-grained AR permissioning system that puts privacy & ethical design first and is built on Metal to render WebXR websites at 60 fps.

WebXR Viewer uses:

• ARKit 2
• WebXR
• Metal
• SceneKit

Project Type: Open-source client work for Mozilla

1984 developed innovative AR interactions and controls for iScape's collaborative landscaping app. iScape utilizes shared world maps, persistent worlds, 3D models, and the novel ability to create custom geometry for visualizing AR textures in the user's world.
My work on iScape was featured by HBO, Axios, and Apple at both WWDC 2019 & in the App Store.

iScape uses:

• ARKit 2
• SceneKit
• AR World Maps
• Collaborative, Multi-User Sessions

Project Type: Client work for iScape

NudgeTV released an AR basketball minigame to accompany their machine vision technology. The basketball game uses Multipeer Connectivity and Collaborative Sessions to seamlessly connect up to 4 players. SceneKit's real-time physics are employed to create realistic, yet otherwise-impossible, gameplay.

NudgeTV's basketball minigame uses:

• ARKit 3
• Collaborative Sessions
• Multipeer Connectivity
• Person Segmentation with Depth

Project Type: Client work for NudgeTV

AR Guitar is an open-source, augmented reality exploration of ARKit 3's body tracking functionality. AR Guitar watches for hand placement and movement in relation to hips. The left hand controls which sound is selected and the right hand controls when sounds are played.

AR Guitar uses:

• ARKit 3
• RealityKit
• Body Tracking
• Reality Composer

Project Type: Open-source

Janksy utilizes the iOS Vision APIs to detect rectangles in real-time. After a user taps a rectangle found by the computer vision algorithms, an ARKit reference image is created on the fly and animated via SceneKit to receive the Banksy treatment.

Janksy uses:

• ARKit 2
• SceneKit
• Vision
• AR Image Anchors

Project Type: 1984 product

Real World Ad Blocker uses tracked AR Image Anchors to replace irrelevant billboards with user-selected video content. Real World Ad Blocker demonstrates augmented reality's ability to fundamentally change the world around us.

Real World Ad Blocker uses:

• ARKit 2
• SceneKit
• AR Image Anchors

Project Type: Proof-of-concept internal research

Float addresses a crucial hurdle for any AR experience: Users don't have access to custom 3D models. Instead, Float taps into content everyone has: Photos and videos. Float enables remixing media in AR and augmented reality captioning of the real world.

Float uses:

• ARKit
• SceneKit
• ReplayKit

Project Type: 1984 product

This GitHub repo serves as a starting point for people interested in image tracking ARKit development. The simple code and examples demonstrate how easy it is to use ARKit to create Harry-Potter-style moving pictures, living movie posters, video postcards, paper-thin displays, and lots more augmented reality fun.

Image Tracking Demo uses:

• ARKit 2
• SceneKit
• GitHub

Project Type: Open-source