🥽 HW 3: AR & XR Foundation

Assignment Deadline

Gradescope assignment due Friday, January 26th, 2024

Submit

via GIPHY

Extended Reality (XR) is an umbrella term encapsulating Augmented Reality (AR), Virtual Reality (VR), Mixed Reality (MR), and everything in between. XR is “reality-plus” tech using any kind of display. XR is VR plus AR. XR stands for “extended reality,” an umbrella term that covers VR, AR, and MR. All XR tech takes the human-to-PC screen interface and modifies it, either by 1) immersing you in the virtual environment (VR), 2) adds to, or augments, the user’s surroundings (AR), or 3) both of those (MR).

The term XR has been around for decades. It first popped up in the 1960s when Charles Wyckoff filed a patent for his silver-halide “XR” film, intended for photographing extremely bright light events, such as nuclear explosions. More recently, the term has moved into the mainstream as device makers struggle to describe the different display upgrades they’re working with. A couple of examples are immersing gamers in the action by putting a screen (a smartphone display or headset) right in front of the eyes (VR) or adding game characters to real-world surroundings like in the popular Pokémon Go (AR).

Augmented Reality

What is AR? Augmented reality is an enhanced, interactive version of a real-world environment achieved through digital visual elements, sounds, and other sensory stimuli via holographic technology. AR incorporates three features: a combination of digital and physical worlds, interactions made in real time, and accurate 3D identification of virtual and real objects.

Augmented reality offers an alternative way to design, curate, and deliver consumable instructions by overlaying digital content in real-world work environments.

• Augmented reality (AR)— designed to add digital elements over real-world views with limited interaction.
• Virtual reality (VR)— immersive experiences helping to isolate users from the real world, usually via a headset device and headphones designed for such activities.
• Mixed reality (MR)— combining AR and VR elements so that digital objects can interact with the real world, means businesses can design elements anchored within a real environment.
• Extended reality (XR)— covering all types of technologies that enhance our senses, including the three types previously mentioned.

There are various types of AR including marker-based and no marker-based.

• Marker-based AR is created using image recognition to identify objects already programmed into your AR device or application. When placing objects in view as points of reference, they can help your AR device determine the position and orientation of the camera. This is generally achieved by switching your camera to grayscale and detecting a marker to compare that marker with all the others in its information bank. Once your device finds a match, it uses that data to mathematically determine the pose and place the AR image in the right spot.

• Marker-less AR is more complex as there’s no point in which your device will focus on. Because of this, your device must recognize items as they appear in view. Using a recognition algorithm, the device will look for colors, patterns, and similar features to determine what that object is and then, using time, accelerometer, GPS, and compass information, it will or orient itself and use a camera to overlay an image of whatever you’d like within your real-world surroundings.

Virtual Reality

Virtual reality is a simulated 3D environment that enables users to explore and interact with a virtual surrounding in a way that approximates reality, as it is perceived through the users' senses. The environment is created with computer hardware and software, although users might also need to wear devices such as helmets or goggles to interact with the environment. The more deeply users can immerse themselves in a VR environment -- and block out their physical surroundings -- the more they are able to suspend their belief and accept it as real, even if it is fantastical in nature.

Some main types:

• Non-immersive. This type of VR typically refers to a 3D simulated environment that's accessed through a computer screen. The environment might also generate sound, depending on the program. The user has some control over the virtual environment using a keyboard, mouse or other device, but the environment does not directly interact with the user. A video game is a good example of non-immersive VR, as is a website that enables a user to design a room's decor.

• Semi-immersive. This type of VR offers a partial virtual experience that's accessed through a computer screen or some type of glasses or headset. It focuses primarily on the visual 3D aspect of virtual reality and does not incorporate physical movement in the way that full immersion does. A common example of semi-immersive VR is the flight simulator, which is used by airlines and militaries to train their pilots.

• Fully immersive. This type of VR delivers the greatest level of virtual reality, completely immersing the user in the simulated 3D world. It incorporates sight, sound and, in some cases, touch. There have even been some experiments with the addition of smell. Users wear special equipment such as helmets, goggles or gloves and are able to fully interact with the environment. The environment might also incorporate such equipment as treadmills or stationary bicycles to provide users with the experience of moving through the 3D space. Fully immersive VR technology is a field still in its infancy, but it has made important inroads into the gaming industry and to some extent the healthcare industry, and it's generating a great deal of interest in others.

Mixed Reality

Mixed reality is the next wave in computing following mainframes, PCs, and smartphones. Mixed reality is going mainstream for consumers and businesses. It liberates us from screen-bound experiences by offering instinctual interactions with data in our living spaces and with our friends. Online explorers, in hundreds of millions around the world, have experienced mixed reality through their handheld devices. Mobile AR offers the most mainstream mixed reality solutions today on social media. People may not even realize that the AR filters they use on Instagram are mixed reality experiences. Windows Mixed Reality takes all these user experiences to the next level with stunning holographic representations of people, high fidelity holographic 3D models, and the real world around them.

Mixed reality is a blend of physical and digital worlds, unlocking natural and intuitive 3D human, computer, and environmental interactions. This new reality is based on advancements in computer vision, graphical processing, display technologies, input systems, and cloud computing. The term "mixed reality" was introduced in a 1994 paper by Paul Milgram and Fumio Kishino, "A Taxonomy of Mixed reality Visual Displays." Their paper explored the concept of a virtuality continuum and the taxonomy of visual displays. Since then, the application of mixed reality has gone beyond displays to include:

• Environmental understanding: spatial mapping and anchors.
• Human understanding: hand-tracking, eye-tracking, and speech input.
• Spatial sound.
• Locations and positioning in both physical and virtual spaces.
• Collaboration on 3D assets in mixed reality spaces.

Resources and Links

• Dual Body Bimanual Coordination in Immersive Environments
• GeneratiVR: Spatial Interactions in Virtual Reality to Explore Generative Design Spaces
• 8th Wall: Wave Hello to Hand Tracking
• 8th Wall: Introducing new and improved Face Effects
• Microsoft: What is augmented reality or AR?
• HP: What Is Extended Reality (XR) and How Is it Changing the Future?
• Lens Studio for Unity Developers
• Microsoft: What is Mixed Reality?
• Tech Target: VR

Submission

Please pick one of the options below. Pick whichever you find the most interesting and fun.

Option 1: Unity AR, VR, MR Core

Install the Unity AR, VR, and MR Core templates. Depending on which Unity version you are on, the name and content of the template may vary. Download the templates and explore what transforms a 3D scene into one that is compatible with AR, VR, or MR.

Then, try building the template to a device! Here, I tried building my AR template to my iPhone. If you are on Mac, you will need to be on the latest version of MacOS and have XCode installed.

Option 2: 8th Wall Effects API

Last semester, we welcomed a guest speaker from Niantic Labs. 8th Wall is a cross-platform development solution for building AR experiences that run on the mobile web and iOS/Android applications developed by Niantic Labs. 8th Wall is a great tool in simplifying the prototyping process for AR applications.

If you are interested in learning more about 8th Wall or Niantic Labs feel free to connect with Josh Mao (he/him) on LinkedIn.

In the meantime, lets give the 8th Wall Face Effects API a try. You can also find and clone the project at the following link: https://www.8thwall.com/8thwall/face-makeup-aframe.

Option 3: Snapchat Lens Studio

This semester, a Design Manager from Snapchat will be attending our playtest for our midterm projects. Feel free to explore Snapchat's Lens Studio for Unity Developers and select any of the "Getting Started" tutorials with Snapchat's Developer Toolkit. For your Gradescope submission, please take a screenshot of what something you tried making with Snapchat Lens Studio.