What is Augmented Reality (AR) and How does it work

Subscribe to our YouTube channel, to get our best video reviews, comparisons and performance tips:

Wondering what a superimposition and haptic communication are? What are smart glasses and head-mounted displays? What is Augmented Reality and Virtual Reality? What’s mobile computing? What do all these digital trends mean? Are you ever going to catch up?… Don’t worry, breathe, keep calm and we are here to help you sort things out.

So you’ve heard something about AR/VR/MR and would like to know more. For most people, it is still a quite abstract and exotic technology, often perceived as the science-fiction out of Hollywood movies. Animate holograms, interactive displays, and virtual 3D models. In fact, all these things do already exist.

Real environment modified by computer-generated objects is present in many areas, from aviation to gaming, we’re just not aware of it as users. Have you tried to catch Pokemon in recent years or to fit furniture in your room via IKEA app? That’s AR, and it has more far-reaching areas of potential use. It is still in development and multiple engineers and tech companies around the world are working to enhance it. In the meantime, let’s find out what is Augmented Reality, and let’s start with this vision by Magic Leap. Exciting!

What is Augmented Reality?

Augmented reality is the technology that expands our physical world, adding layers of digital information onto it. Unlike Virtual Reality (VR), AR does not create the whole artificial environments to replace real with a virtual one. AR appears in direct view of an existing environment and adds sounds, videos, graphics to it.

The term itself was coined back in 1990, and one of the first commercial uses were in television and military. With the rise of the Internet and smartphones, AR rolled out its second wave and nowadays is mostly related to the interactive concept. 3D models are directly projected onto physical things or fused together in real-time, various augmented reality apps impact our habits, social life, and the entertainment industry.

AR apps typically connect digital animation to a special ‘marker’, or with the help of GPS in phones pinpoint the location. Augmentation is happening in real time and within the context of the environment, for example, overlaying scores to a live feed sport events.

There are 4 types of augmented reality today:

• markerless AR
• marker-based AR
• projection-based AR
• superimposition-based AR

Brief history of AR

AR in the 1960s. In 1968 Ivan Sutherland and Bob Sproull created a first head-mounted display, they called it The Sword of Damocles. Obviously, it was a rough device that displayed primitive computer graphics.

AR in the 1970s. In 1975 Myron Krueger created Videoplace – an artificial reality laboratory. The scientist envisioned the interaction with digital stuff by human movements. This concept later was used for certain projectors, video cameras, and onscreen silhouettes. 

AR in the 1980s. In 1980 Steve Mann developed a first portable computer called EyeTap, designed to be worn in front of the eye. It recorded the scene to superimposed effects on it later, and show it all to a user who could also play with it via head movements. In 1987 Douglas George and Robert Morris developed the prototype of a heads-up display (HUD). It displayed astronomical data over the real sky. 

AR in the 1990s. The year 1990 marked the birth of the “augmented reality” term. It first appeared in the work of Thomas Caudell and David Mizell – Boeing company researchers. In 1992 Louis Rosenberg of the US Air Force created the AR system called “Virtual Fixtures”.  In 1999, a group of scientists led by Frank Delgado and Mike Abernathy tested new navigation software, which generated runways and streets data from a helicopter video.

AR in the 2000s. In 2000 a Japanese scientist Hirokazu Kato developed and published ARToolKit – an open-source SDK. Later it was adjusted to work with Adobe. In 2004 Trimble Navigation presented an outdoor helmet-mounted AR system. In 2008 Wikitude made the AR Travel Guide for Android mobile devices. 

AR today. In 2013 Google beta tested the Google Glass – with internet connection via Bluetooth. In 2015 Microsoft presented two brand new technologies: Windows Holographic and HoloLens (an AR goggles with lots of sensors to display HD holograms). In 2016 Niantic launched Pokemon Go game for mobile devices. The app blew the gaming industry up and earned $2 million in a just first week. 

How does Augmented Reality work

What is Augmented Reality for many of us implies a technical side, i.e. how does AR work? For AR a certain range of data (images, animations, videos, 3D models) may be used and people will see the result in both natural and synthetic light. Also, users are aware of being in the real world which is advanced by computer vision, unlike in VR.

AR can be displayed on various devices: screens, glasses, handheld devices, mobile phones, head-mounted displays. It involves technologies like S.L.A.M. (simultaneous localization and mapping), depth tracking (briefly, a sensor data calculating the distance to the objects), and the following components:

• Cameras and sensors. Collecting data about user’s interactions and sending it for processing. Cameras on devices are scanning the surroundings and with this info, a device locates physical objects and generates 3D models. It may be special duty cameras, like in Microsoft Hololens, or common smartphone cameras to take pictures/videos.
• Processing. AR devices eventually should act like little computers, something modern smartphones already do. In the same manner, they require a CPU, a GPU, flash memory, RAM, Bluetooth/WiFi, a GPS, etc. to be able to measure speed, angle, direction, orientation in space, and so on.
• Projection. This refers to a miniature projector on AR headsets, which takes data from sensors and projects digital content (result of processing) onto a surface to view. In fact, the use of projections in AR has not been fully invented yet to use it in commercial products or services.
• Reflection. Some AR devices have mirrors to assist human eyes to view virtual images. Some have an “array of small curved mirrors” and some have a double-sided mirror to reflect light to a camera and to a user’s eye. The goal of such reflection paths is to perform a proper image alignment.

Types of Augmented Reality

1. Marker-based AR. Some also call it to image recognition, as it requires a special visual object and a camera to scan it. It may be anything, from a printed QR code to special signs. The AR device also calculates the position and orientation of a marker to position the content, in some cases. Thus, a marker initiates digital animations for users to view, and so images in a magazine may turn into 3D models.
   
   

   Image source: Augmented Reality trends

2. Markerless AR. A.k.a. location-based or position-based augmented reality, that utilizes a GPS, a compass, a gyroscope, and an accelerometer to provide data based on user’s location. This data then determines what AR content you find or get in a certain area. With the availability of smartphones this type of AR typically produces maps and directions, nearby businesses info. Applications include events and information, business ads pop-ups, navigation support.
   

   Image source: Catchoom

3. Projection-based AR. Projecting synthetic light to physical surfaces, and in some cases allows to interact with it. These are the holograms we have all seen in sci-fi movies like Star Wars. It detects user interaction with a projection by its alterations.
   
   

   Image source: YouTube

4. Superimposition-based AR. Replaces the original view with an augmented, fully or partially. Object recognition plays a key role, without it the whole concept is simply impossible. We’ve all seen the example of superimposed augmented reality in IKEA Catalog app, that allows users to place virtual items of their furniture catalog in their rooms.
   

   Image source: Yamagata Europe

Augmented reality devices

Many modern devices already support Augmented reality. From smartphones and tablets to gadgets like Google Glass or handheld devices, and these technologies continue to evolve. For processing and projection, AR devices and hardware, first of all, have requirements such as sensors, cameras, accelerometer, gyroscope, digital compass, GPS, CPU, displays, and things we’ve already mentioned.

Devices suitable for Augmented reality fall into the following categories:

• Mobile devices (smartphones and tablets) – the most available and best fit for AR mobile apps, ranging from pure gaming and entertainment to business analytics, sports, and social networking.
• Special AR devices, designed primarily and solely for augmented reality experiences. One example is head-up displays (HUD), sending data to a transparent display directly into user’s view. Originally introduced to train military fighters pilots, now such devices have applications in aviation, automotive industry, manufacturing, sports, etc.
• AR glasses (or smart glasses) – Google Glasses, Meta 2 Glasses, Laster See-Thru, Laforge AR eyewear, etc. These units are capable of displaying notifications from your smartphone, assisting assembly line workers, access content hands-free, etc.
• AR contact lenses (or smart lenses), taking Augmented Reality one step even farther. Manufacturers like Samsung and Sony have announced the development of AR lenses. Respectively, Samsung is working on lenses as the accessory to smartphones, while Sony is designing lenses as separate AR devices (with features like taking photos or storing data). 
• Virtual retinal displays (VRD),  creating images by projecting laser light into the human eye. Aiming at bright, high contrast and high-resolution images, such systems yet remain to be made for a practical use.

Possible applications of AR

Augmented reality may complement our everyday activities in various ways. For instance, one of the most popular applications of AR is gaming. New AR games provide much better experiences to players, some even promote a more active outgoing way of life (PokemonGo, Ingress). Gaming grounds are being moved from virtual spheres to real life, and players actually perform certain activities. For instance, a simple gym activity for kids by the Canadian company SAGA, where to crack cubes moving on a wall children hit it with a ball.

AR in retail may act to bring better customer engagement and retention, as well as brand awareness and more sales. Some features may also help customers make wiser purchases – providing product data with 3D models of any size or color. Real-estate can also benefit from Augmented Reality via 3D tours of apartments and houses, that can also be manipulated to amend some parts.

Other potential areas for AR include:

• Education: interactive models for learning and training purposes, from mathematics to chemistry.
• Medicine/healthcare: to help diagnose, monitor, train, localize, etc.
• Military: for advanced navigation, marking objects in real time.
• Art / installations / visual arts / music.
• Tourism: data on destinations, sightseeing objects, navigation, and directions.
• Broadcasting: enhancing live events and event streaming by overlaying content.
• Industrial design: to visualize, calculate or model.