Augmented reality is one of the hottest technologies of this year and last year, and sure, 2017, too. Of course, it is, AR is a staple of modern mobile apps. It allows consumers to virtually try on clothes and makeup, visualize furniture in their homes, catch Pokémon, turn themselves into dogs on Snapchat, and even look at Facebook ads.
We write about AR all the time, but our focus is on how it’s used. Now, we’re going to delve into how it actually works.
Intro to computer vision
In the most basic definition, AR involves superimposing sensory enhancements such as images or audio over what you actually see in the physical world. You probably knew that already, but you may not understand computer vision, where this begins.
Computer vision is an interdisciplinary scientific field that refers to computers’ understanding of digital videos and images. In other words, digitizing the human visual system. Computer vision acquires, processes, analyzes and understands these images, the way your retina would, and extracts their data, turning them into numbers and symbols. This is how Facebook recognizes people when suggesting photo tags, how you’re able to deposit a check with the Bank of America app, and how you can turn yourself into a taco on Snapchat.
Let’s use the Snapchat filter as an example. When you use your camera, every frame is like a tiny image. Computer vision processes those images — in this case, your face. There are a few universal truths to the human face. Our eye regions are always darker than our upper cheeks, while the sockets are always darker than our foreheads.
With that information, computers scan image data and detect contrasts between lighter and darker pixels. Those pixels lead to facial recognition, but only if they’re facing forward. We’ll get back there.
What computer vision means for AR
Computer vision comes into play with AR because, in order for the technology to work, the computer must also understand the context of the physical world. Say you scan a Pizza Hut box to unlock AR content. Computer vision processes the position of the box, any images printed on the box and which direction it’s facing, and what else is in the frame.
Pizza Hut’s unbox AR campaign
Of course, the pizza box will have been embedded with a predefined marker. When the AR system recognizes that box, it captures the image with your smartphone camera. From there, it scans the image with sensors to identify exactly where to overlay the additional information on the box.
Next, the AR solution overlays the content on that spot. This must align precisely over the original frame, which explains why Snapchat filters disappear when you move your head. Together, these form a complete image, a combination of the real world and AR data.
There is also position-based AR, in which the app also utilizes GPS data and the mobile device’s compass, accelerometer, and gyroscope. The latter two determine your direction and orientation, respectively. Putting them in smartphone terms, the accelerometer is how your smartphone knows which direction you’re facing when you look at a map, while the gyroscope allows you to change the device’s orientation by rotating it.
Position-based AR uses geolocation, rather than the embedded marker, to display the content. Instead of pointing your phone at the Pizza Hut box, you can point it at a sign outside the restaurant, which triggers additional information, such as hours of operation or the menu. Last year, Perficient Digital Labs did something like this with Chicago bus stops.
Wrapping up
AR is live so this occurs every time the camera detects a new frame. This should theoretically occur in 16 milliseconds, as smartphones typically work at 60 frames per second. The AR frame is generally delayed by far more than that but it’s also far faster than the human brain can compute.
Either way, it’s helpful to understand because AR is far more popular than its cousin, virtual reality ever was. In March, eMarketer forecasted that 42.9 million Americans — 13% of the population — will use the technology at least once a month in 2019. Meanwhile, AR is expected to have a whopping 60% more users.
AR is also more practical. It doesn’t require a whole new device that blocks out the rest of the world; it’s already part of the device you already use to block out the rest of the world.