Last week I attended a fantastic presentation by Gene Becker (@genebecker) of LightningLaboratories, an augmented reality research and consulting firm. I’ve been fascinated by augmented reality (AR), and after attending Becker’s presentation at the Web 2.0 Expo in San Francisco, I felt I got the full lowdown of the current state of AR, the challenges, and its hopeful future. Much of this post is a summary of what Becker discussed in his presentation. Here are his slides.
Experience Design for Mobile Augmented Reality
What is augmented reality?
As Becker explains, “Augmented reality is the display of computer graphics and media, overlaid and registered on real-world environments, interactively in real time.” That simple explanation requires a lot of wizards behind the curtain. There’s a confluence of multiple technologies and disciplines to make AR appear as a seamless experience. AR is comprised of computer graphics, machine vision, sensing and sensor fusion, geographic information systems, mobile systems, ubiquitous computing, and the web, explained Becker.
While we can realize a certain level of AR today, we’re constantly striving for improvement in these technologies and greater synergy.
History of virtual reality
Becker gave a brief history of augmented reality, covering the work of Steve Mann and his wearable computer in the early 1990s, and the Touring Machine from 1997. Becker claims that modern AR has been around for 15 years. But back then to achieve augmented reality the user had to carry a backpack full of gear. Today, those capabilities that filled a backpack – your orientation, direction, view of the world, and display over the world – can now fit in your iPhone, Android phone, or Symbian device.
Uses for augmented reality
For each of AR’s uses, Becker poses a series of questions that a developer has to contend with when trying to effectively achieve and cope with that level of augmented reality.
Visual search or querying the natural world: You can look at a building and augment it with printed text, or maybe call up a historical photo. Question is how do you go about recognizing things in the world? How do you then translate what you know of the real world and make it interesting?
Augmented Reality ID or Social Augmented Reality – If we can identify people in a room, we can understand more about them and their relations. There’s already an example of this with TAT, a new AR technology that IDs the people in the room through facial recognition. Once you positively identify a person, you can access their social account, see that person’s friends (do you have any in common?) and see personal public information.
Problem is augmented reality touches upon some new social and privacy issues. Becker points out that ARI ID is the collapse of social nuance and the end of personal privacy. Someone can stand in front of you and query your social profile while they’re sitting there. It’s more than just seeing you right here in the moment, but now I have a window into your entire background. Social Augmented Reality redefines what it means to be public.
While having your personal information completely public wherever you go may creep some people out, most are slowly becoming accustomed to this type of recognition. Many embrace it. They’ve chosen to be public online, so it’s only natural that they would want their public information to be portable with them. The question is how will others use that information.
We’ve been creeping to the point of accepting this kind of behavior. Already people are accustomed to having their photo taken from a mobile device, uploaded to Facebook, and then tagged with their name. That’s a manual tagging. Soon it will be done automatically. And once you can make that connection, it opens the path to a plethora of other personal data and connections. Becker argues that we’ll need this technology to be built into glasses before it can be more socially acceptable. 3D movies and TV are trying to move us towards this natural concept. Becker predicts that we’re five to ten years away from Terminator-like vision capability.
What’s available today
Layar – Utilizing your GPS and compass, this system knows where you are and what direction you’re pointing. The majority of iPhone-enabled AR applications work like Layar. For the Layar application they look at the Latitude and longitude coordinates of you and where you’re looking. The dots on houses allow you to call up the address and also the price.
Problem with this application is its use is not realistic. People would use this product for a moment, but not continuously. You can’t expect someone to walk around with their phone out in front of them. Inclusion of this technology in glasses is necessary.
Metaio – This technology lets you see 3D-augmented items on top of physical items. In one example with Lego, you can hold up a box to the camera and it will show you on top, in virtual 3D, what the item inside will look like, once you make it.
Becker stressed that AR is not just a new technology, but rather a new medium of creative expression that expands human experience. It’s a mashup of all sorts of information on top of our physical world: Web, images, sound, movies, games, search engines, databases, advertising, and social media connected to people, places, and things. AR can reveal invisible stories all around us.
What makes a compelling AR experience?
Becker explained that there are three dimensions that create a compelling augmented reality experience. A true AR experience allows for flow across all these issues. Becker talked at great length that success in AR requires complete flow.
When people have a good experience with AR, they talk about these three aspects:
- Challenge/Expression – physical/mental challenge, self-expression, and creativity
- Drama/Sensation – physical stimulation, mental/emotional stimulation, and imagination
- Social – competing with others, bonding, and cooperation
All the tools out now are only good at solving one maybe two of these aspects, said Becker.
The reality of AR experiences today
Here’s Becker’s estimation of what does and doesn’t work with AR today.
What works
- Physical engagement
- Visually captivating
- Emotional “wow” factor
- Invisible stories, revealed
What doesn’t
- Awkward use models
- No flow opportunities – Just point situations. Not continuous.
- No social dimension
- Point experiences only
Mobile AR issues and limitations
Technical limitations
- Sensor accuracy
- Indoor positioning
- Line of sight
- Device constraints – Eats up battery life
- Lack of interoperable standards – No cooperation on what is a “point.” It’s a lot more than just latitude and longitude.
- Relative positioning – Things that we’ve come to expect from an experience on the screen, don’t necessarily work in AR.
- Object recognition and tracking
- Graphics issues
Use model issues
- Unnatural physical position
- Ambiguous social signals – What are you doing that you’re pointing this at me?
- Cognitive model – What am I seeing here? How do I interpret this?
Authoring Limitations
- Platform specific
- Low level tools, complex workflow
- Focus on point interactions. No support for stories or interesting interactivity. How do we create a flow?
An example of a compelling AR experience
Becker showed us this concept video from HP that shows how a great AR experience could play itself out.
The video addresses all the issues of what makes a compelling experience.
- Challenge/Expression – Running to get to your destination. Finding your way through the city.
- Drama/Sensation – There’s a storyline. How should you get there? Will you find the locations in time?
- Social – Racing and competing against other people.
Mobile AR experience design practices
Becker summarized what you need to be thinking about when developing an AR experience.
- Understand limitations and design around them. You’re looking create flow and physical drama.
- Design for inaccuracies in the system.
- Emphasize audio for immersion, continuity, and story.
- Use “magic lens” position sparingly. Don’t need to be looking through it all the time.
- Fail gracefully and transparently.
- Design for challenge, expression, and flow.
- Focus on experience, not technology. Have your experience through the device, not of the device.
- Goals should be challenging.
- Set clear goals, expectations, and rules.
- Encourage focus and concentration. Limit distractions and gratuitous things. Don’t overload the individual with ads.
- Make activities rewarding.
- Provide direct and immediate feedback.
- Design for physical sensation and emotional drama.
- Scout the locations, know the environment and context deeply, sights, sounds, smells, textures, weather, rhythms, and moods. It’s like filming on location.
- Design for “magic moments” – mental immersion, physical/virtual coincidence.
- Emphasize sound design for immersion, continuity.
- Develop a cross-media story.
- Design for social bonding, sharing, and competition.
- Game mechanics drive competitive engagement – Create multi-player competitions, scoring, and leveling systems
- Social media integration to share broadly/vicariously.
- Tight and loose multi-person activities encourage bonding and shared experience.
Bring an interdisciplinary media production mindset
Take all jobs you might have with film, gaming, and web development, mash them together and you’ll have the team for your AR project. To be successful in AR, you’ll need to reach from multiple creative disciplines.
UPDATE 10/2/13: Here’s some recent coverage of the GLAZED Conference and the After show party by Renee Blodgett of We Blog the World.
Photo credit: CC Estaban Trigos
