Augmented Experience

Photo via Reuters / Carlo Allegri

Let’s be honest: Google Glass looks pretty silly. Its appearance is out of time, futuristic, and obnoxiously so. And it’s out of place in daily life—a strange accessory with mysterious purpose, as if someone were to walk around all day with a skateboard on a leash.

But Glass also points to an intriguing future, one in which the line between using a digital device and simply going about daily life is removed. Whereas traditional spectacles have a corrective purpose to see reality more clearly, Glass offers a new category of lenses that promise to augment the reality we see. It opens a vast new frontier for the practice of interaction design that, like the Wild West, is full of lawlessness and danger and promise. And it is the UX community that will shape this landscape; we will determine it’s character, and the impact it will have on people’s lives.

A key question all this raises is: what “reality” is Glass augmenting? At the moment, being a Google product, the augmentation is designed to primarily target the urban economic and social spheres. Looking down the street through Glass, you may see restaurant store-fronts adorned with floating metadata describing the cuisine type and star-ratings by previous diners. Turning your head, an indicator points in the direction of the location of your next calendar appointment. Peering at a product on the shelf, prices for similar products are displayed for easy comparison. You’ll always know where you are, where you need to be, and what you’re looking at. The reality that Glass augments is a realm of people, objects, places of business, and locations. In other words, what can be expressed in a database and efficiently searched.

Toward a better future

At this point in the conversation, the story usually veers into the realm of exasperation and despair. Google Glass represents the death of spontaneity! It will systematize and computerize our lives! Organic experience will be lost! (And, most insidious of all) Google will monitor and monetize every saccade of our eyeball, every step we take!

“Big brother” from the film adaptation of Orwell’s 1984

Given the penchant for technologists to base business models on advertising and “big data” about their customers, it is not surprising that Google Glass can be seen as a kind of portable panopticon. But I think the truth that this device foreshadows is something potentially more benign, and almost certainly beneficial.

The dystopian narrative that depicts a society dominated by machines and ubiquitous surveillance is common, expressed through fiction, film, and even journalism, which tends to draw on the same sinister rhetoric. George Orwell’s 1984 describes the homogenization and suppression of culture through rules, systems, and constant surveillance. In a more recent popular expression, Pixar’s Wall-E imagines a future humanity composed of zombie-like innocents, shuttled along by automated chairs, staring feebly into digital screens, mobilized—and controlled—by machines. The plausibility of these futures is made even more vivid by the unfolding story of the depth of NSA surveillance.

To paraphrase a recent piece by Don Norman, it all depends on how we design and develop augmented reality applications. If we manage to create useful and utility-producing applications with wearable technologies like Google Glass, people will benefit. This seems at first more like a truism than truth. But the obviousness of the statement belies the underlying premise, which is that Google Glass and its future iterations are simply a canvas on which we can write the future of our “augmented” everyday experience. So let’s not leave it all up to Google, shall we?

Big ideas

Ideas for the positive future of augmented reality abound. Augmedix, for example, is a small company with a vision of Google Glass re-shaping the doctor-patient relationship. Increasingly, the burden of the new and fraught world of digital medical records is damaging this interaction. Doctors stare at screens instead of faces, they spend as much time clicking checkboxes and radio buttons as they do examining the bodies and listening to the voices of the people under their care. Augmented reality could turn this scenario on its head by allowing doctors to look at and converse with their patient while simultaneously accessing and transmitting important information through Glass. This will almost certainly lead to fewer errors, an increase in trust, and ultimately better health outcomes.

A doctor wears Glass with the Augmedix app.

Or consider William Gibson’s Spook Country, a novel in which a central character creates “locative art,” what you might call augmented reality sculpture. Imagine looking at a city fountain with your augmentation goggles and seeing a bloom of light and color where others see only water. That we could transform our physical landscape in a way that enhances its beauty—rather than simply enhancing its economic potential—is a stunning notion. Unlike 3D movie glasses or straight-up “virtual reality,” the idea of a physical/virtual mashup offers us a chance to experiment and play in realms previously only available to the world of screens and displays, without losing the notion of being present in a place, something virtual reality cannot avoid. We remain in the real world.

The design of augmented reality

The first attempts to harness the power of Glass-like technology will be “ports,” shoe-horning old functionality into a new form factor. Text and email messages will appear, caller ID will notify you of a phone call, the front-facing camera will take a picture or video on command. But none of these use cases address new goals. They simply make achieving old goals incrementally faster or more convenient. I don’t have to lift my phone and look at the screen to see a text message or know who’s calling. I don’t have to lift my camera and press a button to take a picture. The difference in my experience enabled by porting functionality from my phone to Glass is a difference of degree, not a difference in kind.

More interesting will be the forays into using augmented reality tech to solve previously unmet goals. Augmedix is a good example, because it bucks a trend toward less personal medicine and solves both a doctor and a patient goal. Locative art is similarly interesting, because it provides an entirely new artistic medium and way of experiencing that art. Mapping and orientation in a visually augmented world represents another fundamental change, because it bridges the gap between the abstract 2D map and the immediately actionable—a translation that currently happens in the human brain.

Go get ‘em

Augmented reality is in its infancy. Google Glass still faces some serious challenges, especially on the hardware front—miniaturizing the device and making it less obtrusive is necessary to make it less like pulling a skateboard on a leash everywhere you go. But the frontier for experience design this device opens up is huge, and doesn’t have to remain within the boundaries Google sets. Part of our challenge and calling as a UX community is to think deeply about what an augmented experience feels like, and how it shapes people’s lives. As you would with any user experience, let unmet user goals guide your design.

Your role in this revolution is just beginning.

Engaging Millennials – the UX Boot Camp: Wikipedia

As mobile devices become widely adopted, organizations are increasingly focused on designing engaging experiences across multiple platforms. At Cooper’s UX Boot Camp with Wikimedia, the non-profit took this a step further, challenging the class of designers to create a solution that facilitated content input and encouraged a new group of editors, specifically Millennial women, to contribute through mobile devices.

Read More

UX Boot Camp: Canine Companions for Independence

Join Cooper, Canine Companions for Independence and designers from all over the world, November 19-22 in San Francisco, CA, for the last UX Boot Camp of 2013.

The Goal:

Create interactions and experiences that stimulate memory and cognitive functions to facilitate communication between veterans and their service dogs

The Team:

Designers, Product Managers and Developers from around the world will converge on the Cooper offices for 4 days to immerse themselves in Cooper’s Goal-Directed Design methods and take their design-thinking to a whole new level.

The Client:

Canine Companions for Independence, a California based non-profit, enhances the lives of individuals with disabilities by providing highly trained assistance dogs and ongoing support.

Save your seat now!

Read More

Amber Alert: The Tragedy of Bad Design

If you live in California or New York and you own a cell phone, you probably recently experienced the new Amber Alert capabilities. And by “capabilities,” I mean “the government’s newfound ability to disturb your sleep with non-actionable information.”

In California, the alert that set all this ablaze was in reference to a man, James Lee DiMaggio, who may or may not have killed his friend and her son, burned his house down with them in it, and fled with her daughter. Not that you would have known that from the Amber Alert: “Boulevard, CA AMBER Alert UPDATE: LIC/6WCU986 (CA) Blue Nissan Versa 4 door.” Certainly, Twitter has been all a-buzz about the alerts, and there are dozens of articles on the subject (my personal favorite headline: “Shaquille O’Neal: Yeah I Got That Amber Alert”).

Read More

UX Boot Camp with Marketplace Money

Old School Radio Meets the Digital Age

Take a look inside Cooper’s June, 2013 UX Boot Camp with American Public Media’s Marketplace Money radio show, where students explored the next horizon of audio programming—a paradigm shift from broadcast to conversation-based platforms.

The Challenge
Students rolled up their sleeves to help the show respond to the trend away from traditional radio by finding the right mix of alternative distribution platforms. Marketplace Money came equally ready to take a radical departure from their current format in order to create a new model that redefines the roles of host, show, and audience in the digital age. To reach this goal, students focused on designing solutions that addressed three big challenges:

  1. Engage a new, younger audience that is tech savvy, and provide easy access to content via new platforms, such as podcasts, satellite radio shows, and the Internet.
  2. Inspire audience participation and contribution. Facilitate conversations and inspire people to share their personal stories so that listeners can learn from each other.
  3. Design ways for the host to carry an influential brand or style that extends beyond the limits of the show and engage with the audience around personal finance, connecting with listeners in ways that are likeable, useful, and trustworthy, making the topic of personal finance cool, fun and approachable.

At the end of the four-day Boot Camp, student teams presented final pitches to Marketplace Money, and a panel of experienced Cooper designers offered feedback on their ideas and presentations.In the following excerpts from each day, you can test your own sensory preferences for receiving content as you see, hear and read how design ideas evolved at the Boot Camp, inspiring new relationships between people and radio.

Marketplace Money Class

Read More

Explore New Interaction Paradigms at UX Boot Camp: Wikimedia

Advance and apply your UX design skills to a meaningful real-world problem in this intensive, hands-on workshop


This September, join Wikimedia, Cooper, and design-thinkers from around the world as we find new ways to spread knowledge through mobile Wikipedia. In this four-day workshop, you’ll use new UX skills to make mobile content contribution more approachable, intuitive, and less reliant on traditional input methods like typing. If you’ve wanted an excuse to explore new interaction paradigms and stay ahead of the design pack, this is your chance. Best of all, you get to do all of that in the creative classroom setting of Alan and Sue Cooper’s 50-acre ranch in Petaluma, CA.

Register now: UX Boot Camp: WikimediaSeptember 17-20, Petaluma, CA

What’s in it for you?

  • Learn new interaction techniques and approaches under the guidance of industry leaders, including Alan Cooper
  • Learn how to think through a problem from both a design and business perspective, rather than blindly applying methods by rote.
  • Energize your practice and make new connections by working on a real-world challenge with peers from around the world.
  • Beef up your portfolio with a smart, new design concept
  • Pick up leadership and collaboration skills that will help you better navigate your work environment.

Read More

iZombie? A zombie self-diagnosis and self-destruction app

As Halloween approaches, and the veil between worlds grows wan, threadbare, and permeable, Cooper turns its collective attention to the spirit, spook, and creature population. Last year we sought to understand them from a Goal-Directed perspective. This year we take the next unholy step and design software, devices, and services around these personas. Today we revisit Emily.

Emily is in trouble. She narrowly escaped a horde of flesh eating zombies, but was bitten in the process. Now she’s suffering under the gradual onset of zombification—cognitive decline, neurodegeneration, loss of motor control, and an increased apetite for delicious, raw, human flesh. She wants to stave off zombiism as long as she can, but she knows that once she’s crossed a threshold, she will succumb and attempt to kill her friends and eat her family. What can she do? Enter iZombie?, an app made specifically for zombie-virus-infected humans, distributed by the military for free to all civilians at the first sign of the inevitable plague.

Read More

Cooper helps Chefs Feed launch new social features

Who do you trust for food advice? Review sites like Yelp are bloated and contain a cacophony of opinions. Others just aggregate shallow star ratings. Reviewers often have tastes and preferences that might not match your own. And even if you find a good restaurant, how do you know what is the best thing on the menu?

The idea behind Chefs Feed is that the best food advice comes from experts – professional chefs, and friends with discerning taste.

Currently available in nine US cities, the app doesn’t just tell you where to go, but also what to order, providing an insider’s look at each city’s eateries.

When Chefs Feed approached Cooper, the startup was about to make a big leap. Lots of people were downloading the app, but its functionality was limited to a few features like reading and bookmarking chefs’ reviews. With the user base expanding quickly, Chefs Feed needed a blueprint for making the app a platform for interaction between chefs and foodies. The app also needed features to help friends trade dish recommendations and share their passion for food. In short, the app was to get social.

Read More

Detailing Lyrical Travel

In our last Drawing Board, we explored the desire some travelers have to uncover and experience the authentic spirit of a place. For the Lyrical Travel Drawing Board, we meet Jeanette. She’s in a new town with a day on her hands. Without much to go on and wanting to avoid the same old guidebooks, she turns to Latourex for inspiration.

The Drawing Board Episode 10: Lyrical Travel from Cooper on Vimeo.

Credits: Chris Noessel, Christina Worsing, Greg Schuler

In the video we introduce the high points of the service. In this blog post we’ll go deep in explaining some of our design thinking about how such an app might work to help Jeanette accomplish her goals.

Our Inspiration

This Drawing Board was inspired by “Experimental Travel”, aka Latourex. Latourex is short for LAboratoire de TOURisme EXperimental. Experimental Travel was developed by Rachael Antony and Joël Henry as a series of techniques to make travel more interesting. In this type of travel, people play “games” on the road, to get them off the beaten path. We love this idea where chance operations inspire and help support us in our search for unique and authentic travel experiences. In our Drawing Board, we explore how this approach might translate to a service design opportunity via a mobile experience.

The first thing we realized is that as in Latourex itself, no single game would suffice. So we envisioned a container app called Latourex Travel Games that has an open and extensible architecture. Some game modules would be available at launch, but over time more modules could be added, keeping the content fresh. Lyrical Travel would just be one module within this container app, and the sketch below shows that the Latourex app is what Jeanette would download.

Once downloaded and launched, the app would briefly introduce itself, and then get Jeanette to the action as quickly as possible by providing a default selection from the available modules. If she wasn’t interested in this first randomly selected module, she could “roll the dice” (or shake the phone) to get another one. While fun and in the spirit of Latourex, that mechanism could become tiresome if she was looking for a particular module, so we provided means to select a particular one from a list if she wanted. Since we knew that such an app would grow by word-of-mouth, we also let her replicate the module through which she learned about the app: Lyrical Travel.

In the world of Latourex, Lyrical Tourism is a game where travelers select a song for a given location, and let the lyrics suggest things to do and see. When we imagined what this could be in a mobile app, we recognized we could keep the spirit of the original game and improve upon it by providing:

  • A huge database of songs that might fit given locations
  • Easy access to lyrics
  • Automated local suggestions based on those lyrics
  • Simple ways to capture, curate, and share the experience

We saw that we could facilitate this experience in four steps: getting started, guided suggestions, capturing key moments, and curating/sharing the results.

A Four-Step Process

1. Getting Started

In preparation for her adventure, Jeanette first tells the app some things about her travel: where she wants to explore; what her transportation options are; and how much time she has. We suspect that most users would open the app just before wanting to explore, so the default screen uses her current location, allowing her to select another or even randomize. Rather than make these separate steps, we thought the map screen had enough real estate to let these be simple radio buttons or toggle buttons as overlays to the map. This way it would feel more like a single step even though it has three separate pieces of information.

We’re also big fans of the power of Lyrical Travel to encourage people to rediscover familiar places like their hometowns and suspect setting the current location as the default would encourage this idea when people open the app near home.

With a location set, a song is next needed. By default, Lyrical Travel makes a suggestion, selected at random and based on a search of the location Jeanette has chosen. She has a control that lets her listen to it to see if she likes it. That makes it simple to use. We understand that a default song wouldn’t always be to her liking, so the app enables her to either “re-roll the dice” or allow her to select from her own music. (You’ll notice the interaction is the same from the prior screen. You can read more on the pattern, below.)

These two options—randomization or direct selection—work, but we also wanted to provide her an “in-between” option that would keep the interaction simple and still ensure an ideal match. To accomplish this, we thought that she could permit the app to scan her music collection (on the phone, or possibly in the cloud). In this scan it could look at any ratings she provided, combined with the sheer volume of songs she has of particular categories. From this, Latourex could infer the categories of music she prefers, similar to how Pandora can infer preferences from a small set of choices.

2. Guided Suggestions

With a place and a song in hand, the app next begins to make suggestions on what to do by looking for key words in the lyrics. The algorithm would try to find more unusual words and unique phrases first, so the suggestions would be particular for each song, but gracefully degrade to more common words if those first ones don’t have any results. The app would then display the suggestions it found about local activities and place them alongside the lyrics, highlighting the connections if possible. Suggestions include an image, a title, and a small overview. With a tap Jeanette can get to more details, including directions from her current location and the ability to rate suggestions. She is able to delete any suggestion with a swipe, giving room to other suggestions of greater interest.

3. Capturing Key Moments

Using persistent tools below the suggestions, Jeanette is able to capture moments through text, photos, and video. She’s free, of course, to follow these to the letter or interpret as she pleases. As she collects memories of the day, they stack below the suggestions, next to the lyrics. If she wishes, she can delete or rearrange these with standard iOS gestures.

4. Curating and Sharing the Experience

Jeanette is able to save and share her experiences through curated “video postcards.” We wire-framed three different ways for her to edit: Edit by ratings (the simplest), simple editing, and power editing. Below you’ll see the sketch for the Edit by Ratings screen, which lets Jeanette rate individual elements. The Postcard Player chooses and dynamically displays the elements during the duration of the lyrics, ensuring a minimum time per element, and augmenting elements with “The Ken Burns Effect” slow panning and zooming effects. Though we did not have time to comp it up, we thought that to be in the slightly-random spirit of Latourex, the postcard wouldn’t be a fixed video, but a semi-random display, like slideshows from a photo album app, that plays with the selected song and overlaid lyrics.

To share the postcard, Jeanette has the option to share with others by posting the video to the Latourex site, or sending a link directly to whomever she likes. There are some complexities about getting the media from her phone to the site, but we did not detail these out in this wireframe pass.

Design Challenge 1: Keeping Jeanette in the world

To meet Jeanette’s goals, her experience should involve interacting with technology but not being consumed by it. She shouldn’t walk around the city all day with the phone in front of her face. How do we create interactions that integrate into the flow of a person’s day but not interrupt it? To solve this problem, we made the interface as modeless as possible, letting her drop in and out of it as she needs or wants to.

Design Challenge 2: Avoiding musical burnout

If a single song serves as a guide, how is it offered in a way where the person doesn’t burn out on it by the end of the game? To solve this problem, we chose not to play the song by default. It was tempting to have the song underscore the lyrics as she was interacting with the lyrics, but it felt like it would be too much. It’s always opt-in, and the app would work well with her music player app.

Design Challenge 3: Varying degrees of control

When it comes to planning and travel itself, people want varying degrees of control. From highly supported and structured to free-form and interpretive, Lyrical Travel enables people to make decisions from direct selections to self-guided interpretation. This pattern of enabling people to move from simple to more complex decision-making processes is experienced across the service. For example, in the process of choosing songs people can accept suggestions; choose from random; or make their own, personal selections. In using the lyrics as a guide, people can select and follow specific suggestions in the order in which they appear in the song; jump around and choose at random; or disregard suggestions completely and interpret lyrics using their own internal compass. And finally, when creating a video postcard, as mentioned earlier, people can turn the reins over to Lyrical Travel or take full control of curating their day using the tools provided by the app.

Design Pattern: Random default first & Roll the dice

In several places throughout the app, we noticed a pattern emerging, and once we found it, we opted to stick to it. The pattern is that for choices Jeanette needed to make, we would not present a menu of options. Instead, we would first present a good, default selection. If she was in a hurry or wanted to keep it simple, Jeanette could just press OK and be on her way. And of course we would offer an option to select directly through a full-featured menu, but that didn’t feel complete given the spirit of randomness that’s inherent in Latourex. To meet that spirit, there’s a “roll the dice” option that would select another option randomly. And in the music selection we even went one step further to take advantage of existing data to improve the likelihood of a roll Jeanette would really love.

Service implications

In order to deliver on the promise of the ideas in the video, we realize that there would be not just an app, but a complete service underneath it, to help connect travelers and house the marketplace of modules. Imagining this container service led us to aspects of the design such as hearing about it through an existing member, letting similar travelers’ selections influence suggestions, and giving Jeanette options to meet other Latourex travelers near her.

Wrapping Up

We had a lot of fun making this short video as we explored some fun topics like travel, inspiration, and chance operations. We hope you’ll enjoy the fruits of our labor. We even have a bit of an itch to have this app in hand ourselves when we do our own travel, and though we don’t ever want to design for ourselves, we think it’s a positive sign. What do you think? Would you use this app? Can you think of other ways Travel Experiments can be brought into such an app without losing their magic, and encourage a community of Latourists? Let us know what you think in the comments below.

Contributors: Chris Noessel, Christina Worsing, and Suzy Thompson

Related reading

1 2 3