A better algorithm isn't enough to fix Netflix's recommendations

There has been a lot of hype recently as Netflix announced provisional winners of their million dollar contest to improve their recommendation algorithm. The goal was to improve matching by 10%. Since it took over 50,000 entrants the better part of three years trying to improve past 10% this is probably a trick they can only pull off once. Given that their current recommendation engine does a miserable job of recommending movies for me, even a 10% increase isn't likely to be particularly satisfying.

I've rated just shy of 800 movies on Netflix;and just over 150 items on Amazon, yet Amazon's recommendations are usually satisfying while Netflix struggles to accurately recommend any movies I'd like to see. This isn't a case of esoteric movie tastes, in fact I'm fairly mainstream, largely preferring the entertainment of a summer blockbuster to the intellectualism of an indie documentary. The books I like are the opposite: non-fiction, obscure, expensive, limited runs, or out-out-of-print, in short; not popular. And still, Amazon recommends the right books.

Pandora is a music service which delights me by consistently recommending new music to me which I like. Netflix can't give me great recommendations. Amazon and Pandora do. Why?

Clearly the algorithm is a critical part of any good recommendation engine. But there seem to be limits to what can be accomplished just by tweaking the math. If Netflix can only squeeze a 10% improvement out of the calculation for recommendations, where can they turn to get additional increases in quality?

Tweaking what happens before and after the algorithm seems to be the only other opportunities. Both of these are ultimately interaction design solutions. Let's take a look at a few approaches to recommendations used by Netflix, Amazon and Pandora and see how they lead to different results.

Continue reading...

What does sustainable interaction look like?

In the past few years the design community has taken sustainability from a mere buzzword into a call for action. Eli Belvis, Tony Fry and Cooper's own David Fore have all championed the idea that the practice of interaction design must promote and encourage sustainable decision-making. The Designer's Accord has emerged as a mandate to turn the the goodwill into commitment and a plan of action, improving the role of design in sustainability.

This is all good, it's all needed, but we also need to get down to brass tacks. A cursory survey of the internets reveals a hunger for actionable discourse about sustainable interaction design. What does sustainable interaction design look like in the wild? What does sustainable mean when it comes to designing software? What are practical design choices that encourage sustainable behavior on the part of the end user of software?

Continue reading...

A good design critique

How do you thoroughly critique a design without crucifying the designer? What are ways of critiquing that result in better designs, rather than defensive justifications?

Scott Berkun explores a model for design critique in a detailed post, but I'm interested in the little stuff that works for your design team in day-to-day practice.

At Cooper, our teams often work together for a year or more. It is important for us to create a dynamic of cooperation, but great design often happens when we push on assumptions and challenge the first iteration. We want to encourage this critique, but make sure that it doesn't derail the meeting.

Why is that good?

It's pretty common to hear a skeptical Cooper designer begin a critique with some variant of the question, "Why is that good?" Many ways to express disagreement have negative effects on the meeting or relationship. "That won’t work because," or "But what about." These tend to bring momentum to a halt. Designers must stop, defend their ideas, or chase objections.

As anyone who has faced a blank whiteboard knows, once the ink gets flowing it is important to run with it and see where the idea goes. Communication strategies of design partners can enhance or detract from this process. By asking to see the goodness, we focus on enlightenment, encouraging our partner to help us see what they see. Also, asking an open-ended question is an acceptably naïve way of pushing your design partner to step up and show you what is going on in their mind.

At the core, we want our teams to feel comfortable in expressing healthy disagreement, and to focus on clarifying rather than justifying.

What are ways that your team has developed to critique design while maintaining harmony on the team?

 

What do you think? Join the conversation in Comments

Making people think

Software makes us think. Sometimes, it aids productive thinking by pointing us toward the right things to think about. Other times, it gets in our way, poses confusing choices, and generally frustrates us; this unproductive thinking can be seen as the cost of doing business with an ineffective interface.

Christof van Nimwegen's doctoral thesis focuses on the ways in which software can be used as an aid in creative thinking, and it specifically discusses the trade-offs between requiring users to construct an internal understanding of the system, and externalizing that system in the interface, via menus, dialogs, or wizards.
Bill Thompson, a regular commentator on the BBC World Service program Digital Planet, enthusiastically responded to the paper with the following:

It is also the sort of basic psychological research that we desperately need in the Web 2.0 world where major sites like Facebook are constantly being redesigned on the basis of little real understanding of how people engage with their computers.

I was interested to see someone addressing interface design from a strictly psychological perspective, rather than one rooted in interaction design:

We concluded that relieving a user’s memory and making interactions assisted by externalizing information does not have beneficial effects. It makes users count on the interface and gives them (unrightfully so) the feeling that the task and thinking-work is partly done for them, which seduces users in more shallow cognitive behavior.

Wizards can have the effect of seducing users into shallow cognitive behavior. When users are guided through a simple process, they are often shielded from an underlying complexity. While saving the user time and effort in the short term, the wizard may also make them less capable in the long term because they haven't had to deeply consider their actions.

Nimwegen continues:

... Interaction should facilitate or even persuade users to learn what underlies the task they are doing. The same is true in situations where interruptions are commonplace and where in the meanwhile mastery of what is underlying a task or domain is desired, or when operations come with a cost and direct solutions without deviations are the aim. In designing our interfaces we have to be careful with providing interface cues that give away too much, and must design in such a manner that the way users (should) think is optimally supported, which in turn could help the software to achieve its specific goal.

Not every task is important enough to teach users the mechanisms that support it. Many interfaces benefit from a level of abstraction or decoupling from the underlying processes. The spirit of this research is to point out that the effort to dumb down can go too far. Removing some of the obstacles to learning complicated or deep domain applications may actually do more harm than good for a user.

For example, a beginner may struggle to through the myriad complexities of 3D modeling software, and this struggle may in the end produce more competent users. The software shouldn't erect unnecessary obstacles, but a learning curve that is too shallow may actually hinder their ability to really develop competence in program in the long run.

(Via BoingBoing)

 

What do you think? Join the conversation in Comments

Human motivation as a way to understand user goals

At Cooper we talk a lot about goal-directed design. Usually the term "goal" is used without an explicit distinction between goals and a motivations. The distinction is an important one which can influence design.

Users enjoy the satisfaction of achieving their goals. User goals help us focus our design on solving meaningful problems for the user. If we design with the user's goals in mind, in the best case we will help them achieve their goals, at worst we stand out of the way.

Goals are defined as the "state of affairs that a plan is intended to achieve". Goals are what a person wants to do, achieve, or become. They are boundaries to states that people strive toward, and once they reach either terminate their efforts, or shift into a maintenance of the achieved status. Feeling smart, getting the best deal, and living the good life are all examples of goals. They represent a desired end-state of a set of particular actions.

Motivations are the drivers behind setting and pursuing goals. Motivation is why someone wants to do something. Motivation is what arouses and sustains action toward a desired goal. It gives purpose and direction to behavior.

In researching this, I was disappointed to find that there is little consensus on what are the core motivational needs. Some theorists claim motivation comes from stimulus-response, others from affect, others describe motivation in terms of social or cognitive drive. A survey of the major motivational theories reveals a few commonalities. Needs, desires and wants are the sources of motivation. Motivation directs behavior toward increasing, decreasing, or maintaining a specific state. Dr. William Huitt's list of motivational needs provides an overview of all of the major theories. I have distilled this list into the essentials.

Continue reading...

Better ways to login

With Governor Sarah Palin's very public web-email security breach this week, there are dozens of blogs and websites pointing out how common password reset schemes are broken and debating how to improve the security of password reset tools.

Of course, password reset is just one part of the login experience which could use some improvement. There are a variety of ways to set up a web login system. A survey of many of the leading social networking and web services reveals some best practices and a number of less than satisfactory solutions. The choice of approach to a login ID determines how users must act to recover usernames and passwords and what kinds of verification must be provided.

Continue reading...

Full disclosure: This information has been processed

When we create a persona or a model organization, we're deliberately creating an archetype — a person or company that does not map to any one "real" person or company out there in the world. In creating personas, we need to be up-front with ourselves and our clients about the choices and assumptions we made along the way. We also need to be clear about what questions we asked and what we didn't. When we don't have the data, we need to acknowledge this and rectify it if necessary.

This point may seem like a methodological nuance, but it relates to ethical considerations that in other realms, as I recently discovered.

My design partner Chris Noessel and I just completed three weeks of research travel around the world. Neither of us had been to many of the countries, and we both photographed our adventures obsessively. One morning, he asked me to compare a photo he took to one that I took: Why did they look so different? We were using almost identical cameras and taking photos often of the same views.

Chris's photo.

My photo.

Why does mine look different? Because I adjust the photographs post-capture, slightly adjusting the contrast, lightness, and so on. For me, the unprocessed photos rarely convey my experience of the event or location, and the post-processing is intended to re-create my memory of the experience. I take photographs to share that experience, not to share the exact pixels the camera captured.

Chris admitted that it made my photos "look better," but that I "took liberties" to adjust, and once I started, where would I stop? How much change was too much change? How different could it be from his untouched version and still be the Great Wall of China?

Of course, this is part of a much larger conversation. Photographs appear to be very faithful representations of reality, so one may argue that viewers of photography bring a different set of expectations to them than they do to other visual art. Viewers expect photos to be more "real," more true to life, and therefore post-facto monkeying could be seen as deceiving. On the other hand, who is to say what "real" is, really?

Essayist and photo critic Susan Sontag addresses this argument in the introduction to her book, On Photography.

In deciding how a picture should look, in preferring one exposure to another, photographers are always imposing standards on their subjects. Although there is a sense in which the camera does indeed capture reality, not just interpret it, photographs are as much an interpretation of the world as paintings and drawings are.

Even before taking the shot every photographer has made choices which will affect the captured image — camera and lens, film v. digital, SLR v. point-and-point shoot — and each has an effect on the contrast, color, and depth of field, aspect ratio, and so on. We can continue to split hairs, too; for instance, we accept that the journalist which uses a telephoto lens is "telling the truth" even though it grossly manipulates scale between foreground and background. With so much noise in the system, it seems arbitrary to assign "reality" to the raw output of the camera, doesn't it?

The National Press Photographers Association defines a couple of broad categories in the altering of photographs.

There are technical changes that deal only with the aspects of photography that make the photo more readable, such as a little dodging and burning, global color correction and contrast control. These are all part of the grammar of photography, just as there is a grammar associated with words (sentence structure, capital letters, paragraphs) that make it possible to read a story, so there is a grammar of photography that allows us to read a photograph. These changes (like their darkroom counterparts) are neither ethical nor unethical — they are merely technical ... [However], once the shutter has been tripped and the moment has been captured on film, in the context of news, we no longer have the right to change the content of the photo in any way. Any change to a news photo — any violation of that moment — is a lie." [The emphasis is mine].

The NPPA distinguishes between the technical aspects of making photos "more readable" and "changing the content," and I think that this is an interesting analog to the world of creating design targets (i.e., personas, organizations, environments). In our process, you could look at the transition from research to personas is the process of making the research "readable."

Of course, creating personas from research is a lot different than manipulating contrast and lightness in a photo editing app, but the principles are the same: Altering the content is a lie; each archetype that we create should faithfully reflect the gathered information, and each should bring out the priorities, needs and experience imperatives that affect the design. You can monkey with research just like you monkey with photos. When done well, slight adjustments to the color and contrast of the research more effectively reveals the truth. When done badly, they can lie and deceive.

 

What do you think? Join the conversation in Comments

"Wandering" can be productive during user interviews

Recently, a client who was observing us perform stakeholder interviews made a casual off-hand remark at the end of the day that the interviews had "wandered around a bit." We had explained how our interviews are less survey-driven, and more ethnographic in style, but it's often hard for the uninitiated to see the immediate value of an ethnographic type approach to interviewing, especially when it results in circuitous answers. We were particularly happy with the wandering of our interviews, which had produced visceral clarity which could never have been delivered with an overly structured interview. For example, hearing that the back-end systems are "dog shit" provides an additional layer of information than simply hearing that they're "dated" or "inadequate."

Tommy Stinson, Strategic Director at Cheskin, another Bay Area innovation engine recently blogged: "The goal of the discussion isn't to just get the participant's 'take' on the topic (at least it's not limited to that). The goal is to understand this person (or people) and their culture - the 'webs of significance.'"

We work from structured interview instruments, but as a journalist friend of mine is fond of saying, "the best quotes happen when the tape stops rolling." When we leave the scripted interview and allow someone to lead the interview themselves, often things which we couldn't predict or identify are revealed — and, in some cases, new topic areas can be added to the instrument as a result. Of course it's important to return to the script to hit all of the main questions we have, but it is equally useful and important to allow an interview subject to lead a little, to give them enough time and latitude to wander into areas which are not on the map.

 

What do you think? Join the conversation in Comments