To test, or not to test? You have more than two options

Just as every author needs an editor and every engineer’s code needs QA, every designer’s work can benefit from evaluation. Does it surprise you that I’m saying this? As UIE’s Christine Perfetti once pointed out to me in an interview, Cooper is better known for advocating up-front research, effective process, and skilled designers than for promoting the value of usability testing and other evaluation techniques. It’s true that given a limited budget-which most people have, these days-we think investing in these early-stage activities yields greater value. That said, evaluation is so important that every design project at Cooper has evaluation techniques built in. When, how, and how often we evaluate depends on the nature of the project.Like good programmers, good designers begin a type of design evaluation themselves, constantly assessing the design by throwing scenarios at it and asking their teammates to do the same. When the persona and scenarios are based on good data and applied by skilled designers, this approach identifies the majority of problems within a few minutes after a sketch first appears on the whiteboard. Of course, we all have our blind spots, so this kind of on-the-fly evaluation isn’t enough, especially when you’re several months into a project and getting awfully close to the problem. Another pair of eyes is essential, and many more pairs of eyes are better still.

All of our projects at Cooper include expert reviews. Projects involving interaction design always include scenario walkthroughs. We apply both techniques at multiple stages. Not long after the first framework sketches begin to coalesce or the first design language studies are done, a senior designer examines and critiques each solution based on design principles as well as the goals and skills of the personas. These reviews continue on a regular basis throughout the project, and may involve more than one reviewer. At a minimum, scenario walkthroughs involve product managers and engineers. On any project that involves a complex domain, it’s essential to involve expert users in these walkthroughs, as well. Both expert reviews and scenario walkthroughs are quick and inexpensive. Best of all, because the design team explains their thinking in either case, either technique can catch issues while your sketches are still too vague for even a paper prototype usability test.

Of course, some flavor of usability testing is a good idea if you can do it. However, testing isn’t necessarily the “gold standard” of evaluation that many people believe it is. As a result of a series of comparative usability evaluation (CUE) studies, Rolf Molich has concluded that “There’s no measurable difference in the quality of the results produced by usability tests and expert reviews.” What’s more important, I think, is that Molich and his colleagues also demonstrated that no evaluation technique is perfect-no single evaluator or team of evaluators found every issue, regardless of what technique they used. In other words, although both activities draw upon science, design evaluation is no more a science than design itself.

Does this mean we shouldn’t do evaluation? No! It means that the more essential it is to get the design right, the more evaluation techniques you should apply, and the more often you should apply them. If you have experienced designers on the job and you’ve applied good techniques to arrive at your design, then it’s probably not tragic if you don’t run any usability tests as long as the consequences of design errors are small. If a usability problem in your e-commerce shopping cart might cost millions-and it might just!-then it would be wise to run a test or two. If it’s my loved one lying in a hospital bed being treated by your medication pump, then I hope you’ve tested more than once or twice, and I hope you’ve taken advantage of every evaluation technique available.

2 Comments

David Stewart
I'm surprised not to see some mention of sketching or Paper Prototyping. I've used (just used, actually) Paper Prototyping to design and test UX concepts cheaply and quickly. I've also used what I call, "Dirty Research" to get user research done quickly. I've done it the other way too but I don't know if I would again. Paper Prototyping by Carolyn Snyder and Bill Buxton's Sketching User Experiences have revolutionized my design process and made it possible to fit user centered design in to the existing product dev lifecycles.
Kim Goodwin
David, Thanks for your comment. The books you mention are both good resources. You're absolutely right that sketching is essential. I didn't explicitly mention it because here at Cooper, we couldn't imagine design without sketching, any more than we could imagine, say... living without breathing. Somebody puts a sketch on the board, somebody else asks questions or makes comments (thereby offering some immediate, informal design evaluation), and the sketch immediately evolves until everyone in the room is satisfied that it's working. Thanks for pointing out that not everyone thinks of it that way, though. As Bill Buxton rightly points out, designers (and engineers) learn by trying and failing. Sketching is an inexpensive way to fail early and often. Because it's visible, sketching lets others into your thinking process. By its throwaway nature, sketching implies that this is a work in progress, which invites comments and collaboration. I also didn't mention paper prototyping explicitly because it's just one of many approaches to usability testing, and I didn't want to focus on testing. It's a great way to test certain kinds of design problems, especially forms, navigation, and other interactions that use pretty standard widgets. Anyone interested in testing that sort of thing should definitely pick up Carolyn's book. I find that some kinds of design problems are more reliant on polished visual design or on animated behaviors, though, which makes those a bit tougher to assess using low-fidelity tools.

Post a comment

We’re trying to advance the conversation, and we trust that you will, too. We’d rather not moderate, but we will remove any comments that are blatantly inflammatory or inappropriate. Let it fly, but keep it clean. Thanks.

Post this comment