How good designers can create evil

I’ve been reading Philip Zimbardo’s The Lucifer Effect and thinking a lot about system design as a result. In his words, the book “is a call for a three-part analysis of human action by trying to understand what the individual actors bring to any setting, what situational forces bring out of those actors, and how system forces create and maintain situations.” It’s a rather sobering piece of work, especially as a designer who earns a living designing interactions and systems. The author challenges the common tendency to attribute human failings to an individual’s inner nature, disposition, personality traits, and character and demonstrates how situational and systemic factors seduce ordinarily good people to commit evil acts.

A large part of the book is dedicated to detailed case study of the 1971 Stanford Prison Experiment, tracking the transformation of happy healthy college students playing randomly assigned the role of prisoner or guard in a mock prison.

Stanford Prison Experiment
Stanford Prison Experiment guard in uniform, from lucifereffect.org.

The experiment was terminated early because of the astounding and terrifying impact it had on the participants. The students assigned to the guard role became sadistic and the prisoners became depressed and showed signs of extreme stress. It was clear from notes and diary entries before, during, and after the experiment that the situational and systemic forces resulted in the students doing things they could never have imagined when outside those force fields.The experimental design for this study is wrapped in a dramaturgical narrative, with roles, costumes, props, and stage settings all carefully crafted to create a situation and system that felt authentic so that the psychological effects of being in a prison could be observed. All these elements worked together to deindividualize and dehumanize the prisoners. In the book Zimbardo concludes that dehumanization is one of the central processes in the transformation of ordinary normal people into indifferent or even wanton perpetrators of evil, and the goodness of Everyman and of Everywoman can be transformed and overwhelmed by the accumulation of small forces of evil.

This experiment was explicitly designed to dehumanize, and it was frighteningly effective, but consider for a moment the countless systems that you deal with daily that are (hopefully) unintentionally dehumanizing – from dealing with your healthcare insurance provider, to working in enterprise software. I dealt with this first hand recently when I managed to incur a small cut and ended up in the emergency room one evening for a small but deep cut. I was amazed at how undignified and dehumanizing the whole experience was. Once I had been plucked from the waiting room and processed, I was taken back to sit on a bed where nurses and doctors alike walked past me for over 2 hours without acknowledging my presence.

While I was waiting I got to thinking about what situational and systemic forces may be at work to make healthcare professionals complicit in creating such a dehumanizing experience for the very patients they have sworn oaths to help and care for. How had these systems been put in place? And was any thought given to the experience of the doctors?

Designing for good

We seem to have found ourselves in a situation where we take for granted that individuals continue to struggle to maintain their dignity, autonomy, and morality against the situational power of a mercilessly unjust authority figure or an indifferent, hostile system. As designers, we must ask ourselves we can avoid designing hostile systems? in his essay Human dignity and human rights: thoughts on the principles of human-centered design, Richard Buchanan suggests that we must do this by practicing human-centered design and returning to first principles that “design is fundamentally grounded in human dignity and human rights.”

Practicing human-centered design goes further than conducting usability tests and solving tactical issues. It means taking a step back and considering the broader implications of your work. It means thinking carefully about what human dignity means and how your work impacts the people that come into contact with it. Much of the work we design shapes the structures and systems that provide a framework for our culture. Let’s work to ensure it’s a culture that upholds the dignity of all the people in it.

6 Comments

Janet
Today's New York Times reports Alan Greenspan as saying "Those of us who have looked to the self-interest of lending institutions to protect shareholders’ equity, myself included, are in a state of shocked disbelief." My cynical side hears echoes of Captain Renault in Casablanca: "I'm shocked, shocked to find that gambling is going on in here!" Perhaps Zimbardo's book should be required reading for financial regulators, to help them understand "what the individual actors bring to any setting, what situational forces bring out of those actors, and how system forces create and maintain situations."
Emma van Niekerk
I agree Janet. I think it would be good reading for the regulators as well as all the people working in the banking system. Zimbardo argues that although we cannot escape these forces, being aware of them gives us much better odds of resisting and challenging them. In the last chapter he outlines a set of strategies and tactics to help anyone be more able to resist unwanted social influences. What we need now is a set of strategies and tactics for designing systems that do not result in evil behavior by good people.
Chris Noessel
Not to darken this already dim thread, but I know of a two other experiments that illustrate how easily people can be led to evil: The famous Milgram experiment, in which experiment subjects were willing to "shock" other test subjects past lethal levels because a "scientist" told them to. http://en.wikipedia.org/wiki/Milgram_experiment Jane Elliott foresaged the Stanford experiment with much younger children when she segregated her class into brown eyed and blue eyes students. http://www.pbs.org/wgbh/pages/frontline/shows/divided/ Perhaps, this is just more evidence of the need to consider the systemic effects we design and encourage goodness. (Trying to put a positive spin here.)
Philo
Interesting thoughts - reminds me of the work of Hans Monderman. He was a traffic engineer, and worked with similar ideas of people interacting in deliberately-designed systems. What made him unique was how he valued the humanity that participants brought into traffic systems and accounted for its impact on behavior in his designs. He even proposed that the more humanity there is in a systems, the better it works. He created the "shared space" design approach, which eliminates curbs, lanes, traffic signs, stoplights, etc. Users of the system rely on their own consideration and negotiation instead of systematic enforcement. This made for a more pleasant experience and also smoothed traffic flow, reduced trip times, and improved safety. A great Monderman quote: "We're losing our capacity for socially responsible behaviour, ...The greater the number of prescriptions, the more people's sense of personal responsibility dwindles." Or more succinctly: "When you treat people like idiots, they'll behave like idiots."
Dane Holewinski
Interesting application of the Lucifer Effect. The importance of good design is accentuated by the continued expansion of information technology. IT, for all of its efficiency benefits, certainly has the potential to institutionalize and accentuate dehumanizing proceses and systems. From a consumer perspective, it seems that there are an increasing number of beacons of "human-centered" products, services, interfaces, companies, etc. From my experience, the consumer market is demanding and rewarding pioneers in this area. Within the B2B space where efficiency and return tend to be the primary decision criteria and authority rather than choice often drive technology adoption, there has been less progress. Sadly, this seems to be the area in which the most potential for "evil" exists. Hopefully consumer driven innovation in will find application in the business market as well. I'd recommend reading "Blink" by Malcolm Gladwell (of "Tipping Point" fame). While the book explores how "intuition" and rapid unconscious processing of information in complex situations can lead to better decision making, it also draws on numerous examples of how situational, structural and cultural factors influence rational decision-making, cognition and judgement.
David Fore
By implicating interaction design in acts of evil, Emma forces us to reckon with the moral content of our actions. But that’s so uncool! Hey, aren’t we supposed to be cynical about this sort of thing and disparage do-gooders? But having read this article, one cannot pretend one’s professional conduct is innocent of moral content. Every time a designer makes a choice—or neglects to do so—she is employing judgment about what is right and what is wrong. That’s why designers like to be designers. It’s a hit. But with great power comes great responsibility. Don't let's automate the misery!

Post a comment

We’re trying to advance the conversation, and we trust that you will, too. We’d rather not moderate, but we will remove any comments that are blatantly inflammatory or inappropriate. Let it fly, but keep it clean. Thanks.

Post this comment