How good designers can create evil

I’ve been reading Philip Zimbardo’s The Lucifer Effect and thinking a lot about system design as a result. In his words, the book “is a call for a three-part analysis of human action by trying to understand what the individual actors bring to any setting, what situational forces bring out of those actors, and how system forces create and maintain situations.” It’s a rather sobering piece of work, especially as a designer who earns a living designing interactions and systems. The author challenges the common tendency to attribute human failings to an individual’s inner nature, disposition, personality traits, and character and demonstrates how situational and systemic factors seduce ordinarily good people to commit evil acts.

A large part of the book is dedicated to detailed case study of the 1971 Stanford Prison Experiment, tracking the transformation of happy healthy college students playing randomly assigned the role of prisoner or guard in a mock prison.

Stanford Prison Experiment Stanford Prison Experiment guard in uniform, from

The experiment was terminated early because of the astounding and terrifying impact it had on the participants. The students assigned to the guard role became sadistic and the prisoners became depressed and showed signs of extreme stress. It was clear from notes and diary entries before, during, and after the experiment that the situational and systemic forces resulted in the students doing things they could never have imagined when outside those force fields.The experimental design for this study is wrapped in a dramaturgical narrative, with roles, costumes, props, and stage settings all carefully crafted to create a situation and system that felt authentic so that the psychological effects of being in a prison could be observed. All these elements worked together to deindividualize and dehumanize the prisoners. In the book Zimbardo concludes that dehumanization is one of the central processes in the transformation of ordinary normal people into indifferent or even wanton perpetrators of evil, and the goodness of Everyman and of Everywoman can be transformed and overwhelmed by the accumulation of small forces of evil.

This experiment was explicitly designed to dehumanize, and it was frighteningly effective, but consider for a moment the countless systems that you deal with daily that are (hopefully) unintentionally dehumanizing - from dealing with your healthcare insurance provider, to working in enterprise software. I dealt with this first hand recently when I managed to incur a small cut and ended up in the emergency room one evening for a small but deep cut. I was amazed at how undignified and dehumanizing the whole experience was. Once I had been plucked from the waiting room and processed, I was taken back to sit on a bed where nurses and doctors alike walked past me for over 2 hours without acknowledging my presence.

While I was waiting I got to thinking about what situational and systemic forces may be at work to make healthcare professionals complicit in creating such a dehumanizing experience for the very patients they have sworn oaths to help and care for. How had these systems been put in place? And was any thought given to the experience of the doctors?

Designing for good

We seem to have found ourselves in a situation where we take for granted that individuals continue to struggle to maintain their dignity, autonomy, and morality against the situational power of a mercilessly unjust authority figure or an indifferent, hostile system. As designers, we must ask ourselves we can avoid designing hostile systems? in his essay Human dignity and human rights: thoughts on the principles of human-centered design, Richard Buchanan suggests that we must do this by practicing human-centered design and returning to first principles that "design is fundamentally grounded in human dignity and human rights."

Practicing human-centered design goes further than conducting usability tests and solving tactical issues. It means taking a step back and considering the broader implications of your work. It means thinking carefully about what human dignity means and how your work impacts the people that come into contact with it. Much of the work we design shapes the structures and systems that provide a framework for our culture. Let's work to ensure it's a culture that upholds the dignity of all the people in it.