(This was originally published on Playwell, Alan's personal blog.)

Elisabeth Hendrickson has recently opened a new test-and-development training facility in Pleasanton CA called Agilistry. It’s bright and airy, well-lit and well-stocked, and it feels like home the minute you walk in. In order to publicize her new facility, she very generously hosted a week-long intensive learning exercise.

She invited eleven different people with widely varied skill sets, backgrounds, and interests. She challenged them to build a website in five days using the best practices of interaction design, agile programming, and test-driven-development. We christened it “AgileUpToHere” (#au2h) and it exceeded everyone’s expectations (you can see our results here).

Since it was my 15-year-old homophone web site that was being rebuilt, I nominally played the role of product owner, but I was an observer, an instigator, a goad, and a participant. It’s hard to remember when I had so much fun or learned so much. If you want to learn to be great, I strongly recommend Elisabeth and Agilistry.

Things I learned:

1. After 25 years, it’s time to lose the Windows computer and get a Mac.


2. Good agile developers are self confident; confident enough to trust interaction designers to do interaction design without distrustful oversight.


3. There are lots of programmers who understand that relational databases are not the only approach to solving problems.


4. It is time to build software.


5. Test-driven-development isn’t fully understood. In fact, software testing isn’t fully understood.


6. When even the leanest developer in the room sees really high quality BDUF (big design up front) for the first time, they get all woo-woo and want some for themselves.


7. Getting good software built demands the contributions of many different personalities, competencies, and roles, most of which are new and as-yet ill-defined.


8. Two programmers pairing can create more and better code in less time than one programmer can (I already knew this, but it’s always good to see it in action).


9. Even this jaded old fart can still get excited about changing the world.


10. There are many undiscovered and unfilled product niches on the Web, and one of them is “quality”.


11. People want a leader with a vision.


12. Elisabeth Hendrickson (@testobsessed) is a magical woman. To paraphrase Tom Robbins, “she’s been around the world eight times and met everybody twice.” Like a great chef or symphony conductor, Elisabeth knows how to combine the unexpected to create the sublime. She brought together a dozen people from all over the country, each with different skills, background, desires, and expectations, and then she blended them together into a cohesive, happy, effective team.


13. The pre-written code I arrived with was called “legacy” with a grimace, and was quarantined until discarded. Moral: Non-TDD (test-driven development) code is properly regarded like a ticking time bomb.


14. For interaction design, you can’t have too many white boards, made from porcelain-coated steel, firmly mounted to the wall. For agile development, that isn’t such a big deal.


15. Story-mapping is a major component of the bridge between interaction design and agile development.


16. Story-tracking software isn’t quite there yet.

At the recent Startup Lessons Learned conference in San Francisco I learned a new buzzword for a very old concept. When a startup company discards Plan A and moves on to Plan B, it is called a “pivot.”

Pivoting has some nuanced meaning that differentiates it from simply changing directions. Pivoting is a seek-the-light strategy and it is not seen as fixing a problem. What you were doing might have been good, but what you pivot to do will be better. In fact, a startup that doesn’t pivot can be suspected of rigidity.

When I started my first company way back in 1975, I did contract programming. That lasted less than six months when I pivoted to building turnkey accounting systems. Before I had delivered my first (and only) turnkey accounting system, I had pivoted to just selling the software without the system. Each new business model was better than the last.

When a mature company changes its business model, there are costs associated with it, not the least of which is the dislocation to its people, who were probably very comfortable doing what they used to do. In a small, young startup, the costs are insignificant, and there are few if any extra, comfortable people to be dislocated.

Some people have expressed their doubts about the methods espoused at the SLL conference, and some are surprised to find me enthusiastic about them. But it’s important to understand that it isn’t a one-size-fits-all world. In a four-person web startup it isn’t unreasonable to pivot once a month. In a four-thousand-person mid-size manufacturing company it is insane. Just because methods work, it doesn’t mean that they work everywhere.

Overall, the SLL conference wasn’t so much about entrepreneuring as it was a celebration of today’s incredible web-based entrepreneuring environment. The barriers to entry today are so low that they approach zero.

When the cost to play the startup game is next to nothing, the cost of making mistakes is tiny, too, as is the cost to pivot. Therefore, there is little pressure to be correct or even to have a good idea. You can just keep having and trying ideas at little or no cost, and eventually one of them will be good enough for you to build a business. You can pivot your way to success instead of tediously crafting your way there.

The interesting point to ponder is whether this current web-based startup environment will be around more than a couple of years. Is it a brief anomaly, or is it the new business-as-usual? And should you be pivoting towards it? Should I?

The biggest problem in software today is that programmers and designers simply don’t work well together. They certainly want to, but each craft sees the problem from their own point of view and, with the best intentions, each tries imposing their methods on the other group. But even agile developer’s sharpest tools aren’t going to work well for designers, and likewise, even the designer’s sharpest tools aren’t going to help programmers.

The solution will be to find some common ground where each craft is open to the best contributions of the other, without either side being forced to sacrifice their inherent strengths.

I believe that the solution, like most big things, will be relatively simple in concept, yet getting there from here won’t be easy.

Today, most of the pathologies of both designers and programmers can be traced to their mutual lack of experience working together. Most programmers will tell you their biggest problem is coping with rapidly changing requirements. Most designers will tell you their biggest problem is unresponsive programmers.

In the modern, agile world, programmers defend themselves against changing requirements by showing customers the program as often as possible, and by being able to make rapid changes to suit the customers expressed needs.

Interaction designers defend themselves against uncooperative programmers by doing ever more detailed design and documenting it with greater accuracy, detail, and precision.

But modern, agile programmers can work so flexibly that they don’t need all of that detailed and precisely written design. If designers could just blend into the development team, they could communicate their design directly without the overhead of documentation. They could provide a kind of just-in-time design service to the programmers.

On the other hand, interaction designers can master the driving principles of even the most complex domain so that programmers don’t need to make all of those changes. With a comparatively brief and inexpensive field study, designers can vanquish the changing requirements problem almost completely.

Ironically, the common ground for agile developers and interaction designers is one where the major problem faced by each craft separately is largely solved by the simple presence of the other craft, working collaboratively at a peer level.

That’s really good news for cost-conscious business people (now that’s redundant). Having designers and developers collaborate is very economical. Most of the cost of interaction design is incurred in the documentation and communication of that design. Similarly, most of the cost of software development is incurred in traversing blind alleys trying to elicit useful guidance from the stakeholders. Effective collaboration simultaneously discards the need for the two most expensive parts of product development, while driving quality—and user desirability—through the roof.

Dave Hussman, one of the leaders of the post-agile movement, recently hosted a one-day conference on the topic of “Redesigning Agility”, and invited me to give a plenary talk. The focus of the conference and my talk were how to integrate agile development with interaction design. I was very pleased with how things went.

Here you will find the complete text of my talk, entitled “An Insurgency of Quality”, along with all of the slides I showed. I made a few ad libs, but mostly stayed with the script in order to assure that my message not be misunderstood.

The conference, called “Code Freeze” (due to it being January in Dave’s home town of Minneapolis), was sold out and the audience was razor sharp. The attendees were developers; that is, mostly programmers, but with lots of designers, coaches, testers, and managers, and not a few who wore several of those hats.

This talk is a complement to one with the same title I delivered at the IxDA's Interaction08. That one was directed at designers; this one is for developers.

Lots of ivory tower software experts cheerfully follow their own muse, but in the world of business, the dreams of money-makers are usually in conflict with the dreams of geeks.

In the business world, software developers have always been the whipping boy. In commerce, the delivered-software never matched the envisioned-software, and the technologists got the blame. Executives have always been unhappy about their inability to effectively direct and exploit software development. The only tool that seemed to get results for managers was to keep programmers on a ridiculously short leash by allocating resources in tiny increments. The results weren’t good, but they tended to prevent colossal disasters, which was, apparently, good enough for business.

Over many years, in self defense, programmers increasingly hunkered down to protect themselves. They aggressively lowered the expectations of their managers. They tried to commit to the least possible performance to avoid blame. All they really accomplished was to avoid good performance.

An old friend and former client — let's call him "Paul L." — sent me an email question the other day, asking “Is incremental design the wave of the future, or just a flash in the pan?”

When I finished my response, I thought that it might be of interest to a wider audience. Here is the exchange.

Alan,

My wife teaches computer science at Menlo College. She has been teaching software engineering based upon the traditional cycle of specification, design, development, testing.

I have seen some Research Channel TV shows that talk about Incremental design. Google and Inuit are two companies that seem to be having some success with ID.

My wife and I have been talking about ID as compared to the traditional methods. During the discussion I thought about my friend, the Design Guru. I am wondering if you have any thoughts on ID. Is it the wave of the future or just another flash in the pan?

Thanks,
Paul

Paul,

As agile methods take over the programming world (and they will), EVERYONE else will adjust accordingly. The old paradigm of everyone hunkering down and protecting their turf from everyone else is what gave rise to the "traditional cycle" (which is, by the way, uniquely ill-suited to software construction and design).

The new (agile) paradigm isn't at all defined yet, but it characteristically includes a) Generation Y programmers; b) a refreshing belief in the potential for change; c) the understanding that satisfying human users requires special efforts and probably special skills; d) a belief that software should be built in continuous increments; e) a corresponding belief that everything else in the world relating to software would benefit from such continuous increments; f) that building software is a team endeavor; and g) that nobody has solved these problems before.

There’s a meme floating around the interWeb called “Inbox Zero, the gist of which is that we should not be slaves to our email. That’s a fabulous sentiment and I agree wholeheartedly.

Merlin Mann, the creator of Inbox Zero, has some truly excellent advice on how to think about your email, your inbox, and yourself. In particular, not feeling guilty about deleting messages or sending terse, one-line-replies are golden rules. Not to put too fine a point on this, but I agree without reservation with the principles and practices of Inbox Zero.

Yes, and.

I believe that Inbox Zero is a human operational method for dealing with fundamental shortcomings in the software we are forced to use. The very fact that we have an “Inbox problem” is prima facie evidence that the software bringing our email to us isn’t really designed with our goals in mind.

Behavioral economist, Dan Ariely’s delightful first book, Predictably Irrational, heaps yet one more shovel of dirt onto the fresh but deep grave of traditional, rationalist assumptions about human behavior. The book is a simple, personal, easy-to-read account of Ariely’s research conducted over the past 15 or so years. This research was conducted at his various host universities; all of them paragons of ivy-covered scientific rigor, including MIT, Stanford, The University of Virginia, and The University of California at Berkeley.

The clear and inevitable conclusion of his dozens of research papers summarized in this book is simple: humans don’t make rational decisions. What’s more, the irrationality of their choices isn’t random, but can be predicted and measured. While many of the experiments deal with choices regarding cash, several of them cleverly divorce themselves from money to clearly demonstrate that the goofy human behavior is human-related, not cash-related.

He identifies several predictable forces that act upon humans during decision making, causing them to make irrational choices. These include the distorting effect of similar, but slightly inferior, products offered for sale; the distorting effect of simply thinking about numbers; the distorting effect of items offered for free; the distorting effect of sexual arousal; social norms, ownership, procrastination, self-control, clinging to options, expectations, and being observed.

Bob Martin’s rousing keynote speech at the Agile08 conference in Toronto entitled “Quintessence” proposed a small but significant addition to the Agile Manifesto, a seminal document in the programming world. Uncle Bob, as he is affectionately called, proposed adding the assertion that we value “Craftsmanship over crap” to the manifesto. The idea is excellent, and the wording bold, but it isn’t quite a complete sentiment, and Uncle Bob addressed this issue in his blog.

Shortly after delivering his speech, Uncle Bob stated in his blog,

The problem with my proposal is that it is not a balanced value statement. In the other four statements we value the second item. We just value the first item more. But in my proposed addition, we simply don’t value crap at all.

He goes on to propose rewording it as “Craftsmanship over Execution,” but admits that it still doesn’t capture his meaning precisely. He then asks the blogosphere for help. My response follows.

By day, Brad Brooks is a technology executive in Vancouver, BC. By night, he is a popular local talk radio host. Brad recently read my book, The Inmates are Running the Asylum and became a convert to the concepts I wrote about a decade ago.

He quickly asked to interview me on his show. Brad clearly sees the problem and its solution, and the interview neatly recaps the basic ideas in the book. The sad thing is that so little has changed. It all just means that we have to continually beat the drum for design otherwise we will drown in hard-to-use high-tech products.

You can listen to the interview on the Brad Brooks Show Web site.

Some attendees at the recent Agile08 conference were put off when it appeared that I was reading my speech rather than delivering it offhand. (If you're interested, you can find my slides and speakers notes here.)

It’s true; I was reading my speech.

When I speak to groups of interaction designers or business people I often address them extemporaneously. It’s a style I enjoy very much and feel that I can do well.

However, the Agile08 audience demanded special treatment. Not only was it large, but it consisted primarily of programmers, agile coaches, and product managers. These professionals are bright, knowledgeable, critical, and opinionated. They do not suffer fools lightly. I was coming to them as something of an outsider; not having programmed for a living for years, and never having programmed in a canonically agile shop.

I was asked by the leadership of the Agile 2008 Conference to give the closing keynote address at their annual conference in Toronto. The audience at Agile08 consisted of about 1500 programmers, engineers, product managers, and others involved in the creation and deployment of software primarily using Agile methods.

My belief in the value of detailed written design has often led enthusiasts of Agile to assume that I am an adherent of the obsolete, and justly maligned, waterfall method of software construction. I was pleased to have this opportunity to state my position with clarity and precision, not to mention making the case for effective collaboration between interaction designers and Agile programmers.

Here are the slides and accompanying speaker's notes for my talk. Some in the audience noticed that I was reading from my notes during the presentation. If you're interested in why I chose to do this, read this Journal entry.

If you would like to discuss this presentation with me, either post a comment to the Cooper Journal blog or email me directly at alan@cooper.com.

Is interaction design done by consultants or employees?

When Cooper was launched as an interaction design consulting firm in 1992, the answer to this question wasn't at all clear. However, as the 90s drew to a close, I confidently predicted that the bulk of the interaction design done in the world would be done by consultants. I based this conclusion on the proliferation and success of interaction design consulting firms. I assumed that the industry would follow the model of building architecture, where major design projects are typically performed by outside consultants. Architects on corporate staff would act primarily as liaison and project management. And for the first few years of the 21st century my prediction appeared correct. Today, I wonder if I called it wrong.

More and more I see corporations both large and small with their own in-house interaction design staffers. In fact, in a broad sense, my company competes with our own clients for qualified designers. There are still many successful interaction design consulting firms, but I see an ever increasing number of design projects handled completely by internal design talent, and successfully at that.

This, of course, brings up the thoughtful question of "Whither interaction design consulting firms?" What will their role be in the next decade? Will the pendulum swing the other way, and clients find that it is less expensive to hire designers on a project-only basis instead of keeping them on staff full time? Or will the consultants find themselves working only on fringe projects that are too large, too small, too complex, or too unique?

I don't yet know the answer to these questions, but I'm leaning towards the idea of an ever-more specialized role for interaction design consultancies. What do you think?

Software construction is slow, costly, and unpredictable.

Slow is a big problem because markets and technologies are always in rapid flux and our ability to maintain pace is critical to our competitiveness.

Costly haunts business people because every precious dollar spent on software design and construction is typically a dollar that is difficult—if not impossible—to directly attribute to a measurable benefit.

Unpredictable is by far the nastiest of these three horsemen of the software apocalypse. Unpredictable means 1) you don't know what you are going to get; and 2) you won't get what you want. In badness, slow and costly pale in comparison to unpredictable. While well-tempered business people are loath to part with either time or money unnecessarily, exchanging time and money for an asset that generates an offsetting future flow of revenue is the essence of business, and "slow" and "costly" are relative terms. If something costs millions and takes years it can still be considered excellent business if the return is tens of millions annually over dozens of years. However, exchanging time and money for something that doesn't generate an appropriate flow of revenue is bad. Very bad.

The industrial age is over. Manufacturing, the primary economic driver of the past 175 years, no longer dominates. While manufacturing is bigger than ever, it has lost its leadership to digital technology, and software now dominates our economy. We have moved from atoms to bits. We are now in the postindustrial age.

More and more products have software in them. My stove has a microchip in it to manage the lights, fan, and oven temperature. When the deliveryman has me sign for a package, it's on a computer, not a pad of paper. When I shop for a car, I am really shopping for a navigation system.

More and more businesses are utterly dependent on software, and not just the obvious ones like Amazon.com and Microsoft. Thousands of companies of all sizes that provide products and services across the spectrum of commerce use software in every facet of their operations, management, planning, and sales. The back-office systems that run big companies are all software systems. Hiring and human resource management, investment and arbitrage, purchasing and supply chain management, point-of-sale, operations, and decision support are all pure software systems these days. And the Web dominates all sales and marketing. Live humans are no longer the front line of businesses. Software plays that role instead. Vendors, customers, colleagues, and employees all communicate with companies via software or software-mediated paths.

In my recent travels I have noticed a growing malaise in the community of programmers. Sadly, it is the best and most experienced of them who are afflicted the worst. They reflect cynicism and ennui about their efforts because they know that their skills are being wasted. They may not know exactly how they are misapplied, but they cannot overlook the evidence. Many of the best programmers have actually stopped programming because they find the work frustrating. They have retreated into training, evangelism, writing, and consulting because it doesn't feel so wasteful and counterproductive. This is a tragic and entirely avoidable loss. (The open-source movement is arguably a haven for these frustrated programmers—a place where they can write code according to their own standards and be judged solely by their peers, without the advice or intervention of marketers or managers).

Programmers are not given sufficient time, clear enough direction, or adequate designs to enable them to succeed. These three things are the responsibility of business executives, and they fail to deliver them for preventable reasons, not because they are stupid or evil. They are simply not armed with adequate tools for solving the complex and unique problems that confront them in the information age. Now here I am sounding like I'm slamming people again, only this time businesspeople are in my sights instead of programmers. Once again, to solve the problem one must deconstruct it. I'm questing after solutions, not scapegoats.

The Inmates Are Running the Asylum, published in 1998, introduced the use of personas as a practical interaction design tool. Based on the single-chapter discussion in that book, personas rapidly gained popularity in the software industry due to their unusual power and effectiveness. Had personas been developed in the laboratory, the full story of how they came to be would have been published long ago, but since their use developed over many years in both my practice as a software inventor and architectural consultant and the consulting work of Cooper designers, that is not the case. Since Inmates was published, many people have asked for the history of Cooper personas, and here it is.

In their book, Fire in the Valley, authors Paul Freiberger and Mike Swaine, credit me with writing the “first serious business software for microcomputers” as far back as 1975. Like so much software of the time, it was terribly hard to use, and its real power was in demonstrating that making software easy to use was harder than everyone thought. Despite my commitment to making software more user-friendly, it wasn’t until 1983 and about 15 major business and personal applications later that I began to develop a more effective approach.

When all you have is a hammer, everything looks like a nail. If you have never seen a wrench or a screwdriver you will have a hard time seeing what you need, even once you discover that your hammer does not work very well on bolts or screws. This makes it hard to break away from tools that do not serve you. Under pressure, companies tend to fall back upon what they know, so they often end up trying to solve problems with the same tools that got them into trouble in the first place. When this tactic threatens to choke an organization, we call it "breathing your own exhaust."

Right now, many companies see an opportunity to approach product creation from a fresh perspective. With the frenzied dot-com "business model" no longer a distraction, and the recession apparently easing, these companies are looking for ways to benefit from their painful experiences and create a better crop of products and services. They want to nurture customer loyalty by building products that please their customers, rather than following fads or stacking up long lists of features that no one really wants. Everyone knows pleasing customers is the right thing to do, but how do you really do it?

The artless Websites created during the Web's infancy were of necessity built only with simple HTML tags, and were forced to divide up their functionality and content into a maze (a web?) of separate pages. This made a navigation scheme an unavoidable component of any Website design, and of course, a clear, visually arresting navigation scheme was better than an obscure or hidden one. But many Web designers have incorrectly deduced from this that users want navigation schemes. Actually, they'd be happy if there were no navigation at all.

Computers and their software participate in every aspect of the precious and delicate relationship between the company and the customer. The typical customer first learns about your product from an email advertisement or a computerized mailing. He visits your website to find out more about it. He buys it from your online store, and you ship it to him by FedEx, where he uses software to track it on his PC. Once delivered, even if your product isn't 100% software, it very likely has some silicon intelligence inside it. When your customer can't figure out how to work it, he calls your company on the telephone (which is itself a computer). The first thing he hears is the stilted and artificial voice of your automated call distribution system, instructing him "for technical assistance, press one now." Finally, the software puts him in contact with a real human, only for him to find that this person-trained by software-is merely echoing instructions from a problem report database software program running on his computer!

Even though "wireless" is the hot buzzword on the lips of every high-technologist, the effects of the technology hold far more interest than does the technology itself.

Wireless freedom is intriguing: It isn't hard to imagine a world of perpetually perambulating people with cell phones clamped to their ears and styli firmly gripped in their fingers doing at the cinema or the next table over at Il Fornaio what they could formerly do only at their desks.

But this flexibility to work where you want is just the first order of change wrought by these new tools. Far more interesting are the second-order effects - those unintended consequences of a new technology which often have a more powerful impact on society than the more obvious first-order changes.

High-tech companies are in a hurry—as well they should be—but many hurt themselves by trying to move products out the door too quickly. I often hear executives repeat homilies like "Ship early, ship often," and "Launch and learn." They assume that there is no penalty for simply slapping something together, shipping it, and then upgrading their product or site in a rapid iteration cycle. Unfortunately, there is a big, hidden cost associated with this tactic.

Rapid development environments like the World Wide Web have promoted the idea of simply iterating many versions of a product or service until something works. Arguably, the Web is in its nascent stage and companies are still experimenting to see what works and what doesn't, yet this should not be an excuse for iteration without planning, nor should "speed to market."

Which is harder to change: a program with 1000 lines of code or a 1000 square foot slab of concrete? The concrete is ten inches thick and has steel reinforcing rods criss-crossing within it. Every cubic foot of it weighs almost 100 pounds. The software has almost no physical existence at all. It weighs nothing. It consumes no space. A few microamps and those bits flip from zero to one without a second glance. The answer to my question seems a simple one, doesn't it?

Which is the best medium for designing software: Visual Basic or a sharp pencil and a couple of sheets of paper? Visual Basic is a powerful, flexible integrated development environment. It is on its way to becoming the most widely used language ever. It has won every industry award there is. Paper is not interactive. Paper offers no palette of pre-made controls. It just lays there and you have to do all of the work. The answer to my question seems a simple one, doesn't it?