Software construction is slow, costly, and unpredictable.
Slow is a big problem because markets and technologies are always in rapid flux and our ability to maintain pace is critical to our competitiveness.
Costly haunts business people because every precious dollar spent on software design and construction is typically a dollar that is difficult—if not impossible—to directly attribute to a measurable benefit.
Unpredictable is by far the nastiest of these three horsemen of the software apocalypse. Unpredictable means 1) you don't know what you are going to get; and 2) you won't get what you want. In badness, slow and costly pale in comparison to unpredictable. While well-tempered business people are loath to part with either time or money unnecessarily, exchanging time and money for an asset that generates an offsetting future flow of revenue is the essence of business, and "slow" and "costly" are relative terms. If something costs millions and takes years it can still be considered excellent business if the return is tens of millions annually over dozens of years. However, exchanging time and money for something that doesn't generate an appropriate flow of revenue is bad. Very bad.
The key, it seems, is vanquishing unpredictability, which means determining in advance what the right product design is, determining the resources necessary to build it, and doing so. As the airline pilots say, "Plan your flight, and fly your plan."
Now as many of you know, I'm a big proponent of interaction design. For years, I've been describing how good interaction design done in advance of programming can help to vanquish unpredictable—not to mention slow and costly—better than any other tool. Interaction design determines who the users are, what goals they are trying to achieve, and then creates a tangible, visible plan for how the product will look and behave to get those users to achieve those goals. Over the last few years, more and more companies have adopted interaction design methods. They have flight plans. Why, then, are so many organizations still repeatedly visited by those three ugly horsemen of software apocalypse?
As in all things related to software the answer is complex, obscure, counter-intuitive and shaded with nuance. Throwing money or time at the problem doesn't work. Interaction design is a big part of the solution, but without making structural changes to the programming organizations, interaction design alone can't solve the problem. Over the last 15 years my company, Cooper, has provided hundreds of detailed interaction design solutions to a wide variety of companies. How successful these companies are at building those solutions doesn't appear to have anything to do with the complexity of the design, and it doesn't necessarily depend on the skill of the programmers. I have concluded that software builders struggle to integrate design into their process for two basic causes: 1) programmers have never learned to follow a design, and 2) their day-to-day responsibilities forbid them from doing so. I also believe that these reasons can be understood and vanquished.
Of course you can see how both of these problems would stem from the same root: if a programmer has never learned to follow a written design, then he would structure his daily work to do without. He would attempt to do the necessary design himself, concurrent with the construction effort. And that is exactly what programmers at all levels and in all sub-disciplines of computer programming do: they design code at the same time as they build it. If we could untangle these two parts of the programming job, we could begin to defeat the apocalyptic horsemen.
Software professionals have a long, well-established history of designing as they build. In every university, on every project and every thesis, the two tasks are performed simultaneously by the same person. Certainly there is some rudimentary sketching of class boxes or data flow before coding begins, and this is often thought of as "design" but contrast this to the level of detailed design work that goes into the construction of something much simpler than software, say, a modern office building, where thousands of pages of drawings and notes are completed, checked, distributed, bid-upon, and otherwise vetted long before the first shovelful of dirt is turned.
If we can find a way to motivate programmers to integrate planning into their normal engineering tasks we can make significant progress in the battle against the three nasty horsemen. Planning, not plans, is the key to visibility. We already have clear and detailed plans, but with engineers who can and want to follow them, the software construction process becomes clear even to outsiders (like managers), and the whole process becomes more predictable. With this predictability comes a significant reduction in sunk costs and elapsed time, a better quality product with a longer lifespan, and our market opportunity is maximized.
Programming has always been viewed as just that: programming. Only two distinctions
have ever been made about the various characteristics of writing code. Programming
has been divided up by its arena of tools and practice, such as COBOL versus
VisualBasic versus Java versus XML, or systems programming, or server programming,
or user interface programming, or embedded programming. And from the instant
the first op code was manually entered into the first ENIAC by plugging in
a jumper cable, programming has been divided up by skill level. Programmers
are nothing if not competitive, and they share a universal delight in the
personal achievement of solving the innumerable complex technical problems
served up by any programming project.
Programmers rate themselves almost exclusively by these two rubrics: how
challenging is the field and toolset, and how virtuoso are their talents.
I propose that the answer to our software construction predictability problems lies not in these two rubrics (which are excellent as far as they go), but in correcting their inherent exclusion of a vital third rubric: the personal goals of the programmer.
Personal goals are our chief motivators in life, and these motivators drive our behavior in the workplace as well as at home. The personal goals of programmers vary widely, and when these goals are ignored, programmers find themselves working unproductively—and often counter-productively.
Of course each programmer is an individual, and there are as many motivating goals as there are programmers. However, I think that there exist at least two large identifiable categories of programmer-goals, and the population of software writers neatly cleaves to one or the other camp.
The first camp is composed of builders: those programmers who, like the many carpenters and masons who preceded them in history, take a sublime joy in seeing their handiwork take form in the real world where it can be—and is—used and appreciated by others. They may be hammering together packing crates or they may be painstakingly crafting Steinway pianos, but seeing their products of wood and steel assume a tangible form that gets things done in the wider world provides them with a sense of accomplishment and well-being. It achieves their goals.
The second camp is composed of designers: those programmers who, like the many visionaries who preceded them in history, take a sublime joy in seeing complex, apparently intractable problems dissolve in the face of their creative thinking. They may be arranging utensils in their kitchen drawers or they may be painstakingly calculating how to shape a girder to support a mile-long highway bridge, but seeing that the solution they imagine is the best and most efficient one possible provides them with a sense of accomplishment and well-being. It achieves their goals.
Of course, the builders share in the joy of devising clever solutions just as the designers revel in seeing their creations take real form, but if a mutually exclusive choice between the two options ever arises, each will be happy to focus on the type of work that best satisfies his goals.
Generally, in the industrial age, a person who designed industrial tools or mechanisms was called an "engineer" while the men and women who actually built the design were referred to by the name of their trade specialty: carpenter, stonemason, thatcher, pipe-fitter, welder.
In the software world, anyone who writes code is typically referred to as an "engineer" and this word has lost much of its precision and all of its usefulness. However, it is still the name of the profession by which many thousands make their livelihoods, so discarding it or denigrating it is counter-productive. Instead, I will prepend descriptive words to "engineer" to differentiate between those programmers who like to build and those who like to design. I call them "Production Engineers" and "Design Engineers" respectively. Production engineers are good programmers who are personally most highly motivated by seeing their work completed and used for practical purposes by other people. Design engineers are good programmers who are personally most highly motivated by assuring that their solutions are the most elegant and efficient possible.
Interaction designers' motivations are very similar to those of design engineers, but interaction designers are not programmers. Although most programmers imagine that they are also excellent interaction designers, all you have to do to dissuade them of this mistaken belief is to explain that interaction designers spend much of their time interviewing users.
The common expression "Engineers build bridges" is actually a misnomer. Engineers build mathematical models of bridges and draw little pictures of bridges on paper or inside computers. Ironworkers are the people who really build bridges. This inexact, industrial age metonym has led to much confusion in the post-industrial age, where it's all too easy to confuse software designers with software welders because they both use the same tools and raw materials for their very different work.
Software is the only medium where the construction materials are entirely the same as the design materials: source code. When a design engineer works out a complex problem for how a program will work, she uses source code. When a production engineer later uses that code as a design document and produces his own, release-quality, shippable software, he uses source code. In no other medium is this true and it is this watershed more than any other that defines the post-industrial era.
Virtually every industrial age product is first designed on paper or in some other cheap, disposable, high bandwidth, easy-to-iterate medium. In fact, it is these characteristics of the tools that allow design to exist at all. If designers of 15th century European cathedrals had to use huge blocks of limestone and marble to design with, they simply could not do it, and large or complex structures would not exist. Cathedrals were designed on paper (or parchment or vellum), using scaled-down, two dimensional images of the huge three dimensional buildings. Sometimes scaled-down models—usually made of wood—were used to better communicate the design to illiterate masons.
Yes, software can be a cheap, disposable, high bandwidth, easy-to-iterate medium. Advocates of Agile methods would have you believe that all software is so. If only it were true. Programs written so as to be cheap, disposable, and easy to change are simply not fit for commercialization. They lack the robust completeness necessary for good behavior. Any business person with the slightest experience in software management (and obviously any programmer) knows this simple truth about software and strives to make his or her code complete enough and robust enough to have value in the marketplace. But a remarkable thing happens here. When a conscientious programmer begins the laborious process of adding robustness to his code, the medium very quickly ceases to be a cheap, disposable, high bandwidth, easy-to-iterate medium. Instead it becomes expensive, cherished, obscure, and ponderous. It transitions almost instantly from a design medium to a production medium. And yet, it is still source code, indistinguishable from its design sibling without lengthy study (and if you are not a programmer, don't even bother trying).
As you might imagine, determining the difference between cheap, disposable, high bandwidth, easy-to-iterate "design code" and expensive, cherished, obscure, and ponderous "production code" is critical to the very existence of successful software. I contend that few people in the software business pay attention to this difference, and this is the primary cause of the familiar litany of software toxicity: programs are expensive to write and maintain, don't please their users, inexplicably fail in known marketplaces, cause dissension and frustration in the programmer ranks, demand expensive technical support, intractably resist business redirection, and are closed off to corrective design efforts.
The solution is elegantly simple in concept, if arduously difficult in implementation.
First, as stated earlier, we need to segregate engineers who like to design
software from engineers who like to build software. Second, we need to give
those design engineers a cheap, disposable, high bandwidth, easy-to-iterate
design medium that happens to be source code. Third, we need to give those production
engineers sufficient design support (both technical and interaction) so that
when they begin crafting their masterpiece in the expensive, cherished, obscure,
and ponderous medium that is production code, they never have to iterate, back-track,
redesign, wonder, or guess at what the correct decision might be as they laboriously
erect the huge code ziggurat of which a product of shippable quality consists.
Of course, when you give a design engineer a cheap, disposable, high bandwidth,
easy-to-iterate technical medium, you must also provide her with the big picture
guidance needed to apply that design medium in the most effective way. Actually,
she will demand this from you. Before, when the design engineer was tasked
with concurrent design and production coding, she was forced to dive in and
solve problems big, small and in-between all by herself with little help from
anyone. Always under a tight schedule, she would hunker down and refuse admittance
from any others. In her situation, merely listening to someone's suggestions
could derail days or weeks of effort. The secret was to pick a direction and
As soon as the burden of production programming is removed, the design engineer
can code up practically anything in a day or two. The problem then becomes,
"What do I code?" Our intrepid design engineer will now come asking
that question of the interaction designers (or business and marketing folk).
This is what Extreme programming is all about: Getting the chunks of code
down to pieces small enough to be useful as design tools.
It is vital that the design engineer not try to solve interaction design
issues. Not only do design engineers lack the aptitude, training, tools, and
time to do so, but they are already quite busy enough doing their own job.
The only way to prevent them from attempting to tackle interaction design
questions is to assure that interaction designers are present and available
at all times. The great bulk of interaction design can be done in advance
of design engineering, but never all of it. There are always many, many human
interaction issues that arise during engineering that must be solved by those
competent to do so.
While interaction designers design program behavior that will allow users
to achieve their goals, they are constantly considering technology. They need
to know if the software can accomplish presentation style A; can the server
deliver performance level B; is the benefit worth the cost of coding up widget
C? Interaction designers need the help of design engineers to answer such
questions. Freed from the burden of production code, a skilled design engineer
can answer such questions typically in a matter of hours. The answers are
determined by writing snippets of lean code bereft of error checking, internationalization,
multiple browser support, print capability, security measures and the thousand
other elements that turn design code into production code. In other words:
cheap, disposable, high bandwidth, and easy-to-iterate. "Cheap"
is really a benefit to the business, while "disposable" is mostly
a state-of-mind, but a state-of-mind that makes "easy-to-iterate"
a reality. If the programmer knows that the code won't go out the door and
won't be subjected to the quality control necessary for shippable, production
code, he knows that he can write it exceedingly spare, lacking any frill or
breadth. Just a few lines here and a few lines there are all it typically
takes. Short code is easy to discard and replace, which empowers the design
engineer to iterate. The essence of design (beyond the epiphanic nature of
design vision, which you either have or you don't have) is iteration. You've
got to try it out this way and that way and the other way to assure yourself
that you've got it right. If, as in the old way, your "try it out"
medium is ponderous and opaque, design iteration simply won't happen. Worse
yet, when the programmer, not having had the luxury of finding the correct
path, sets off down the incorrect path and eventually perceives his error,
will be forced to bend the code towards the light. It is this "bending"
that causes much of the expense of production coding. It also is the prime
cause of recalcitrant software that cannot be grown or adapted to a changing
marketplace over its lifetime.
It is vital that the production programmer not have to answer design engineering
questions during production coding. The only way to avoid this is to assure
that all engineering design questions have been answered before production
begins. Interaction designers cannot perform this feat, nor can product managers,
marketers, CEOs, or program managers. Only design engineers can do it, and
only those engineers with a personal proclivity for determining the correct
and most efficient answer can apply the discipline necessary for writing large
quantities of code that they know must be 1) correct, and 2) thrown out.
Throwing out disposable code is very good. It's like plowing under a cover crop to enrich the soil while fighting weeds. Conversely, throwing out production code is very bad. It's like throwing out brand new automobiles simply because their gas tanks are empty.
Most business executives believe that writing production code is a good thing.
They assume that getting to production coding early is better than getting
to it later. This is not correct. Writing production code is extremely expensive
and extremely permanent. Once you've written it, it tends to stay written.
Any changes you might make to production code are 1) harmful to the conceptual
integrity of the code, and 2) distracting and annoying to the programmers.
Annoying your programmers is more self-destructive to a company than is annoying
the Board of Directors. The annoyed programmers hold your company's operations,
products, and morale in the palms of their hands.
All of this is to say that if you are engaged in a productive process of
learning what your product is supposed to be doing and how it is supposed
to be doing it, then the longer you can delay the beginning of production
code the better. The wise executive should spare no effort to ask the most
detailed questions and demand the most detailed answers, and demand that these
questions and answers be made on cheap, disposable, high bandwidth, easy-to-iterate
media, such as paper and disposable code. The more paper design and snippets
of illustrative code you can provide to the production engineers the better.
Please note that I am not advocating indecision. Endless wrangling and opinion
battles get nowhere. The time you invest in delaying the start of production
programming must be time you invest in performing productive design work,
both interaction and engineering.
I have deliberately avoided using the highly tainted term "prototype."
This word has no useful value in the effort to change and improve our software
construction process. A "prototype" has come to mean any piece of
code that is not yet considered the final release, even though it has probably
already been shipped to the customer. This is why I stubbornly adhere to the
clumsy phrase "disposable code." The design engineer must know that
this code will never, ever leave the doors of the shop. The design engineer
must know that his cheap, disposable, high bandwidth, easy-to-iterate code
will be used solely for two purposes: 1) to determine the best way to build
the product, and 2) to communicate this method to the production programming
The design engineer's role is vital. Any sufficiently skilled software engineer
can create a product right, but only the design engineer allows you to create
the right product. The irony is that the design engineer is not strictly responsible
for determining what the right product is. Rather, that's the job for the
interaction designer, who interprets the goals of marketing and business stakeholders
along with those of the users. But when design and production engineers' roles
are conflated there is simply no effective point for the work of the interaction
designer to enter into the process. When difficult, technical design questions
are being answered at the same time that big slabs of software concrete are
being poured, the software construction process becomes a zero-sum game with
everyone fighting for scarce resources.
By disentangling design engineering and production engineering, you create
the perfect entry point for interaction design. Design engineers—when not
compromised by production—deeply desire to create the correct technical solution,
but they are generally clueless about what the human problem is they are solving.
Interaction designers provide that guidance, and the design engineers are
very eager to hear it. The interaction designer becomes the design engineer's
best ally and most valuable resource.
As for production engineers, they must be reassured that they don't have
to begin writing any code until they know what that code must do and how it
must do it. The interaction designer provides the answer to the former, and
the design engineer provides the answer to the latter. Once again, we have
an organization whose structure impels collaboration from within, rather than
one that imposes it from without (always an ineffective way).
Interaction design is design for humans, design engineering is design for
computers, and production engineering is implementation. Recognizing these
three separate divisions and organizing the work accordingly is something
I call "The Triad." While it cannot exist without interaction designers,
it depends utterly on teasing apart the two kinds of engineering which today,
in most organizations, are almost inextricably linked. It will take some heroic
efforts to segregate them.
Once we solve the predictability problem, we find that the forces work to
our benefit in other ways. The more visibility we have into the future merit
of our software construction efforts today, the more time and money we will
be willing to invest in them. If we are confident that the product will achieve
all of its business goals, there is no need to tighten our belts unnecessarily.
In fact, if we could map individual efforts to their expected returns, we
would aggressively fund those efforts with the brightest future. Instead of
imposing draconian, across-the-board spending limits, we'd be able to increase
spending on the portions of the product that will drive up its desirability
in the marketplace.
The more predictable any construction process becomes, the less time and money
it requires simply because you traverse fewer blind alleys. In my 30 years experience
designing and building software, I have learned that the most expensive software
to build is the wrong software. You might think that I'm describing the situation
where you build the wrong software, have to throw it away, and then build the
correct software. Actually, that's not a bad thing to do at all, as long as it's
disposable code. The most expensive wrong software you can build is the wrong
software that is shipped to customers because you promised it to them or because
your senior executives or financial backers insist—erroneously—that
shipping the wrong software that you do have is better than waiting
until you have the correct software.
When you divide your programming team into two squads based on their personal
goals you will find that both groups get a lot happier. You will also find
that happier programmers are more individually productive and more productively
communicative. The design engineers will gladly answer tough technical problems
by coding a half-dozen different test modules when they are not simultaneously
burdened by the needs to convert that code into something that ships to users.
The production programmers will quickly learn to bring design questions to
the design engineers to solve collaboratively rather than hunkering down and
inventing a quick solution that may or may not be correct.
With a construction team properly organized as a Triad, design engineers will be doing what they love to do: rapidly iterating to solve complex technical problems. Production engineers will be doing what they love: building huge, complex systems that make end users happy. Interaction designers will be doing what they love: determining who the users are, what their goals are, and what the best possible software behavior would be to help those users achieve their goals. Product managers will be doing what they love: supporting and facilitating the team towards their common goal of a successful product. And senior executives will be much happier now that they have visibility into what their software construction teams are up to. By getting the right people doing the right jobs in the right sequence, the whole process becomes more predictable, faster, and less costly.
Currently there is a pitched battle raging in the programmer world between
conventional engineering methods and Agile methods. What neither side sees
is a path of reconciliation; where Agile and conventional methods can effectively
coexist. Providentially, the Triad reconciles them very well. The lean, iterative,
problem-solving work of the software design engineer is the archetype of Agile
programming. The purposeful, methodical construction work of the production
engineer is the quintessence of conventional software engineering, particularly
the type espoused by disciples of Grady Booch's Rational Unified Process,
or RUP. Both methods are correct, but only when used at the correct time and
with the correct medium.
The three horsemen of the software apocalypse—slow, costly, and unpredictable—are largely just a single problem: lack of visibility into the process of software design and construction. If we knew what we were doing, we'd be more willing to pay full price for it, and because the returns on successful software are so generous, full price isn't too much to pay. By creating the role of design engineer we empower both interaction designers and production engineers to fulfill their jobs, and the whole process opens like a flower to the examining eyes of the business folks.