Strategies for early-stage design: Observations of a design guinea pig

Where do you start when you're approaching a complex software design problem? If you work on a large development team, you know that software engineers and UX designers will often approach the same design problem from radically different perspectives. The term "software design" itself can mean very different things to software architects, system programmers, and user experience designers. Software engineers typically focus on the architectural patterns and programmatic algorithms required to get the system working, while UX designers often start from the goals and needs of the users.

In the spring of 2009, I participated in a research study that looked at the ways in which professional software designers approach complex design problems. The research study, sponsored by the National Science Foundation, was led by researchers from the Department of Infomatics at the University of California, Irvine. The researchers traveled to multiple software companies, trying to better understand how professional software designers collaborate on complex problems. At each company, they asked to observe two software designers in a design session. At my company, AmberPoint, where I worked at the time as an interaction designer, I was paired with my colleague Ania Dilmaghani, the programming lead of the UI development team. In a conference room with a whiteboard, the researchers set up a video camera, and handed us a design prompt describing the requirements for a traffic control simulation system for undergraduate civil engineering students. We were allotted two hours to design both the user interaction and the code structure for the system.

Jim-and-Ania-at-the-whiteboard.jpgJim Dibble and Ania Dilmaghani at the whiteboard in their research design session

As I'm sure you're thinking right now, two hours is not enough time to fully grasp the intricacies of traffic control, let alone to design the mission-critical software architecture and the user interaction model of a traffic control simulation system. As designers who approach problems from the user's perspective, Ania and I spent most of our allotted time considering the user's experience. We explored the real-world problems involved in traffic control and the tools that users would need to construct maps, set traffic light timings, seed traffic, and evaluate the results of a simulation run. We touched briefly on the way in which the simulation engine might work, but definitely did not provide sufficient detail to enable programmers to get started on the innards of the system.

SPSD-Whiteboard.jpgThe final whiteboard diagrams after the two-hour design session

Our team was selected for study

Several months after our design session, the researchers wrote to tell us they were planning a conference on studying professional software design at UC Irvine. They chose our design session as one of three to be analyzed and interpreted by invited participants. The organizers put the call out to researchers from a wide variety of disciplines, including software engineering, design studies, human-computer interaction, cognitive science, and psychology. Participants were given access to both the videos and transcripts, and were encouraged to analyze the design from any of a number of perspectives, including design strategies, design notation, interpersonal communication, decision-making strategies, comparisons to established software design methodologies, and design outcomes. Because the organizers felt that the conference participants would appreciate the ability to discuss their observations and theories with design practitioners, they invited us to attend.

In February 2010, we traveled to UC Irvine to attend the conference. Walking into a lecture hall with more than 50 researchers who had viewed our videos repeatedly, we had a small sense of what Brad Pitt and Angelina Jolie might feel like when walking into a restaurant in LA. Heads turned, and people pointed and whispered.

Researchers from multiple disciplines came from around the world, from as far away as Australia, Brazil, Japan, and Europe. Over a period of three days, we heard over 30 papers analyzing our design strategies, our communication patterns, and our final design solutions. And at the end of each day, we heard closing thoughts from luminaries in the fields of computer science, design studies, and psychology, such as Frederick Brooks (professor at UNC Chapel Hill and author of "The Mythical Man-Month"), Nigel Cross (design studies professor at The Open University, editor of Design Studies, and author of "Designerly Ways of Knowing"), and Michael Jackson (computer science professor at The Open University and author of "Software Requirements and Specifications").

SPSD-Panel.jpgThe panel of luminaries speaks at the close of the conference: (L to R) Michael Jackson (The Open University), Mary Shaw (Carnegie-Mellon University), Clayton Lewis (University of Colorado, Boulder), Nigel Cross (The Open University), Barbara Tversky (Stanford and Columbia), and Frederick Brooks (University of North Carolina, Chapel Hill)

The three design teams had taken very different tactics in approaching the design problem. Because we came from a user interface background, our design strategies focused on exploring the problem from the user's perspective: how does traffic signal timing work, and what tasks do civil engineering students need to perform to gain a better understanding of traffic signal timing? The other teams, composed of software architects, worked from the inside out: how does traffic signal timing work, and what architectural patterns can be used to implement the traffic simulation system? While one of those teams touched briefly on the user interface, the other team completed their architectural design in an hour, waving their hands to indicate that all that remained was a basic user interface design.

How our design fared

So, how did the researchers assess our designs? Well, their evaluations depended on their research perspective. Several computer science researchers evaluated the design session based on the technical skills exhibited: the ability to accurately frame the technical problem based on the explicitly stated requirements and the ability to choose an appropriate architectural pattern for the implementation. Design studies researchers tended to evaluate the design on the "soft skills" associated with design: interpersonal communication, ability to articulate the reasoning behind design choices, and the ability to examine and evaluate requirements based on the end-users' needs. Other researchers, rather than being prescriptive, examined the processes and strategies used by the design teams: how did the designers structure their design sessions, how did they explore the breadth and depth of the problem, and how does this compare to the way that the academy teaches software design students to approach problems?

Not many designers have the opportunity to hear a team of cross-disciplinary researchers examine their work. We heard much in those three days, reminding us of the importance of explicitly planning a design session, of framing the design problem, and of conversation as part of the design process. Throughout the workshop, we also passed on our observations to the researchers, letting them know that real-world design problems seldom begin from a complete problem statement or requirements document, and that architects and developers can easily jump to the system's architectural structure while overlooking the user's goals, needs, and outcomes.

SPSD-Discussion.jpgMary Shaw and Frederick Brooks discuss the conference proceedings during a break

After the conference, Ania and I discussed how differently the design session might have turned out if either one of us had been paired with a software architect. We definitely would have spent more time addressing the system's architecture, but we were unsure whether we would we have been able to frame the problem around meeting the user's needs. In engineering-led organizations, designers are seldom included in early-stage design sessions, which can lead to products that don't meet the target users' needs. Collaborating with engineers and architects in early-stage design can be incredibly difficult because we often don't share the same criteria for evaluating design decisions. While many engineers are starting to express requirements in more user-centered ways, for example through the use of agile user stories, they often have difficulty moving these beyond low-level narrative that describes the purpose of technical features.

Read more

Several publications have come out of the work of the conference, including a special issue of Design Studies (Volume 31, Issue 6, November 2010), a special issue of IEEE Software (Volume 29, Issue 1, January/February 2012), and a forthcoming book on studying professional software design. Because of our participation in the conference, the researchers encouraged us to write an article based on our experiences in software design, for the special issue of IEEE Software. Our article, Strategies for Early-Stage Collaborative Design, (Ania Dilmaghani and Jim Dibble, "Strategies for Early-Stage Collaborative Design," IEEE Software, vol. 29, no. 1, pp. 39-45, Jan./Feb. 2012, doi:10.1109/MS.2011.124), offers a set of ten ground rules for conducting collaborative design sessions. In our article, we encourage engineers and architects to develop the "soft skills" of design, and to evaluate design solutions against user needs and scenarios. These ground rules, based on our experience and our learnings from the conference, will be helpful to anyone running a collaborative design session, whether UX designers or software architects.

The experience has us thinking about early collaboration outside the research laboratory. UX designers and software architects each bring unique understandings to problem-solving. Depending on how an organization structures design work, these differing perspectives can be either complementary or conflictual. We're familiar with how it works in our business, and curious about how it works in yours. How does early-stage design work in your organization? Do your interaction designers and architects work collaboratively from the outset to understand the problem? How do you establish a feedback loop to bring understandings from each design approach back to the overall solution?


Janet McDonnell, Central Saint Martisn
Great to hear from you and that the experience of taking part in the study was quite a career developing one! My study of you and Ania was published later in Design Studies ( than the Special Issue) because I had to re-transcribe the data I used as the transcriptions from the workshop were not fine grained enough for my purposes – so I could not meet the SI deadline. It is just out in Design Studies 33 (2012) 44-63doi:10.1016/j.destud.2011.05.003. It will appear in the same form in the book. A couple of years before the NSF study I organised, with a colleague, Peter Lloyd, a similar ‘shared-dataset’ study of design meetings (architect with client (2 meetings 8 months apart) and a bunch of technical folk designing a product to exploit some novel print technology ( again 2 meetings)) – Fred Brooks mentioned this earlier study at the workshop – we, too , have a book called About Design: Analysing Design Meetings. Our study was of ‘real life’ meetings rather than the type you did where you were given a researcher-constructed brief. I think it was great in the NSF study that you and Ania were able to come along and take part – that was a great innovation on what we did in our earlier workshop, as is the invitation to you to write a piece on your experience of taking part. There is often criticism of researchers/academics not having anything to say to practitioners/for practice so it is immensely important (and we academics are immensely grateful) that practitioners like you are prepared to let us study what you actually do - it is rarer and terrifically more difficult to set up than anyone who hasn't tried to do it can imagine. So it is doubly welcome to hear - and that you have written about what the experience did for you. BTW I do know that ‘our’ architect would completely relate to your Brad Pitt moment . We invited him to the conference dinner in the British Museum, he was overwhelmed to arrive to find 50 strangers to him who ‘knew him very well indeed’ in some respects, all wanting to shake his hand and grab a photo with him. One of the wonderful later developments from that project for me has been that the building he was designing has now actually been built and a couple of weeks ago I went to see it and to be shown round by him and the client – so I had my own miraculous moment of incredulity as we drove round a bend in the drive and there was that design that we had poured over as a virtual building – only present in the thoughts and drawings of the architect and his client – suddenly for me right there in concrete, glass and stone. Best wishes, Janet McDonnell Central Saint Martins, University of the Arts London
Ania Dilmaghani
Participating in the study was a great experience. During the workshop number of researchers echoed my own observation that there is not enough opportunities for interactions and collaborations between the academics and the practitioners, thus the conference was indeed an unique opportunity for Jim and I. It was most interesting and illuminating to view our own video and to read and watch the presentation of the papers analyzing and dissecting our every move and word. As a developer I’m always chasing deliverables, deadlines and bugs, not very often do I have an opportunity to stop and analyze the multitude of influences which affect how software, and design in particular, happens. I think the study and the workshop provided a great insight into software design and an opportunity for me to reflect on my own practices and approaches. Ania Dilmaghani Oracle
Kirsten Robinson
Hi Jim! I didn't know you were at Cooper. Really enjoyed your article. We've had good results when we can get product managers (representing high-level business requirements), UX folks (representing user needs) and developers all working together on collaborative design. But it's hard to get all the right folks together in one room....
Your post captures the issue peftcerly!

Post a comment

We’re trying to advance the conversation, and we trust that you will, too. We’d rather not moderate, but we will remove any comments that are blatantly inflammatory or inappropriate. Let it fly, but keep it clean. Thanks.

Post this comment