1. INTRODUCTION
RationaleFootnote 1 research in software development is a challenging area because although there is no shortage of advocates for its value, there is also no shortage of reasons for why rationale (for design or other phases of software development) is unlikely to be captured in practice. There are numerous proposed uses for rationale: providing additional documentation, assisting new personnel in learning about the design, supporting software maintenance, and many more. There are also many barriers to its capture and use: the effort involved in capturing it, potential liability issues if decisions can be tracked, the potential for disrupting design, and so forth. Despite more than 30 years of research there still remains much uncertainty: how useful are the potential benefits and how insurmountable are the barriers?
1.1. Rationale capture and project failure
Grudin (Reference Grudin, Moran and Carroll1996) describes the danger of adding additional costs to earlier (“upstream”) phases of software development and, as part of a discussion of the benefits of upstream investment when many projects are never completed, offers this rather off-putting statement “And any project, by diverting resources to capture design rationale, may reduce its likelihood of surviving or succeeding.” This dramatic assertion provides an especially gloomy picture of the future of rationale research.
Software project failure rates are indeed high. A frequently cited source for software failure information is the CHAOS Report, generated every 2 years by the Standish Group. The 1994 report (Standish Group, 1994) states that 31.1% of projects are cancelled prior to completion and 16.2% are completed on time, within budget, and with all features implemented. The remaining projects (“challenged” projects) are completed, but with time and cost overruns and often with not all of the originally planned features implemented. The situation in 2006 is somewhat improved, with 18% having failed and 29% having succeeded (Hartmann, Reference Hartmann2006; Standish Group, 2006). Still, it is interesting to note that while only 29% can be viewed as having succeeded, the projects in the challenged category (53% in 2006) are still delivered to the customer (although with possibly reduced functionality). If the usefulness of rationale is dependent on the system being delivered, it has the potential to become useful on 82% of software projects.
1.2. Barriers to rationale
Horner and Attwood (Reference Horner, Attwood, Dutoit, McCall, Mistrík and Paech2006) investigated barriers to design rationale (DR) use and divided them into four types of limitations: cognitive, capture, retrieval, and usage. The cognitive limitations arise because of human information processing limitations that make it impossible to exhaustively capture rationale. The capture limitations involve the need to collect rationale along with its context, the difficulty of eliciting tacit knowledge, lack of incentives for capturing rationale, the cost of capture versus predicted benefits, and if the collected rationale may pose a risk to the individual (exposing a bad decision) or the corporation (possible liability issues). Retrieval limitations concerned the need to determine what the content of the rationale being retrieved should be and the method of retrieval. When rationale is captured for future use, how do you predict what information will be useful? The final limitation, usage limitations, was concerned with if the applicability of rationale reuse might be constrained by the uniqueness of design problems and that there is a deficiency of methods for assessing how effective rationale is.
Exhaustive rationale capture is certainly unlikely, and is probably also undesirable. It would make far more sense to optimize resource allocation by having some rules or heuristics for which information is most likely to be useable in the future. Another option would be to follow an “incremental formalization” approach (Shipman & McCall, Reference Shipman and McCall1997) where information is captured in an unstructured format and formalized as needed. Capture remains a key difficulty to rationale adoption. There is little data available to indicate if the costs are, indeed, greater than its benefits. This is an area where data collection would be critical. Retrieval is an area where technology can be of assistance: integration with development tools and methodology will help capture the context along with the rationale although no technology will help us see the future. The final issue, usage, has similar issues to any reuse of information or technology yet these issues have not discouraged reuse research in the software domain. Software reuse remains a critical goal of software development.
1.3. What use at what cost?
In their article “Argumentation-Based Design Rationale: What Use at What Cost” Buckingham Shum and Hammond (Reference Buckingham Shum and Hammond1994) examined a variety of argumentation-based DR approaches to determine if they were able to meet two claims: “Argumentation-based DR is useful” and “Argumentation-based DR is usable,” that is, the use and cost of DR. The first claim was examined by looking for three types of evidence: evidence that designer reasoning and deliberation were assisted by working with argumentation, evidence that argumentation recorded earlier was useful, and evidence that using the notation impeded reasoning.
For the first claim, usefulness, they were able to find some of each type of evidence, including evidence of impeding reasoning. The two examples of impeding reasoning were cases where capturing the rationale as argumentation caused designers to get sidetracked onto issues that were not important or not controversial. The examination of the second claim, usefulness, provided some evidence that the semiformal argumentation was not always easy for the designers but the overall result was that more information was needed about how designers might use this kind of an approach. The need for more empirical evidence was highlighted by this article, and the authors expressed concern that technology was being built on “a mutually constructed and self-perpetuating folklore.”
1.4. So why bother?
Why should we press on despite these obstacles? Some of the doom and gloom predictions appear somewhat speculative, such as Grudin's assumption that the effort spent in capturing rationale could be the factor delaying development just enough to cause cancellation (Grudin, Reference Grudin, Moran and Carroll1996). There are reports of unsuccessful attempts at rationale use but those results have not been published so there is no way for those who were not involved to understand what happened.
Software projects vary widely in size, scope, and process followed. Developers in companies operating at the higher levels of the capability maturity model (SEI, 1997) are used to spending time on documentation, metrics, and other types of information gathering that go beyond simply writing and debugging code: is rationale collection more arduous than those tasks? What do we need to know to move from guessing about rationale's benefits and barriers toward determining what direction is best for our research?
The remainder of this article looks at what we actually know and do not know in rationale research. Section 2 describes what techniques have been used in the past to evaluate rationale and rationale tools. Section 3 discusses why this is the right time for a renewed interest in the field. Section 4 describes the challenges we face in evaluating our research. Section 5 discusses the importance of obtaining the perspective of practitioners in the field and presents the results of a pilot survey of software practitioners to get their opinions on how, when, and by whom rationale could be used and what they perceive to be the most significant barriers. Section 6 discusses conclusions and future research.
2. EMPIRICAL EVIDENCE
What do we really know about design or other forms of rationale in use? There have been many different approaches to rationale capture and use proposed but most of them did not receive formal evaluation. Table 1 lists 56 approaches and how they were evaluated. For nearly half, no evaluation was described, and of the remaining approaches, the most common evaluation method was the case study, most of which were informal.
Table 1. Design rationale approach evaluations
There have been a couple of studies to investigate DR use that were not intended to evaluate a specific approach (although they did use specific notations). Karsenty (Reference Karsenty1996) investigated usefulness of DR by showing DR captured by a scribe observing design meetings that was formalized into the QOC notation (MacLean et al., Reference MacLean, Young, Bellotti, Moran, Moran and Carroll1996). All documents were captured on article with one problem per page. Karsenty (Reference Karsenty1996) conducted an experiment where six designers external to the project were asked to perform a series of tasks given DR documents captured earlier by a scribe who observed design meetings. The data collected from the experiment consisted of the questions asked by the designers (168 questions total) that were grouped into six categories. Of these categories, the majority fit into the category of DR Questions. Of these questions, the DR answered 41%. The unanswered questions were analyzed to determine why the DR was insufficient. In some cases, the DR did not capture all the issues discussed at the design meetings, in some cases the questions were not answered because it involved misconceptions about the designed artifact, whereas in other cases the questions were not answered because the issues asked about were not issues recorded in the rationale because they had not come up at the design meetings.
QOC was also used in the first of three studies performed by Haynes (Reference Haynes, Dutoit, McCall, Mistrík and Paech2006) to investigate the use of DR as explanation. The first study generated DR retrospectively from design meeting transcripts and other design artifacts. They found that it was difficult to capture the chain of reasoning and that the DR did not appear to be complete. Two additional studies were performed using scenarios and claims analysis (Carroll & Rosson, Reference Carroll and Rosson1992). These studies appeared to indicate that DR in the form of scenarios would be effective in assisting with technology transfer.
Tang et al. (Reference Tang, Babar, Gorton and Han2006) point out that there is a lack of empirical research studying practitioner opinions on DR, how they capture and use DR, and what the barriers are to capturing DR. They conducted a survey that focused on how software architects viewed rationale and received valid responses from 81 architects. The survey investigated the capture and use of nine rationale elements: constraints, assumptions, weaknesses of designs, costs of designs, benefits of designs, certainty of the design, certainty of the implementation, and trade-offs considered. All of these were considered important by their respondents. The ones with the highest support were benefits, certainty, and constraints. These types were also identified as being the most frequently used. The types with the least use were rationales describing weaknesses in the design decisions (design weakness, costs, and complexity). When looking at how frequently they captured the different design elements, the respondents reported that constraints and assumptions are documented frequently. The least documented types were design weaknesses, certainty of design, and certainty of implementation. When investigating barriers, cost and time were the most frequently given reason for not documenting rationale. The overall conclusion of the article was that practitioners do believe that DR is important and should be captured.
3. RENEWED INTEREST: HOW AND WHY
Despite the many doom and gloom predictions, there is still a significant amount of interest in this area. There are numerous ongoing research projects, a recent book on rationale management in software engineering (Dutoit et al., Reference Dutoit, McCall, Mistrík and Paech2006), another book, Rationale-Based Software Engineering (Burge et al., Reference Burge, Carroll, McCall and Mistrík2008), and several DR-based workshops. Has the time come for rationale to move beyond being an interesting research problem into something that can become useful in practice?
3.1. Incentives
There are still strong feelings, at least among researchers, that this is an important avenue of research and shows great promise for aiding software development. There are a number of incentives for continuing to pursue this research.
3.1.1. Capability maturity model integration (CMMI)
One incentive for reconsidering rationale use for software engineering can be found in the CMMI (CMMI Product Team, 2006) and its decision analysis and resolution (DAR) process area. The DAR is required for Level 3 of the CMMI and consists of defining a “formal evaluation process” for decision alternative evaluation. The elements of this evaluation include alternative identification, evaluation criteria determination, evaluation method selection and use, and finally, selecting the alternatives based on the identified criteria. The CMMI recommends that part of this evaluation process is defining which decision categories need this formal evaluation and how the evaluation should be performed. The formal evaluation is especially important for decisions that are identified as “high risk.”
3.1.2. Knowledge management (KM)
One of the uses of rationale is the ability to capture “corporate knowledge.” This is also a goal of KM. KM is critical in software engineering because “intellectual capital” is the key asset for a software organization (Russ & Lindvall, Reference Rus and Lindvall2002). The challenges faced by the rationale community are similar to those faced by KM: encouraging knowledge sharing, providing incentives, and making knowledge capture part of business practices (Smith & Farquhar, Reference Smith and Farquhar2000).
The importance of KM has been acknowledged by a number of organizations. Schlumberger (Smith & Farquhar, Reference Smith and Farquhar2000) uses their intranet and a “knowledge hub” to support “communities of practice,” groups of personnel within the organization where information sharing is necessary. This process is supported by a “knowledge champion” who encourages participation. This recalls the success of IBIS, where the presence of a champion-supporting rationale was essential (Conklin & Burgess-Yakemovic, Reference Conklin, Burgess-Yakemovic, Moran and Carroll1996). NASA is also promoting KM initiatives to capture case studies, share “stories,” describe “lessons learned” and to provide an “expertise locator” (Liebowitz, Reference Liebowitz2002). DR research and KM research share many of the same benefits and barriers and should study the approaches used in KM research.
3.1.3. Value-based software engineering (VBSE)
VBSE is another synergistic field. The idea behind VBSE is to move away from “value-neutral” approaches to software decision making (Boehm, Reference Boehm, Biffl, Aurum, Boehm, Erdogmus and Grünbacher2006). Not all software requirements are of equal value to the stakeholder so it makes sense to take value into account when making decisions. Boehm's win–win Theory W (Boehm & Ross, Reference Boehm and Ross1989), a methodology supported by capturing rationale, lives at the center of his 4 + 1 theory of VBSE (Boehm & Jain, Reference Boehm, Jain, Biffl, Aurum, Boehm, Erdogmus and Grünbacher2006), which also includes utility theory, dependency theory, decision theory, and control theory. Using value as a key driver in SE decision making requires methods for capturing the criteria that indicate what “value” is as well as methods for evaluating the value of each decision alternative. These are not trivial issues. There may be uncertainty involved in accessing value: costs may be unknown, stakeholders may have conflicting beliefs, intangible benefits are difficult to quantify (Erdogmus et al., Reference Erdogmus, Favaro, Halling, Biffl, Aurum, Boehm, Erdogmus and Grünbacher2006). The potentially numerous criteria affecting decisions may interact requiring tradeoff analyses (Vetschera, Reference Vetschera, Biffl, Aurum, Boehm, Erdogmus and Grünbacher2006). Much of the information needed to assess value could be provided by the rationale.
3.2. Technology and tool advances
The previous flurry of rationale research activity occurred when hypertext systems became more popular. Hypertext was a key factor in many of the systems described in Moran and Carroll's (Reference Moran and Carroll1996) human–computer interaction-focused book. Now we have new tools promising more integrated development. One example is the Eclipse Framework (www.eclipse.org). The ability to create “plugins” that add on to an existing development environment was used in the SEURAT system (Burge & Brown, Reference Burge and Brown2004) so that rationale could be captured and used in the same environment that is being used to develop software. Eclipse is also an open-source project, which means that it is easily accessible to commercial and research organizations. It is no longer necessary to have a large software tool budget to get powerful tools. Eclipse is only one of many potentially useful open-source development environments and development tools that could be extended to support rationale.
3.3. Success stories
Although much of the evidence for rationale has been collected rather informally, there have been some notable successes. The NCR field trials (Conklin & Burgess-Yakemovic, Reference Conklin, Burgess-Yakemovic, Moran and Carroll1996) demonstrated that IBIS and gIBIS could be useful in an industrial setting. The Compendium project (http://www.compendiuminstitute.org/; Buckingham Shum et al., Reference Buckingham Shum, Selvin, Sierhuis, Conklin, Haley, Nuseibeh, Dutoit, McCall, Mistrík and Paech2006) has been used on a number of different projects and has created a community large enough to sustain an annual workshop. The Compendium Institute e-mail group (http://tech.groups.yahoo.com/group/compendiuminstitute/) has over 1000 members as of December, 2007.
Another success story has been the DREd system (Bracewell et al., Reference Bracewell, Ahmed and Wallace2004). Not only did the designers at Rolls-Royce feel that using DREd helped their design process, it was also given the Rolls-Royce Research and Technology Director's Creativity Award for 2004 (http://www.eng.cam.ac.uk/news/stories/2005/rollsroyce_award/). Using DREd has been made mandatory for design scheme reviews on at least one Rolls-Royce project. The acceptance of DREd by the designers suggests that resistance to the capture of DR may not be as significant as predicted.
4. CHALLENGES IN RESEARCH AND TECHNOLOGY TRANSFER
How do we gather evidence for rationale's success or failure? Controlled experiments collecting and using rationale are challenging. The SEURAT evaluation ran into four types of problems (Burge, Reference Burge2006): task problems, where tasks had to be limited to those that could realistically compare using rationale to not using rationale (eliminating tasks that would require rationale to be successful); subject availability problems, where large numbers of subjects could not be recruited for experiments that needed to be performed after working hours and that could take up to 4 h; time problems, where tasks had to be simplified to be possible for all levels of user (making them so simple that advanced users would not need assistance provided by rationale); and noise problems, where the short duration of the experiment meant that the learning curve introduced by using the rationale tool could eclipse the time saved. Task, time, and noise problems could be mitigated by using the system on more complicated tasks over a longer duration, but the subject availability problems remain. In addition, longer term tasks cannot be performed in a controlled environment that introduces the possibility of more noise.
Ideally, we would like to gather additional data on how rationale could be used in industry. The challenge is to convince practitioners that using rationale will be beneficial. Although it would be nice if developers would participate in evaluations “in the interest of science,” the reality is that if they suspect that the evaluation will add to their work burden they are likely to decline participation.
Of course, that does not mean we should give up. Technology transfer is a difficult process for all new innovations in software, not just rationale. Redwine and Riddle (Reference Redwine and Riddle1985) studied technology in the mid-1980s and discovered that it took 15 to 20 years for a technology to become mature. Redwine and Riddle also identified a set of critical factors influencing technology adoption: it must be well developed, it must fill a well-defined and recognized need, it needs to be adaptable to fit user practice, there need to be reports on “prior positive experience,” management must be committed, and training must be provided. Of these, two stand out: user needs and positive experience. Does rationale meet a well-defined need or is it a solution in search of a problem? Many uses have been proposed but which ones are really needed? How can we provide “prior positive experience?” gIBIS was shown to help developers avoid mistakes and provided considerable cost savings but the researchers were not able to find other “disciples” to promote it (Curtis, Reference Curtis2000).
5. SURVEY OF SOFTWARE DEVELOPMENT PRACTITIONERS
Numerous articles on technology transfer point out the criticality of knowing what the technology adopter needs (Redwine & Riddle, Reference Redwine and Riddle1985; Larson et al., Reference Larsson, Wall, Norström and Crnkovic2006; McGill et al., Reference McGill, Deadrick, Hayes and Dekhtyar2006), yet, as Tang et al. (Reference Tang, Babar, Gorton and Han2006) point out, there has been little research that studies practitioner views of rationale. DR has been an active area of research for many years, yet it still has not been adopted in practice except for a few instances. As described earlier, there are many theories why this has been the case but much of this is speculation.
As Redwine and Riddle (Reference Redwine and Riddle1985) pointed out, if a technology is going to transition into practice we need to know what the users need. To get an initial assessment of how software development practitioners viewed rationale, we decided to conduct a pilot survey of software developers to gather opinions.
5.1. Survey design
We had two primary goals for the survey: to determine how software practitioners felt rationale could be the most useful, and to determine what they thought the barriers to rationale adoption were. For the first goal, we looked at how potential uses were valued, when rationale would be used, when rationale would be captured, who would be most likely to provide it and who would be most likely to use it. For the second goal, we asked questions about which barriers were most significant, and what would make rationale more appealing (and therefore motivate its capture). We also asked some questions about rationale presentation and tool integration.
In addition to the questions on rationale, we also collected demographic data about the company/industry of each respondent, their job role, and their level of experience.
5.2. Survey administration
We planned on two methods for survey administration: hard-copy surveys administered and collected at the annual Miami University Computer Science and Systems Analysis (CSA) Alumni Conference and identical Web-based surveys. Participants at the Alumni Conference were invited to attend a talk describing the research and were then invited to fill out the survey. Attendees were also given a hard copy of the talk so that those who were not able to attend the talk could still participate in the survey. This approach proved to be less than successful with only one survey filled out at the conference.
The Web-based survey invitations were sent by e-mail to 168 Miami CSA alumni, 20 professionals who had either participated in SEURAT evaluations or attended demonstrations, and were also sent out with an electronic newsletter to alumni from the author's undergraduate school, Michigan Technological University. Survey invitees were also encouraged to forward the survey to any coworkers whom they felt would be interested in participating. The online survey approach proved to be more successful than the paper one and resulted in 35 responses. The Web-based survey included a text description of what rationale was and how it could be used and a link to the presentation created for the Alumni Conference. Respondents were instructed to view the presentation prior to taking the survey. The average time taken to complete the online survey (not counting viewing the presentation or reading other information provided about rationale) was 14 min.
5.3. Demographics
The end of the survey asked a series of questions to provide demographic information about the respondents companies and level of experience. This resulted in the following information about the respondent's employers:
1. Nearly 45% of the respondents reported the average lifetime for a product developed by their company as 5–10 years. Only 14% reported longer than 10-year lifetimes.
2. Fifty percent of respondents reported the average time from conception to product release as 1–2 years. Less than 17% reported lifecycles longer than 2 years.
3. New releases of a product typically happened at least once per year (41.67% 6 months to 1 year, 38.89% less than 6 month intervals between releases).
4. Seventy-five percent of the organizations maintained the software they developed.
5. Only 25% of the respondents gave the biggest risk of software failure as mission failure (16.67%) or lives lost (8.33%). The largest number of responses was 36% where business process disruption was the greatest risk.
6. Team sizes varied with the largest number of respondents having teams of 6 to 10 people (30.56%), the next 3 to 5 people (27.78%), 2-person teams were at 16.67% and 1-person teams at 8.33%. Less than 17% had teams of greater than 10 people.
Many respondents (41.67%) did not feel their organization fit into any of the listed types (defense at 22.22%, aerospace at 2.78%, financial at 8.33%, educational at 2.78%, computer technology at 22.22%, and entertainment at 0%).
We also asked the respondents about what type of development they did: developing systems sold by their organization (50%), systems used by their organization (19.44%), and providing consulting services to other organizations (30.56%). Their current job roles were primarily technical (55.56%), followed by management (33.33%), and with 11.11% giving their role as “other” (two project managers, one VP of government sales, and a fourth in Web development and systems administration). Our respondents had an average of 16 years of experience, with the lowest being 1 and the highest being 30.
The survey also asked if the respondents had any prior experience using rationale tools. Most had not (86.49%). The three respondents who used rationale tools indicated SEURAT, self-made decision-tree documentation programs, and a system the respondent had built for him or herself.
5.4. Survey results
We designed the survey to receive input on two categories of questions: uses of rationale to investigate if practitioner needs matched our research goals and the perceived barriers of use to compare practitioner barriers with those described in the literature.
5.4.1. Uses of rationale
Our first set of questions looked at how and when software development practitioners thought rationale would be useful and used. The first question on the study asked the respondents to rate the importance of several uses of rationale. Table 2 gives the results shown in the same order presented to the respondents. The last column of the table shows the combined percentages of the top two levels.
Table 2. Potential rationale uses
No rationale uses were ranked as being not useful by more than one person. Many were ranked in the upper two levels of usefulness. For the top levels combined, 7 out of 17 uses were given a combined score of greater than 50%. Uses that appear to be the most compelling to the respondents were capturing assumptions, providing traceability to functional requirements, and assisting in impact assessment when goals change. The item of least interest was post mortem reporting.
The next five questions asked for additional information about how rationale could be captured and used. For these questions, we had the respondents rank the different options so we could determine relative importance or value with a value of one being the most likely. We give the percentages for each ranking in Tables 3–6.
Table 3. Most likely use time: When do you think rationale is most likely to be used?
Table 4. Most likely capture time: When do you think rationale is most likely to be captured?
Table 5. Rationale provider: Who is most likely to provide rationale?
Table 6. Rationale user: Who will find rationale the most useful?
This question, on use time, yielded some of the most surprising results of the survey. Many researchers, including the author of this article, predicted maintenance as the aspect of software development where rationale would be the most valuable. Instead, nearly half the practitioners surveyed ranked it as the least. It is possible that if the respondents did not have a preference between the last three elements they simply ranked them in order of appearance. It is interesting, however, that less than 20% of respondents ranked maintenance as first or second. During design appears to be the most likely with during requirements development coming in second.
These results were not surprising; most respondents felt that design would be the phase when rationale was most likely to be captured.
These results are consistent with the earlier question, and suggest that the designer is most likely to provide rationale.
The consensus appears to be that rationale is most useful during development by developers wanting to understand why someone else made a decision or in looking back on their own decisions. They felt that rationale would be the least useful for managers to review system progress.
The survey also investigated when rationale should be presented to the user. Some rationale systems, such as JANUS (Fischer et al., Reference Fischer, Lemke, McCall, Morch, Moran and Carroll1996) act as critics and present rationale when there appears to be a problem with the decisions being made; other systems, such as SEURAT (Burge & Brown, Reference Burge, Brown, Dutoit, McCall, Mistrík and Paech2006), display, or indicate, rationale when the associated artifact is being viewed or modified. Other systems store rationale but only present it to the user upon request. Nearly half (45.95%) preferred to see the rationale when they were editing or viewing an associated artifact with the remainder of the vote split between when they take an action that signifies a potential problem (24.32%) or upon request (29.73%).
There are also different mechanisms available for visualization. Some of the most successful approaches, including Compendium (Buckingham Shum et al., Reference Buckingham Shum, Selvin, Sierhuis, Conklin, Haley, Nuseibeh, Dutoit, McCall, Mistrík and Paech2006) and DREd (Bracewell et al., Reference Bracewell, Ahmed and Wallace2004), use a graphical format. Others, such as SEURAT (Burge & Brown, Reference Burge, Brown, Dutoit, McCall, Mistrík and Paech2006), use a tree format. Of the respondents of this survey, 67.57% preferred the tree structure and 27.03% preferred the graph. Two respondents chose “other” for their selection. One was not sure what representation would be preferable, the other said that a tree would be insufficient and that they would want to see indented free text with hyperlinks. It is possible that because the materials provided to them describing rationale used SEURAT as an example they may have been biased toward the tree structure. Alternatively, they could find a tree representation more natural because software developers are used to working in development environments that use a tree structure to represent code directories.
5.4.2. Barriers and assistance
When designing the survey, we were especially interested in determining if the barriers to rationale were as described in the literature. Table 7 ranks several factors that have been proposed as potential barriers to the use of rationale.
Table 7. Potential barriers to rationale use
The first two items, cost and tediousness of rationale capture, were ranked as first or second by the majority of respondents. A quarter of those responding put reluctance to record decisions as being the greatest barrier. The concern that documenting reasons behind decisions hampers the process was ranked as last by over a third of those surveyed.
We were very curious as to practitioners’ opinions on what could be done to make rationale capture and use appealing. The results of that survey question are given in Table 8.
Table 8. Possible appeal: What would make capture and use more appealing?
Augmenting code comments with rationale scored quite high. This is not an approach that has been taken by many researchers, with one exception being the Archium project (van der Ven et al., 2006). Traceability matrixes were also of interest to the practitioners. Using rationale to produce post mortem reports was ranked low which is consistent with results reported in Table 2.
There has been some discussion among researchers about how integration with development tools could assist in rationale capture and use. Table 9 ranks several types of tools by how important it would be to integrate them with rationale.
Table 9. Integration priorities: Importance of integrating rationale with different tools
The first three types of tools on the list, interactive development environments (IDEs), design tools, and requirements management tools were ranked as the most important. There was little interest in integrating rationale with the other types of tools.
5.5. Experiment summary
Some survey results were what one might predict. One example is the barriers to rationale: the greatest barriers involved the tediousness, the cost, and designers reluctant to record their reasons. The biggest surprise was when developers thought rationale would be the most useful. Grudin (Reference Grudin, Moran and Carroll1996) cautioned that rationale capture required resources “upstream” while providing savings “downstream” during maintenance that would weaken incentives for capture. The results of the survey described here indicated that our respondents felt the rationale would be most likely to be used during requirements development and design and would be least likely to be used during maintenance.
The 36 developers who responded to the survey are not a representative sample of the industry as a whole. In particular, it would be interesting to receive responses from more developers working on safety-critical and mission-critical systems. One question that, in retrospect, should have been on the survey was if the developer's organization followed the CMM or CMMI. The CMMI process area on decision analysis and resolution requires formal evaluation of alternatives, yet less than half the respondents rated evaluating support for alternatives in the top two value categories.
Another interesting observation from the experiment is that more than half the respondents listed managers reviewing system progress as being those who had the least use for rationale. Informal discussions with developers had suggested that providing that kind of insight to managers might be helpful in promoting rationale acceptance yet these results suggest otherwise.
6. SUMMARY AND CONCLUSIONS
Rationale, design and otherwise, has been an active research area for many years yet we still have had only limited success in putting rationale into practice. Although technology transfer rates in software engineering are typically slow, we should be concerned that rationale has not received wider adoption. On a more positive note, advances in software tool technology are making it possible to provide support for integrating rationale into the development process in ways that were not possible before. There is also an increasing acknowledgement of the importance of knowledge as a “key asset” to corporations (Liebowitz, Reference Liebowitz2002). Rationale has the potential to make a key contribution toward that knowledge. There remain two major obstacles toward rationale's acceptance that need to receive more attention by the research community. One is that we need to understand the needs and problems of the practitioners we are trying to support. The other is that we need to provide more concrete evidence of the value of our solutions through formal empirical evaluations of both existing and new approaches. Until then, we will remain researching under uncertainty.
Janet E. Burge is an Assistant Professor in the Miami University Computer Science and Systems Analysis Department. She has worked in industry as a researcher and software developer for over 20 years. She received her PhD and MS in computer science from Worcester Polytechnic Institute and her BS in computer science from Michigan Technological University. Dr. Burge's major research interests are in software engineering and artificial intelligence. Her primary research area is in DR, with a focus on DR for software maintenance. She is a coauthor of the book Rationale-Based Software Engineering.
