2. CRITICAL REVIEW AND MOTIVATION

The manner in which decision making is performed (i.e., how decisions are made) can have a significant impact on the effectiveness of a decision (Dean & Sharfmann, Reference Dean and Sharfman1996). There is strong agreement on the basics of how decision making is executed, which involves formulating a problem, generating potential solutions, and selecting the preferred solution for subsequent implementation (Simon, Reference Simon1960; Brim et al., Reference Brim, Glass, Lavin and Goodman1962; Mintzberg et al., Reference Mintzberg, Raisinghani and Theoret1976), its iterative, “wicked” nature (Rittel & Webber, Reference Rittel and Webber1973), and the types of routines that can take place within it (Mintzberg et al., Reference Mintzberg, Raisinghani and Theoret1976).

At a deeper level, a variety of approaches have been identified that characterize decision making in terms of, for example, the overarching principles that govern it. Rational (Lindblom, Reference Lindblom1959; Etzioni, Reference Etzioni1967; Keen & Morton, Reference Keen and Morton1978), administrative (Simon, Reference Simon1993), successive limited comparisons (Lindblom, Reference Lindblom1959), mixed scanning (Etzioni, Reference Etzioni1967; Wiseman, Reference Wiseman1979a, Reference Wiseman1979b), and “garbage can” (Cohen et al., Reference Cohen, March and Olsen1972) approaches are examples of these principles. Such approaches begin to account for the impact of resources within decision making. For example, the rational approach is an idealized form of decision making that seeks to obtain an optimal decision (Tarter & Hoy, Reference Tarter and Hoy1998), whereas Simon's administrative approach is a descriptive form based on direct observation of how decisions are made. It is a bounded, “satisficing” approach that recognizes limitations associated with resources, including, for example, the cognitive abilities of human decision makers. Thus, although it is less rigorous than its rational counterpart, a cognitively demanding approach that seeks to obtain an optimal solution from an exhaustive search of possible alternatives, it is simpler to execute. It does not produce optimal decisions in the same vein as rational decision making, but instead tends to focus on producing decisions that will prove effective. Likewise, the successive limited comparisons approach involves making decisions that result in a series of small, incremental changes as a means of accounting for the limitations associated with resources directed towards predicting the impact of changes.

In a similar vein, models of decision making have been developed that provide characterizations of decision making in greater detail. Nutt (Reference Nutt1976) performed a series of studies to characterize decision making in terms of, for example, types of processes employed and their individual variations across all phases of decision making (Nutt, Reference Nutt1984) together with a focus on tactics employed during problem formulation (Nutt, Reference Nutt1993). Resources are partly included within some of these characterizations, and although they are only discussed at an abstract level of detail, they do imply that resources have an impact on how decision making occurs. Likewise, Arroba (Reference Arroba1977) derived six styles of decision making by analyzing decisions and the usage levels of each style, which revealed that decision makers were both able and willing to use different styles based on their perception of the situation, although some styles were used more often than others. In addition, Shrivastava and Grant (Reference Shrivastava and Grant1985) identified four different types of decision making, featuring partial high-level descriptions of the resources employed within each type. Again, these results imply that resources impact on how decision making is executed.

Further research has sought to identify what decision-making approaches are appropriate in certain situations, where resource considerations are part of the definition of situations. For example, Tarter and Hoy (Reference Tarter and Hoy1998) developed a series of rule-based propositions containing 10 premises with corollaries describing which approach would suit a particular situation. These premises included consideration of resources and their characteristics (e.g., information quality) and how they influence the decision-making approach. In a similar vein, Sherpereel (2006) developed a decision order methodology that matched decision problems to generic decision-making approaches (specifically, heuristic, probabilistic, and deterministic decision making) by analyzing the language used to describe a problem and the resources. For example, first-order problems feature high levels of simplicity, and their descriptions include language such as “certain” when characterizing information quality. Such problems would typically be solved using deterministic approaches characterized by the use of direct evidence. The contribution of such research is that it begins to define how a decision-making activity should be performed given the resources available. The research described thus far has not typically extended to a consideration of decision-making methods that can be applied within the identified approaches, but does consider the types of analyses that may be appropriate (e.g., Scherpereel, Reference Scherpereel2006).

Other studies of decision making have sought to highlight the role that resources play within it and how those resources influence the manner in which decision making is executed. Such studies have been conducted within individual and team-based contexts. Within individual decision making, for example, empirical studies have found clear evidence that experience allows decision makers to make more effective decisions more quickly than those with less experience by pattern matching current decision situations to previous experiences (Klein et al., Reference Klein, Wolf, Militello and Zsambok1995; Klein, Reference Klein2008). This experience enables decision makers to reach effective decisions in situations characterized by uncertainty through direct matching rather than by concurrent choice, which involves generating and comparing a number of potential options (Lipshitz et al., Reference Lipshitz, Klein, Orasanu and Salas2001). Other research has examined the mechanisms that can be employed during decision making conducted by decision makers with expertise-based intuition (Salas et al., Reference Salas, Rosen and DiazGranados2010).

Within a group context, empirical studies have examined the diversity of decision makers and its link to decision making. Elements of diversity include the different knowledge, and the experience, cognitive, and value approaches of group members (Martin-Alcazar et al., Reference Martin-Alcazar, Romero-Fernandez and Sanchez-Gardey2011), as well as age and gender differences (Jackson et al., Reference Jackson, May, Whitney, Guzzo and Salas1995). Studies have investigated how diversity can affect decision outcomes (Olson et al., Reference Olson, Parayitam and Bao2007) and negotiation within decision making (Martin-Alcazar et al., Reference Martin-Alcazar, Romero-Fernandez and Sanchez-Gardey2011). Diversity has been found to have both positive and negative impacts (Austin, Reference Austin1993), by, for example, increasing innovation and group creativity (positive impact) and increasing conflicts (negative impact). Additional studies have examined the use of other resources (e.g., external representation formalisms) as means of enhancing negotiation within the decision-making process (Beers et al., Reference Beers, Boshuizen, Kirschner and Gijselaers2006). Decision makers' emotions have also been studied to identify the generic goals and strategies they induce into the decision-making process during negotiation between decision makers (Martinovski & Mao, Reference Martinovski and Mao2009).

The manner in which uncertainty attached to resources (e.g., information uncertainty) impacts on decision making has also received attention. Lipshitz and Strauss (Reference Lipshitz and Strauss1997) identified five approaches that decision makers incorporate into their decision-making process as means for handling uncertainty, including suppressing or reducing uncertainty. In addition to observed methods by which decision makers alter their decision making to handle uncertainty (e.g., heuristic-based reasoning; Griffin et al., Reference Griffin, Koehler, Gilovich, Holyoak and Morrison2012), other methods, with stronger theoretical bases, have been developed, including probability-based reasoning (Fenton & Neil, Reference Fenton and Neil2011) and utility analysis (Yang, Reference Yang2001). A review of all decision-making methods lies outside the remit of this article, but interested readers are directed to Belton and Stewart (Reference Belton and Stewart2002) and Salo and Hämäläinen (Reference Salo, Hämäläinen, Kilgour and Eden2010) for further details on the types of methods employed. An important feature of decision making, however, is that the selection of the methods used within it should account for the competencies of the individuals involved in the decision-making context (Salo & Hämäläinen, Reference Salo, Hämäläinen, Kilgour and Eden2010), again illustrating that resources can impact on how decision making is performed.

Overall, there is strong evidence that resources (both active and passive) have the potential to and do impact on how decision making is executed. For example, Simon's (Reference Simon1993) administrative decision-making model illustrates how decision making is bounded to account for cognitive limitations associated with human decision makers. There is a lack of clarity, however, on the precise scope of the impacts that resources can have. A particular instance of decision making can be described at the approach, method, and process levels. The approach level is concerned with the overall guiding principles being adopted within the decision making. At a deeper level, the method level is concerned with the particular method(s) used within the approach, and, finally, the process level is concerned with the specific decision-making tasks that must be executed to solve specific problems. These processes can be defined in terms of tasks and their goals, activities, and resources (Duffy, Reference Duffy2002). Obviously, the approach influences the method(s) used, which in turn influences the definition and execution of the process. Although it has been established that resources do have an impact (e.g., measuring their impact on the quality of decisions; see Martin-Alcazar et al., Reference Martin-Alcazar, Romero-Fernandez and Sanchez-Gardey2011), it is less clear what the impacts will be on the approach, method, or process, or how these effects vary. For example, if uncertainty is introduced into a decision-making context, will the impact occur in one or a combination of the decision-making approach, method, or process, and if so how? What is the timeline of these impacts, for example, do they occur at once or over a period of time?

To provide greater clarity on the impact that resources can have, the remainder of this article describes an experiment that sought to identify and provide clear evidence of how resources can impact on decision making in terms of the approach, method, and process adopted within it. The value of such experiments is that they have the potential to inform the development of intelligent DSS by providing clear evidence of particular support that is required. It is important to note that studies have also investigated how the incorporation of DSS has impacted on decision making. Focusing on the use of information technology within a group context, Dennis (Reference Dennis1996) investigated the impact of a group support system. Although the decision making featured increased levels of information exchange as a result of the technology, there was no significant evidence that it provided decisions of higher quality. In a similar vein, Baltes et al. (Reference Baltes, Dickson, Sherman, Bauer and LaGanke2002) found that computer-mediated communication during group decision making reduced group effectiveness and increased task times compared to face-to-face decision making. An interesting feature of such studies is that they highlight the need to maintain a close relationship between the study of decision making and the development of DSS to help ensure that systems provide the appropriate decision support. Thus, in addition to describing the experiment and its findings, this article will also proffer suggestions as to how the results can be used to guide the development of intelligent DSS.

3. EXPERIMENTAL METHODOLOGY AND RESEARCH QUESTIONS

The experiment, initially detailed in Boyle et al. (Reference Boyle, Duffy, Whitfield and Liu2009), was conducted within the BAE Systems and United Kingdom Engineering and Physical Sciences Research Council funded Network Enabled Capability Through Innovative Systems Engineering project as part of a wider set of demonstration activities concerning architectures, control and monitoring, decision support, and through life systems management within the domain of network-enabled capability development. The focus of this experiment was on investigating the requirements for providing decision support within such a domain characterized by both resource variability as well as the collaborative nature of decision making. This article provides a further analysis of the initial results reported in Boyle et al. (Reference Boyle, Duffy, Whitfield and Liu2009).

The experiment centered on an observational analysis using scenario-based role play in which a co-located decision-making network (Figure 2), consisting of decision makers and passive information resources, was presented with a decision problem to solve. The resources were subject to variation in terms of the expertise of the decision makers and the availability and quality of the information resources. The decision making of the role play participants was captured and analyzed to examine the impact of dynamically changing resources within the network upon the decision making.

Fig. 2. The decision-making network (Boyle et al., Reference Boyle, Duffy, Whitfield and Liu2009).

The scenario used in the experiment was that a coalition of organizations would have to work together and engage in team decision making to develop a repair plan for a naval defense vessel. Such coalitions are used within the naval defense industry to provide routine maintenance support for vessels and have defined processes, supplier agreements, and communication protocols. This particular scenario, however, put the coalition in unfamiliar territory in that they were asked to develop a repair plan for a vessel that fell outside of its routine maintenance remit. As Figure 2 illustrates, the coalition consisted of both decision makers and passive resources. There were four decision makers, each of whom had different roles and responsibilities. DM_1A represented organization A and acted as the project manager. DM_2B was the joint head partner of the coalition and represented the customer organization (i.e., the vessel owner/operator). DM_3B was the finance representative from the customer organization, responsible for ensuring any proposed solution was cost effective. DM_4C was a production representative from organization C who would be responsible for the physical repair work. At the start of each role play, each participant was provided with a briefing pack containing a description of their roles and responsibilities, together with information specific to the problem they were to solve.

Three specific research questions drove the experiment and, to obtain answers, the experimental variables highlighted in Table 1 were used. Specifically, to examine the impact of the passive resource quality, information of varying quality was fed to the participants dynamically as the role play progressed, and the impacts were observed. The specific types of information quality examined were completeness and certainty. For example, the participants were presented with incomplete information during a particular decision. The manner in which participants executed decision making based on incomplete information was captured. During a later stage of the role play, they were provided with complete information for a similar type of decision, and again their decision making was recorded. The decision making from both occasions could then be compared to identify any impacts caused by varying information quality.

Table 1. Research questions and experimental variables

To examine the impact of changing resource availability, the technical information resource was not initially available to the network at the start of the role play and the decision-making process captured in this scenario. Thereafter, the resource was added to the network and was, therefore, available to the participants; the decision making employed by the participants after its introduction was captured. Again, the decision making prior to and after the introduction of the resource was compared to identify any impacts its availability had on the manner in which decision making was executed.

To examine the impact of the expertise level of the decision makers, the role play was executed twice. The first role play was run using domain-novice decision makers; the second run used domain-experienced participants. Domain-novice decision makers were experienced decision makers, but they were not experienced in the domain that was the subject of the experiment. In contrast, domain-experienced decision makers had experience of making decisions within the subject domain. The decision making employed during both role plays was captured and compared to identify the impact of the differing levels of domain expertise.

4. DATA SETS AND OBSERVATIONS FROM THE EXPERIMENT

The executions of both role plays were captured in video and audio formats to provide an initial set of raw data, augmented by the material the participants produced during the role play. This data was processed to produce three key data sets that were subsequently analyzed to determine the impact of resources, according to the research questions listed in Section 3. Section 4.1 presents details of the processed data sets, and Section 4.2 presents the results of the analysis.

4.1. Processed data sets

The primary data sets produced were decision, task, and interaction timelines. These processed data sets are too numerous to produce here in their entirety, but samples of these timelines are presented to illustrate the data used when answering the research questions. Figure 3 presents an excerpt of the decision timeline from one of the role plays. Decision timelines detail for a particular decision an identification ID, when the decision was made, the decision content, and also what type of decision it was. Drawing on O'Donnell and Duffy's (Reference O'Donnell and Duffy2002) distinction between design and design management activities, decisions were classified according to whether they were concerned with the goal of the decision problem (i.e., a “problem” decision) or if they were concerned with managing the decision making to ensure its satisfactory performance (i.e., a “management” decision). For example, decision D7 in Figure 3 describes a problem decision that the shaft bearing on the ship should be replaced, whereas decision D9 was a “management” decision about the need to create and include a risk register within the decision-making process as a means of managing risk during the development of their plan.

Fig. 3. A decision timeline sample (excerpt from role play 2).

Figure 4 presents an example of the decision process timeline, which contains a list of the tasks performed. Tasks were identified based on the approach outlined in Mizoguchi et al. (Reference Mizoguchi, Vanwelkenhuysen, Ikeda and Mars1995). Tasks have an identification ID, an owner (i.e., who is performing the task), classification of the type of task, the goal of the task, its motivation, a general description, the means by which the task was completed, and the final outcome. Figure 4 presents an excerpt from a decision process timeline, in which task ID46 is involved with clarifying a piece of information's quality shortfall. The outcome of the task is that a piece of information is found to be incomplete and resolution of the poor information quality is required, and subsequently addressed by task ID47.

Fig. 4. A decision-process timeline example (excerpt from role play 2).

Figure 5 presents an abstracted example of the interaction timeline from the first role play. It details the level of interaction between the decision makers just after the introduction of the technical information resource (one of the experimental variables). Specifically, it lists the general level of interaction between participants in the role play, with whom they are interacting, and the start and end times of the observation. For example, IO5 in Figure 5 states that between 41 and 47 min after the start of the role play, DM_1A and DM_4C were not actively involved in any interactions with other decision makers and were working autonomously. Their level of interaction during that period was nil, whereas DM_2B and DM_3B were interacting equally with each other.

Fig. 5. An interaction timeline sample (excerpt from role play 1).

4.2. Observations with regard to the research questions

The observations from the experiment are discussed in the following three sections for research questions RQ1, RQ2, and RQ3, respectively. Detailed results to research questions RQ2 and RQ3 have already been published in Boyle et al. (Reference Boyle, Duffy, Whitfield and Liu2009), but a précis of those results are presented in Sections 4.2.2 and 4.2.3 because they are directly relevant to Section 5, where the results are synthesized and subsequently used to identify areas for future DSS research.

4.2.1. Observations on the impact of passive resource quality (RQ1)

To gauge the impact of passive resource quality on the participants' decision making, the variable used was information quality within the decision-making network. Specific examples investigated were incomplete and uncertain information and noticeable process impacts were identified from the participants' handling of information. The introduction of the experimental variable commenced once the technical information resource was part of the decision-making network. Although the variables did not impact on the overall decision-making approach, some impacted on the decision-making method and process, and others impacted on the decision-making process alone.

The novice decision makers (role play 1) made a guiding principle decision regarding their decision making, which was to adopt a “worst-case” approach. This was retained throughout their role play and had a significant impact on the decisions they subsequently made regarding the repair plan being developed for the naval vessel. The observed manifestation of this guiding principle decision was that they typically tended to incorporate simplifications: if there was an information quality problem with a particular decision option they were considering (e.g., a piece of information was incomplete), then it was classified as “high risk,” and, under the worst-case principle, was to be avoided. Consequently, the option was not considered again.

An example of the impact of information quality on the decision-making process occurred during decision D18, which was concerned with determining how the damaged propeller of the naval vessel should be repaired. Three options were under consideration: replace the propeller with a direct replacement (option A), mend the existing propeller (option B), or fit an alternative design of propeller (option C). The participants were provided with incomplete information regarding the new propeller. Specifically, the information provided to the participants was “New propeller can be ordered from supplier, delivery time is unknown.” As a result of the earlier guiding principle decision made to classify unknowns as high risk, option A was discarded as a potential solution. As Figure 6 illustrates, this introduced additional tasks into the decision-making process:

• • an abstraction task to clarify that option A had an information quality problem,

• • a clarification task to identify the nature of the information quality problem,

• • a mapping task to identify and retrieve an earlier decision that could be used to address the information quality problem,

• • an applying task to implement that decision and therefore classify option A as high risk,

• • a mapping task to identify and retrieve an earlier decision that could be used with regard to handling high risk, and

• • an applying task to implement that decision regarding high risk that resulted in option A being discarded as a potential solution.

Fig. 6. Dealing with incomplete information based upon earlier decisions (decision D18).

Dealing with incomplete information in this manner through the direct matching of incomplete information with high risk (regardless of whether in reality it did represent high risk) helped to keep the decision making tractable, which was a feature observed in both of the role play executions. The decision makers wished to reduce the complexity of the problem caused by the information quality. An alternative approach that could have been considered would have been to perform a “what-if” scenario and ask “if this task takes so many days, what is the impact upon the plan?” Does it matter if the duration is unknown, or at what particular value of duration does it become a high-risk option? Such an approach would have represented a more rigorous decision-making process, but the preferred option was to simplify.

An example of where uncertain information impacted on the decision making through the addition and execution of new tasks occurred during decision D24, when the domain-novice decision makers were deciding how best to repair the shaft. They were presented with three basic options (replace, mend, fit an alternative), but for the mend option the information concerning the duration of the mending task was uncertain: it would take 6 to 8 days. An earlier decision had been taken that the worst-case scenario was held to be the longest duration for a task; thus, the domain-novice decision makers refined this piece of information by assuming eight days for repairing the shaft, which represented the worst-case scenario. As Figure 7 illustrates, the uncertain information impacted on the decision-making process through the need to incorporate additional abstracting, clarifying, mapping, and applying tasks.

Fig. 7. Dealing with uncertain information through refinement (decision D24).

A feature of the two examples presented in Figures 6 and 7 is that the impacts were confined to the decision-making process alone. On other occasions, however, information quality resulted in the need to incorporate new methods within the decision making as well as additional tasks. For example, when incomplete information was encountered (e.g., a repair task that had no duration attached to it), the participants on occasion tried to determine what the missing part of the information might be by looking at the information they had that was complete and employing an additional method in the form of analogical reasoning. Thus, for example, if the duration of removing a bearing was unknown, but it was known that fitting the same bearing took a day, the novice decision makers made the assumption that it would be reasonably safe to assume that removing the bearing could also be done in a day. In addition to impacting on the decision-making method through the participants' use of analogical reasoning, it also impacted on the decision-making process through the addition of additional abstracting, clarifying, mapping, and assigning tasks (Figure 8). The mapping task in this particular example concentrated on identifying and retrieving a similar piece of information from the technical information resource within the decision-making network, and resulted in the participants employing analogical reasoning to infer that the removal of a propeller was sufficiently similar to its installation to justify their directly reusing the duration of the propeller removal as the installation duration. This was in contrast to the domain-experienced decision makers, who were able to use their own experience to fill in the missing elements of a particular piece of information. Nevertheless, they still had to incorporate the same types of additional tasks, the difference being the manner in which they performed the mapping. Whereas the domain-novice participants mapped to a passive resource within the network (the technical information resource), the domain-experienced participants were able to map more frequently to their own expertise.

Fig. 8. Dealing with incomplete information using analogies.

The use of analogies in this manner raised the interesting question of the different types of bias that can occur within group decision making (Kahneman et al., Reference Kahneman, Lovallo and Sibony2011), and their occurrence within the observed role plays. The groups did make a concerted effort to address confirmation bias by evaluating different decision options. For example, when deciding how to deal with damaged components of the ship, they explicitly considered alternatives, including direct replacement, repair, or nonstandard equipment replacement. In other respects, however, there were limited attempts to recognize the existence of and mitigate bias. Considering their use of analogical reasoning, little effort was made by the decision makers to ascertain whether any analogy they drew was subject to some form of bias (i.e., was it an appropriate analogy to draw) through identifying, for example, what the common features of similarity in the analogy were, what the differences were, and how those differences affected the relevance of the analogy. An additional feature noted in the role plays regarding the information that was provided to the decision makers was that, regardless of whether it was individual pieces of information or a collection of information (e.g., the technical information resource), its credibility was never questioned by the participants: the information was assumed to be true. Although the information provided to the participants was realistic, there was no evidence that the decision makers incorporated tasks within their process to assess the credibility of the supplied information or to mitigate the potential impacts caused by information with low credibility.

4.2.2. Observations on the impact of passive resource availability (RQ2)

To gauge the impact of passive resource availability on the decision-making process, the variable used was the presence of the technical information resource within the decision-making network (Figure 2). It was absent from the network at the start of the role play; thus, the participants neither had access to it nor were they aware of its existence, and it was introduced part way through the role play. Significant impacts caused by this variable were observed. In particular, two major impacts were identified relating to the rigor of the decision-making process and the interactions between the decision makers within the network.

In terms of decision-making rigor (which, for the purposes of this article, applies to both the problem identification and solution phases of decision making), the introduction of the technical information resource resulted in a significant increase in both role plays. An examination of the types of tasks that the domain-novice participants undertook prior to its introduction illustrated that they were not engaged in activities geared toward decision making. There was general discussion concerning familiarization with the problem, some brainstorming to generate a list of likely repair tasks, and abstract evaluation of the identified repair tasks.

After examining the number of decisions made prior to the introduction of the technical information resource after 28 min, only three decisions were made (Figure 9). There were, however, resources available to the participants that would have enabled the decision making to progress more, such as drawings of the vessel's drive system. For example, the participants knew this system was damaged and required repair. They could have examined the drawing to identify the components it was likely they would need to remove and replace, and an examination of the physical layout of the drive system would have helped them determine the order in which that would need to be done.

Fig. 9. The accumulative decision count of the domain-novice participants before and after the introduction of the technical information resource (Boyle et al., Reference Boyle, Duffy, Whitfield and Liu2009). [A color version of this figure can be viewed online at http://journals.cambridge.org/aie]

It was not until the introduction of the technical information resource, however, that the participants began to adopt a more rigorous decision-making process. This resulted in the number of decisions being made by the group increasing such that within the next half hour seven more decisions were made. The nature of the tasks the participants were engaged in was considerably more focused compared to those they undertook prior to the introduction of the resource, and this represented a significant increase in the rigor of their decision-making process. Specifically, they were engaged in the following:

• • choice from a known set of decision options;

• • detailed and quantified evaluation of those options based upon criteria including cost, duration, and available resources, resulting in the production of a detailed repair plan for the vessel;

• • detailed definition of guiding principles to be followed throughout their decision making (e.g., refinement of the worst-case decision-making approach they chose to adopt throughout the role play);

• • definition of the risks associated with the final repair plan; and

• • assignment of responsibilities and resources within the final plan.

The second major impact related to the introduction of the technical information resource was the manner in which the interactions between the four decision makers changed (Figure 10). Prior to its introduction, the levels of interaction between them were uniform. After its introduction, however, the two decision makers with access to the resource (DM_1A and DM_4C) withdrew completely from the group activity and worked individually in isolation, focusing exclusively on absorbing its contents. Thereafter, these two interacted together, but remained separate from the two remaining decision makers, before rejoining them to dominate the decision-making activity. Thus, the introduction of the passive technical information resource had a significant impact in terms of disrupting and altering the interactions between the four decision makers.

Fig. 10. The interaction impact of technical information resource addition.

4.2.3. Observations on the impact of expertise (RQ3)

To gauge the impact of expertise on the decision-making process, the variable used was the domain experience of the participants. The first role play involved domain-novice participants, and the second involved decision makers who were domain-experienced (see Section 3). Similarities and differences between the decision making were observed in both role plays.

In terms of similarities, the addition of the technical resource caused the decision making to become more rigorous in both role plays and the observed decision making progressed to a greater level of detail. In the case of the domain-experienced participants, however, this impact was less pronounced because their decision making was more robust to the absence of the technical information resource, and they were able to draw on their domain experience to make progress in their decision making. This was an expected result, given, for example, the findings of Klein et al. (Reference Klein, Wolf, Militello and Zsambok1995). As illustrated in Figure 11, they made considerably more decisions during the first 20 min of the role play, including technical decisions (e.g., deciding to repair the propeller rather than replace it), that their domain-novice counterparts were unable to make until the introduction of the technical information resource. Overall, the domain-experienced participants completed the exercise in 1 h 30 min, which was 66% of the time it took the domain-novice participants to complete it. Nevertheless, the addition of the resource did cause them to increase the level of rigor within their decision making through, for example, shifting from qualitative to quantitative assessment of alternative decision options.

Fig. 11. A decision count comparison for both role plays. Adapted from Boyle et al. (Reference Boyle, Duffy, Whitfield and Liu2009) with permission.

Another similarity between the two role plays was that the definition of how they would make decisions regarding the repair plan was developed in real time. Thus, problem and management decisions were made concurrently. For example, the domain-novice decision makers did not decide to use a GANTT chart as a means of representing and working on their work plan until 1 h 5 min into the exercise, which is when they needed some form of representation. Hence, the need for their reactive management decision.

A further example highlighted that decisions themselves were subject to evolution. To illustrate, the domain-novice participants decided to adopt a worst-case approach after the introduction of the technical information resource, but its definition evolved as the decision making progressed and they acquired a greater understanding of the problem they were addressing. At 47 min, they decided to adopt a worst-case approach (D6) in which the worst case corresponded to highest cost associated with an element of the repair plan. At 1 h 3 min, a further decision (D12) modified this definition so that the worst case now corresponded to either the longest duration or the highest cost. Changes to earlier decisions (such as that illustrated by D12), however, were seldom related back to decisions taken during the interim period of time regarding the contents of the repair plan being developed. For example, decisions D8, D9, and D10 in Figure 12 were taken after D6. Decision D12, however, negated the validity of D6 (on which decisions D8, D9, and D10 were dependent); thus, there should have been an explicit check to determine if those decisions remained valid given decision D12, which updated and revised D6.

Fig. 12. The observed evolution of decisions.