1. INTRODUCTION
Engineering design is a multifaceted activity of which a key component is to achieve trade-offs between competing criteria to deliver quality products to a demanding market. In automotive engineering, for example, the market's demand for new vehicles with more comfort and convenience onboard pulls the weight of cars upward, while skyrocketing gasoline prices make the fuel economy a new priority leading to the requirement of light cars. Engineers must constantly explore new avenues to keep their products up to date with the expectations of the fast-paced market. To do so, effective teamwork is essential. Engineering teams composed of experts in different technical areas work together to identify requirements, generate design alternatives, make both interactive and joint design decisions, and eventually arrive at a final design. Such a process requires not only efficient communications but also proper means to facilitate mutual understanding, agreement making, and generation of new ideas.
Collaborative engineering support systems have been developed with the primary goal of achieving seamless information flows among designers and engineering systems. Database systems and various communication and workflow tools have been developed to support information sharing, design change propagation, and process management. Few systems provide means for engineers to negotiate their decisions for the benefit of the overall design, and little work has been done to quantitatively assess how given negotiation methods may impact on the collaborative design process and results.
In our research, we take an argumentation-based negotiation (Parsons et al., Reference Parsons, Sierra and Jennings1998) approach to supporting collaborative engineering design. Our research goal is to develop a negotiation framework that links designers and engineering systems together at the decision level, facilitates understandings among them, and helps designers expand their search space and subsequently generate better alternatives. In our previous work (Jin & Lu, Reference Jin and Lu2004; Jin & Geslin, Reference Jin and Geslin2009) we proposed an argumentation-based negotiation framework for engineering design, called ANED. ANED is composed of an argumentation model specifying the process of collaborative negotiation, a negotiation protocol providing communication message formats, and a design context model mapping design information to the protocol and the process. To assess how the ANED approach may influence designers' design process and design results, we conducted an experimental study. In this paper, we present the design of the experiment and discuss the results and implications obtained.
Negotiation is a process in which a joint decision is made by two or more parties (Pruitt, Reference Pruitt1981). The parties first verbalize contradictory demands and then move toward an agreement through both trade-offs and searching for new alternatives. For collaborative design, negotiation can be a way for multiple designers to exchange information, learn about other designers' perspectives and intents, and identify new opportunities based on the newly learned information and knowledge. Therefore, negotiation in collaborative design should not be merely a way for designers to reach agreements through simple give-and-take interactions. It should facilitate designers' exploration of a wider range of solution space through influencing each others' understanding of the problem, knowledge, perspective, and judgments.
Negotiation processes can be analyzed from two different perspectives. The value analysis views negotiation as a multiparty joint decision-making process (Raiffa et al., Reference Raiffa, Richardson and Metcalfe2002) and attempts to comprehend the negotiation situation in more numerical terms such as buyer's/seller's true and revealed prices, preferences, and zone of possible agreements (ZOPA). In this analysis, it is often the case that the “propose-reject/accept” negotiation structure is assumed, and the choice space for each party is relatively clear. By translating the contents of the negotiation into numerical values, the analysis can uncover potential win–win directions, the efficient frontier (or Pareto frontier), and how compromising or modifying one's preference can lead to more desirable agreements.
In contrast, the negotiation process can be analyzed from a linguistic perspective. This analysis focuses on the structure and process and attempts to understand how the use of the different communication language and domain language may impact the process and outcome of negotiation. The communication language is usually composed of locutions or speech-acts (Searle, Reference Searle1969) that the parties can use for their negotiation. It defines the structure of interaction and determines what intentions and associated information can or cannot be communicated. For example, if only the locutions of propose, reject, and accept are allowed for negotiation, then one will not be able to request the other party to provide justification for a given proposal. The domain language for negotiation determines what concepts and associated information of the domain can be communicated and negotiated. In engineering design, the domain language may cover only the design parameters and parameter values; or it may further include constraints, functional requirements (FRs), and design objectives (DOs).
To support engineering collaboration through negotiation, we need to understand what negotiation structures and processes are most effective in encouraging designers to explore their design space and generate good design alternatives. In this research, we follow the linguistic analysis and attempt to clarify the roles that our argumentation-based negotiation framework may play in providing an effective negotiation structure and processes for collaborative design. Our research question is thus “how will the application of ANED negotiation protocol and strategies (enforced by the ANED tool) impact the collaborative design process and outcomes compared with the cases where such protocol and strategies are absent?”
To address this question, we conducted a design experiment study in which human subjects were engaged in solving collaborative design problems with and without the use of our ANED tool. The results showed both the positive effects and limitations of the ANED approach. In the following sections, we review the related work in Section 2 and then provide a brief overview of the key concepts of the ANED approach in Section 3. The experiment design and the performance measures are described in Section 4. The experimental results are presented and discussed in Section 5 and concluding remarks are made in Section 6.
2. RELATED WORK
Extensive research on negotiation has been done in diverse areas from social psychology and social sciences where the focus is on human interaction (Toulmin, Reference Toulmin1969; Gulliver, Reference Gulliver1979; Pruitt, Reference Pruitt1981; Rahim, Reference Rahim1986) to distributed artificial intelligence whose goal is to achieve better collaborative work among computer systems (Bond & Gasser, Reference Bond, Gasser, Bond and Gasser1988; Sycara, Reference Sycara, Gasser and Huhns1989; Ronsenschein & Zlotkin, Reference Ronsenschein and Zlotkin1994). Decision theorists have proposed normative models of negotiation based on decision and game theories (Raiffa et al., Reference Raiffa, Richardson and Metcalfe2002).
Gulliver (Reference Gulliver1979) proposed an eight-phase model negotiation process that describes the progress of negotiation from the initial recognition of the dispute to some kind of outcome. The eight phases are search for arena, agenda setting, exploring the field, narrowing the difference, preliminaries to final bargaining, final bargaining, ritual affirmation, and execution. Pruitt (Reference Pruitt1981) proposed a strategic choice model of negotiation, stating that parties involved in negotiation must make strategic choices at every point in time. The choices include conceding unilaterally, standing firm, or collaborating with other parties in search of a mutually acceptable solution. Toulmin (Reference Toulmin1969) introduced a simple model of argument structure for negotiation based on the exchange of “claims,” “data,” and “warrant” among the participants to assert and justify their negotiation stance.
Researchers in the distributed artificial intelligence community have investigated the issue of negotiation by creating agent-based support systems that collect data from the participants and reconcile their disparities to achieve optimal decisions. Sycara (Reference Sycara, Gasser and Huhns1989) proposed a negotiation process that uses a case-based reasoning mechanism together with a restricted protocol to support agents resolving their goal conflicts. Jennings et al. (Reference Jennings, Parsons, Noriega and Sierra1998) proposed argumentation-based negotiation to support negotiation among distributed agents. Through argumentation, the parties can exchange information pertaining to the negotiation situation, explore mutual option spaces, and eventually arrive at a solution acceptable to all (Parsons et al., Reference Parsons, Sierra and Jennings1998).
Raiffa et al. (Reference Raiffa, Richardson and Metcalfe2002) proposed a taxonomy of group decision making and suggested negotiation as a way to make joint decisions. Extending multiobjective decision theory and game theory, he examined the dynamics of win–lose, win–win, and multiparty negotiations and proposed novel approaches and analysis methods for successful negotiation.
Although the advances of the above-mentioned negotiation research have been applied in business management activities and networked computer systems, few have been introduced to the field of engineering design. Researchers in the field of engineering design have been attempting to facilitate engineering negotiation by providing information and technology supports. Some treated the problem of negotiation in design as an issue of information imprecision and developed formal mathematical models to incorporate the imprecision into design computation (Antonsson & Otto, Reference Antonsson and Otto1995; Scott, Reference Scott1997). Others formulated collaborative design problems as games and treated negotiation as a process of playing various types of games, for example, collaborative and noncollaborative (Lewis and Mistree, Reference Lewis and Mistree1998). Viewing negotiation as a conflict resolution process and devising ways to support conflict identification and resolution is another direction of engineering negotiation research (Klein, Reference Klein2000). CONVINCER (Peña-Mora & Wang, Reference Peña-Mora and Wang1998) is a computer program that facilitates the negotiation process in large-scale infrastructure projects by integrating the concepts of game theory and negotiation forms and guiding negotiations toward sustainable outcomes. One common feature of the existing approaches to negotiation in engineering is that they treat negotiation as a process of single level information exchange and conflict resolution and attempt to reduce the negotiation problem into a multiobjective optimization problem so that a convergent solution can be found. Because these approaches usually require prior knowledge of evaluation criteria and available alternatives, they have only limited use for the early stage of engineering design where defining problems and exploring alternative spaces have to be part of the negotiation process.
There have been experiment studies of negotiation in the literature, but few of them are specific to the engineering design field. Some experiments conducted in the fields of social and management sciences study the impact of personality on the negotiation outcome (Evan & McDougall, Reference Evan and McDougall1967), and others explore the difference between individual versus group negotiators (Polzer, Reference Polzer1996). In the field of engineering design, Kirshmann and Greenstein (Reference Kirschman and Greenstein2002) tested the influence of groupware on a design project. Their approach is similar to ours in its implementation, but the two differ in the focus of study. They investigate the impact of video and audio connectivity and the sharing of various applications, whereas our research is focused on understanding the impact of ANED negotiation protocol on collaborative design.
3. ANED: AN ARGUMENTATIVE NEGOTIATION MODEL FOR ENGINEERING DESIGN
ANED was developed based on the argumentation-based approach to negotiation (Jennings et al., Reference Jennings, Parsons, Noriega and Sierra1998; Parsons et al., Reference Parsons, Sierra and Jennings1998). The basic idea is that negotiation should not be reduced to a mere give-and-take/reject process. Instead, it should be viewed as an opportunity for participants to argue about their respective positions and expectations, influence each other, and eventually achieve mutually beneficial agreements. To ensure that negotiation is efficient and moves toward the right direction, designers should do more than simply “agreeing” or “rejecting” a proposal. They should make “arguments” for others to understand “what do you want” and “why.” Our ANED model is composed of three key components: an argumentation model, a communication language composed of specific speech-acts, and a design context model consisting of the concepts of engineering design and serving as domain language for negotiation. In the following we briefly describe the three components. The details can be found in Geslin (2006) and Jin and Geslin (Reference Jin and Geslin2009).
3.1. Argumentation model
Following Toulmin (Reference Toulmin1969), we model argument as a structure depicted in Figure 1. In this model, negotiation starts when a designer makes a “Claim,” for example, “Hinge position h g should be 20 cm < h g < 25 cm.” If the claim is challenged by another designer, then the designer is required to provide “Data,” for example, “Door size D s = 60 cm,” to defend it. If the challenger is still not satisfied with the data, then a “Warrant,” for example, “If sports car, then h g < 0.5 D s,” can be supplied by the designer, either voluntarily or at the request of the challenger. Note that the datum “This is a sports car,” is in the designer's (the claimer's) mind and is not presented in Figure 1.
Fig. 1. The ANED argumentation model based on Toulmin (Reference Toulmin1969).
A “Warrant” can be a rule that states the relation between the “claim” and “data,” as shown in Figure 1, or a related higher level concept, such as a function requirement (e.g., if function1 is required, then parameter2 needs to be above 10). In the latter case, if the challenger starts to challenge the “Warrant,” that is, the higher level concept, the negotiation moves to a higher level in which the “Warrant” becomes a “Claim” and negotiation continues.
The argumentation model shown in Figure 1 benefits the parties by providing them with a common argument format, with which they can unambiguously exchange information about their negotiation stance, argue about them, and resolve their differences in an effective and efficient way.
3.2. Communication language
The communication language determines the protocol of negotiation by specifying what actions can be taken in the process. The speech-acts of ANED were chosen from Ballmer and Brennenstuhl's (Reference Ballmer and Brennenstuhl1981) speech-act dictionary based on our analysis of engineering negotiation needs (Jin & Geslin, Reference Jin and Geslin2009). Figure 2 illustrates the ANED protocol and use of speech-acts. Following is a brief description of the speech-acts and local actions used in the ANED protocol (Geslin, Reference Geslin2006; Jin & Geslin, Reference Jin and Geslin2009).
• Propose <claim>: Introduce an initial <claim> and initiate negotiation process. The <claim>, expressing the stance of the proposing party, is passed to the other party.
• Agree <claim>: Declare that an agreement is reached on the <claim> and the agreeing party is committed to the agreement.
• Dissent <claim>: Declare that the <claim> under negotiation cannot be agreed, resulting in a disagreement that indicates the conflicting stances of the two parties.
• Defend <claim> AS <data> (or SINCE <warrant>): Introduce <data> and/or <warrant> to defend the <claim> challenged by the other party. Either or both <data> and <warrant> are passed to the other party as additional information.
• Critique NOT <claim> (or <data> or <warrant>) AS <c-data> (or SINCE <c-warrant>): Introduce a critique of <claim> (or <data> or <warrant>) by providing <c-data> and possibly <c-warrant> to justify the critique. Additional information, that is, <c-data> and/or <c-warrant>, is passed to the other party.
• Compromise <claim>: Propose a <claim> that is a compromised version of a previously proposed claim. The <claim> is made based on the previous claim and the newly received information.
• Counter-Propose <claim>: Introduce a new <claim> going against another claim proposed by the other party earlier. The <claim> expresses the stance of the counterproposing party.
Besides the above speech-acts, ANED protocol also includes several local actions including Evaluate, as shown in Figure 2. Furthermore, the parties in the negotiation can choose to acquire more information, wait, or terminate negotiation by providing proper data and warrants.
Fig. 2. ANED negotiation protocol.
3.3. Design context model
The design context model in ANED is an information model that categorizes design product and process concepts and subconcepts for designers and computers to describe their design situation and compose negotiation arguments. Following are the key concepts included in the model.
• Design entity (DE): refers to the elements generated during the design process to satisfy certain FRs, for example, solution concepts, components, assemblies, and parts. A DE is usually characterized by a number of design parameters that can be given specific parametric values.
• Design constraints (DCs): specify relations and bounds of certain design parameters of the overall system or certain DEs. For a given design problem, DCs can be either given by customers or imposed by designers during the design process.
• FR: refers to the functional specific requirements that can be fulfilled by a DE that characterize a physical embodiment.
• DOs: defined as a statement of some aspect associated with the design product that the designer desires to achieve. For example, in designing bicycle frames, maximize strength and minimize weight can be two important objectives for a designer.
3.4. Multilevel issues and negotiation strategies
In collaborative design, negotiation usually starts from identification of conflicts. The conflicts can be task related, such as entity conflicts and constraint conflicts, or they can be value judgment related, such as objective conflicts and preference conflicts. Conventional negotiation begins from identifying ZOPA. If there is no ZOPA between the two participants, then the negotiation can be deadlocked. In our research, we propose a multilevel argumentation approach, as shown in Figure 2. The basic idea is that most issues being negotiated belong to a hierarchy of related issues. Usually, a “superissue” governs the “range” and “behavior” of its “subissues.” If two participants cannot agree at the level of certain “subissues,” then they should be able to move to a “higher level” and negotiate about the related “superissues.” The agreement at the level of “superissues” may lead to an innovative and unforeseeable agreement at the “subissue” level. We call this process multilevel integrative negotiation.
Given the model of argument and communication language, the efficacy of negotiation depends on how the participants decide on strategic actions, proposals, and arguments. The question is related to negotiation strategy: whether to explore the solution space of the current issue, identify new issues at the same level, or move to a higher level of relevant issues. In ANED, three generic strategies are devised based how negotiation is directed vertically in the multilevel space shown in Figure 3.
1. Solution exploration: Try to stick to the current issue and explore its solution space extensively.
2. Issue exploration: Try to move to, or create, new issues at the same level in order to avoid conflicts.
3. Hierarchy exploration: Try to move to a higher level of the design entity hierarchy to resolve conflicts. The hierarchy includes (from lower to higher level) parameter-value → parameter → parameter constraints → FRs → DOs (evaluation criteria).
Fig. 3. Multilevel issues and negotiation.
Using the above concepts, designers can clearly describe the current design situation, their claims, and their justifying data and warrants. The details of the use of the ANED protocol and design context model in a simple negotiation support tool can be found in Geslin (Reference Geslin2006) and Jin and Geslin (Reference Jin and Geslin2009).
4. RESEARCH METHOD
Our objective in this experimental study is to evaluate the effectiveness of the ANED negotiation protocol and strategies and investigate how they influence negotiation processes and design results. More specifically, we intend to compare how using and not using the ANED protocol and the hierarchical strategy leads to different collaborative design processes and outcomes and to identify what needs to be done to further improve ANED. Prior to the experiment, we formulated the following hypotheses:
• Hypothesis 1: The ANED protocol can improve the performance of collaborative design, because focused exchange of arguments (i.e., claims, data, and warrants) may help designers better understand each other's situation and hence be able to find more suitable solutions.
• Hypothesis 2: The ANED negotiation protocol can help designers explore more design alternatives, because the better understanding of others through argumentation and the attempt to maintaining one's own stance may lead designers to searching for more alternatives.
• Hypothesis 3: The ANED protocol can make the collaboration process more efficient, that is, fewer message exchanges are needed for solving a design problem, because of more restricted and guided communications among the designers.
• Hypothesis 4: Following the “hierarchical exploration strategy” can improve the thoroughness of design space exploration, because moving to a higher level provides a better view of subissues in the lower level.
• Hypothesis 5: The strategy support leads to more proposals and agreements, because having the options of moving into higher levels provides more opportunities to find proposals.
To attain the research objective and validate the hypotheses, we need a proper experiment design, a suitable collaborative design problem for testing, and adequate performance measures of design processes and results.
As shown in Figure 4, the design of the experiment in this research involves one independent variable, collaboration support, one control variable, design problem, and four different performance measures. Figure 4 also indicates the possible values for the independent and control variables. More details of the variables are described in the following subsections.
Fig. 4. Design of experiment.
4.1. Subjects and design teams
The experiment involved 24 subjects who were divided into three treatment groups: a control group (CG), a protocol group (PG), and a protocol plus strategy group (PSG). Each group had 4 teams, and each team had 2 participants working together to solve a common design problem. All 12 teams worked on the same design problem and were given the same information. The subjects were recruited among the students attending senior level design class AME410 (Engineering Design Theory and Methodology) offered at the University of Southern California. Participation was strictly on a voluntary basis, and no coercive process was used. The 12 two-person teams were created randomly. The students were all undergraduates in their senior year and majoring either in mechanical engineering or aerospace engineering. Prior to conducting the experiment, the authors went through several testing sessions with the help of a group of graduate students majoring in mechanical engineering.
To ensure that all communications between the two subjects of a team are correctly monitored and there is no unmonitored communication (such as those through voice volume, body language, and gestures), we divided the two subjects into two rooms and they could communicate only through a keyboard and text-based computer connection that we provided. All communication logs were saved and used for analysis.
The control variable Collaboration Support in Figure 4 has three possible values, corresponding to three treatment groups. “Ad-hoc” corresponds to the “CG,” “with ANED protocol” to the “PG,” and “with ANED protocol & strategy” to the “PSG.” As mentioned above, the 12 two-person teams were randomly divided into the above three groups. All teams in the groups worked on the same design problem and were given the same information and directions for design. The CG, PG, and PSG groups are different in the following ways.
• CG: The CG teams were given an ordinary chat tool so that they could exchange text messages freely using any communication language and design information as they collaborate on solving the common design problem.
• PG: The PG teams were asked to use the ANED tool so that they were forced to use the ANED communication language and design context model for communicating and describing their claims and design situations.
• PSG: The PSG teams used the ANED tool and applied the “Hierarchy Exploration” strategy described above.
Each experiment sample, that is, one team solving the given design problem, lasted about an hour and the process included the following three phases:
• Instruction (t = 0–15 min): The subjects sit through an automated PowerPoint slideshow of the design exercise that explains the subject's tasks and responsibilities.
• Practice (t = 15–25 min): This is a brief practice time for the subjects to familiarize with the problem, the data, and the use of the ANED tool for design and communication.
• Design (t = 25 min−1 h): The subjects work collaboratively to solve the design problem.
4.2. Design problem
The design problem for the experiment should be simple enough so that the subjects can comprehend and solve it within the allowed time frame. In contrast, the problem should also be rich or complex enough so that the effect of applying the ANED protocol is observable. We created a problem of designing a manufacturing line for the production of a water filter composed of a grid and a filter body, as shown in Figure 5. Each subject is responsible for a part of the process: Designer A is in charge of the fabrication of the filter body, whereas Designer B is in charge of the grid production and assembly processes.
Fig. 5. Water filter to be manufactured.
The task of each subject is to select the required operations for fabricating the water filter and the needed machines to carry out the selected operations. All of the possible operations for producing and assembling Part1 and Part2 are predefined. Each operation has three alternative corresponding machines. Each machine has two attributes: the cost ($) of using the machine and the space (m2) the machine occupies. Table 1 summarizes the DOs, tasks, and design information for each designer.
Table 1. Design tasks, objectives, and information
To add needed complexity to the manufacturing line design problem, we framed the following concepts as part of the problem definition:
• Local incompatibility: Two machines may be locally incompatible so that they cannot be applied simultaneously by one designer in one manufacturing process. For example, M 32(thread rolling machine) and M 41(column drilling machine) are locally incompatible, so Designer A cannot select both in his solution set.
• Global incompatibility: Two machines may be globally incompatible so that they cannot be applied by the two designers in a team simultaneously in the overall process. For example, M 11(sand casting machine) and M 61(band saw cutting machine) are globally incompatible; Designer A cannot select M 11in his solution set if Designer B selects M 61, and vice versa.
• Issue: Two machines may have a shared issue. In this case, they can be simultaneously applied only if the issue is addressed by selecting an option. For example, M 22(CND milling machine) and M 61(band saw cutting machine) have an issue (#2): “Cut grid must be checked dimensionally to match NC high quality.”
• Option: An option is an item that can be selected from the option list to resolve an issue encountered by the subjects during their machine selection task. For example, Option #11 in Designer B's options list “Dimensional Control Station,” which costs $3 and takes up two blocks of space addresses the aforementioned issue #2.
The incompatibilities and issues were devised as part of the design problem definition to prevent the subjects from selecting the cheapest or the most compact set of machines. This way, the subjects are forced to make decisions over local and global trade-offs. Each of the two team members had a different list of options. The lists were designed to provide the subject with some of the solutions to his/her own issues and some of the solutions to the issues of his/her teammate. Therefore, the only way to properly resolve some of the issues was to discuss them and collaboratively search for suitable solutions.
Using the terms of the design context model introduced in Section 3.3, we can map this design problem to the design context model as follows. The fabrication operations needed to make the water filter components shown in Figure 4 are the FRs that designers must identify, the machines are DEs that designers need to select for achieving required operations, the local and global compatibilities and issues described above are the DCs, and the DO is set to be “minimize machine cost and space usage” as indicated in Table 1. For this given design problem, the PG designers are expect to negotiation mostly about their machine (DE) selections so that the overall compatibility can be maintained. The PSG designers are expected to go to a higher level beyond discussing merely machine selections. They are supposed to exchange their local compatibility or constraints (DCs) and address the “issues” mentioned above. They can sometimes go to an even higher level to question each other's “fabrication operation” (FR) selections. This multilevel negotiation process is realized by teaching the PSG designers how to take advantage of multilevel negotiation through presenting our intentionally designed slide shows to them prior to the experiment.
A machine layout tool, illustrated in Figure 6, is given to each of the subjects during the design session. Besides the computer based communication tool, each subject can also see the other team member's machine layout screen. The following guidelines were given to the subject:
• The space is shared between the two sets of machines selected by each designer and machines cannot overlap.
• Machines must be laid out from left to right following the order of operations.
• Designer A must position machines in the top half of the factory and Designer B in the bottom half.
Fig. 6. The machine layout tool interface.
These guidelines were enforced to give the subjects another opportunity to collaborate about the layout, explore possibilities and possibly create some win–win situations.
4.3. Performance measures
One major task of this research is to develop meaningful performance measures to assess the effectiveness and efficiency of the collaborative design process and results. The following indices are introduced as design performance measures.
4.3.1. Score-based design performance index (SDP)
This index is computed using two metrics: cost performance S c and space performance S s. The maximal score S c = 100% was assigned to the cheapest design observed (m c), whereas the score of S c = 0% was assigned to the design with the highest possible cost (M c). A linear grading scheme was used. The score S c can be represented as
where A c is the cost of the machine set selected by the team.
The space is measured along the horizontal direction. The space score is computed as
where M s is the maximum number of cells used, m s is the minimum number of cells used, and A s is the number of cells evaluated in the experiment.
The SDP index is computed using weighting factors:
4.3.2. Design space exploration index (DSE)
When there is an issue associated with an incompatibility, resolving the issue may need new solutions or options. The DSE index measures the “exploration” quality of the design process and is computed by counting the number of issues discussed (A I) and the number of options considered (A O) to resolve these issues. For each of these two measures the highest number recorded throughout the experiment (M I and M O, respectively) are considered as full scores and scaled to 100%. The lowest values for each were both 0. We have
where
4.3.3. Negotiation content distribution (NCD)
This term refers to the occurrence of each speech-act (Fig. 2) in a given experiment. For each team, the number of occurrences of the following utterances are collected: plan proposals (propose/counterpropose), solution proposals (propose/counterpropose), arguments (critique, defend, dissent), and information requests (acquire-info).
Tracking the speech-acts used provides an overview of the negotiation contents that can be used to assess dominant communication activities in each team.
4.3.4. Negotiation process distribution (NPD)
In this study, a collaborative design process is divided into three consecutive phases.
1. Planning: During the strategic planning phase the subjects strategize about how to address the design problem.
2. Resolution: During the design resolution phase the subjects generate solutions for the common design problem.
3. Optimization: During the design optimization phase, the subjects try to improve their design.
For each team sample, the NPD index measures the ratio of the number of utterances devoted to each of the phases. For example, for the planning phase, we have
Similarly, we can calculate NPDResolution and NPDOptimization.
5. EXPERIMENT RESULTS AND DISCUSSION
With three treatment groups (CG, PG, and PSG) and four sample teams in each group, our experiment yielded 12 samples. A one way analysis of variance (ANOVA), equivalent to a t test, was performed for pairs CG versus PG and PG versus PSG as independent variables for four dependent variables described in Section 4.3. The level of significance was chosen at p = 0.05 as a matter of convention. The ANOVA assumptions (i.e., normal distribution of data, same variance for different treatments, randomness of samples, and independence of samples) were validated for all ANOVAs performed in this research by performing residual analyses. Pearson's correlation coefficient was also used to support a number of observations.
In the following subsections, we first introduce the data encoding scheme and then, based on the experiment results, discuss the impact of our ANED protocol on the design results and process.
5.1. Communication data encoding
After each experiment session, the following design materials were collected:
• the final machines and options selected,
• the final layout of the machines, and
• the transcript of the communication between the two subjects.
The communication logs collected from the design sessions were encoded using the communication language described in Section 3.2. Because the ANED tool was employed by the PG teams, the encoding of their communications was straightforward. For the CG teams, we developed standard definitions for each locution in the communication language and coded their transcripts by mapping the communication transcripts to the definitions of the locutions. The encoding was performed by one coder but was spot checked by the second coder to ensure consistency. Table 2 shows the definitions and some examples of the coding.
Table 2. Locution definitions and coding examples
Based on the selected machines, the options, and the encoded communication transcripts, the values of the four performance measures described above were obtained. The results and their implications are discussed in the following subsections.
5.1.1. Impact of ANED protocol
In the course of this research we developed Hypothesis 1 (protocol leads to better results), Hypothesis 2 (protocol leads to more alternatives), and Hypothesis 3 (protocol leads to more efficient processes), which address how the argumentative negotiation protocol may influence collaborative design results and processes.
An ANOVA, equivalent to a t test in this case, was performed with the negotiation type [two levels: ad hoc (−1) and ANED-protocol (+1)] as the independent variable and performance measures as dependent variables.
5.1.2. Protocol and design performance
In Table 3 and the following experiment result tables, T1, T2, T3, and T4 indicate the four testing teams in a single testing group. From the data shown in Table 3, the average SDP of the CG is 81.38% versus 85.66% for the PG. Although the difference is subtle, the tendency of improvement from using the protocol can be seen. Because the standard deviation is relatively large in both groups, the one-way ANOVA with the experiment type (CG vs. PG) as the factor and the SDP as the response did not yield a significant result [F (1, 6) = 1.05, p = 0.344] and thus could not conclusively validate our Hypothesis 1.
Table 3. Protocol and score based design performance (SDP)
The insignificance might be due to the definition of the design problem. Further analysis of the design problem revealed that the problem was created such that the score differences between the good solutions and the bad ones are small compared with the total scores. Therefore, the chance for the subjects to achieve significantly better scores by uncovering win–win situations was relatively low.
5.1.3. Protocol and design space exploration
An effective negotiation process should lead to exploration of a larger design space, because the final agreement is only as good as the best of the agreements explored during the negotiation. Using DSE as the response and the CG/PG as the factor, the experiment results are shown in Table 4. The ANOVA result shows that the ANED protocol has a significant effect on design space exploration [F (1, 6) = 38.21, p = 0.001], supporting our Hypothesis 2. Another interesting analysis can be done by looking at the correlation between the experiment type (with or without protocol) and the number of issues discussed. The computed Pearson's coefficient value is r = 0.961 (p = 0.000), indicating a very strong correlation.
Table 4. Protocol and design space exploration (DSE)
When ANED was developed, one of the initial considerations was that negotiation is not merely a communicative process but a stimulating and creative one, during which the parties exchange information as well as argue with, and attempt to influence, each other. Conflicts between two parties are not only the problems to deal with but also the opportunities that the parties can take to explore new solutions. This basic principle is adopted by TRIZ (Altshuller, Reference Altshuller1998). In ANED, the argumentative protocol allows the parties to preserve and then explore the conflicts once they are identified. In addition, the negotiation tendency of “maintaining one's own position” embedded in the protocol leads the parties to strive for more alternatives for resolving their conflicts. As shown in Table 4, unlike the teams in the CG, who tended to agree on the solutions they found in the first place, the teams in the PG kept their conflicts “alive” longer and reached agreements only after exploring more alternatives through discussing issues and deciding on options. Our results indicate that the ANED approach has the potential to enhance designers' behavior of generating more alternatives.
5.1.4. Protocol and NCD
One objective of this experiment was to observe the impact of the ANED protocol on the collaboration process in engineering design. By analyzing the NCD data shown in Table 5, we notice a significant difference between the two treatment groups in the type of activities that dominate the negotiation process.
Table 5. Negotiation content distribution (NCD) index
The one-way ANOVA for the total number of nonplanning proposals (i.e., “Proposal-other” in Table 5) shows that the protocol has a significant impact on subjects' proposal making behavior [F (1, 6) = 8.21, p = 0.029]. Using the ANED protocol leads the subjects to generating more resolution- and optimization-related proposals. This result was expected because proposals and counterproposals are the locutions introducing possible agreement points: generating more proposals expands the range of the possible agreements. This supports our Hypothesis 2.
The analysis of the number of information request utterances indicates that the protocol reduces the need for information requests [F (1, 6) = 5.90, p = 0.051]. This can be explained as the result of two combined effects. First, the higher number of proposals is balanced by a lower number of information request/passing loops because proposing and arguing assume the information passing function in the form of data and warrants (see Fig. 1). Second, the efficiency of argumentative negotiation enhances the mutual understanding of their stances and reduces the need for information requests.
The analysis of the number of planning related proposals shows a conclusive result [F (1, 6) = 7.58, p = 0.033]: the ad hoc group does more planning related exchanges than the protocol supported group. We will discuss this interesting result in the following subsection.
The average amount of utterances used by each group validates our Hypothesis 3, that is, the protocol improves collaboration efficiency, as the teams in PG used an average of only 69 utterances to complete the design task, whereas the CG teams needed an average of 118.
5.1.5. Protocol and NPD
In addition to NCD, we assessed the impact of the protocol on NPD by counting the numbers of utterances used in each of the three phases, planning, resolution, and optimization. The experiment results are shown in Table 6.
• For the planning phase, the CG used 23% of the utterances, whereas the PG used nearly 0%.
• For the resolution phase, 42% are used by the CG versus 87% by the PG.
• For the optimization phase, the CG had 35% and the PG had 12%.
A statistical analysis supports the observations. Although the significance is not as strong for the resolution phase [F (1, 6) = 4.25, p = 0.085], the data leads to significant results for planning [F (1, 6) = 13.33, p = 0.011] and optimization [F (1, 6) = 6.45, p = 0.044].
Table 6. Negotiation process distribution (NPD) index
The data and analysis revealed two interesting results. First, the teams in the PG spend little effort of their communication on planning, whereas the CG teams devote almost a quarter of their effort in planning. Planning-related communications are needed when two designers try to decide on the strategy and process to solve a problem. The ANED protocol was designed with a focus on the argument exchange, and the exchange process is predefined. This restriction to some extent relegates the need for planning. Using the protocol, the subjects first identify their stances and go directly into the argumentation process. In the ad hoc CG teams, however, after the subjects get together, they spend a long time on deciding what needs to be done and how to do it. In other words, they try to “optimize” the way to solve the problem. This planning “optimization” often leads to an “easy way out” to solve the problem. As a result, the solutions found from the “easy ways” are considered as the solutions. Fewer additional explorations are pursued. The discussion in the following paragraph further supports this observation.
The second interesting result is that the PG had twice the resolution-related communications than the CG. Although this appears to be inconsistent with our Hypothesis 3 (protocol leads to more efficient processes) when only “resolution” phase is considered, it actually reveals the change of problem solving dynamics when the ANED protocol is used. Without the guidance and restriction of the protocol, the ad hoc teams tend to find solutions and then stick to the found solutions rather than try to argue for, and maintain, their own stances. As a result, any solution is a good solution, leading to less effort in the resolution phase. In contrast, the PG dedicated most of their communication exchange to problem resolution. The argumentation-based negotiation protocol contributes to a richer communication content among the subjects and let them spend more efforts arguing about their positions, exploring new alternatives, and proposing compromises during the problem resolution phase. This more thorough design space exploration often results in a convergence to desirable solutions, reducing the need for postresolution optimization, as visible in the data in Table 6.
5.2. Impact of multilevel negotiation strategy
It is worth mentioning that our designers are all self-interested, trying to maximizing their own design scores (Klein et al., Reference Klein, Sayama, Faratin and Bar-Yam2003), and collaborative, attempting to help achieve the best global scores based on their understanding of the design situation. To that extent, the negotiation process is not only trying to find a middle ground between the two designers but also attempting to create new understanding of the global design situation and generate new alternatives. Therefore, our negotiation is more integrative and a joint decision-making process (Raiffa et al., Reference Raiffa, Richardson and Metcalfe2002). This integrative negotiation process is limited by the designers' willingness to follow the other designer to continue the process. A competitive designer may strongly insist on his or her positions (Klein et al., Reference Klein, Sayama, Faratin and Bar-Yam2003). In this case, the negotiation will mostly be a process of identifying the “best middle ground” at the solution exploration level shown in Section 3.4 and Figure 3. This issue is not explicitly addressed in this research. Thus, the effect of the competitiveness was treated as randomness in the experiment and was not explicitly analyzed.
The PSG treatment group was exposed to the “hierarchical exploration strategy” described in Section 3.4. Prior to the test, the subjects were given a slideshow of a number of case examples of how to apply the strategy. Our intent was to assess how effective this exposure to the strategy can be in addition to the use of the ANED protocol. Because the “hierarchical exploration strategy” involves the concepts included in the ANED protocol, it was natural to compare the strategy group with the PG instead of the CG. Two hypotheses, Hypotheses 4 (multilevel strategy leads to more design exploration) and Hypothesis 5 (multilevel strategy leads to more proposals and arguments) described in Section 4, were postulated.
5.2.1. Strategy and score-based design performance
We can draw a number of conclusions based on the raw costs and space results from Table 7. The average cost scores vary (PG, 90.52% vs. PSG, 97.03%) between the two treatment groups. Furthermore, the teams from the PSG did not get space scores as high (average 49.5%) as the teams of the PSG (average 66.25%), which reveals a more thought-out process focusing on high cost score and compromising on the space score (consistent with Hypothesis 4). The strategic support has thus been instrumental in keeping the design effort in line with the design requirement shown in Eq. (3).
Table 7. Strategy and score based design performance (SDP)
The analysis indicates, however, that the cost difference shown in Table 3 is not statistically significant (F = 1.97, p = 0.21), fending off any conclusion. Nevertheless, the standard deviation drops from σ2 = 2.66 to σ3 = 0.25, denoting a higher consistency of the design results among the PSG teams. This observation corroborates the average number of issues selected by the teams of each group in their final design (2.25 for PSG vs. 0.75 for PG).
The SDP values follow comparable trends, as they are based on the scores along the cost and space performance measures. The average SDP shows a progression from PG to PSG. However, the statistical significance is not clearly established. Therefore, the contribution of the strategy on the design outcome quality is important but not as far reaching as expected according to this experiment. The reasons can be the limited exposure to the strategy received by the subjects of PSG. It can also be the limitation of the problem definition. Further research is needed.
5.2.2. Strategy and design space exploration
For design space exploration, Table 8 shows a progression in the average numbers of issues and options discussed from PG to PSG teams (consistent with Hypothesis 4), even though the statistical significance is not reached because of large standard deviation values [F (1, 6) = 1.49, p = 0.269].
Table 8. Strategy and design space exploration (DSE)
A careful examination of the data indicates that although T2 of PSG did not exhibit significant efforts to explore the design space thoroughly, they achieved a high scoring design. This “singularity” may be due to the design problem's insufficient intricacy to require extensive and thorough design space exploration to achieve a good design. In a real-world design task, the complexity stems from the fact that the solution space is continuous and not discrete as in the problem used in this experiment. There are virtually thousands of solutions for each task leading toward a design solution, and the likelihood of achieving a good design by chance is essentially annihilated. Further study is needed to include more real and complex design problems.
5.2.3. Strategy and NCD
The analysis of the NCD data of the PG and PSG teams in Table 9 reveals conclusive results of the impact of the negotiation strategy. PSG teams generated a significantly higher number of strategic proposals [F (1, 6) = 5.93, p = 0.051] and total number of other proposals [F (1, 6) = 8.40, p = 0.027]. In addition, the increased number of proposals is echoed by a direct increase in the number of agreements reached as seen in Table 3 [F (1, 6) = 6.79, p = 0.040].
Table 9. Strategy and negotiation content distribution (NCD)
The results are consistent with our Hypothesis 5. The implication can be drawn that the hierarchical exploration strategy has a distinct effect on the types of utterances employed by the subjects. The density of the total argumentative content does not change; however, more proposals are exchanged. The subjects are conscious that the discussion should not be limited to the machine selection and machine layout, but it should spread over the higher levels of machine issues and incompatibilities. In this way, they can generate proposals over a larger scope, leading to more proposals and agreements.
5.3.4. Strategy and NPD
The NPD data in Table 10 indicate that PSG teams share the same behavior in strategic planning with those in the PG teams: the planning phase is totally missing for the same reasons described in Section 5.4. Nonetheless, the distribution over the other two phases, that is, resolution and optimization, is appreciably different. One-way ANOVAs over the ratio of utterances used for resolution and optimization yield both [F (1, 6) = 13.37 and p = 0.011].
Table 10. Strategy and negotiation process distribution (NPD)
The PSG teams spent an average of 66% of their communication efforts over the design problem resolution phase and the remainder 34% optimizing the design solution. This redistribution of the two activities is an indication of a more effective problem resolution phase, because the total amount of utterances used is comparable in the two groups. This observation agrees with the higher efficiency of collaboration for PSG teams observed through the number of agreements reached. The team members, who adhere to the same strategy, achieved better understanding of each other's intentions and negotiation stances.
6. CONCLUSIONS AND FUTURE WORK
This experimental study yielded several results supporting our initial hypotheses and showed that negotiation outcomes in a collaborative design process can be significantly affected by the ANED negotiation protocol and strategies. The findings can be summarized as follows:
• The use of ANED's argumentative negotiation protocol and hierarchy exploration strategy affects the dynamics of the negotiation/collaboration process and has the potential of improving the results of collaborative design. Future research is needed to verify the process benefits and link them to the improvement of the design results.
• By imposing argumentative interaction, the protocol leads the subjects to making more efforts on design space exploration and alternative generation, avoiding the general human tendency of “plan, quick solution, and finish.”
• Furthermore, the restrictive exchange of information of the argumentative negotiation protocol makes the overall collaboration process more efficient because the communication is more focused and well guided. However, it is conceivable that the restriction may become an obstacle when design problems become more complex.
• Little planning occurred in protocol- and strategy-supported teams. This implies that the designers should have a good understanding about the design problem and the design process when they come to work together. Future research is needed to verify if adding more speech-acts may help planning interactions.
• The hierarchical exploration strategy propels the designer to explore a wider range of design spaces more thoroughly, both vertically over different issue levels and horizontally across each issue level.
• The hierarchical exploration strategy provides a larger space and more opportunities for designers to generate more proposals and thus more agreements. As a result, the number of arguments exchanged being equal, the strategy-supported teams are able to reach a final design faster and spend more time optimizing their results.
The experimental study described above has several limitations. First, the experiment is restricted to interactions between two designers. Our future work will investigate how the insights gained from this study can be applied to three or more party interactions. Second, the experiment was not set up to address the issue of multidisciplinary collaboration. Although it can be speculated that being able to enhance design space exploration can be positively linked to being able to facilitate better understanding between the designers of different disciplines, further study is needed to verify this link. Third, the results obtained thus far are limited to the types of design problems and the subjects tested. Future experimental research is needed to test various types of design problems and to include professional designers as subjects. Our ongoing research is attempting to address these limitations.
ACKNOWLEDGMENTS
We thank the USC 2005 AME410 students who participated in the experiment. This research was supported in part by a Toyota Research Grant, and we are grateful to Toyota for their support. We also thank three anonymous reviewers who provided insightful and helpful comments for improving the paper.
Yan Jin is a Professor of aerospace and mechanical engineering at University of Southern California. He received his PhD in naval architecture and ocean engineering from the University of Tokyo. Prior to joining the USC faculty in 1996, he worked as a Senior Research Associate at Stanford University. Dr. Jin is the recipient of the NSF CAREER Award (1998), TRW Excellence in Teaching Award (2001), and Xerox Best Paper Award (ASME Design Theory and Methodology Conference, 2002). He currently serves as Associate Editor of the ASME Journal of Mechanical Design and previously served as Conference Chair of the ASME Design Theory and Methodology Conference.
Mathieu Geslin is a Design Engineer at Honda R&D Americas Inc., where he manages a team of digital modelers and leads R&D projects for Honda's automotive and motorcycle design studios. Prior to joining Honda, Dr. Geslin performed his PhD study at the Department of Aerospace and Mechanical Engineering of the University of Southern California. His PhD research was about studying argumentation-based negotiation methods for improving collaborative engineering design. With his years of industrial and academic experiences, Dr. Geslin has accumulated extensive expertise in computer-aided design, collaborative engineering, and engineering innovation.
