2. GOAL OF DESIGN RESEARCH AND ITS CURRENT STATUS

According to Blessing et al. (Reference Blessing, Chakrabarti, Wallace and Cross1992, Reference Blessing, Chakrabarti and Wallace1995) the goal of design research is to develop knowledge to make design more effective and efficient, to enable design practice to develop more successful products. Pederson et al. (Reference Pedersen, Emblemsvåg, Bailey, Allen and Mistree2000) and Seepersad et al. (Reference Seepersad, Pedersen, Emblemsvåg, Bailey, Allen, Mistree, Chen, Lewis and Schmidt2006) have similar a view, as they see scientific knowledge in engineering design as having “usefulness with respect to a purpose.” In design research, the work has to be both academically (i.e., challenging and unsolved) and practically worthwhile (i.e., related to its usefulness to a purpose, i.e., the success of the product). Hence, design research has two related objectives (Blessing & Chakrabarti, Reference Blessing and Chakrabarti2009):

1. 1. formulation and validation of models and theories about the phenomenon of design, and

2. 2. development and validation of support founded on these models and theories to improve design practice and its outcomes.

However, design research is not often carried out to address these objectives in a logically linked manner, leading to several major issues, as identified by Blessing and Chakrabarti (Reference Blessing and Chakrabarti2002):

• • Lack of an overview of existing research: It is hard to obtain an overview of the results of design research. There is no agreed terminology (Lowe et al., Reference Lowe, McMahon, Shah, Culley, Culley, Duffy, McMahon and Wallace2001), and little contradiction among the findings of empirical research; all address something different, and few have tried to consolidate the results. Design research is highly fragmented (Horvath, 2001; Samuel & Lewis Reference Samuel, Lewis, Culley, Duffy, McMahon and Wallace2001). There have been attempts to create an overview of design research (e.g., Finger & Dixon, Reference Finger and Dixon1989a, Reference Finger and Dixon1989b; Horváth, Reference Horváth, Culley, Duffy, McMahon and Wallace2001), but these are still far from providing an overview that could help identify major subsets of research (Blessing et al., Reference Blessing, Chakrabarti and Wallace1995). Bringing the results together is a prerequisite to developing comprehensive models and theories.

• • Lack of use of outcomes in practice: As design research aims to improve design, its outcomes should influence practice. However, as Cantamessa (Reference Cantamessa, Culley, Duffy, McMahon and Wallace2001) found in his analysis of the papers in two large engineering design conferences, industrial implementation issues are addressed in only a minority of the papers on support development. Research results rarely translate into practice (Reich, Reference Reich1994; Upton & Yates, Reference Upton, Yates, Culley, Duffy, McMahon and Wallace2001). Many proposals for support have weak empirical foundations, both in development and evaluation. Validation is seen as crucial; how this should be carried out in design research is a topic of current research (Pederson et al., Reference Pedersen, Emblemsvåg, Bailey, Allen and Mistree2000; Olewnik and Lewis, Reference Olewnik and Lewis2005; Seepersad et al., Reference Seepersad, Pedersen, Emblemsvåg, Bailey, Allen, Mistree, Chen, Lewis and Schmidt2006). A more rigorous research approach is needed for better realisation of support and their successful utilization in practice.

• • Lack of scientific rigor: There is often a lack of scientific rigor, in particular, in applying research methods, interpreting findings, developing support, and validation of results. Scientific rigor requires, among others, falsifiability (Frey, Reference Frey2004) and generality (Pederson et al., Reference Pedersen, Emblemsvåg, Bailey, Allen and Mistree2000; Seepersad et al. Reference Seepersad, Pedersen, Emblemsvåg, Bailey, Allen, Mistree, Chen, Lewis and Schmidt2006) of research outcomes. The multifaceted nature of design is one reason for the diversity of research topics and methods used. Design researchers “are yet to properly grapple with the overwhelming complexity of the discipline” (Samuel & Lewis, Reference Samuel, Lewis, Culley, Duffy, McMahon and Wallace2001), which requires a variety of methods to be applied, often from disciplines unfamiliar to design researchers, leading sometimes to incorrect use, resulting in biased data or conclusions. Blessing and Andreasen (Reference Blessing, Andreasen, Clarkson and Huhtala2005) found many students involved in design research for 1 to 2 years who were unclear about what constitutes design research and how to carry this out. This partly explains the lack of methodological rigor observed.

  Overall, the approach to doing design research should clearly connect its goals and deliverables: the development of support to improve (some aspects of) design must be based on a sound understanding of design as currently carried out (i.e., models and theories of the relevant aspects), which must be clearly related to the goals (i.e., criteria for success), to ensure that development and evaluation of support can be more relevant, effective, and efficient. DRM had been proposed to help achieve this (Blessing, et al., Reference Blessing, Chakrabarti, Wallace and Cross1992, Reference Blessing, Chakrabarti and Wallace1995, Reference Blessing, Chakrabarti, Wallace, Frankenberger, Badke-Schaub and Birkhofer1998; Blessing & Chakrabarti, Reference Blessing and Chakrabarti2002, Reference Blessing and Chakrabarti2009). The methodology is presented in the following section.

3. DRM

This methodology stems from what we view as the goal of design research: to inform and support practice by developing knowledge that can improve the chances of producing successful products. As discussed in our earlier papers, this goal raises a number of major questions:

1. 1. What is meant by a successful product?

2. 2. How is a successful product created?

3. 3. How can the chances of being successful be improved?

The first question leads to issues such as what criteria should be used to determine whether a research work has been successful. The second question leads to issues such as what the influences on success are, how these interact with one another, and how these can be assessed. Investigating these will increase our understanding of design, as it currently is, which forms the basis for improving design. The third question leads to issues related to the use of this understanding to develop design support, and its evaluation to determine whether the application of the support leads to more successful products as represented by the criteria for success.

A research methodology is needed to address these issues in an integrated manner; it should aid researchers identify key research issues and select appropriate research methods to address these.

As discussed in Blessing and Chakrabarti (Reference Blessing and Chakrabarti2002), two characteristics of design research require the development of a specific research methodology. First, the selection of research areas is hard because of the numerous influences from the many facets of design, and the interconnectivity among them. Design research has to be multidisciplinary. Second, design research involves both understanding the phenomenon of design and using this understanding to change the way design is carried out, requiring both a theory of what is and a theory of what should be. Because this cannot be predicted, design research involves design (creation of support) and their validation, which require research methods from a variety of disciplines. A variety of research methods have been used to address the multitude of issues involved in design research. DRM has been used to piece these issues together into a generic design research methodology that links their research questions together and systematically addresses them. Figure 1 shows the main stages of DRM and proposes research to proceed through the following steps:

• • clarify what is meant by success [research clarification (RC) stage];

• • understand relevant aspects of the phenomenon of design and its influences on success [descriptive study I (DS-I) stage];

• • based on this understanding, envisage what aspects of design can be improved and develop proposals, referred to as support, for improvement [prescriptive study (PS) stage];

• • evaluate proposed support for its influence on success [descriptive study II (DS-II) stage];

• • if the results are not acceptable, modify the support (go back to PS stage); and

• • if still unacceptable, improve understanding of success and its links to relevant aspects of the phenomenon of design (go back to DS-I or RC stage).

An example from Blessing et al. (Reference Blessing, Chakrabarti and Wallace1995) is used to clarify the above process:

• • RC stage: A reduction in time to market is identified as a criterion for success. This is used as the metric against which to judge a design support.

• • DS-I stage: A descriptive study, involving observation and analysis, shows that insufficient problem definition relates to high percentages of time spent on modifications, which is assumed to increase time to market. This description provides the understanding of the various factors that influence the criterion, in this case, time to market.

• • PS stage: Based on the outcome of the previous stage and introducing assumptions and experience about an improved situation, a support is developed to improve problem definition. As discussed before, developing support is a design process in itself. Support at this stage needs to be evaluated for its internal consistency, as also stressed by several others (Pederson et al., Reference Pedersen, Emblemsvåg, Bailey, Allen and Mistree2000; Olewnik and Lewis, Reference Olewnik and Lewis2005; Seepersad et al., Reference Seepersad, Pedersen, Emblemsvåg, Bailey, Allen, Mistree, Chen, Lewis and Schmidt2006).

• • DS-II stage: The support is applied and is evaluated using a descriptive study. This includes two tests. The first test is whether problem definition is supported. The second test is whether less time is spent on modifications, and whether this reduced the time to market. There might be reasons as to why the second test fails, for example, side effects of the support. These tests evaluate how well the support serves its purpose, its external validity (Pederson et al., Reference Pedersen, Emblemsvåg, Bailey, Allen and Mistree2000; Seepersad et al., Reference Seepersad, Pedersen, Emblemsvåg, Bailey, Allen, Mistree, Chen, Lewis and Schmidt2006), where statistical intervention approaches such as those used in medicine can sometimes be useful (Frey & Dym, Reference Frey and Dym2006).

Fig. 1. The stages of the design research methodology: the downward arrows connecting the stages show natural progress, the upward arrows indicate possible iterations, and the horizontal arrows indicate means for and outcomes of stages.

Descriptive studies should reveal a network of causes and effects, connecting influencing factors with the success criteria. Support developed in the PS stage would directly address some of these influencing factors, so as to indirectly affect the success factors. However, interference in this network, because of the influence of the support, may result in a different network of causes and effects, making the effects different than expected: side effects may occur. Evaluation of support (DS-II) should therefore take this into account and at least test all the causal links between the influencing factors directly addressed by the support and the success criteria.

Not every step of the methodology will be executed in depth in every single project, because of resource constraints, or availability of existing research. In all cases, however, the methodology should be taken into account, and at least the links between the stages under focus of the research, and the stages related to these should be addressed. A study of some influences on the design process (DS-I) should at least indicate how this can be used to improve the design process.

The specific features of DRM are as follows. First, it integrates all major stages of design research in a systematic way, and provides a nomenclature with which a given piece of design research can be systematically categorized, thereby enabling an overview and comparison of apparently disparate pieces of design research cases. In Blessing et al. (Reference Blessing, Chakrabarti, Wallace, Frankenberger, Badke-Schaub and Birkhofer1998), all the papers presented in a design research workshop were possible to be categorized in terms of the stages of DRM. Wood and Greer (2002) used DRM to classify the papers they reviewed on functional reasoning. An exhaustive set of design research types, using DRM stages and their order of execution as the scheme for categorization, has been proposed in Blessing and Chakrabarti (Reference Blessing and Chakrabarti2009).

Second, the DRM framework provides an overarching, logical order of stages to follow in design research. For instance, if the relationships between the intended area of focus for research (say manufacturability of products) and success criteria (say, customer satisfaction) is not yet understood well, it is not advisable to start developing support to improve manufacturability of products with the goal of increasing customer satisfaction. That is, unless the understanding (as developed in DS-I) is adequate, one should not start carrying out support development (PS). Comparison of the current status of research in the area of focus with the possible design research types helps determine which research type is logical to follow in a particular research.

Third, DRM aims to address each major stage of design research systematically, by providing, for each such stage (e.g., RC, DS-I, etc.), a specific methodical process for carrying out that stage. A list of alternative research methods for use in each major research stage and guidelines for their selection to address various questions and hypotheses have also been provided.

DRM has already been evaluated for its efficacy in various ways. The developers of DRM and their research students have found it useful in their own research, both for reviewing literature and for structuring the research process (e.g., see Nidamarthi, Reference Nidamarthi1999; Ahmed, Reference Ahmed2001; Chakrabarti et al., 2004; Sarkar, Reference Sarkar2007; Vijaykumar, Reference Vijaykumar2009). As assessed from the feedbacks of the European Summer School participants, DRM has been found useful by its participants (Blessing & Andreasen, Reference Blessing, Andreasen, Clarkson and Huhtala2005). DRM has also been used to structure research proposals (Blessing & Chakrabarti, Reference Blessing and Chakrabarti2009).

4. OBJECTIVES OF THE DRM COURSE AND APPROACH TO ITS DEVELOPMENT

The overall objective of the DRM course is to help students understand the design research process, and a variety of approaches and methods on how to carry out design research systematically. This course also aims to help students to carry out the Design and Society course more systematically and rigorously than currently possible.

The specific objectives are the following:

• • to teach a course on methodology for carrying out design research to postgraduate students and

• • to teach DRM effectively (to help assimilate and apply DRM and associated materials to research problems) and efficiently (in reasonable time and effort) as part of the course.

The overall approach followed to develop and teach the course has been the following:

• • Develop the course based on DRM material, now published in Blessing and Chakrabarti (Reference Blessing and Chakrabarti2009): The course was first introduced in 2002, using earlier papers and unpublished material.

• • Teach the course: It has been offered to seven batches of over 30 Master's and PhD students.

• • Evaluate the course: Informal discussion with students after the course; formal, end of course evaluation forms from students; and analyses of student grades are used.

• • Improve the course: Based on the feedback obtained, various modifications have been made.

5. FACTORS TO CONSIDER

The vuDAT guidelines of Michigan State University (http://vudat.msu.edu/design_factors/) recommend the following factors to be considered for designing an effective course:

1. 1. Motivation for the student: Why learn this? Where and when is this used? What are the payoffs for learning? This is discussed in Section 6.

2. 2. Motivation for the teacher: What are the reasons for developing this course? How can you create the best learning experience for the students? This is also discussed in Section 6.

3. 3. Teaching material: What teaching material is to be provided? This is discussed in Section 7.

4. 4. What approaches are undertaken to enhance its assimilation?

   • • Pedagogical considerations: What pedagogical models and learning theories will be incorporated into the teaching? These are discussed in Sections 7, 10, and 12.

   • • Orientation: Help the student adjust to the environment or content being taught. This is discussed in Section 7.

   • • Information: The content the student needs to master. This is discussed in Section 7.

   • • Application: How will students demonstrate learning? This is discussed in Section 7.

   • • Evaluation: Assessment of what the student learned, the relevance of the content, and appropriateness of the instructional method. Sections 8, 9, and 11 discuss these.

5. 5. Technical competency: Is the instructor comfortable with the technology used to deliver the course content? Will he need training or support? Because the course was delivered using Microsoft Powerpoint^TM presentations and white boards, no training or support was needed.

According to these guidelines, the efficiency issues to be addressed are, How efficient is the course for the students and how efficient is it for the teacher? Both are discussed in Section 9.

6. MOTIVATIONS OF STUDENTS AND THE INSTRUCTOR

This course is considered by the department as an essential input for its research students and is therefore offered as a core course for these students. After attending the course, students are expected to be able to apply DRM to carry out their own research in a systematic way. For MDes students, the motivation to attend this course is not as strong, because their career goal is to become designers rather than design researchers. The moderate motivations are the ability to use DRM for research in the initial stages of design, exposure to developing design support as a product, or exposure to research as a career. The experience of teaching this course justifies this: the ratio of the number of research students to MDes students in the course has been about 2:1.

A major motivation for the instructor for developing this course has been the lack of regular courses, and until recently, lack of textbooks on “how to do design research.” It was unclear what the “best learning experience” should be, except that teaching a subject as complex and open-ended as design research should use real research examples, situated research problem solving as a hand-holding journey with experts, and peer learning and peer and expert feedback.

7. TEACHING MATERIAL AND APPROACH

This section gives an overview and details of the course material and the approach to its delivery.

7.2. How the course is taught and why: An overview

The structure of the course is shown in Table 1. In this 15-week course, each week has two time slots of 2.5 h. In the first 12 weeks the course material is taught and associated exercises are carried out; the last 3 weeks are spent on an individual student project as part of their evaluation.

Table 1. Distribution of time to the teaching modules and project

Traditional theory-based courses with little laboratory component are taught using model I (Fig. 2). Every few lectures will be followed by working out examples in the class. In pure project-based courses (e.g., in a research-based Masters thesis) students will carry on a portion of the project on their own (described in Fig. 3 as “assignment within project”) and obtain feedback from the instructor/supervisor based on presentation or a write up. A theory-based course with project component typically will follow model III (Fig. 4) where theories are taught through the course using examples and exercises, and at some point (typically at the end) there is a project to apply all taught material. In a typical theory course with laboratory component (Fig. 5), the instructor presents a portion of the theory, uses some examples and exercises to illustrate the theory; students then work under instruction and guidance of the teacher to carry out the corresponding laboratory component, and often write a report on their findings that will be evaluated by the instructor. This cycle continues through the course.

Fig. 2. Model I: teaching in typical theory courses.

Fig. 3. Model II: teaching in typical project-based courses.

Fig. 4. Model III: teaching in typical theory courses with a project component.

Fig. 5. Model IV: theory courses with a laboratory component.

The model followed in the DRM course (model V, Fig. 6) is similar to model IV. A portion of the theory is introduced in the class, followed by illustrative examples. Instead of a laboratory, here a research assignment that is relevant for the theory introduced is taken up in the class, where the instructor acts as a coresearcher and research leader to work with the students to take the research assignment forward. The distinction between model V followed in DRM and model IV is that in model V, this assignment is completed not in the class, but by the students working outside the class, who in the next class would individually present his/her portion of the assignment (even when done in a team). The presentation in both content and form is evaluated by the instructor and the other students; if the assignment is yet to be completed, it is taken further forward by the instructor along with the students. The cycle goes on until the assignment is completed. Then the instructor introduces the next portion of the theory, followed by illustrating with examples, and so on, as before. The other feature of this model is that for much of the course (except in the literature review module) the assignments are part of a running project, where the same research problem is explored through the various stages of the DRM. Typically, a theory portion, an associated assignment and presentation are completed each week.

Fig. 6. Model V: the teaching model used in the DRM course.

Theory portions are taught using traditional lecture methods; the presentation slides are made available to students after each class. Probing questions are used as a way of gauging attention and superficial understanding of the students during the lectures. Working together with students on the running assignment is carried out in a dialogue with students, and using a chalkboard. The students complete the assignment outside the class typically in teams of three, and often use chalkboards, PowerPoint presentations, and discussion as the means. The presentation of completed work is presented individually (but as part of the team) by each student, using a PowerPoint presentation. Both the students and the instructor give feedback, and together they engage in further dialogues to modify the content as necessary subsequent to the discussion.

The unique features of the model of instruction used are the following:

• • The teacher works as a coresearcher and research leader to take the students through a common research project as a linked set of assignments that run through all the DRM stages; the assignments act as running, hands-on applications of the theories taught. Use of the same project across all DRM stages demonstrates continuity and development of research results.

• • The model uses the following cycle (Fig. 6): (1) the instructor presents a theory with illustrative examples; (2) the instructor and students work together in the class on a running assignment to apply the theory; (3) the students progress or complete the assignment outside the class and prepare a presentation; (4) the students present their progress in the next class before working together further; and (5) they proceed to step 4 or 1, depending on the status of completion of the assignment.

• • The problems for class project and final project remain unchanged each year, but a different set of research papers are used as literature for carrying out the assignment in each year.

• • Presentation on assignments is used as a central element in the learning process to enhance student presentation skills (essential for a researcher) and as a focus for discussion feedback.

• • Discussion and team-based learning is encouraged through discussions in the class, assignments outside the class, and feedback on presentations by both students and instructor.

• • Questioning is encouraged as another central element of research, and each student is encouraged to question the ideas presented in lectures, presentations, and discussions.

• • Developing skills for reading research papers, which is another central element of research, is encouraged through review assignments, running assignments, and a final project.

• • Application focus is a unique feature of this course, encouraged through class project assignments, involvement in real research via method assignments, and the final projects.

8. EVALUATION OF PROGRESS OF STUDENTS IN THE DRM COURSE

Students are evaluated in two ways: informal evaluation and formal evaluation. The purpose of informal evaluation is to give students feedback for improvement only; these evaluations are not marked. Formal evaluation is used for feedback for improvement as well as for marking.

Students are formally evaluated on a total of 100 marks. Fifty marks are on student performance in assignments and class tests; typically 30% are on six assignments distributed to the first four modules as shown in Table 2, and 20% are on two class tests during the first 12 weeks. The remaining 50 marks are on a final research project carried out individually by each student over the last 3 weeks of the course. The marks are converted to a grade using this grading system: S (8 out of 8 points, meaning “outstanding” performance) through A–D (7–4 out of 8, meaning “excellent,” “very good,” “good,” and “satisfactory” respectively), to F (3–0 out of 8, meaning “fail”).

Table 2. What is learned in the course and how it is evaluated

Note: K, knowledge; C, comprehension; Ap, application; An, analysis; S, synthesis; E, evaluation. According to Bloom's taxonomy (Reference Bloom1956).

The following are what the students are evaluated for in the course:

• • Grasp of basic concepts in design research: This includes concepts such as design, design research, research methods, design research methodology, research questions/hypotheses, and evaluation.

• • Grasp of various design research methods, their combination into alternatives for carrying out research, their evaluation for suitability, and their application in realistic research projects.

• • Grasp of the research processes: overall DRM process and processes for each DRM stage.

• • Ability to present and defend results: This includes guidelines for presentation, for example, making slides, preparing a talk, and defending research results.

• • Ability to document and publish results: This evaluates how well the student uses the material taught on how to document research into a research report.

The evaluations are carried out using the following means (see Table 2).

• • Questions/discussions in the class during lectures: This is used for informal evaluation of students’ understanding of basic concepts, for example, research process, methods, and so forth, and relating these to examples. The evaluation is used to test progress and give feedback for improvement. Using Bloom's (Reference Bloom1956) taxonomy of cognitive dimensions, this constitutes (informal) evaluation of knowledge and comprehension of materials taught in the course.

• • Class tests: Two class tests are taken in the DRM course, where students individually answer, in a brief and to the point manner, 15 questions in an hour, as a test of their knowledge of the concepts, processes, and methods taught and how well they can relate these to simple examples. Using Bloom's (Reference Bloom1956) taxonomy, this constitutes (formal) evaluation of knowledge and comprehension of the concepts, processes, and methods taught in the course.

• • Literature review and running project assignments: Literature reviews are done individually by the students, whereas running project assignments are done in teams. Each student makes a presentation on each assignment. Each student is formally evaluated for his/her ability to apply the basic concepts and processes taught to research (about methods, processes, etc). The students are informally evaluated for and given feedback on their enthusiasm, skills for presentation, and defense of their work, and for teamwork and individual effort. Using Bloom's taxonomy, this constitutes evaluation of the application of materials taught in the course. Because students analyze research methods, combine them into alternative ways, and evaluate them before selecting the most appropriate ones for use in answering or evaluating specific research questions or hypotheses, for research methods, the evaluation additionally encompasses the cognitive dimensions of application, analysis, synthesis, and evaluation.

• • Method assignment: In this assignment, each student applies at least one empirical research method to carry out a part of the project of an existing research student. This is formally evaluated to assess the student's in-depth application of that method. Using Bloom's Taxonomy, this constitutes evaluation of the application of materials taught in the course.

• • Individual final project presentations, slides, and report: During the last 3 weeks of the course after completion of all lectures and assignments, students individually carry out a small research project. The same research problem and the same set of research papers are given to each student for carrying out the project. Students are asked to use only the given papers for literature review to construct the outcome of their understanding of the “current” design situation (i.e., carry out a review-based DS-I and develop a reference model); use the conclusions from DS-I to develop a support to address some of the issues raised in DS-I, and develop an impact model describing the desired, improved situation as a result of using the support (comprehensive PS); and develop a plan of how they would evaluate the support to assess the influence of the support in attaining the improved situation. The same research project is given each year, but the four papers given are changed. Students individually meet the instructor each week to present their progress, to ensure they work on a regular basis on the project. Based on the presentation and report, the research is evaluated for its quality of results, documentation, and presentation. This tests each individual student for application of all course materials in a realistic project. Using Bloom's taxonomy, this constitutes evaluation of application of all the materials taught in the course.

9. ASSESSMENT OF THE EVALUATION METHODS USED IN THE COURSE

This is assessed using three metrics as discussed below.

10. MODES OF LEARNING IN THIS COURSE

How does learning take place in this course? The major instruments of learning used are by applying: from the act of applying via individual assignments, for example, literature survey, team assignments, individual project, and method assignment; in teams: from other students, in assignments, projects, and so forth; with experts: from instructor in running assignments, and from research students in the method assignment; with DRM as a framework for design research: from the stages, steps, guidelines, and methods of DRM, because in every assignment carried out, DRM is used as the guiding framework; learning from presentations: from the immediate feedbacks from instructor and other students, both as questions and proposals for improvement.

11. EVALUATION OF THE DRM COURSE

The evaluation of the course is carried out in three ways. The first source of evaluation was a face-to-face feedback: after the DRM course is completed, the instructor arranges a group discussion with the outgoing students. The students are asked to respond to these questions: What did you find useful in the course and wish to be retained in the next year and why? What did you find not so useful and wish to be modified, why, and how? Although the approach is potentially unreliable in commenting on highly negative aspects of the course or the instructor (as the students may be reticent in criticizing the course or the instructor), it usually provides more detailed feedback on both positive and moderately negative aspects. The following are some responses.

• What should be retained? The high degree of interaction between instructor–student and between students themselves; one-to-one discussions between instructor and students; presentations followed by discussions and dialogues, particularly because of the real-time evaluations and modifications suggested by the instructor during these; project assignments, because of their practicality and interestingness which are of current interest; the assignments on reviewing papers, the effectiveness of midterm tests for reminding students of concepts that should be known well, the enlightening subject matter for students of design, and the structure of the course.

• What should be modified? Final project should be given much earlier; some students felt that as various stages of DRM were taught they should carry out corresponding parts of the final project (it was, however, explained to students that this would potentially clash with the running project assignments; the reason the final project is given after introduction of all teaching material is to give an opportunity to students to carry out the project with knowledge and experience of carrying out the entire research process via the running project); rather than giving all students the same running or final project, different aspects of the project should be given to different students, so that a variety in what is carried out by each individual student is increased (it was pointed out to them that this would reduce the amount of learning that each student can currently have because of sharing of the same projects across all students, which enables comparison of their literature reviews, research methods, results, etc.); some of the slides of the instructor are highly textual (this comment has been used to modify the specific slides and associated lectures); rather than stopping at some point during PS in terms of support development, it would be nice to carry out the project in enough detail to be able to develop and evaluate a support (this is a nice idea that has so far not been implemented because of time constraints); the course is very intense (note that this comment and the comment immediately before are in some sense contradictory, which can be because of different people making the comments and the difference in their level of commitment to the course; however, this prompted the instructor to check if the number of hours spent by the students in the course is greater than the maximum allowed, and it was found to be within limits); please procure appropriate reference books for the library (this is a valid comment; now that a book on DRM has been published, this should be possible to address); excellent course even though at times the student who commented this felt he was being subject to interrogation while giving his presentation (the latter part of the comment made the instructor aware of the intensity of questions and feedback during the presentations, and a discussion was undertaken with subsequent batches to ensure they took the questioning in the right spirit).

The second source of evaluation was via a standard, anonymous evaluation form through which each course offered in IISc are evaluated by the outgoing students of the course. The instructor has access to these only after the student grades for the course are finalized and submitted to the office. These forms are long (one student commented on this form: “This questionnaire is tiresome”), often not filled by many students, and are easier for producing statistical conclusions about the course and the instructor than about open-ended evaluation. However, if a student is highly dissatisfied with some aspects of the course or the instructor, this is vented out through comments made in the form. In this sense, it nicely complements the feedback obtained from the face-to-face discussion discussed above. The form has two sections. The first section is on the quality of the course material and how well it is carried out. The specific elements under this section are: test standards, coverage of the syllabus, course organization, how well fundamentals are taught, how up to date the topics covered are, whether text books are available on the course, whether tests are taken regularly, and their usefulness. In this section, the DRM course consistently received an average rating between 4 and 5, where 5 in a scale of 1 to 5 is considered perfect. The second section has a set of questions on how good the instruction quality was. The specific elements under this section are: pace of teaching, command of the instructor on the subject, clarity of expression, level of preparation, quality of interaction, and accessibility of the instructor. In this section also, the course consistently received an average rating between 4 and 5. However, one comment consistently made by the students has been the lack of availability of textbooks that can be used for the course, as also found in the comments made by students in the face-to-face discussion elaborated before.

The third source of evaluation was a comparison between the average grades in the Design and Society course obtained by the MDes students who were taught DRM formally through the course (as well as those who were taught DRM informally, as part of being supervised by the instructor of the DRM course, in the research projects carried out by them in the Design and Society course), with students who were not taught DRM at all. This was to see whether knowledge and application of DRM made a positive difference in the performance of the students in the subsequent Design and Society research course. It was found that, among the 30 students in a span of 5 years who took the Design and Society course, those who learned DRM had a grade point average (GPA) in the Design and Society course of 6.69 (in a scale, as discussed before, of 1–8, where 6 is considered very good, 7 excellent, and 8 outstanding), whereas those who did not learn DRM had a GPA of 5.75 in the Design and Society course, indicating that knowledge of DRM may have had a significant positive effect on the quality of their research. This is further strengthened by the fact that the average overall GPA, across all courses in the MDes program, for those who learned DRM was 6.68 (i.e., they did slightly better in their Design and Society research project than their average performance in all the courses undertaken in the 2 years of their MDes). In contrast, those who did not learn DRM had their average program GPA at 6.33 (which means that they did much worse in their Design and Society research project than their average performance in all courses taken together in the 2 years). There is also a strong correlation (0.82) between the grades obtained in the DRM course and those in the Design and Society research project (p < 0.02).

This correlation is particularly significant in the context of the number of papers in refereed international conferences and journals that came out of the Design and Society Projects taken up by the students. Out of over 30 students who took up the Design and Society research project, 5 put in an additional average of 6 weeks work over and above their Design and Society coursework effort and went on to publish a paper in an international conference or journal (4 in International Conference on Engineering Design conferences, and 1 in Journal of Engineering Design). Because the students involved in these papers had directly or indirectly used DRM in their approach, their success may have been partly because of the efficiency that following of DRM brought to their research.

12. RELATED WORK

The content, structure, and methods of evaluation of the DRM course are related to existing work.

13. SUMMARY AND CONCLUSIONS

This paper highlights some of the major issues about the status of design research. It describes DRM, a methodology developed earlier to address some of these issues, and details the instructional material, model of delivery, and evaluation of a credited, postgraduate DRM course designed to train students in using DRM for carrying out their research.

Overall, the DRM course is a unique one; the author is not aware of other regular courses offered in training students in design research. The evaluations to assess the effectiveness of the course seem positive: responses from students are highly positive, their grades in this course correlate well with their grades in a subsequent research course, and those who use DRM in their research seem to do in the research course not only better than those who do not use DRM, but also better than their average grade in their whole Masters Program. The author's research students, who had earlier undertaken the DRM course, regularly use DRM in their work and find this useful. On the whole, the course seems to have been successful in imparting some basic knowledge for carrying out design research among students who never did any research before.

However, there are some limitations in the way the current course is structured and conducted. Because of the size of the department, the course currently needs to handle only small batches of students. If it were to be offered to a larger class, scaling would be an issue because of these:

• • Scaling the theory and example portions are not an issue, but carrying out a running assignment with a much larger class would be an issue, because for collective progress in the assignment, the views of all students should be brought together into a coherent whole.

• • Individual student presentations and feedbacks become an issue, as this can be very time consuming for a single instructor; perhaps the number of instructors have to be increased.

• • Project presentations and evaluation would require a lot more time and major restructuring.

There are other limitations. In the running assignments, students do not carry out a full-scale study for developing understanding, support, or its evaluation; in the method assignment, students are exposed in depth to a single research method only; and there is difficulty in finding appropriate example research problems for use in the short span of the course, which are sufficiently open-ended for students to see the richness and ill-structured nature of research problems.