III. Findings
We received responses from a variety of different sized departments, with varying numbers of majors, and from all of the six regional accrediting bodies operating in the United States. Before reviewing the characteristics of responding departments, which are summarized in Table 1, it should be noted that the researchers suspect a bias in the pattern of responses. The perception is that departments that have been at least somewhat focused on learning assessment are much more likely to have returned the survey than departments that have not had such a focus. If you are interested in learning assessment, or believe your department has done good work in this area, you are more eager to participate in a study of this kind. However, this suspicion cannot be confirmed on the basis of the data presented here.
Turning to Table 1, then, the majority of our responses (64.2%) came from departments offering only undergraduate programs. Departments offering graduate programs accounted for 34.4% of our respondents. The largest group of respondents (30.2 %) was departments offering only an undergraduate major and having five or fewer faculty members. Another 18.9% of the respondents were departments where political science is combined with other disciplines. A further 15.1% offer undergraduate programs only but have more than five faculty members. A small majority of our respondents (51.5%) had more than 75 political science majors.
The respondents came from all six regional accreditation areas but with large differences in the distribution among the accrediting regions. A full 33.5% came from the North Central Association of Colleges and Schools. Conversely, as Table 1 illustrates, only 4.7% came from the Northeastern Association of Schools and Colleges. To some degree this reflects differences in the number of four-year colleges and universities that fall within the boundaries of each accrediting region, but it may also reflect the degree to which the accrediting body for the region in question has made assessment of student learning a priority.
The survey asked respondents to indicate whether or not the department had formally adopted a set of specific learning objectives for its students. Just over 16% stated that there was not even a discussion about learning objectives taking place within the department. Another 12.3% said that there was a department-level discussion taking place, 16.1% said that learning objectives were being formulated, 38.9% said that learning objectives had been adopted, and 14.2% said that a previously adopted set of learning objectives was under review.
What is the association between the type of department and that department's stage in this process? The largest single group (17%) of responding departments that had formally adopted (or had adopted and were now reviewing) learning objectives is those departments that offer only an undergraduate program and have five or fewer faculty members. The second largest group (10.5%) is the combined department (where political science is but one discipline under the umbrella of a single department). The third largest group (9%) is the remaining exclusively undergraduate category—departments with more than 5 faculty members. To summarize, 58.5% (or 79 out of 135) of undergraduate-only departments are at the advanced stages of the learning objective development process. Just over 45% of departments offering graduate work are at this same stage in this process. Such a breakdown is not surprising; one would expect departments exclusively focused on undergraduate work to be more focused on the assessment of undergraduate learning, and the survey did not ask questions about the assessment of student learning at the graduate level.
Thus far, the focus of this paper has been on the presentation, more or less, of the learning assessment context. What about the specifics of learning assessment being carried out by the respondents? The data shown in Table 1 begin to illustrate these specifics.
Table 2 presents the percentage of respondents that have adopted the various specific learning objectives listed. Each objective was treated as a separate nominal variable, thus allowing for multiple objectives. The objective adopted by more respondents than any other is “writing skills” at 57.1%. In other words, 57.1% of the responses were from departments that have formally (but see below) adopted the objective “students should develop writing skills.” Critical thinking received the next highest percentage of “yes” responses at 55.7%. Fifty-four percent of responses came from departments that have as a learning objective “familiarity with major theories and analytical approaches in political science.” No other single objective has been formally adopted by a majority of the responding departments. Objectives such as becoming familiar with the major subfields of political science, understanding the international dimensions of political problems and policies, and the ability to design and conduct political science research projects all received between 40 and 46% “yes” responses. Exclusive of the “other” category (8%), the least frequently adopted learning objectives were “develop a fundamental understanding of cognate disciplines” (15.6%) and “students should acquire practical experience in politics or government” (22.2%). This means, of course, that the most popular learning objectives are not those specific to the discipline of political science.
Finally, 18.9% of respondents indicated that the question did not apply to them. The assumption is that this 18.9% came primarily, if not exclusively, from two groups of respondents: those that have not yet adopted any learning objectives and those departments offering only graduate programs. Actually, this percentage seems somewhat low given that a full 28.1% of our respondents indicated that they were not yet even in the formulation stages of developing a set of departmental learning objectives. This discrepancy is probably the result of some respondents answering the set of learning objectives survey questions based upon implied rather than formally adopted objectives (or even the individual respondent's sense of good objectives for political science majors).
A similar set of survey questions focused on the development and type of assessment instruments being used by the respondents. Table 3 shows the percentage of departments using each of the listed learning assessment instruments. Again, each possible instrument was treated as a separate nominal variable to allow for the use of multiple methods.
The assessment instrument used by the largest percentage of departments is the senior capstone course. Over 39% of respondents require such a course for student learning assessment purposes. Twenty-five percent of respondents use (either alone or in conjunction with other tools) faculty observations to assess student learning. The tool receiving the lowest percentage of positive responses (i.e. the department uses the tool) is the pre-test/post-test (9.9%). A somewhat larger group of respondents (14.2%) use a post-test only.
The percentage of respondents (17.5%) who indicated that this set of questions was not applicable to them is particularly interesting. Over 31% of these respondents indicated that their departments were not yet at the formulation stage of establishing learning assessment tools. In order to fully explore this discrepancy, a follow-up study is necessary. However, it is possible to speculate about the reason for this disjunction. Two different explanations might account for this. First, tools which the survey listed as assessment instruments might be in place in a department but not used for assessment purposes in any formal way. Thus, a respondent might have indicated that his or her department was not even talking about learning assessment tools but still might indicate that the department requires a senior seminar (rather than simply marking “not applicable”). Second, assessment tools may be in place and may be used by a department for assessment purposes, but without the knowledge of the department member responding to the questionnaire. This second explanation seems less plausible since the surveys were sent to department chairs but must be noted as one possibility.
The final set of closed-ended questions included on the survey concerned the availability of resources to support departmental learning assessment efforts. Overall, 5.2% of responding departments stated that there were substantial resources available to support learning assessment, 32.5% said that there were some resources available, and 31.6% stated that there were a few resources available. The final 25.9% of our respondents stated that there were no resources available for this purpose. Oncampus workshops are the most widely available (39.6%) and widely utilized (25.9%) resource in support of learning assessment. Off-campus workshops and on-campus teaching centers follow with 29.7% of respondents stating that off-campus workshops were available and 28.8% indicating that on-campus teaching centers were available. However, only 17.9% of respondents indicated that they or their departments had utilized any off-campus resources, and only 16% stated that the available on-campus teaching center was utilized.
Interestingly, but perhaps not surprisingly, the least widely available types of resources, course releases and financial rewards, were also the most likely to be used by departments. The full 5.7% of respondents who indicated that course releases were available also indicated that they used that option. Of the 7.1% of departments that indicated that some sort of financial reward was available in support of learning assessment, 5.2% indicated that they had taken advantage of that resource. In addition, 11.8% of respondents indicated that the issue of resource exploitation was not applicable to them.
In order to further explore the relationship between resource availability and departmental assessment efforts, we conducted two crosstabs. There is a substantial correlation between the amount of resources available and departmental stage in the process of developing learning objectives.
In 91% (10 out of 11) of the cases where substantial resources were available, the department had either adopted or adopted and was also reviewing a set of formal learning objectives. Out of the 68 cases where the respondent indicated that some resources were available, 65% also indicated that they were in one of these two final stages of learning objective development. Conversely, in the 55 instances where the respondent indicated that there were no resources available at all, 38.25% were not even having discussions about learning objectives, while only 18% had actually adopted learning objectives and 13% were reviewing a formally adopted set of objectives.
There is a similar association between the availability of resources and the department's stage in developing learning assessment instruments. One hundred percent of departments (n = 11) that reported there were substantial resources also reported that they had either adopted or had adopted and were now reviewing an assessment instrument or instruments. Conversely, of the 55 departments that reported there were no resources available to support learning assessment, 36.25% stated there was no discussion of assessment instruments, 20% stated there was a discussion but no adoption, 9% stated they were formulating instruments, 27% reported that they had adopted instruments, and finally, 4% reported that they were reviewing a previously adopted instrument or instruments. While no formal causal relationship has been established here, this data does clearly indicate that increased resources in support of learning assessment are likely to have an impact on the development of learning assessment in departments.
Respondents were also asked a series of open-ended questions. These questions asked for summaries of the most significant conclusions departments had reached after analyzing data generated by their learning assessment instruments, and summaries of changes that departments had made in majors, course offerings, or course-level pedagogy as a result of learning assessment. Respondents whose departments have not developed or implemented learning assessment programs generally did not answer these questions. Also, departments in the early stages of developing and implementing learning assessment programs frequently did not respond to these questions. A total of 127 departments (59.75%) provided responses to the open-ended questions.
Departments were asked to indicate the three most significant conclusions they had reached as a result of analyzing data generated by learning assessment. It is striking that departments indicated a wide range of conclusions based on their analyses of data generated from learning assessment instruments. A substantial number of departments (17.5%) reported either that it was too early to report conclusions, or that the question was not applicable to them (4.75%). Fifteen percent of these departments indicated conclusions that either no other department listed or that were not directly related to learning objectives. Such responses were coded as “other.” A few somewhat surprising results appear here. First, although critical thinking is a very common learning objective, few departments have reached conclusions about their students' critical thinking skills, with two reporting satisfaction with student achievement and one reporting dissatisfaction. One might imagine that learning assessment programs would provide more frequent conclusions about such a popular learning objective. Two hypotheses might account for this observation. One, although many departments value critical thinking as a learning objective, few departments have been able to develop instruments to measure student achievement in this area. Second, departments may be evaluating critical thinking skills when they evaluate their students' writing skills.
It is also noteworthy that relatively few departments (4%) have reached conclusions about their students' information technology skills, despite its popularity as a learning objective and the frequency (discussed later) with which departments have adapted courselevel pedagogy to emphasize information technology.
A second open-ended question asked departments to list significant changes they have made in their majors as a result of learning assessment. A large number of departments (37%) report no changes in their major program, either because it is too early in the assessment process to implement changes in majors, because departments have not implemented learning assessment, or because what they have learned from learning assessment has not led to changes. For departments that have made learning-assessment related changes in their programs, a number of results are prominent. First, learning assessment has led several departments either to a general revision of major requirements (19%) or to create new majors, tracks, or emphases (4%). Second, several departments have tried to address perceived student deficiencies in analytical techniques and research methods. Some departments have done this by adding a required methods course or by requiring students to take a methods course earlier, usually as sophomores. Some of the departments (16.5%) adding or revising a senior seminar or capstone course are doing so to address deficiencies in analytical techniques and research methods.2
Of course, senior seminar and capstone courses are popular instruments for assessing student achievement of learning objectives. It is not possible to distinguish whether a department is using a senior seminar or capstone course to help students address analytical and research deficiencies, to assess student learning, or both.
A third open-ended question asked departments to list significant changes made in course offering as a result of learning assessment. For example, had departments added courses to address gaps in student achievement? A first observation is that departments were less likely to report that they had not made changes in course offerings than they were to report that they had not made changes in their major programs. Departments were more likely to report that they had added courses (31.5%) than that they had dropped courses (12.75%). When departments observe deficiencies in student learning, it may be almost a reflex response to add courses to deal with the deficiency. For example, several departments reported that their students had gaps in international relations and comparative politics and added courses in those areas to address the deficiencies. Dropping courses may result from a perception that students overemphasize a subfield, most likely American politics.
Another common response to this question is that departments have added or significantly revised a research methods course. This is consistent with responses to the previous two questions, which also revealed departments' concerns about their students' deficiencies in research and analysis.
Finally, departments were asked to list three significant changes in specific courses. The number of departments reporting that learning assessment has not or has not yet led to change drops off again. It would seem that the results of learning assessment are first realized at the level of the individual course. Generally, this results in a greater emphasis of skills: research and analytical skills (21.5%), writing skills (14%), communication skills (4%), and critical thinking skills (1.5%). The order of priority is consistent with responses to other questions. Departments are most likely to emphasize analytical and writing skills and least likely to emphasize critical thinking skills. Again, the disparity between the popularity of critical thinking as a learning objective and departmental efforts to emphasize critical thinking skills stands out.
Other course-level changes emphasize how courses are delivered. By far, the most popular change in course presentation (24.5%) is by increasing the use of information technology. Responses to this question are unclear as to what extent information technology is being deployed to help students enhance their own information technology skills. The alternative is that using information technology is seen as a useful way to convey content or to help students achieve other learning objectives.
Results from the four open-ended questions indicate ongoing uncertainty about the process of learning assessment and a concern with students' learning skills. Many departments that have implemented learning assessment programs have also implemented changes at the levels of major programs, course offerings, and individual courses. Changes are most commonly directed at more complete coverage of the discipline and development of student learning skills, especially analytical and research skills.
