Sample Bioethics Question

To demonstrate how a rubric would be constructed in bioethics, consider the following question, which might appear on a master’s-level bioethics examination. Note that four “domains” are marked in the question for future reference.

Throughout the remainder of this manuscript, we will refer to this question in order to demonstrate how to apply the theoretical principles of rubric development and interpretation to bioethics education.

General Principles of Creating a Scoring Rubric

There are various designs for scoring rubrics, but we focus only on a common matrix-style grid that exemplifies the key features and benefits of using a rubric. The grid uses horizontal and vertical axes with one construct shown on each axis. Cells on the grid represent the intersection of those two constructs. “Descriptors” are text descriptions of student work that would be characteristic of that point of intersection of the horizontal and vertical constructs.

When designing a matrix rubric, the matrix is commonly laid out with the criteria (i.e., assessment domains) listed on the vertical axis and the levels of performance on the horizontal axis. The criteria may be listed in any order; however, the levels should be ordinal by increasing performance quality. For our example, the criteria are as follows: Defense 1 (Warren), Defense 2 (Singer), critique, and implication. The horizontal axis follows the SOLO categories^{Footnote 20} from left to right: prestructural, unistructural, multistructural, relational, and extended abstract. Each cell formed by the intersections of the rows and columns should contain a short description of the characteristics of an answer related to that criterion (row) and of that quality (column).

The decisions that need to be made during rubric development are as follows: What criteria should be used in the assessment domains? How many performance levels should be specified? What are the defining characteristics for each performance level in each domain, and how will those characteristics be stated (i.e., how should the descriptors be worded)? Careful consideration of these three decisions is critical in order to create a rubric that achieves its goals of producing consistent scores for student work and that reinforces the learning outcomes of the educational program.

Rubric Domains

The assessment domains should be discrete, so that as raters make judgments about various attributes of students’ work, they are clear about in which domain those attributes should be reflected. For our purposes, the easiest way to accomplish this was to ensure that both students and faculty were clear about the topics students should include in a thorough response. We achieved this by adding parenthetical references to indicate the discrete domains of the question, as marked in the preceding sample question.

Based on the principles of structure described previously and the sample question we posed, the assessment domains in our rubric would be as stated in the example in the previous section. Note that each part of the question represents a unique domain for which a performance level must be assigned. Identifying the relevant domains is significant work. In our case, the process helped us to re-evaluate what we were truly asking of the students and often prompted us to rework the question itself. In this way, rubric creation is often bidirectional—careful attention to scoring requires careful attention to wording the question in a manner that minimizes misinterpretations by students or faculty.

The number of assessment domains in the rubric depends on the purpose of the assessment. For example, a holistic rubric with a single domain may serve well when the assessment is intended to provide a global assessment of the student. If the assessment’s goal is to provide students with feedback on strengths and weaknesses, then an analytic rubric with a larger number of domains is appropriate. As a guide, analytic rubrics typically employ three to five discrete domains. Aggregating scores from multiple domains into a single grade is discussed later in this article.

Levels of Performance

To develop a rubric for assessment in bioethics education, the SOLO taxonomy can straightforwardly be applied to each of the domains being assessed. For example, a student who demonstrates knowledge of some bioethics concepts would be at the unistructural level. A student who knows basic concepts and how those concepts can be applied in various cases would be at the multistructural level. If a student can consider a case and explain how she or he would apply bioethics concepts and reasoning to the case and can also demonstrate how those concepts connect to each other, then that would be at the relational level of performance. The extended abstract level would be achieved only by a student who can extend concepts and applications to hypothetical bioethics situations that have not previously been considered, who can recognize the broader implications of her or his claims and reasoning, and who can critically evaluate the validity of her or his own claims and reasoning.

The levels of student performance, shown on the horizontal axis of a scoring rubric, should be ordinal but not necessarily a scale. Whether the lowest level is placed at the left, with increasing levels toward the right, or vice versa, with the highest level at the left, is a matter of preference. There is one major caveat for defining the performance levels: we recommend that the levels not be assigned numeric values. Granted, the levels are ordinal; however, associating a number with a level imposes arithmetic relationships among those levels. For example, a paper rated as being “level 4” is twice as good as one rated as “level 2.” The temptation for rubric users to overinterpret numeric results is almost inescapable. The descriptors used in rubrics for bioethics education are composed of qualitative observations. Imposing quantitative principles on qualitative characteristics is not justifiable and leads to confusion or misinterpretation.

Domain-Specific Descriptors of Performance

The descriptors that are written for the cells of a rubric are the core of the rubric’s value to educators and students. These descriptors exemplify the traits that should be evident at each performance level within each domain. In order to realize their full value, descriptors must be specific and evident and not contain judgmental terms such as “good” or “excellent.” (It should be emphasized here that overall evaluation of students’ performance, when judgments such as “good” or “excellent” are appropriate, ensues after measuring students’ abilities using the scoring rubric.) Descriptors should be concise and should identify observable features of student performance. These features should be stated in precise language such that raters can readily find evidence of the features in the student work—or can note the absence thereof. The reliability of a rubric as an assessment tool depends on the consistency with which raters score student work. Although training the raters might increase reliability, we contend that well-written descriptors are vital to ensuring that raters draw similar conclusions about the quality of students’ work.^{Footnote 21,Footnote 22}

For example purposes, Table 2 demonstrates what descriptors could look like for one of the domains of assessment that we previously introduced. In the same manner, faculty would need to compose descriptors for each of the other assessment domains for the question. Note that in Table 2 we include very specific descriptors throughout, including itemizing the components necessary to constitute a complete answer. Doing so minimizes confusion among graders, because it provides them with an exact list in the rubric of what is expected. An added benefit is that disagreements among faculty members regarding a “correct” answer are negotiated during rubric construction rather than while trying to score students’ responses. It is impossible to anticipate all of the ways in which students’ answers may deviate from what is expected, but we also highlight both the likely deficiencies and the elements of excellence in students’ answers, so that faculty will recognize where an answer belongs on the rubric scale.

Table 2. Bioethics Rubric with Minimum Pass and Student Feedback Information

Note. The far left-hand column lists domains for assessment. The top row lists categories of student performance. Descriptors of performance for each level of each domain are shown in the remaining cells. The shaded areas represent the categories that we considered to be failing performance.

Building on the matrix model described previously with criteria listed on the vertical axis and SOLO levels of quality on the horizontal, we propose the following descriptors for the first domain of assessment (Defense 1 [Warren]):

• • Prestructural: Fails to identify any of the criteria for personhood; confuses Warren’s argument with others’.

• • Unistructural: Recognizes that Warren’s theory depends on constitutive traits; identifies two or fewer traits correctly.

• • Multistructural: Correctly describes at least three traits of Warren’s theory: (1) consciousness and the capacity to feel pain, (2) reasoning, (3) self-motivated activity, (4) the capacity to communicate, and (5) the presence of self-concept and self-awareness.

• • Relational: Describes all five traits of Warren’s theory and describes the relationships among those traits.

• • Extended abstract: Discusses contemporary implications of Warren’s theory (such as findings about consciousness) and presents hypothetical situations to test applications of that theory.

As mentioned previously, although we do not provide detailed examples here, descriptors for each of the other criteria and levels of performance need to be composed and agreed on by the faculty.

We readily recognize that some readers may disagree with the domains or descriptors in our rubric. In the opinion of some, we may have failed to include the most important features of an answer, or we may have included too much, or we set the descriptors at the “wrong” performance level. Yet this is precisely the point of creating a rubric through a consensus process. The only acceptable rubric for any individual or program will be the one that is congruent with the program’s curriculum and outcomes. We have provided an example here simply to demonstrate the process of applying a theoretical framework, not to suggest a singular, definitive instrument for use in all contexts.

Score Interpretation

Up to this point, we have presented the process for developing a scoring rubric that will measure student performance in a valid and reliable way. To complete the educational assessment process, student performance must be interpreted and a decision must be made about the students’ achievement of desired outcomes. After students’ products have been scored using a rubric, those scores need to be interpreted, often using a scale of A-B-C-D-F or simply pass/fail. The interpretation of scores has a significant impact on the students’ educational trajectory, which is a compelling reason for dedicating a substantial amount of time and effort to developing a valid and reliable rubric for scoring students’ work and interpreting their scores. For example, the descriptors presented in the five levels suggested by the SOLO taxonomy give clear and specific guidance to the reader regarding into which performance level an essay should be categorized.

As stated previously, we advise against assigning numeric values to performance levels on a rubric. Although it is not uncommon for raters to assign numbers using a rubric, convert those numbers into percentages, and then interpret the percentages as letter grades, we strongly advise against this practice. We noted previously that doing so requires the assumption that the descriptors for performance levels represent arithmetic relationships (e.g., half as good, 25% better, etc.), and this assumption is practically impossible for the nonquantitative outcomes of bioethics education. By way of contrast, a rubric used to score a musical performance in which one domain is “intonation” could maintain arithmetic properties for performance levels by using descriptors such as “all notes were in tune” and “90% of notes were in tune.”

Score interpretations could be made using common grading scales, such as A–F, or categorically, such as categorizing a student’s ability as “emerging,” “proficient,” or “outstanding.” Because our goal was to assess a high-stakes examination, we were most concerned with a pass-or-fail decision. The principles of setting a cut score, described subsequently, may be extrapolated for use in discriminating among other categories as well.

Setting Cut Scores

Interpreting rubric measurements in terms of grades, particularly pass/fail decisions, involves setting a minimum standard for performance.^{Footnote 23} We will refer back to the SOLO taxonomy to provide a theoretical basis for setting minimum standards appropriate to bioethics education. A basic tenet of setting the “cut score” for minimum passing is that the cut point should be congruent with the stated outcomes of the course or program that is being assessed. Because bioethics programs are usually offered at advanced baccalaureate, professional, or graduate school levels, faculty will likely consider minimum acceptable performance to be at the multistructural level or the relational level of the SOLO taxonomy. This is illustrated by the shaded areas on Table 2. The choice of cut points should be made and justified by each program in consideration of the program’s outcomes and student population. Techniques and additional considerations are described elsewhere in excellent detail.^{Footnote 24} For our own exam, we set the minimum passing standard at the multistructural level. Thus, a student scoring at the multistructural level would pass, whereas a student at the unistructural level would fail.

The most important element to successfully using a rubric is to write descriptors that precisely state the characteristics that distinguish minimally passing work from failing work. Apart from making that critical distinction, the number and arrangement of descriptors may be varied according to the purposes of the assessment. Thus, there are two approaches to aligning cut scores with descriptors. On one hand, if the assessment is strictly to identify substandard performance with no concern for discrimination between excellent work and minimally passing work, then only two performance levels are needed, along with the descriptors for those two levels. Specifically, only descriptors for the borderline between the lowest passing level and the highest failing level would need to be written. To create such a rubric from our example, only the titles on the horizontal axis would change. Rather than using all of the SOLO categories as titles, only two columns—failing and minimally passing—would need to be included. Given our passing standard elucidated previously, the rubric would retain only the criteria and descriptors in the unistructural and multistructural columns.

On the other hand, if the faculty has already done the work described previously using the SOLO taxonomy, then they simply need to reach consensus about which category is the lowest acceptable level to pass the exam. An educational advantage of using this full-scale approach is that, in addition to making pass/fail decisions, faculty can provide feedback to the learners about relative areas of strength and weakness. To construct such a rubric, at a minimum, descriptors must be defined for the extremes of each domain as well as for both sides of the cut point between pass and fail. Any additional points on the quality scale could be left for interpolation by the rater.

Aggregating Scores from Multiple Domains

Using analytic rubrics that have multiple domains requires one more step for interpretation than is needed for holistic rubrics. In order to represent a student’s overall performance with a single score, such as a grade, the student’s performance on the various domains of measurement needs to be aggregated. This can be done using either a compensatory approach or a conjunctive approach. In the compensatory approach, a student can compensate for low performance in one domain with high performance in another. In the conjunctive approach, the student must achieve the minimum standard on each domain in order to pass the overall assessment. The decision on which model to use depends entirely on the purpose of the assessment and the consequences, such as participation in a remedial program, that are dispensed to the students according to their results.

For our own comprehensive exam,^{Footnote 25} students responded to several questions with a separate rubric created for each question. We used a compensatory approach when aggregating the domains of an individual question, so a student who did poorly in one domain could recover with a high score in another domain for that question. But we used a conjunctive approach to arrive at a pass-or-fail decision for the exam as a whole. Thus students needed to achieve a passing score for each question they answered.

Additional Considerations for Rubric Creation

Creating a scoring rubric is a laborious process. Making decisions about the student attributes that should be rewarded by the program will bring to the forefront discrepancies in values among faculty as to what a graduate should know, be, and do. Creating a scoring rubric to be used for high-stakes, program-level assessments forces faculty members to confront their own disagreements and negotiate common ground. In essence, creating the rubric forces faculty members to discuss and compromise on these differences, differences that otherwise could fester or flare up into departmental infighting. We propose that bringing such disagreements to the forefront, as is necessary to develop a scoring rubric, is not always pleasant but is more collegial and better for students and the program as a whole than compelling students to continually adapt to the varying expectations and standards that may be held by various instructors.