2.1 Claims for computerised essay-rating programs

Most essay-grading software functions by analysing sentences and paragraphs, looking for keywords as well as relationships between terms. The Intelligent Essay Assessor (IEA), for example, uses latent semantic analysis (LSA) for the major part of its analysis (Foltz et al., Reference Foltz, Kintsch and Landauer1998). Through training with LSA, a matrix of words and documents is produced. Pieces of writing are then scored to see how well they match the matrix.

The Project Essay Grade (PEG) system, having identified elements such as sentence length, number of paragraphs and elements of punctuation, uses regression to determine how well the different variables correlate with the scores of human raters.

E-rater identifies and examines specific linguistic categories such as grammar, usage, mechanics, style, organization, development, lexical complexity, and prompt-specific vocabulary usage. It functions on similar lines to PEG, using regression to predict performance. It compares features in the essays against a bank of sample essays (Attali & Burstein, 2006).

The Bayesian Essay Test Scoring sYstem (BETSY), draws on a broad set of essay features for analysis. After a set of parameters is specified, the system functions by:

• • assigning occurrences of content and stylistic features to a set of super-specified levels, according to probability;

• • determining the probability of features in input essays falling into the levels stipulated (see Rudner & Liang, Reference Rudner and Liang2002).

As BETSY is used in the current study, a more detailed description is presented below.

Several studies have reported favourably on different computerised essay-scoring programs with IEA studies reporting that its scores typically correlated as well with human raters as raters did with each other (Chung & O’Neil, Reference Chung and O’Neil1997). Foltz et al. (Reference Foltz, Laham and Landauer1999) report a study with essays written on psycholinguistics by American university students where a correlation of 0.8 was obtained between the program and human raters. Likewise, studies with PEG consistently reported relatively high correlations between PEG and human raters as compared to correlations between human raters (Page et al., Reference Page, Poggio and Keith1997). Concerning e-rater V.2, Attali and Burstein report correlations with a human rater as high as 0.97 (2006: 22).

In a Graduate Record Examinations (GRE) essays study, Powers et al. (Reference Powers, Burstein, Chodorow, Fowles and Kukich2001) concluded that the e-rater system was not ready for use by itself. In high-stakes assessment situations, it needed to be paired with human raters. Powers et al. (Reference Powers, Burstein, Chodorow, Fowles and Kukich2002) also report significant, although small, correlations between non-test indicators (self-reported accomplishments and success with various kinds of writing, and self-evaluation of writing ability). Weaker relations emerged with automated scores, however, than with human rater scores.

2.2 Criticisms of computerised essay-rating programs

Against the potential advantages of computerised rating programs and their reliability, counter opinions are now considered.

A major criticism is that the computer rating process is a essentially a “black box” (see Weigle, Reference Weigle2002: 118ff). This same criticism was levelled at “general impression marking” procedures of the 1970s and 80s (Hay, Reference Hay1982), since rating criteria were not explicit (Weigle, Reference Weigle2002: 112). The assessment of writing by humans has moved, in the past twenty years, from a single impressionistic, holistic scale to scales and descriptors that more clearly define the constructs being assessed. Clearly-defined domain descriptors allow for clearer inferences to be made about test taker abilities. Consequently, computer scoring programs, while returning high correlations with a human rater, still lack transparency as to how they achieve their grades. They are objected to by some teachers of writing. Drechsel (Reference Drechsel1999), for example, complains that automated essay scoring (AES) systems do not read and understand essays like humans. As Attali and Burstein (2006) comment:

Two further issues need to be considered. First, most pieces of writing are currently produced by students writing by hand, i.e., using a pen. If scripts are to be graded by a computer, they need to be machine-readable. This requires clerical assistance to input the scripts. It is not a cost-free option. Secondly, as with speech recognition, a considerable amount of training needs to take place on the program itself before it is operational. In speech recognition systems, users need to train systems to their voice (Coniam, Reference Coniam1998). Likewise, computer rating programs may require hundreds of scripts on a single topic to develop their own potentially reliable rating metric (Dikli, Reference Dikli2006). This initial priming of the program is only feasible with large examination bodies such as ETS, with access to large numbers of scripts. Training is also a time-consuming affair as a representative set of scripts across a range of abilities needs to be available to be fed to the program for training. In the current study, for example, training BETSY with approximately 200 scripts on each of three essay topics (due to the different iterations recommended) took approximately five hours per topic.

Major studies of computerised assessment of written essays have also been skewed: firstly, there has been a focus on native speaker writing; secondly, rating has tended to examine different subject content areas, e.g. using the GRE; thirdly, many have been with tertiary level students. Warschauer and Ware (Reference Warschauer and Ware2006) provide a useful discussion of the place of automated writing evaluation in the ESL classroom; in general, however, there has been little research with secondary ESL students and automated writing assessment. The current study therefore aims to extend the scope of investigation. Firstly, the data is drawn from the writing of secondary school (Year 11, age 17) ESL students. The data comes from the Writing Test of the 2005 Hong Kong Certificate of Education examination (HKCE). Further, as the HKCE Writing Test consists of three topics (each in a different genre), all three are examined.

Data is presented from a number of perspectives. One focus of analysis is the issue of reliability. In addition. the paper compares the program’s performance on a range of genres, with some discussion given to a brief qualitative analysis of why certain scripts were over- or under-graded.

To restate, the current study is not a validation of a particular computer essay scoring program (i.e., BETSY) per se, but rather an investigation into how well such a computerised rating program performs when rating ESL students writing in different genres.

3.1 The Hong Kong Examination situation

Hong Kong’s major public examination is the Hong Kong Certificate of Education (HKCE) examination, administered by the Hong Kong Examinations and Assessment Authority (HKEAA) at the end of Secondary 5 (Grade 11) with a 2005 candidature for English language of 82,078 (HKEAA, 2006). There are four components in the English language HKCE: (1) Writing; (2) Reading; (3) Speaking; and (4) Integrated Reading, Writing and Listening. The 2005 HKCEE Writing paper offered test takers three prompts. Test takers select one prompt and write approximately 300 words within 70 minutes. Overall grades awarded on the HKCE English language paper were A to C (credit), D and E (pass), F and U (Fail).

The HKCE Writing Test has four subscales and descriptors (HKEAA, 2007: 104–105):

1. 1. Relevance and adequacy of content for purpose

2. 2. Accuracy and appropriacy of punctuation, vocabulary, language patterns

3. 3. Planning and organisation

4. 4. Appropriacy of tone, style and register; appropriacy of features for genre.

The subscales each comprise six levels, ranging from 1 (indicating weakness) to 6 (indicating good ability). A level “0” also exists, essentially indicative of “no or minimal performance”.

Figure 1 (from the 2005 HKCE examination) presents the three prompts:

Fig. 1 2005 HKCE Writing Paper.

Prompt 1 is descriptive, prompt 2 argumentative and prompt 3 imaginative. Offering a choice from a range of such genres is a longstanding practice in the HKEAA’s English language examinations.

3.2 BETSY – the Bayesian Essay Test Scoring sYstem

3.2.3 Output

BETSY produces two major sets of results. The first decides which of three levels a script is assigned: “certainty”, “I think” and “I don’t know”. The possibility of a script falling into each of the five criterion levels is presented as a percentage. Where only a single probability is manifestly high, i.e., above 0.9, BETSY declares “certainty”. Where there are competing probabilities, BETSY opts for the most likely, declaring that she “thinks”. In circumstances where all probabilities are very similar, BETSY declares that she in unable to come to a decision “Therefore I do not know”. Figure 2 elaborates.

Fig. 2 BETSY’s output – degrees of certainty.

Secondly, for whatever features have been selected – words, phrases, arguments – BETSY provides tables with the relative probability that a feature would fall into a particular level as against the default possibility of 20% for each level (in the five-level mode). In Figure 3, level 1 indicates least able, and level 5 most able. Figure 3 presents a sample from Script 240 referred to in Table 10.

Fig. 3 BETSY’s output – relative probability of occurrence of different features.

3.3 Hypotheses

The working hypotheses in the current study are:

1. 1. Ratings obtained from the program BETSY will not reach interrater correlation levels achieved between “good” human raters. In previous Hong Kong public examinations such as the English language HKCEE, good interrater correlations (i.e., 0.8 or above) are desired.Footnote 2

2. 2. Discrepancies between the computer rater and the human raters will be greater than between human raters.

3. 3. There will be no difference between the levels of correlation reached among the three genres of writing, i.e., genre will not influence scores.

3.4 Methodology

4.1 Background analyses – raters and scripts

The HKEAA’s rating scales effectively consist of seven levels, ranging from 0 to 6. Table 1 presents the distribution of test takers’ Fair Average scores across levels for the 300 scripts in each topic.

Table 1 Distribution of scripts across Fair Average levels

The distribution of scores is generally consistent across the three topics, although Topic 3 is slightly skewed towards the weaker end, with fewer test takers scoring at Level 5. Only one test taker obtained a Level 6 score – on Topic 1.

To provide BETSY with comparative sets of training scripts, the optimal would have been an even spread of Fair Average scores across the five levels. This was not the case, however, as Table 1 indicates; so, to provide BETSY with a representative set of scripts for the five criterion levels, the following procedures were adopted.

Approximately two thirds of the scripts identified at the five levels were selected as training scripts. These scripts were selected from the middle of the level range (where they might be more representative of the level) rather than at the boundary points (where they might be closer to a higher or lower level).

Level 0 scripts were excluded from the training, since Level 0 occurred well below the midpoint of the lowest level, Level 1, as such scripts could skew the definition of Level 1. Level 0 scripts were, however, included for analysis (where they would be expected to be rated as Level 1, the lowest level). The Level 6 script was likewise excluded from the training but included for analysis.

Following the decisions above, the sets of scripts were identified for training and analysis. Table 2 elaborates.

Table 2 Distribution of scripts across levels – training and analysis

Table 2 shows just under 200 scripts as criterion training scripts for each topic – 188 for Topic 1, 189 for Topic 2 and 193 for Topic 3.

On the HKEAA writing paper, interrater correlations of at least at 0.8 would be expected in terms of good rater reliability [see footnote 2].

Table 3 presents interrater correlations for each of the three prompts. These were produced as the correlation between the total rating (i.e., a maximum of 24 points) given by each rater on the four rating scales, and are presented in Row 1. Data was also made available by the HKEAA for test takers’ performance on other components of the examination for comparative purposes (such as their performance on the HKCE examination as a whole). Column 2 presents the correlation between raters and test takers’ overall score on the HKCEE examination.

Table 3 Interrater correlations

** = p < .01.

Given the target of 0.8 or better, Table 3 shows acceptable rater levels, indicating a high degree of agreement between them on all three topics. This reliability is confirmed by the raters also correlating highly with the overall subject grade, with correlations in the high 0.8 range or better. All correlations were significant at the 1% level.

High interrater correlations are, however, only one measure of reliability. In fairness to test takers, we must consider amounts of disparity between raters for any individual test taker. In assessing writing, re-marking – involving a third rating – is often required when major discrepancies between raters occur. The usual trigger for this in public examination bodies is differences in raters’ grades by one subscale level or more.Footnote 5

In the current study, the trigger for re-marking was calculated as more than one point of difference on each of the four scales, i.e., 5 or more points out of 24 (1.25/6). Given this potential level of re-marking between human raters in the current study, Table 4 presents the results for the three topics. The data in the Table indicate the number of scripts that fell within the acceptable level of no more than 4 points of difference out a maximum of 24.

Table 4 Potential amount of remarking required

As can be seen, with current raters, re-marking of between 16.7% and 24.3% scripts would have been required, with Topic 1 triggering considerably more re-marking than the other two topics. The results echo those reported by Rudner and Liang (Reference Rudner and Liang2002: 9) for American high school students writing on a biology topic, where an accuracy rate of 80% was achieved between BETSY and two good raters’ results.

We now move to an examination of BETSY’s performance with the HKCE Writing Test scripts.

4.2 BETSY analyses

In examining the three possible decisions concerning degree of certainty, Table 5 presents BETSY’s decisions on the three writing topics.

Table 5 BETSY’s decisions according to degree of certainty

BETSY’s decisions were either definite or likely, with definite decisions emerging in approximately 80% of the scripts analysed for each topic. No uncertain decisions were recorded.

Table 6 presents the correlations between the “expected” grades (the Fair Average level scores obtained from the two human raters) and the grades awarded by BETSY. Note that in the following tables, results are included for all scripts analysed for the three topics (as in Table 5), irrespective of whether the decision was “I am very certain” or “I think”. As all results form part of a rater’s statistics in a public examination (that is, no test taker’s scores could be omitted because the rater was unsure about the student), the same modus operandi is retained.

Table 6 Correlations between Fair Average scores and BETSY’s scores

** = p < .01.

Table 6 shows that BETSY achieves strong correlations at the 1% significance level on all three topics. Correlations at the low to mid 0.8 compares very favourably with the interrater correlations achieved between good raters (included from Table 3 above). Topic 2 correlations are slightly lower than that of human raters but Topic 3 has a higher correlation. The results echo those reported by researchers such as Foltz et al. (Reference Foltz, Laham and Landauer1999) where correlations of 0.8 were obtained between computerised scoring and human raters.

Given that the baseline for re-marking was established earlier (Table 4) as 5 or more points out of 24 (1.25/6), Table 7 below now presents the differences between BETSY’s scores and the Fair Average level scores for all scripts that BETSY rated. In Table 7, a positive score indicates that BETSY was more lenient than the level score derived from the two raters; i.e., BETSY rated test takers higher. A negative score indicates that BETSY awarded test takers a lower score. The last row in the Table incorporates interrater discrepancy levels from Table 4.

Table 7 Differences between BETSY’S scores and calibrated human rater scores

Table 7, with 1.25 points of difference being taken as criterial for acceptability, shows comparable levels of acceptable marking from BETSY as for two human raters. Topics 1 and 3 are in fact slightly higher than their human discrepancy counterparts, with Topic 2 slightly lower. All three topics are very close to the human results, and further illustrate a lower level of discrepancy. Topic 1 would require most third marking, with Topic 2 again requiring least.

BETSY tends towards stricter rather than more lenient ratings. On Topic 1, the strictest prompt, 5.3% of test takers were rated more than two whole levels lower than by the human rater. Between 1.25 and 2 points of difference, BETSY was between 11.7% and 18.3% stricter. Only a few cases of leniency were recorded – 3.6% of leniency above the 1.25 level on Topic 2. No instances of leniency by more than two levels emerged.

To examine, in greater depth, why BETSY misgraded, each prompt will be discussed in terms of the most overgraded and most undergraded script (see Table 8 below). Since Topic 1 emerged as most divergent on various statistical measures, scripts from this prompt will be examined. Data will be presented for one low ability test taker (scoring below Level 3) who was overrated by BETSY and one high ability test taker (scoring Level 5) who was underrated by BETSY. First, some background textual details of these scripts will be examined, followed by a discussion of BETSY’s output.

Table 8 Most overgraded and undergraded test takers

The most overgraded script was Script 98; it is presented in Figure 4, along with BETSY’s certainty rating.

Fig. 4 Overgraded script – Script 98.

The script above achieved a fair average score of 3.2, with BETSY’s rating a “very certain” 5. Against the Level 3 average of 35.3 t-units, this script consisted of 35 t-units, of which 34.2% were free of errors. It did, however, contain many infrequent words, many of which were misspelt (sanscity [presumably, “sanctity”], tumultous, comtemplation, unplesent). While in the training process BETSY supposedly purges infrequently-occurring items, BETSY’s analysis is partly predicated on the occurrence of specific words and phrases (see section 3.2). These words may have had an effect on the overall rating. They would not have affected the human raters, who would not have been impressed by misspelt words incorporated into often ungrammatical sentences (reflected in the low percentage of error-free t-units), e.g., “Thought that they live in an unplesent environment, They are kind”. It is a possible indicator, nonetheless, as to how computerised rating programs may err, and although it is not the focus of the current study, one issue which could be investigated further in computerised rating is the extent to which a computer may be fooled by writers (Page, Reference Page2003).

Table 9 presents a sample of the output produced by BETSY for Script 98. For each of the five levels (“5” indicating most able), BETSY provides the possibility that a particular “argument” is indicative of a particular level. In Table 9, frequencies above 30% (as against the average of 20%) have been bolded to highlight where the highest amount of decision-pointing frequencies occur among the five levels.

Table 9 Most frequent “arguments” – Script 98

As Table 9 illustrates, a higher percentage of frequencies above 30% occur in the Level 4 and 5 columns even though the data in the Table essentially contains collocations between function words, and as such is not particularly informative. It is presumably partly for this reason that BETSY is “very certain” that this is a Level 5 script.

In contrast with an overgraded script, the most undergraded script is now presented – Script 240 in Figure 5. Against the Level 5 average of 41.7 t-units, Script 240 consisted of 34 t-units, fewer even than Script 98 with its 35 t-units. However, Script 240 contained 94.1% error-free t-units.

Fig. 5 Undergraded script – Script 240.

In contrast to Script 98, where the test taker attempted to produce elaborate structures with flowery vocabulary beyond their ability level (as evidenced by the low number of error-free t-units), the writer of Script 240 writes in a style containing simple, mainly accurate structures, using vocabulary that shows a certain, although limited, range. One feature that BETSY examines is sentence length, which is comparatively shorter in Script 240, a possible reason for the Level 2 score.

There is little low-frequency vocabulary in Script 240. The style is simpler and less flowery than script 98. Consequently, whereas Script 240 achieved a human level score of 5.2, BETSY was “very certain” that the script was a “2”. Table 10 presents the most frequent arguments.

Table 10 Most frequent “arguments” – Script 240

Script 98’s arguments consisted only of function words whereas the arguments in Script 240 show many more content word collocations, although substantially more occur in levels 1 and 2 than in the upper levels. It is, presumably, on this basis that the script has emerged as a “2”.