MATERIALS SELECTION

Makers of size and levels tests have noted that such instruments could be helpful in selecting appropriate materials for language learners (Nation & Beglar, Reference Nation and Beglar2007; Schmitt et al., Reference Schmitt, Schmitt and Clapham2001; Stoeckel & Bennett, Reference Stoeckel and Bennett2015). A useful paradigm for considering whether existing tests can adequately perform this function is Nation’s (Reference Nation2007) four strands. In this framework, effective language programs are described as giving approximately equal attention to fluency development, meaning-focused input, meaning-focused output, and language-focused learning. The fluency development strand aims to help learners become better at using elements of the language that are already known. Therefore, in principle no new vocabulary should appear in materials for this strand. In meaning-focused input, learners’ attention is focused on receiving and understanding a message rather than on learning new elements of the language, so reading passages in this strand should be written at a level that can be understood independently. Nation (Reference Nation2007) has stated that learners should know at least 95% and preferably 98–99% of the running words in written materials for this strand. The optimal level of coverage for language-focused learning is less clear. Hu and Nation (Reference Hu and Nation2000) observed that there was no meaningful comprehension when learners knew 80% of the words in a text, suggesting that this level of coverage would be too low even when the focus of instruction is on the language. Schmitt et al., (Reference Schmitt, Jiang and Grabe2011) speculated that 85–95% coverage might be appropriate for this kind of learning.

Research indicates that differences between these levels of coverage are indeed significant. When learners know 98–99% of the words in a text, vocabulary knowledge is probably not a limiting factor in comprehension (Hu & Nation, Reference Hu and Nation2000; Schmitt et al., Reference Schmitt, Jiang and Grabe2011); however, the level of comprehension decreases rapidly with each successive 1% loss in coverage (Schmitt et al., Reference Schmitt, Jiang and Grabe2011). Thus, if vocabulary tests are to assist in selecting materials for these different purposes, they need to be capable of distinguishing between whether learners know (a) all the words in a set of materials, (b) 95–99% of the words, (c) somewhere in the range of 85–95%, or (d) less than that. If a test is not accurate enough to do this, it is of limited use in materials selection.

TRACKING VOCABULARY GROWTH

A second commonly cited purpose of size and levels tests is to measure lexical growth for research or for assessing program efficacy (Nation & Beglar, Reference Nation and Beglar2007; Stoeckel & Bennett, Reference Stoeckel and Bennett2015; Webb et al., Reference Webb, Sasao and Ballance2017). For a test to serve this function, it must be able to establish an accurate baseline and be sensitive enough to track meaningful changes in lexical knowledge over time. For overall vocabulary size, Milton (Reference Milton2009) has summarized research showing that among L2 learners of English, the rate of growth tends to be approximately four words per classroom hour, with specific programs achieving gains from 300 to 500 words per year. This implies that tests need to be accurate enough to detect changes of this size. For high-frequency words, studies have estimated annual growth rates from 18 to more than 250 words (Agustin-Llach & Canga Alonso, Reference Agustin‐Llach and Canga Alonso2016; Stoeckel, Reference Stoeckel2018; Webb & Chang, Reference Webb and Chang2012), suggesting that levels tests need at least that degree of precision. This comes with the caveat that estimates of vocabulary growth in much of this research were made using instruments that have the very limitations that this article describes.

GOAL SETTING

A third commonly cited purpose for size and levels tests is to help learners set goals for their vocabulary learning (Nation, Reference Nation2012b; Stoeckel & Bennett, Reference Stoeckel and Bennett2015; Webb et al., Reference Webb, Sasao and Ballance2017). Considering the previously discussed importance of knowing at least 98% of the words in a text, high-frequency vocabulary—defined here as the first 3,000 words of the lexicon (Schmitt & Schmitt, Reference Schmitt and Schmitt2014)—is essential. This relatively small number of ubiquitous items provides a very large proportion of coverage of any natural text. Nation (Reference Nation2006b), for example, found that the first 1,000-word families provided 77.9% coverage of a wide range of written text types, with the second and third 1,000-word bands adding 8.2 and 3.7%, respectively. If tests are to help learners establish meaningful learning goals, they should be able to identify which high-frequency words, if any, are unknown. For learners who have mastered the first 3,000 words, rather than using frequency as a guide for further learning, it may often be more efficient to focus on specialized or technical vocabulary depending on specific learning needs (Nation, Reference Nation2013). Size tests do not separately assess knowledge of specialized vocabulary, but some levels tests include an academic vocabulary level for learners who need English for higher education.

In sum, when we consider some of the specific purposes for which size and levels tests have been developed, we get a better idea of how accurate these instruments need to be. Size tests ought to be able to detect changes in overall lexical knowledge at least as small as 500 words, preferably 200 words or less, if they are to be used to track annual lexical growth. Presumably, greater precision would be needed for shorter periods. Levels tests should be capable of (a) distinguishing between learners who have small differences in coverage of specified texts, namely four levels defined by 100%, 95–99%, 85–95%, and below 85% coverage; (b) detecting annual changes in knowledge of specific word bands at least as small as 250 words; and (c) identifying which words within a high-frequency or specialized word level are unknown so that they can be targeted for study.

VOCABULARY LEVELS TEST

The VLT was introduced by Nation in 1983 as a way to determine whether learners had gained mastery of high-, mid-, and low-frequency words as well as words that are common in academic discourse. Schmitt, Schmitt, and Clapham revised the VLT in 2001, creating two parallel forms of the test and conducting a preliminary validation study. This is the version most commonly in use today and the one described in the text that follows.

The VLT uses a matching format. Test items, referred to as “clusters,” each contain six words and three definitions (Figure 1). The test has five levels, each with 10 clusters assessing 30 target words sampled from family-based lists (the distinction between word families, flemmas, and lemmas is discussed in the section "Word Grouping Principle"). The first level assesses knowledge of the combined first and second 1,000-word bands, and this is followed by levels that test the third, fifth, and tenth 1,000-word bands and one assessing knowledge of academic vocabulary drawn from Coxhead’s (Reference Coxhead2000) Academic Word List. Within each level, stratified sampling was used to ensure that 15 nouns, 9 verbs, and 6 adjectives were assessed. Two parallel forms of the test were developed, and the target words in each level of each form were selected to represent the average item facility of that level from a bank of items that was trialed in the first stage of test development. Regarding score interpretation, scores of 26 or higher (out of 30) were put forth to represent mastery of a test level. However, no theoretical or evidence-based justification for this was given.

FIGURE 1. An example cluster from the Vocabulary Levels Test. From Schmitt et al. (Reference Schmitt, Schmitt and Clapham2001).

VOCABULARY SIZE TEST

Whereas the VLT was designed to determine whether learners had mastery of certain word bands, the Vocabulary Size Test (VST; Nation & Beglar, Reference Nation and Beglar2007) was developed to estimate overall written receptive vocabulary size. Stratified random sampling was used to select target words from each of the first fourteen 1,000-word frequency levels of family-based lists. As determining mastery of any given frequency level was not a goal of the test, the number of target words from each word band—just 10—was considerably smaller than in the VLT. The VST employs a multiple-choice format in which each item presents the target word followed by an example sentence which uses the word in a nondefining context. The four options include a definition of the target word plus definitions of three other words of similar frequency (Figure 2).

FIGURE 2. An example item from the monolingual 14,000-word version of the Vocabulary Size Test (Nation & Beglar, Reference Nation and Beglar2007) available at https://www.wgtn.ac.nz/lals/about/staff/paul-nation.

Nation later produced a version of the VST that includes five items for each of the first twenty 1,000-word frequency bands, which allowed for research with native speakers at secondary schools in New Zealand without having a ceiling effect (Coxhead et al., Reference Coxhead, Nation and Sim2015). In addition to these two versions of the test, other variants have been produced including several bilingual forms in which answer choices are rendered in examinees’ first language (Elgort, Reference Elgort2012; Karami, Reference Karami2012; McLean et al., Reference McLean, Ishii, Stoeckel, Bennett and Matsumoto2016; Nguyen & Nation, Reference Nguyen and Nation2011; Zhao & Ji, Reference Zhao and Ji2018), a version with an added “I don’t know” answer choice for each item (Zhang, Reference Zhang2013), and a version using a serial-multiple-choice item format (described in Item Format; Stoeckel & Sukigara, Reference Stoeckel and Sukigara2018). Regarding score interpretation, for versions of the test that are based on a target word sampling rate of 1:100, raw scores are multiplied by 100 to estimate overall vocabulary size. Similarly, raw scores are multiplied by 200 for the versions which use a 1:200 sampling rate.

NEW GENERAL SERVICE LIST TEST

Stoeckel and Bennett introduced the New General Service List Test (NGSLT) in 2015. This instrument assesses knowledge of only high-frequency vocabulary, defined as the 2,801 entries in Browne et al.’s (Reference Browne, Culligan and Phillips2013) flemma-based New General Service List. Each of the five levels represents a 560-word frequency-based band. Approximately 40 words were randomly sampled from each band, data were collected to determine item difficulties, and then 20 items were selected for each level to represent the average and range of difficulty of the items in the level. The NGSLT employs a multiple-choice format like that used in the VST. In addition to the original monolingual format, there is a Japanese–English bilingual version (Stoeckel et al., Reference Stoeckel2018b). The authors of the NGSLT did not describe score interpretations in terms of mastery but instead stated that scores below 17 or 18 (out of 20) are meant to flag the level for intentional vocabulary study.

UPDATED VOCABULARY LEVELS TEST

Webb et al. (Reference Webb, Sasao and Ballance2017) made an updated Vocabulary Levels Test (UVLT). This is similar to Schmitt et al.’s (Reference Schmitt, Schmitt and Clapham2001) VLT in that it has 30 target words per level (following a 15:9:6 ratio of nouns, verbs, and adjectives), words are sampled from word family lists, and it uses a matching format with three definitions and six words per cluster. Reflecting a shift in focus toward mastery of higher frequency words first, the UVLT differs from the VLT in that it assesses knowledge of words sampled from each of the first five 1,000-word frequency levels rather than from nonconsecutive levels or from a list of academic vocabulary. Additionally, words were selected from more up-to-date British National Corpus/Corpus of Contemporary American English frequency lists (BNC/COCA; Nation, Reference Nation2012a) to reflect current English. The mastery criterion is 29 (out of 30) for the first three levels and 24 for the fourth and fifth levels. This difference was intended to reflect the greater relative importance of the higher frequency words in terms of the coverage they provide.

ITEM FORMAT

In tests of written receptive vocabulary knowledge, two broad categories for item format are meaning-recall and meaning-recognition.

Score Interpretation

The fundamental shortcoming relates to score interpretation. Correct answers are assumed to represent known words for the purpose of reading, and raw scores are interpreted in terms of the actual number of words that are known or mastery of the words in a test level. Between meaning-recall and meaning-recognition tasks, however, it has long been argued that the former is more similar to the aspect of lexical knowledge needed in reading because both meaning-recall and reading require word meaning to be evoked from memory without reference to a list of choices (Kremmel & Schmitt, Reference Kremmel and Schmitt2016; Nation & Webb, Reference Nation and Webb2011; Stewart, Reference Stewart2014; Stoeckel et al., Reference Stoeckel, Stewart, McLean, Ishii, Kramer and Matsumoto2019). This view is empirically supported by McLean et al. (Reference McLean, Stewart and Batty2020), who found that meaning-recall test scores are significantly better predictors of reading proficiency than meaning-recognition scores.

There are two principal factors that make meaning-recognition item formats dissimilar from the lexical knowledge needed in reading. First, meaning-recognition requires a lower threshold of lexical knowledge than meaning-recall (Laufer & Goldstein, Reference Laufer and Goldstein2004; Nation, Reference Nation2013; Nation & Webb, Reference Nation and Webb2011; Schmitt, Reference Schmitt2010), so words whose meanings are recognized from a list of choices may not be known when reading. Second, meaning-recognition items can be correctly answered through random guessing or the use of construct-irrelevant test strategies (Gyllstad et al., Reference Gyllstad, Vilkaitė and Schmitt2015; Kremmel & Schmitt, Reference Kremmel and Schmitt2016; McLean et al., Reference McLean, Kramer and Stewart2015; Stewart, Reference Stewart2014; Stewart et al., Reference Stewart, McLean and Kramer2017; Stoeckel & Sukigara, Reference Stoeckel and Sukigara2018). These factors lead to overestimation of lexical knowledge for reading.

Although numerous studies have compared meaning-recall and meaning-recognition formats to ascertain the extent to which scores differ, some have potential sources of inaccuracy. In some cases, learners were instructed to skip unknown items or use an explicit “I don’t know” option (e.g., Zhang, Reference Zhang2013), but these conventions are sometimes used differently (Bennett & Stoeckel, Reference Bennett and Stoeckel2012; Zhang, Reference Zhang2013), which results in different scores for learners of similar ability (Stoeckel et al., Reference Stoeckel, Bennett and McLean2016; Stoeckel et al., Reference Stoeckel, Stewart, McLean, Ishii, Kramer and Matsumoto2019). In several other studies, the meaning-recognition task was administered first (e.g., Gyllstad et al., Reference Gyllstad, Vilkaitė and Schmitt2015; Kremmel & Schmitt, Reference Kremmel and Schmitt2016). This is useful for retrospective insight into test-taking behavior, but it exposes learners to the word form and meaning together, which can inform performance on the subsequent recall task. Finally, meaning-recall responses have been judged by comparing them to a predetermined bank of potential correct L1 translations (Aviad-Levitzky et al., Reference Aviad-Levitzky, Laufer and Goldstein2019); this approach has real potential (as discussed in the following text), but incorrect responses (i.e., those not listed in the bank of correct answers) may need to be validated by comparison to human judgments, as there can be many ways to express the same concept.

We are aware of three studies with a total of five learner groups in which the recall test was administered first, examinees were required to answer every item, and each recall response received human scrutiny (Gyllstad et al., Reference Gyllstad, McLean and Stewart2019; Stoeckel et al., Reference Stoeckel, Stewart, McLean, Ishii, Kramer and Matsumoto2019; Stoeckel & Sukigara, Reference Stoeckel and Sukigara2018). In these studies, the score difference between meaning-recognition and meaning recall was as if examinees had correctly answered 13.9–62.7% of the meaning-recognition items that assessed words that were unknown at the meaning-recall level of knowledge (Table 1).Footnote ¹ Both item type and test language were factors that affected score differences. The SMC format and monolingual options produced mean scores that were more similar to a criterion meaning-recall measure than traditional multiple-choice and bilingual options, respectively. Although other factors also affect the relationship between meaning-recall and meaning-recognition scores (e.g., learner proficiency relative to the difficulty of the tested words, Kremmel & Schmitt, Reference Kremmel and Schmitt2016; Stewart & White, Reference Stewart and White2011), the range of values in the final column of Table 1 are useful for broadly understanding the potential extent of overestimated meaning-recall knowledge in size and levels tests.

TABLE 1 Meaning-recall and meaning-recognition score differences when the recall test is administered first, a response is required for each item, and all responses receive human judgment

Note: SMC = serial multiple choice; MC = multiple choice.

^a Calculated as (correct recognition responses—correct recall responses)/incorrect recall responses (for explanation, see Footnote note 1).

On the basis of this previous research, what impact does item format have on estimates of vocabulary size? Taking the VST as an example, Table 2 displays estimated vocabulary sizes under meaning-recall and meaning-recognition formats at levels of meaning-recall knowledge from 1,000 to 8,000 words for three tests of the same length as those used in Beglar’s (Reference Beglar2010) validation of the VST: 40, 80, and 140 items to assess knowledge of the first 4,000, 8,000, and 14,000 words, respectively. The differences between recall and recognition scores shown in Table 2 reflect the highest and lowest proportion of unknown items answered correctly as reported in Table 1 (13.9% for a bilingual SMC format and 62.7% for a bilingual multiple-choice format) as well as a rate of 25%, which would be expected due to blind guessing under a four-option multiple-choice format. For example, the first row of data shows that meaning-recall performance indicating knowledge of 1,000 words is estimated to correspond with knowledge of 1,417 words under the SMC format (1,000 + 417 [i.e., 13.9% of the 3,000 words not known to the level of meaning-recall]), 1,750 words due to blind guessing of items not known to the level of meaning-recall under a four-option multiple-choice format, and 2,881 words when test strategies are also factored in with a multiple-choice format. Because there is less room for overestimation of meaning-recall knowledge as vocabulary size approaches the test ceiling, Table 2 shows that recall and recognition scores become more similar as they increase. Nonetheless, the difference between these two modalities can be quite large, often exceeding a thousand words.

TABLE 2 Differences in estimated vocabulary size at three levels of meaning-recognition knowledge for unknown items at the meaning-recall level

Note: SMC = serial multiple choice; MC = multiple choice.

For levels tests, the problems associated with item format are smaller, but not insignificant. Whereas size test scores are interpreted in terms of the actual number of words a learner knows even when many items are unknown, levels test are usually interpreted as to whether individual levels are mastered, with a high proportion of correct responses needed for mastery. As with size tests, when meaning-recall knowledge approaches the test ceiling, there is little room for overestimation. However, it has been theoretically posited (Stewart & White, Reference Stewart and White2011) and practically observed (Kremmel & Schmitt, Reference Kremmel and Schmitt2016) that the use of test strategies can result in a higher proportion of unknown items being answered correctly when a learner’s knowledge approaches level mastery. With a four-option multiple-choice format, we might therefore expect a level of success on unknown items similar to that shown in the rightmost column of Table 2. If so, roughly 60% of learners whose meaning-recall knowledge is one item below the mastery threshold would be able to achieve a mastery score, and about 36% whose recall knowledge is two items below this threshold could also do so. Using the four-option NGSLT as an example, even if the level mastery threshold were set at 20 out of 20 (a 100% success rate), many learners with actual knowledge of 18 or 19 items, which corresponds with 90% and 95% success rates, respectively, would be identified as having achieved mastery. Considering the substantial differences in comprehension that occur with small changes in coverage, the use of meaning-recognition formats with levels tests is not without consequences, even when the mastery threshold is set very high.

WORD GROUPING PRINCIPLE

Another characteristic of size and levels tests that can impact the accuracy of estimated employable vocabulary knowledge is the principle by which related word forms are grouped. In test development, target words are normally sampled from corpus-derived frequency-based wordlists. Such lists typically group word forms into lemmas, flemmas, or word families. A lemma consists of a headword together with inflected forms that are the same part of speech (POS) as the headword. For instance, the lemma for the verb center includes that headword and verbal uses of the inflected forms centers, centered, and centering. A flemma is similar to a lemma except the flemma takes no account of POS. That is, any use of the form center is included together with any use of the forms centers, centered, and centering. A word family, as used in L2 English vocabulary tests, generally refers to a headword together with all its inflectional and derivational forms up through Bauer and Nation’s (Reference Bauer and Nation1993) level 6 affix criteria (hereinafter WF6). The WF6 for center in Nation’s (Reference Nation2012a) BNC/COCA lists includes the flemma constituents above as well as centrist, centrists, central, centralization, centralize, centralized, centralizes, centralizing, centralism, centralist, centralists, centralities, centrality, centrally, centeredness, and centric plus alternative spellings for 10 of these words.

In existing L2 vocabulary tests, learners are assessed on just one member of the lemma, flemma, or WF6 (usually the base form), and a correct response is assumed to indicate knowledge of the entire group of related word forms. When WF6 is used, this approach overestimates vocabulary knowledge, as all current L2 English research indicates that for EFL learners of a range of proficiencies, receptive knowledge of a headword is a poor indicator of knowledge of related derivational forms (Brown, Reference Brown2013; McLean, Reference McLean2018; Mochizuki & Aizawa, Reference Mochizuki and Aizawa2000; Schmitt & Meara, Reference Schmitt and Meara1997; Ward & Chuenjundaeng, Reference Ward and Chuenjundaeng2009). Overestimation is less extensive for flemma-based instruments because correct responses do not assume knowledge of derivational forms. However, there is evidence that receptive understanding of a word form in one POS does not always correspond with knowledge of the same word form in another POS (Stoeckel et al., Reference Stoeckel, Ishii and Bennett2018a), suggesting that flemma-based tests can also overestimate employable word knowledge.

What does the existing literature tell us about the amount of overestimation when tests assume a WF6 level of knowledge? To explore this question, Brown (Reference Brown2018) estimated the proportion of coverage of the complete set of texts in the BNC that is provided at each 1,000-word level of Nation’s (Reference Nation2006a) BNC word lists by different levels of affixation knowledge and knowledge of irregular forms and alternative spellings. Summarizing Brown’s findings for the first 1,000 WF6s, Table 3 shows that base forms alone provide about 67% of the total coverage that the 1K level offers. When irregular forms, alternative spellings, and compound forms (i.e., the “other” forms in Table 3) are added along with Level 2 affixes, this increases to 87.9%. These forms basically correspond with the flemma. Levels 3 through 6 of Bauer and Nation’s (Reference Bauer and Nation1993) framework provide the remaining 12.1% of coverage at the 1K level. This represents WF6 level of coverage.

TABLE 3 Estimated proportion of coverage of the texts in the British National Corpus provided by knowledge at different levels of affixation for the first 1,000 words of Nation’s BNC Word Lists (adapted from Brown, Reference Brown2018)

^a Includes irregular noun and verb forms, abbreviated forms, alternative spellings, and compound forms.

Brown’s analysis enables us to estimate the amount of employable coverage that a learner possesses relative to their test score, depending upon their level of affix knowledge. For instance, if a learner with a score of 98% on the 1K level of a WF6-based test has receptive knowledge of affixation up through only level 2 forms, the 1K test score (.98) could be multiplied by the coverage provided up through level 2 forms at the 1K level (.879, Table 3) to estimate that the employable coverage of 1K words for this learner is approximately 79.1%. This is substantially lower than the raw test score.

Although this is a good starting point for quantifying the extent to which use of WF6 overestimates employable coverage, real-world learners do not have all-or-nothing understanding of the affixes in each of Bauer and Nation’s (Reference Bauer and Nation1993) levels. Additionally, demonstrated knowledge of one word using a given affix is no guarantee that another word using the same affix will be known when the base word is known (McLean, Reference McLean2018; Mochizuki & Aizawa, Reference Mochizuki and Aizawa2000; Ward & Chuenjundaeng, Reference Ward and Chuenjundaeng2009). This is because the known word may have been learned independently rather than understood as a base word plus affix. Data from two recent studies provide a more nuanced account of actual employable knowledge for learners of typical proficiency levels in one EFL context, Japanese universities.

In the first study, McLean (Reference McLean2018) examined knowledge of 12 base forms and 87 inflectional and derivational forms among 279 Japanese university students ranging from beginning to advanced proficiency. When learners knew a base form, they understood an extremely high proportion (.968) of related level 2 (inflected) forms but not so (.544) for levels 3 through 6 (derivational) forms (Table 4). This study also found very good comprehension (.980) for one irregular verb form, representing Brown’s “other” category.Footnote ²

TABLE 4 Written receptive knowledge of irregular, inflected, and derived forms for known base words in McLean, Reference McLean2018

^a Based on just one irregular verb (taught).

In the second study, Stoeckel et al. (Reference Stoeckel, Ishii and Bennett2018a) found that Japanese university students (N = 64) who understood a high-frequency word form in one POS were able to derive the correct meaning from the same word form when it occurred in another POS just 56% of the time. For example, learners who understood the use of extra as an adjective were often unable to understand its use as a noun. If the purpose of a written receptive vocabulary test is to ascertain when a word form can be understood in actual receptive use of the language (Schmitt et al., Reference Schmitt, Nation and Kremmel2019), then this research suggests that knowledge of a base form in one POS should not be assumed from demonstrated knowledge of the same word form in another POS. Because many base forms from the first 5,000 WF6s have more than one POS (Table 5), this could substantially influence estimates of employable coverage.

TABLE 5 The proportion of content words with multiple parts of speech in each of the first five 1,000-word bands

Note: We calculated these proportions based on frequencies and POS designations for the British National Corpus from Lancaster University (http://ucrel.lancs.ac.uk/bncfreq/flists.html).

Using Brown’s (Reference Brown2018) BNC analysis together with the data from McLean (Reference McLean2018) and Stoeckel et al. (Reference Stoeckel, Ishii and Bennett2018a), the employable coverage at each of the first five 1,000-word BNC levels for Japanese university students can be estimated. An example for the 1K level is shown in Table 6. Column A shows the proportion of coverage offered by specified word forms as reported by Brown (Reference Brown2018), with the base form category separated into function words, content words that have one POS, and content words with multiple POS. Column B shows the estimated proportion of each category that is known when knowledge of the base form is known, as reported by McLean (Reference McLean2018) and Stoeckel et al. (Reference Stoeckel2018a). Column C, the product of A and B, shows the estimated employable coverage for these learners when knowledge of a base form has been demonstrated in one POS in a WF6-based test.

TABLE 6 Estimated employable coverage from base word knowledge on a WF6-based test of the first 1,000-word BNC Level for Japanese university students

^a From Brown (Reference Brown2018) and from Table 5.

^b Calculated from data published in Stoeckel et al. (Reference Stoeckel, Ishii and Bennett2018a).

^c Calculated from data published in McLean (Reference McLean2018).

In this analysis, function words and the base form of content words that have just one POS were assumed to be known. Therefore, the values for these categories in column B are 1.000, and the values in columns A and C are identical, showing no loss in employable coverage. For the base forms of content words with multiple POS and levels 3–6 derived forms, however, there is substantial lost employable coverage. The final row of Table 6 shows that when interpreting test results of the 1K level, the Japanese university learners in these two studies, on average, possessed only about 86% of the coverage indicated by test scores. The employable coverage for each of the first five 1,000-word bands of the BNC, calculated in the same way, is summarized in Table 7.

TABLE 7 Estimated employable coverage from base form knowledge in a WF6-based test of the first five 1,000-word BNC Levels for Japanese university students

Note: Calculated based on data from Brown (Reference Brown2018), McLean et al. (Reference McLean, Stewart and Batty2020), and Stoeckel et al., (Reference Stoeckel, Ishii and Bennett2018a).

From Table 7, we can estimate the usable word knowledge at each 1,000-word level from the scores in a WF6-based levels test for Japanese university students. Take, for example, a hypothetical student with scores of 99, 98, 90, 80, and 70% on a test of the first five 1,000-word bands. These scores suggest very good knowledge of the first 2,000 WF6s. However, when these raw scores are multiplied by the corresponding values in Table 7 to account for lost employable coverage due to incomplete WF6 knowledge, a vastly different picture emerges (Table 8). The estimated employable coverage for the first two 1,000-word bands is just 85.1 and 78.4%, respectively. If, for example, the raw scores were used for materials selection, reading passages intended for use in the meaning-focused input or fluency strands of a language program would be nearly unintelligible without some assistance from a dictionary or glossing.

TABLE 8 Adjusted coverage figures for a hypothetical Japanese university student

This analysis was based upon two studies whose participants were mostly representative of university students in Japan (with a small number of learners in McLean’s Reference McLean2018 study who were of unusually high proficiency). As these were average scores, some such students would certainly possess greater employable knowledge of word family constituents, but others would possess less, and the extent of overestimation for any given learner is not readily knowable from a WF6-based test score. There is, perhaps surprisingly, little research comparing base word knowledge to knowledge of derived forms, which is at the heart of the assumption underlying WF6-based tests, but the other study we are aware of is consistent with this analysis. Ward and Chuenjundaeng (Reference Ward and Chuenjundaeng2009) investigated knowledge of 16 base words and 16 related derived forms with two groups of Thai university students, finding that when learners had knowledge of a base form, they had knowledge of related derived forms 59.2% (high group) and 39.6% (low group) of the time.Footnote ³ Other investigations of L2 learners’ receptive knowledge of derivational morphology have also shown far from complete understanding (Mochizuki & Aizawa, Reference Mochizuki and Aizawa2000; Sasao & Webb, Reference Sasao and Webb2017; see also Gardner, Reference Gardner2007).

Laufer and Cobb (Reference Laufer and Cobb2019) recently argued that WF6 remains usable because most derivational word forms utilize only a small number of common affixes, but this position relies on learners knowing such derivational forms. Unfortunately, when data from previous research is reanalyzed to include only words with frequently occurring affixes, the results are much the same (Table 9). For instance, when participants in McLean (Reference McLean2018) knew a base form, they could demonstrate knowledge of related derivational forms that used frequently occurring affixes just 62.1% of the time, which is just a few percentage points higher than their performance across all affixes. Though the precise amount of overestimated word knowledge probably varies with learner proficiency and proximity of the L1 to English, there is no evidence to assume that use of WF6-based tests would be unproblematic for score interpretations with many, if not most, L2 learners of English.

TABLE 9 Written receptive knowledge of derived forms using common affixes when the base form is known

Note: Includes the 10 most frequent affixes in the Morpholex database developed by Sánchez-Gutiérrez et al. (Reference Sánchez-Gutiérrez, Mailhot, Deacon and Wilson2018). To the best of our knowledge, no studies have examined the ~y, ~ate, in~, or ~ant affixes under the same conditions as Ward and Chuenjundaeng (Reference Ward and Chuenjundaeng2009) or McLean (Reference McLean2018).

It is important to note, however, that in the studies cited in the preceding text, decontextualized test formats were used to assess word knowledge. Because context can support lexical comprehension (Nation, Reference Nation2013), the extent to which a more naturalistic reading task might enable learners to work out the meaning of derivational forms for known base words is an area in need of research. This would further clarify the appropriateness of using WF6 with tests of vocabulary for the purpose of reading. We also stress that our focus is on assessment, not pedagogy. The organization of word forms according to WF6 can be valuable for deciding how to treat related word forms in the classroom. A wonderful example is Gardner and Davies’s (Reference Gardner and Davies2014) Academic Vocabulary List, which is organized into both lemma- and family-based lists, the latter useful for helping learners to see the morphological relationship among family members.

TARGET WORD SAMPLE SIZE

We now turn to the issue of target word sample size. In the development of vocabulary size and levels tests, target items are sampled from large sets of words, often 1,000-word frequency-based bands. To achieve high estimates of internal reliability and a strong correlation with a criterion measure, it appears that a target word sample size of 30 items is sufficient (Gyllstad et al., Reference Gyllstad, Vilkaitė and Schmitt2015; Schmitt, et al., Reference Schmitt, Schmitt and Clapham2001).

A separate question, however, is how well a sample represents the population of words from which it was drawn. Vocabulary knowledge as a tested construct differs from assessment in many other areas of learning. Lexical knowledge is not a skill that, once acquired, can be applied to understand unrelated new words. Knowledge of lexis is built word by word, differing for example from basic addition and subtraction, areas of learning for which demonstrated mastery in a well-made test can be interpreted as the ability to solve a universe of similar problems. Thus, there are limitations regarding the inferences that can be made from knowledge of a random sample taken from a larger set of words (Gyllstad et al., Reference Gyllstad, Vilkaitė and Schmitt2015). A high estimate of internal reliability on a size or levels test indicates that the instrument can reliably measure knowledge of the target words in the test. Likewise, a strong correlation between scores on a vocabulary test and a criterion measure of the same words indicates that there is a strong relationship between measures of only the sampled words. This is separate from whether test scores accurately represent knowledge of an entire word band. Such distinctions are important because if the small number of words that are assessed is significantly more (or less) likely to be known than the population of words from which they were sampled, results will systematically over- (or under-) estimate vocabulary knowledge even if estimates of internal reliability or correlations with a criterion measure are high.

Perhaps this issue has received little attention because of an implicit assumption that the items within a frequency-derived word band are of similar difficulty. If this were the case, it would not matter which words were selected; they would be equally representative of the entire band. The empirical evidence does not support this assumption, however. Though mean scores on large, frequency-based word bands decrease with frequency (Aizawa, Reference Aizawa, Murata, Minamide, Tono and Ishikawa2006; Beglar, Reference Beglar2010; Brown, Reference Brown2013; Milton, Reference Milton, Daller, Milton and Treffers-Daller2007), there is considerable variation in difficulty for individual words within frequency bands (Beglar, Reference Beglar2010; Bennett & Stoeckel, Reference Bennett and Stoeckel2014). This calls into question the accuracy with which small samples represent the average difficulty of a large population of words.

Though vocabulary levels and size tests are commonly used to estimate the total number of words known (e.g., Webb & Chang, Reference Webb and Chang2012) or to determine level mastery (e.g., Chang, Reference Chang2012), confidence intervals (CI) for individual scores are rarely if ever reported, and the number of items required for desired levels of confidence is underexplored. Using meaning-recall data from all 1,000 words at the 3K frequency level, Gyllstad et al. (Reference Gyllstad, McLean and Stewart2019) found that for a test consisting of 20 randomly selected items, the 95% CI was as much as 400 (i.e., ±200) of the 1,000 words assessed. Thus, a learner who correctly answers 10 out of the 20 items (50%) might know anywhere between 300 and 700 items in the 1,000-word band. The maximum 95% CIs for 30, 50, and 100-item tests were 326, 248, and 172 words, respectively.

Confidence intervals can also be estimated based on sample size and the proportion of correct responses. Using this approach, we calculated Clopper-Pearson (Clopper & Pearson, Reference Clopper and Pearson1934) CIs for a hypothetical test with different target word sample sizes from a 1,000-word band. The Clopper-Pearson method is appropriate when the proportion of correct responses approaches or reaches 1 (McCracken & Looney, Reference McCracken and Looney2017), which corresponds with common mastery criteria for levels tests. We calculated CIs for two scoring outcomes. The first is at a score of 50% because this is where the CI is widest, revealing the largest potential difference between test scores and actual knowledge. The second is where scores approach and reach 100% because this is where mastery has been defined in levels tests, and it is where CIs are smallest.

The values in Table 10, consistent with Gyllstad et al. (Reference Gyllstad, McLean and Stewart2019), indicate that when levels tests are used to estimate the number of words that a learner knows in a level (rather than complete mastery), massive error is possible for scores around 50%. Although the CIs for perfect or near-perfect scores are narrower, they are unsatisfactory when we consider the importance of knowing 95 or 98–99% of the words in a text. The most items per level in any existing levels test of the form-meaning link is 30 in both the VLT and UVLT, with the strictest mastery criterion set at 29 for the 1K and 2K levels of the UVLT. The 95% CI for that score (828–999 words) or even for a perfect score (884–1,000 words) casts doubt on whether a mastery score consistently corresponds with knowledge of 95% or more of the words in the level. The values in Table 10 also suggest that the current approach to test construction, in which items are randomly sampled, may be untenable for the needed level of precision for many testing purposes, even when a 90% CI is used. When a learner achieves a perfect score on a test of 100 items sampled from a 1,000-word band, the 90% CI falls outside of the level of 98% coverage. It is hard to imagine how a multiple-level test with 100-plus items per level could be practically used in most educational settings. For existing instruments, combining items from multiple test forms and, when time is limited, administering fewer than the full complement of levels, is a good way to increase precision.

TABLE 10 Clopper-Pearson confidence intervals for the number of known words for specified scores and sample sizes from a 1,000-word band

Note: CIs were calculated at openepi.com/Proportion/Proportion.htm.