2.1 Level

DDL was largely initiated by Johns, who originally had in mind “a particular type of student (adult: well motivated: a sophisticated learner with experience of research methods in his subject area)” (1986: 161). He later reported that fellow teachers often objected that DDL “may be all very well for students as intelligent, sophisticated, and well-motivated as ours at Birmingham University, [but] it would not work with students as unintelligent, unsophisticated and poorly-motivated as theirs” (Johns, Reference Johns1991: 12). But he goes on: “what I suspect, however, is that most students given the opportunity to show what they are capable of might be (almost) as remarkable.” Work by Sealey and Thompson (e.g. Reference Sealey and Thompson2007) shows how even primary school children can exploit corpora in their mother tongue, suggesting that, in the right conditions, no great level of sophistication is necessary. But when it comes to learning a foreign or second language, very little DDL research ventures outside the university environment (Braun, Reference Braun2007). Virtually all of the 39 empirical studies of DDL surveyed in Boulton (Reference Boulton2007a) focus on university students, the two exceptions being Ciezielska-Ciupek (Reference Ciezielska-Ciupek2001) and Sun and Wang (Reference Sun and Wang2003), conducted in secondary schools in Poland and Thailand respectively.

Similarly, it is also often assumed that DDL will not work with learners at lower levels – even to the extent that the objection seems to apply to level of sophistication and language proficiency simultaneously, as if the two were inextricably connected. Again, the objection seems to be based on gut reaction rather than on empirical evidence: Hadley (Reference Hadley2002), for example, was assured by colleagues in Japan that any attempt to use DDL with beginners was “doomed to failure”, even though these same colleagues had never tried. Boulton’s (Reference Boulton2007a) survey found only a handful of studies which claim to evaluate DDL at lower levels. As at other levels, the results are mixed, although in no case is it found to be completely useless. In one instance reported by Yoon and Hirvela (Reference Yoon and Hirvela2004), DDL was apparently more successful with intermediate than advanced learners, although the experimental design may be partly responsible for this (O’Sullivan & Chambers, Reference O’Sullivan and Chambers2006: 61).

One commonly accepted objection is that learners at lower levels might simply not have sufficient analytical and linguistic skills to cope with the complexity and fuzziness of authentic data of a foreign or second language – “authentic” in Holec’s (Reference Holec1990) sense that they were not originally created for purposes of language teaching or learning. On the other hand, learners might benefit from exposure to that complexity at an early stage rather than living in the false expectation that clear and simple rules can always be devised. Given the strong arguments that learners can and should be confronted with authentic documents from the beginning (e.g. Holec, Reference Holec1990), the case against using corpora with lower levels is severely weakened in the absence of evidence to the contrary.

One might also wonder why a new approach should be reserved for learners who are already advanced, those who are already successful using existing methods. Perhaps DDL might be appropriate for “less successful” learners as an alternative to traditional methods which, by definition, have not worked in their case. But in the face of failure, traditional methods tend to inflict more of the same, perhaps on the assumption that poor learners will be poor learners whatever the method, so there is little point in attempting anything new. Responsibility and initiative (in DDL or otherwise) are thus seen almost as rewards for learners who have proved they can cope without; poor learners are not to be trusted with them.

Traditional teaching, especially at lower levels, tries to take as much of the burden off the learners as possible, partly by pre-formulating the rules to learn. One problem though is that such explicit rule-learning is an ‘artificial’ and ‘difficult’ process demanding considerable intellectual rigour. DDL, on the other hand, exploits processes that humans have evolved to be naturally good at: exposure to information, detection of patterns, extrapolation to other cases (e.g. Scott & Tribble, Reference Scott and Tribble2006: 6; Gaskell & Cobb, Reference Gaskell and Cobb2004). It might be argued that this is particularly appropriate at lower levels when learners already have considerable cognitive demands made upon them.

One major aim of this paper is, then, to see to what extent lower level learners can benefit from aspects of DDL.

2.2 Training

It has frequently been claimed that lack of sufficient training is a major barrier to the implementation of DDL, whether for teachers or for learners, or indeed both (e.g. Breyer, Reference Breyer2006). Lack of training has on occasion been cited as a factor where empirical research has not produced the hoped-for results, but the training itself has generally been an incidental element rather than a main research question in such studies. More often still, it is simply taken as given. We are perhaps beginning to see something of a retreat on this strong insistence on training. For Bernardini (Reference Bernardini2001: 243), “the difficulties should not be overestimated; learners should quickly acquire the skills needed”; and for Sinclair (Reference Sinclair2004: 297), “any teacher or student can readily enter the world of the corpus and make the language useful in learning.” Of course, the more experience and training learners have, the more responsibility and autonomy they can assume, and the more sophisticated the use of corpora can be, but “both teacher and student can make use of a corpus right away, with only a modest few hours of orientation” (Sinclair, Reference Sinclair2004: 288). A number of studies, however (e.g. Boulton & Wilhelm, Reference Boulton and Wilhelm2006), have shown that learners can prove quite sophisticated, even with complicated tasks such as building and analysing their own corpora, with comparatively little training.

Insistence on training is particularly associated with hands-on corpus work, but may be less important when working with prepared materials. However, it could be argued that students who are used to a deductive approach will require “extensive guidance” in using inductive learning strategies needed for interpreting concordances (Sun, Reference Sun2003: 612). But the same could be said for dictionaries, for example: “The dictionary is an excellent tool in the hands of a skilled learner. An unskilled user wastes time and comes away frustrated from dictionary consultations” (Roby, Reference Roby2005: 59). Indeed, Frankenberg-Garcia (Reference Frankenberg-Garcia2005) found that learners were just as good with minimal training on concordances as they were with dictionaries they had been using all their lives. With corpora as with dictionaries, it seems obvious that learners cannot be expected to gain maximum advantage by merely being given access to the tools. Yet for either resource, that is not to say that learners cannot benefit from at least some aspects with little or perhaps even no training.

One potential problem lies in the “sometimes startling physical appearance of concordances” (Lamy & Klarskov Mortensen, Reference Lamy and Klarskov Mortensen2007), with the key word in the middle of truncated texts for “vertical” reading; this may be confusing for some, hence the need for a book such as Sinclair’s Reading Concordances (Reference Sinclair2003). Specifically regarding L2 learners, Johns (Reference Johns1986: 157) first cited the appearance of “unfinished sentences” as a common learner complaint over twenty years ago. Gavioli (Reference Gavioli2005: 29) claims that the concordance “is not a type of text whose reading and digesting can be taken for granted… The type of (linguistic) information a concordance gives to the student-analyst is not obvious.” All three major problems cited in Koosha and Jafarpour (Reference Koosha and Jafarpour2006: 206) relate to the difficulty of interpreting concordances. Yoon and Hirvela (Reference Yoon and Hirvela2004: 270) are among the few to quantify this, with 62% of their intermediate learners citing the “cut-off sentences in concordance output” as a difficulty. Otherwise the evidence remains largely anecdotal: Johns’ (1986: 157) experience is that “students fairly soon overcome this first aversion”, while for Lamy and Klarskov Mortensen (Reference Lamy and Klarskov Mortensen2007), “our experience is that you cannot overestimate the students’ need for familiarity with the appearance of concordances, and their need for guidance as to how to derive conclusions from lists of citations.”

The second main aim of the present study is thus to see whether learners can derive useful information from concordances without training.

3.1 Population

The participants were first-year students enrolled at an engineering college in the north east of France. The vast majority were male (86%) and native speakers of French (94%), average age 18.5. They had been studying English for an average of 6.6 years at school before coming to the college, though few could be qualified as advanced, as can be seen from the results of an in-house placement test organised for streaming purposes. Based on a full-length TOEIC of 100 listening and 100 grammar/reading questions, their average score was 51.29%, which corresponds to about 450 on the TOEIC scale, towards the lower end of their “intermediate” band (405–600).Footnote 1 Motivation levels for English are generally quite low in this highly traditional environment (see Brown, Reference Brown2007): the classes are compulsory, and seen by most as something to be endured rather than as preparation in a useful vocational skill. Discounting data from repeat students and those who missed any of the test sessions left 132 participants.

3.2 Linking adverbials

The language point chosen for the present study was linking adverbials, described as follows by Biber, Johansson, Leech, Conrad, and Finegan (Reference Biber, Johansson, Leech, Conrad and Finegan1999: 875) in the Longman Grammar of Written and Spoken English: “the primary function of linking adverbials is to state the speaker/writer’s perception of the relationship between two units of discourse”. Such a general definition would appear appropriate as it derives from a corpus-based pedagogical description of English, and has the advantage of avoiding terminological disputes: the same items are referred to under many names, including linkers, connectors, conjuncts, conjunctions, cohesive markers, connecting adverbials, and so on.

A number of corpus studies (e.g. Conrad, Reference Conrad1999; Aijmer, Reference Aijmer2003) have been devoted to linking adverbials as their use is markedly complex. This also suggests they are likely to be difficult to teach, support for which can be found in several studies of learner corpora showing that they continue to pose problems even at advanced levels, as measured by overuse, underuse, and misuse (Aarts & Granger, Reference Aarts and Granger1998; Altenberg & Tapper, Reference Altenberg and Tapper1998; Cheng & Warren, Reference Cheng and Warren2000; Crewe, Reference Crewe1990; de Haan & Van Esch, Reference De Haan and Van Esch2007; L. Flowerdew, Reference Flowerdew1998; Granger & Rayson, Reference Granger and Rayson1998; Granger & Tribble, Reference Granger and Tribble1998; Granger & Tyson, Reference Granger and Tyson1996).

Probably in part because they are difficult to describe, linking adverbials seem to be one of the areas “generally ignored, neglected or misrepresented in standard works of reference and course materials”, and thus most suited for DDL (Johns, Reference Johns1997). J. Flowerdew (Reference Flowerdew2001) finds differences between corpus data and the way linking adverbials are traditionally taught, and Garton (Reference Garton1996: 8) uses them as an illustration of a “gap between simplified textbook models and authentic native-speaker usage”. For successful mastery of such items, learners would seem to need something more than what can currently be found in standard materials.

It may be possible to learn by analogy from data even without being able to put that knowledge into words in the form of clear and simple rules; or as Widdowson (Reference Widdowson1998: 715) puts it, “a lot of time is wasted in trying to teach things that can only be learned by experience”. In this way, the learners’ findings “may show a far greater degree of abstraction and subtlety” (Johns, Reference Johns1991: 3) than that found in attempts at formal descriptions. For all of these reasons, linking adverbials would seem to be an appropriate language area for investigation in DDL.

Linking adverbials as a whole, according to Biber et al. (Reference Biber, Johansson, Leech, Conrad and Finegan1999: 880), are “considerably more common in conversation and academic prose than in fiction and news”. As the experiment described here was based on newspapers for reasons outlined below, the focus was on the most common type of linking adverbials in this register, namely contrast and concession (Biber et al., Reference Biber, Johansson, Leech, Conrad and Finegan1999: 882). Table 1 shows the ten items selected for the experiment, along with their frequency per million words in the news and spoken registers compared to the entire 100 million words of the British National Corpus.Footnote 2

Table 1 Frequency per million words of linking adverbials in the British National Corpus: in the news and spoken registers, and in the entire 100 million word corpus

For this experiment, these items represent a suitably wide variety of linking adverbials of contrast. They have varying rates of frequency overall, with the least frequent more typical of written registers. Even these were likely to be familiar to the participants in this study, as all feature explicitly in the relevant sectionFootnote 3 of the usage manual available to our learners (Swan, Reference Swan2005: 138–157); however, none were in the list of “130 common mistakes” (p. xxvi–xxix). Some of them are apparently quite simple (e.g. but), others more complex (e.g. besides). Some are frequently classed as “faux amis” (e.g. actually vs. actuellement), and others are deceptively transparent: on the contrary is formally similar to French au contraire, but its use is sufficiently complex for Lake (Reference Lake2004: 138) to advise learners to avoid it altogether. All of these items have posed problems for our learners in the past.

3.3 Authentic language

The language items for the tests and the corpus information sheets were gathered on a single day from WebCorp (Renouf, Kehoe & Banerjee, Reference Renouf, Kehoe and Banerjee2007). The main disadvantages are slow download time (a minor drawback for only ten items), and lack of stability as the corpus changes from day to day. This is true of any internet “corpus”, and although it might be desirable to know the size of the data-base and the relative frequency of the items, these were not essential for this experiment. On the other hand, WebCorp has a number of relevant advantages: in particular, it is freely available on the internet, and allows domain-specific searches of up-to-date sources sufficient for current purposes.

The chosen options were for Google, plaintext output and thirty words to left and right. This last criterion allowed for easier downloading, and meant that the same data could be used for different question formats. Although linking adverbials are not particularly frequent in newspapers (Conrad, Reference Conrad1999), the search was limited to this genre in English, partly as newspapers are familiar to most students, partly to gain up-to-date samples, and partly to ensure a certain minimum quality. Specifically, the search was limited to “British broadsheets”, as students in France are less likely to be familiar with American newspapers and tabloids. The total results for each are given in Table 2.

Table 2 Frequency of target items in corpus

To keep the language authentic it was not altered or edited in any way: each concordance line retained was kept exactly as it was (except for formatting for pagination purposes). However, the 30 concordance lines needed were selected from a random pool of 50, firstly to eliminate doublets and non-cohesive text – an essential stage with the typically messy web-as-corpus (Kilgarriff, Reference Kilgarriff2001). Of the remainder, the most apparently transparent or useful for the experiment were retained – the “most illustrative”, in Stevens’ words (1991: 51). Such selection of concordances is common practice – Kennedy and Miceli, for example, talk of “quality control” in their small samples (Reference Kennedy and Miceli2002: 187) – and can be justified on the grounds that a concordance consists of samples not examples (Gavioli, Reference Gavioli2005: 7): teachers routinely select examples of all kinds. There is no reason to believe it should affect the basic research questions, viz whether lower level learners with no previous corpus training can make sense of a concordance and draw useful information from it.

3.4 Experimental condition

For the experimental condition itself, the students were randomly assigned to one of four groups (Table 3) to test their ability to interpret, apply and recall different information types.

Table 3 Group distribution

The information sheets, each about three pages long, were distributed after the first test, and collected at the end of the session:

1. a) The short context (SC) sheets consisted of five short contexts for each test item gathered using WebCorp, grouped together and with the test item highlighted for ease of reference. Where the search had returned more than one full sentence within the specified span (thirty words either side), all full orthographic sentences were accepted, giving an average of 2.4 full sentences for each sample, or nearly 40 words.

2. b) The keyword in context (KW) sheets featured eight concordance lines for each test item gathered using WebCorp, grouped together and with the test item highlighted for ease of reference. The concordance format provided an average of 46 characters either side, or just over eight whole words.

3. c) The bilingual dictionary (BD) entries for each test item were taken from the Collins-Robert Senior, a fairly large desk dictionary of nearly 2,000 pages of entries. Like most bilingual and unlike most monolingual dictionaries, it is not substantially corpus-based, but is popular in France, familiar to many learners, and large enough to provide sufficient information for our purposes. The information was presented in alphabetical order for the entries in exactly the format of the dictionary.

4. d) The grammar/usage (GU) notes for each test item came from Swan’s Practical English Usage (2005), which uses “simple everyday language” (p. ix). Although not explicitly corpus-based, the recent edition used claims to be “thoroughly checked against large electronic databases (‘corpora’) of authentic spoken and written English” (p. ix). The relevant entries from the section on “discourse markers” formed the basis, supplemented by other entries for the target items where they existed. The information was presented in exactly the format of the manual.

3.5 Test instrument

The participants completed three separate tests: (a) a pre-test; (b) a test where they could consult the information sheets; (c) a recall test conducted ten days later. All students had the same testing instrument in each test regardless of which group they were assigned to; the format was identical between the three testing sessions, only the actual language content was new each time. Only answers which corresponded to the original data were marked correct; other responses might have been judged acceptable in many of the full-sentence contexts, but this necessarily involves an element of subjectivity, and more importantly would bias the scoring as alternative responses would be less likely in the multiple concordances. For the purposes of the experiment, it was considered sufficient that all groups completed both question types and were subjected to the same scoring procedure.

Each test sheet included two exercises, each comprising ten questions corresponding to the ten target items. One set of questions presented ten sets of four concordances; the other presented ten short contexts, each of between one and three full sentences (Figure 1). In both cases, the target words were blanked, and participants had to choose from the list of target items (a to j); a completed example was provided for each exercise type. It seemed plausible that the concordance group might have an unfair advantage if the entire test instrument was based on KWICs, while the others might do better given only full sentences; the combination of the two question types was designed to reduce bias in favour of one or the other.

Fig. 1 Example test questions.

4.1 Results by test and question type

Figure 2 shows some of the clearest findings: the lowest scores were obtained on test 1; there was significant improvement in test 2; and although the scores in test 3 were lower than in test 2, they were still significantly higher than in test 1 (p < 0.05). Overall, more than half of all participants scored better in tests 2 (59.0%) and 3 (50.8%) than in test 1. This is perhaps not surprising, partly because test 2 was administered immediately after test 1 while test 3 took place ten days later, but more importantly because the information sheets were available for consultation during test 2 but not in test 3. It seems, then, that these learners can use the various types of information as a reference (test 2), as well as for recall at a later date (test 3).

Fig. 2 Correct responses by test (T1/T2/T3) and question type Concordance questions; full sentence questions; average of both question types.

As regards the test design, it is significant (p < 0.01) that students scored higher on the concordance questions than they did on the full sentence questions: 19.47% and 11.15% respectively over the three test sessions. This confirms Stevens’ (1991) results, and the pattern holds however we look at the data, for each level and for each group, so the overall low scores cannot be attributed solely to the use of concordances as a test instrument. A more likely cause is the use of authentic language, as it was used in both test types, agreeing with the findings of a number of researchers (e.g. Koosha & Jafarpour, Reference Koosha and Jafarpour2006; Cobb, Reference Cobb1999a).

4.2 Results by level

As it is often alleged that corpus data are only appropriate at higher levels, we also need to examine the effect of proficiency. The sample population was divided into three equal groups according to level determined by their placement test scores, the highest averaging 64.49% (at the lower end of TOEIC’s “basic working proficiency”), the others 50.34% (“intermediate”) and 40.27% (“elementary”) respectively. Figure 3 shows that the highest level group fared best in the experiments on all counts, though it should be remembered that even they were only intermediate in level.

Fig. 3 Correct responses by level (L1=highest) over the three tests (T1/T2/T3) Concordance questions; full sentence questions; average of both question types.

This basic pattern is to be expected, but the differences are fairly small: there was no significant difference between levels 2 and 3, and the difference with the highest level was only just significantly different at the p < 0.05 level. It is important too to note that all levels showed the same pattern of evolution over the three tests: lowest on test 1, highest on test 2, with a drop on test 3 (although the results were still significantly higher than in test 1; p < 0.05).

4.3 Information types

The crucial stage is to examine the effect of the different types of information for tests 2 and 3. This can be seen clearly in Figure 4, which shows that the two groups that had corpus information (SC and KW) managed to use this information more effectively as a reference source in test 2 than the groups which had the traditional pedagogical information (BD and GU). When it comes to recall, however, in test 3, the differences are no longer significant between any of the four groups (p > 0.05): there is thus no indication that having had corpus data as a reference is better or worse than traditional sources for learning itself.

Fig. 4 Change in correct responses between tests (T1/T2/T3), by data type SC = full-sentence group; KW = key-word-in-context group; BD = bilingual dictionary group; GU = grammar/usage manual group.

Table 5 provides the scores for each group in each test, along with the percentage rate of change between tests. On average, students scored 37% higher in test 2 than in test 1, with the corpus groups showing the biggest improvements: +43% for the SC group and +91% for the KW group. On the other hand, the group with the bilingual dictionary information (BD) showed a more modest improvement (+26%), while the group with the grammar/usage information (GU) actually decreased by –1%, although this last result is not significant (p > 0.05). Between tests 2 and 3, as we have already seen, there was a substantial drop: −12% on average over all groups. The biggest drop was in the two corpus data groups (SC and KW), which had previously recorded the biggest increases; the changes for the other groups were not significant. A more useful point of comparison for test 3 is against test 1. All groups show an improvement here, on average +21%. The GU group increased by only +6%; this insignificant difference may be accounted for by familiarity with the test design alone. The other groups increased by around +25%, a significant improvement (p < 0.05) although there was no significant difference between these three groups.

Table 5 Correct answers and percentage changes between tests (T1/T2/T3), by information type

SC = full-sentence group; KW = key-word-in-context group; BD = bilingual dictionary group; GU = grammar/usage manual group.

An alternative approach is to look at the number of participants in each group who improved their scores. The group where most scored better was the KW group: 79% of them scored higher in test 2 than in test 1; 65% also scored higher in test 3 than in test 1. In the GU group, however, only just over a third improved their scores.