2.1 Generative theory of multimedia learning

Mayer and Sims (Reference Mayer and Sims1994) state that one of the most important functions of instructional materials is to assist students in constructing referential connections between two forms of mental representation: the verbal representational system and the visual representational system. These referential connections are most easily built when both verbal and visual materials are presented concurrently. Their experiments found that predominantly spatial learners could devote more cognitive resources to building referential connections between visual and verbal representations of the presented material (op. cit.).

Mayer's (Reference Mayer2001) generative theory of multimedia learning posits that learners engage in three major processes – selecting, organising, and integrating – when they are presented with visual and verbal information, such as illustrations and text. When first presented with a text, the learner must select the relevant words to be retained as a text base in the verbal working memory; when first presented with illustrations, the learner must select relevant images to be retained as an image base in the visual working memory. The learner must then organise the text base into a coherent verbal representation and the image base into a coherent visual representation. The last phase of the generative theory is that the learner must integrate the verbal and visual representations by making one-to-one connections between features of the two representations. According to this theory, meaningful learning is enhanced when a learner can construct and coordinate visual and verbal representations of the same material (Plass, Chun, Mayer, & Leutner, Reference Plass, Chun, Mayer and Leutner1998).

Chun & Plass (Reference Chun and Plass1996b) tested how reading comprehension can be facilitated through the application of multimedia tools in a language learning program. In their study, 160 university students were enrolled in a second-year German language program, which used the multimedia application CyberBuch. The results indicated that a dynamic, visually advanced multimedia tool did facilitate overall comprehension and that annotating individual vocabulary items with both visual and verbal information was more useful than providing only verbal information. Their findings support the dual-coding theory and its extension to multimedia learning; they also emphasise the importance of both visual and verbal information as reinforcement in both top-down (knowledge-based strategy) and bottom-up (text-based strategy) processing in reading in a foreign language.

Mayer & Moreno (Reference Mayer and Moreno1998) tried to test a straightforward prediction of a dual-processing theory of working memory. Their assumptions of dual-processing theory are as follows: (1) working memory includes an auditory working memory and a visual working memory, which are analogous to the phonological loop and visual-spatial sketch pad, respectively; (2) each working memory has a limited capacity; (3) meaningful learning occurs when a learner retains relevant information in each memory store, organises the information in each memory store into a coherent representation, and makes connections between corresponding representations in each memory store; and (4) connections can be made only if corresponding pictorial and verbal information are in working memory at the same time. Mayer and Moreno conducted two experiments to determine whether there was evidence for dual processing systems in working memory. They found that, after viewing these two animations, when describing the major steps in the process of lightning forming or the operation of a car's braking system, learners who had had concurrent narration outperformed those who had had concurrent on-screen text. They concluded that multimedia learners can integrate words and pictures more easily when the words are presented auditorily, rather than visually.

Plass et al. (Reference Plass, Chun, Mayer and Leutner1998) extend the generative theory of multimedia second-language learning. They claim that two aspects of language learning are targeted, namely the learning of individual vocabulary items and the overall comprehension of text reading. The application of the generative theory of multimedia learning to vocabulary learning suggests, first of all, that learners of a second language have two separate verbal systems and a common imagery system. Second, in addition to translations of words linking the two verbal systems, storage in the second verbal system has an additive effect on learning. Words that are coded dually in two modes (verbally and with pictures) would be learned better than those coded only verbally. In contrast to vocabulary learning, which involves rote learning on a low level, reading comprehension is a constructive process that involves the construction of meaning on a higher level. According to the generative theories of comprehension, when reading a text, learners must build referential connections in working memory between the mental representations of ideas or propositions that have been presented in different modes. Comprehension occurs when these connections are stored in long-term memory, but storage may be hindered if learners are not able to build long-term memories. We conducted an experiment to determine the visual and verbal preferences in a second-language multimedia learning environment. The participants were 103 English-speaking college students who were enrolled in second-year German language courses. The students were asked to read a 762-word German language story presented by a computer program. As a tool to assist learning key words in the story, students could choose to view a translation on the screen in English (i.e., verbal annotation) or view a picture or video clip representing the word (i.e., visual annotation), or both. The researchers observed that students were able to remember word translations better when they had selected both visual and verbal annotations during learning than when they selected only one or no annotation; in addition, students comprehended the story better when they had the opportunity to receive their preferred mode of annotation. These findings are consistent with the generative theory of multimedia learning, which assumes that learners actively select relevant verbal and visual information, organise the information into coherent mental representations, and integrate these newly constructed visual and verbal representations with one another.

2.2 Literature review on captions

As discussed by Markham (Reference Markham1999), subtitles refer to “on-screen text in the native language combined with the second language soundtrack”, and captions refer to “on-screen text in the second-language combined with the second-language soundtrack” (op. cit.: 321). The literature review here is presented in two parts, which discuss captions and subtitles separately. Reviewing the studies of subtitles and captions can provide us with an understanding of their independent effects on the first language (L1) and the second language (L2).

Some subtitle studies provide evidence that subtitles are a useful tool for learning a second language (Borràs & Lafayette, Reference Borràs and Lafayette1994; Koolstra & Beentjes, Reference Koolstra and Beentjes1999). Borràs and Lafayette (Reference Borràs and Lafayette1994) found that multimedia presentations including English subtitles facilitated better general comprehension of material by native English speaking university students who studied French as a foreign language. Investigating subtitled television programs viewed by Dutch children in grades 4 and 6, Koolstra and Beentjes (Reference Koolstra and Beentjes1999) designed three experimental conditions: (1) watching an English television program with Dutch subtitles; (2) watching the same English program without Dutch subtitles; (3) watching a Dutch television program (as a control group). Their results showed that the subtitled group significantly outperformed the other two groups in vocabulary acquisition. Furthermore, the results showed that there was no interaction effect found between conditions. Koolstra and Beentjes (Reference Koolstra and Beentjes1999) conclude that both sixth and fourth grade Dutch children learned English words from a subtitled English television program. The results of these studies showed that L1 on-screen text might contribute to L2 learning when subtitles are presented in the L1.

The positive effects of captions on L2 learning have been documented in earlier research. Captioned video material improves L2 learners’ reading vocabulary knowledge (Neuman & Koskinen, Reference Neuman and Koskinen1992), listening word recognition (Markham, Reference Markham1999) and reading and listening comprehension (Garza, Reference Garza1991; Markham, Reference Markham1989). In these studies, video materials were presented with on-screen text in the second language. In general, these studies provide evidence that L2 captions promote L2 learning. However, captioning may not be useful for enhancing all L2 learning. Taylor (Reference Taylor2005) indicates that the length of time a learner has been studying the L2 probably has an impact on the effects of captions. By testing 85 beginner students of Spanish, Taylor (Reference Taylor2005) found that there were no significant differences between the captioning and no-captioning groups on either free recall of, or multiple-choice tests on, the information from the video. The conclusion from Taylor's study is that “captioning might not be as effective for enhancing beginning learners’ comprehension as it [was] for more experienced learners” (op. cit.: 426). From Taylor's study, we see that the learner's L2 proficiency may play a role in determining the effects of captions. In this study, the learner's proficiency in English was treated as an independent variable.

Given the positive effects of L1 and L2 on-screen text, another research path focuses on investigating which type of on-screen text better promoted L2 learning. Using L1 and L2 as on-screen text, Markham, Peter, and McCarthy (Reference Markham, Peter and McCarthy2001) examined the effects of using L1 (English) captions, L2 (Spanish) captions, and no captions with an L2 (Spanish) soundtrack on university-level Spanish as a Foreign Language students’ comprehension of DVD material. The results of a written summary and a multiple-choice vocabulary test showed that the group with English captions significantly outperformed the group with Spanish captions, which in turn, significantly outperformed the group without any captions. As shown in Markham et al. (Reference Markham, Peter and McCarthy2001), L1 captions yielded better outcomes than L2 captions.

There are some empirical gaps in what has been tested. First, the languages investigated were in the same language family. Second, in the studies reviewed, only one language with captions was used in an experimental condition. Third, except in the studies by Taylor (Reference Taylor2005) and Markham (Reference Markham1989), few studies examined the effects of captions by considering learners’ target language proficiency. In contrast to earlier studies, the present study uses two languages from two different language families with captions and presents them simultaneously on the screen to examine whether proficient and less proficient learners were significantly different in their L2 learning.

3.1 The participants

A total of 32 fourteen-year-old students from five classes were selected from a cohort of eight classes at a public junior high school in Taiwan. At the junior high school stage in Taiwan, students have experience in learning English from elementary school level. The participants were divided into two proficiency groups: sixteen students with high proficiency in English and sixteen students with low proficiency in English. Their proficiency in English was measured by scores from achievement tests, which had been conducted in the previous semester. The selection criteria for the two groups were as follows:

1. 1) Students with high English competence had average marks that were above 80 in the mid-term and final examinations in grade seven.

2. 2) Students with low English competence had average marks of approximately 60 in the same tests in grade seven.

Students differing in their proficiency in English were selected and assigned to each of the four captioning groups based on their performance level and the male-to-female ratio. The four groups included no captions (M1), Chinese captions (M2), English captions (M3), and Chinese + English captions (M4). There were eight students in each group, four with high marks and four with low marks; four males and four females.

3.2 Study tools

A multimedia reading program for testing and observations was designed by the first author and programmed in Action Script.

3.2.1 The multimedia reading program

This multimedia system has two lessons aimed at the two different student levels described in section 3.1: the first is “Sleeping animals” (Fig. 1), and the second is “Snails” (Fig. 2). Four types of learning modalities with different captions were made from the two articles using Flash software.

Figure 1 scene from “Sleeping animals”

Figure 2 scene from “Snails”

The article in the first lesson is called “Sleeping animals” (O'Neil, Reference O'Neil1999b). The main scenes are divided into four learning units, which are about a lizard, a snake, a bat, and people, respectively. Descriptions of the habitat and mode of sleeping for the first three learning units are provided. The sentence pattern is simple, and there is a high degree of similarity.

The first scene is about a lizard and contains three sentences, which are designed as follows:

1. 1) This is a lizard.

2. 2) It can sleep on a branch.

3. 3) It can sleep hanging on.

The second scene is about a snake and proceeds similarly:

1. This is a snake…

The article in the second unit is called “Snails” (O'Neil, Reference O'Neil1999a). The main scenes can be divided into two acts about the snail's organs and its way of life. Compared to the first unit, this unit has more complex sentence structures, and there is no repetition in the sentence patterns.

The first scene is about the snail's organs and contains three sentences that are designed as follows:

1. 1) Snails are animals with shells on their backs.

2. 2) Their eyes are on the end of their feelers.

3. 3) The breathing hole is under their shell.

The second scene is about the snail's way of life and includes sentences such as the following:

1. 1) This snail makes slimy mucus…

3.3 Study procedure

3.4 Data analysis

Quantitative and qualitative data were collected in this study. For the quantitative data, SPSS for Windows was used to carry out the statistical analysis and processing. This included descriptions of the sample, frequency distribution, and percentage analysis for each variable, means, standard deviations, and three-way ANOVA. For the qualitative data, the researcher wrote out the interview verbatim to complement the parts that were unclear or needed further exploration in the quantitative data about the students’ understanding and attitudes. Then an analysis was performed using the three qualitative analytic phases developed by Humberman and Miles (Reference Humberman and Miles1994): data reduction, conclusion drawing, and verification.

5.1 In L2 multimedia programs, animated images and voice-overs had more impact than captions

This study found from the post-testing scores that the students’ vocabulary recognition and application did not show a significant difference between the four caption types after viewing the two animations.

Neither the presence or absence of captions nor the inclusion of Chinese and English captions together or separately resulted in any difference in cognitive understanding. The group that had voice-overs but no captions produced the same effect described by Mayer and Moreno (Reference Mayer and Moreno1998). That is, the combination of visual animation and audio voice-overs produces enough learning inputs for L2 programs.

There was no significant difference, therefore, with respect to learning across the four caption types. This is different from the results of most related research, which has found captions to be more useful and that voice-overs are more helpful than texts (Chun & Plass, Reference Chun and Plass1996a; Plass et al., Reference Plass, Chun, Mayer and Leutner1998).

We postulate that the possible reasons for this difference include the following. (1) The present research adopted a sentence-by-sentence and scene-by-scene approach, with viewing, recording, further viewing, and so on. The process was slow and full of interruptions, which might have affected the students’ overall understanding and thinking. (2) As Kost, Foss, and Lenzini (Reference Kost, Foss and Lenzini1999) discovered in their research on foreign language reading, “visual aids are most beneficial in terms of reading comprehension of a foreign language”. We postulate, therefore, that animated images will have a more significant impact on the level of comprehension than the presence or absence of captions and caption types. (3) The participants might have selected the information they needed at the right time, though there was an abundance of information presented at the same time.

5.2 The effect of captions in L2 multimedia on reading comprehension varies, depending on the student's L2 proficiency

During the retelling process for the first lesson, with easier sentence patterns, we found that having either English captions or Chinese + English captions was more helpful than not having any captions for less proficient learners. This was also the case for the second lesson with more complex sentence structures, as the less proficient group also did better with Chinese + English captions than with no captions, when asked to repeat in English the sentences that were in the animations. This result is consistent with Koolstra and Beentjes’ (Reference Koolstra and Beentjes1999) finding that young children could acquire elements of a foreign language by watching subtitles in video programs; the result is partially similar to Markham, Peter and McCarthy's (Reference Markham, Peter and McCarthy2001) finding that the group with English (L1) captions performed better than the no-captions or Spanish (L2) caption groups. Lin's (Reference Lin2005) experiment showed that the combination of voice-overs in the L1 and captions in the target language can alleviate the cognitive burden of learners. It provided evidence of an effect with either L1 captions or narration in L2 multimedia.

This result shows that students of different proficiency levels show different responses to different caption types; this is especially evident in the less proficient group. With simple sentence structures, having Chinese + English captions or English captions paired with animations was sufficient to give the support needed for less proficient students. As for the more complex sentence structures, the less proficient group had their best rate of correct responses in recalling the sentences in the animations when having Chinese + English captions.

Seen in the light of the modality effect and generative theory (Mayer, Reference Mayer2001), we then postulate that a less proficient student is more likely to selectively take in material that they notice. They do not seem to be affected by the presence of too much information; therefore, with simple sentence structures, they can select the information they need when viewing animations with Chinese + English captions or English captions. Both of these caption styles allow them to understand foreign sentences. In the more complex sentence structures, there is a better outcome in repeating a sentence they have just heard only when they are given Chinese + English captions. These differences do not occur for more proficient students. As Katchen (Reference Katchen1996) concluded, for the more advanced student, native (Chinese) captions might become a hindrance because it is easy to become dependent on them, and they can even make the student slower by distracting him.

In analysing the above two findings, it seems that there was no obvious effect on reading comprehension. There was only a small difference between students of different levels of proficiency. For more proficient students, there was no significant difference in their learning, regardless of what type of captions they saw. For the less proficient students, providing English or Chinese + English captions was helpful for comprehending simple sentence structures; with complex sentence structures, only Chinese + English captions had a positive effect on the correct repetition of the sentences.

5.3 The effect of a multimedia L2 program with captions on vocabulary acquisition is very slight

Based on post-test scores, students’ vocabulary recognition and vocabulary use did not show a significant difference across the four captions types after viewing the two animations. In this study, the group with voice-overs but no captions did not perform better or worse than the groups with both voice-overs and captions. That is, the presence or absence of captions in the target or the native language have no significant effects on the outcome of vocabulary learning when learning a foreign language.

The concurrent presence of both Chinese and English captions had no effect on students’ vocabulary learning. Based on the generative theory, combining Chinese captions (as supplemental visuals in Chinese) with English ones can alleviate the cognitive burden on the students. However, this was not the case in this study. In this study, all of the four learning styles had voice-overs and animations; therefore, the difference between each group was not very significant. The most important information had already been transmitted to the working memory area through the visual animations and voice-overs, resulting in the difference between different caption types not being significant.

5.4 Animations were most beneficial when headings affected choice of imagery

When the students were asked what they noticed as they watched the animations, all of them said they saw the animation images. Up to 93.8% of participants thought that animations formed a deep impression; all participants thought that animations were beneficial in learning English.

At the beginning of each L2 lesson, headings were presented to show the title of the lesson. In the animation “Sleeping animals”, the background (showing flowers and a girl) attracted the viewer's attention the most, whereas in the animation “Snails”, apart from the moon in the last scene, which was the most conspicuous, the main character, the snail, was most able to attract the viewer's attention. This shows that headings, which are a key visual element in multimedia, not only deliver messages about the content, but also influence the viewer's choice of image memory.

The feedback compiled following student interviews demonstrated that the highest proportion of students liked the animations and the content of the stories, followed by those who liked the voice-overs and captions.