Subtitle speed

Subtitle speed, also referred to as “presentation rate” or “subtitle rate” and even “reading speed” (although the latter is obviously problematic), is a measure of the length of time the subtitle stays on screen, as a factor of the amount of text that has to be read during its display. Researchers in the field of audiovisual translation (AVT) are well aware of the fact that “subtitle speeds are not set in stone; they differ from country to country and even from company to company” (Szarkowska & Gerber-Moron, Reference Szarkowska and Gerber-Morón2018; pp. 2). Previous studies on this topic have often included a comprehensive overview of different traditions regarding subtitle speed (e.g., d’Ydewalle, Van Rensbergen & Pollet, Reference d’Ydewalle, van Rensbergen, Pollet, O’Regan and Levy-Schoen1987; Szarkowska & Bogucka, Reference Szarkowska and Bogucka2019; Szarkowska & Gerber-Moron Reference Szarkowska and Gerber-Morón2018; and Romero-Fresco, Reference Romero-Fresco2019). Some countries such as Australia (ACMA, 2016), Canada (CRTC, 2016), France (CSA, 2011), Spain (AENOR, 2012), the UK (Ofcom, 2017), and the US (FCC, 2014) have developed official rules regarding some technical parameters to be followed by subtitlers. The recommended subtitle speeds range from 10 to 20 cps depending on the guidelines’ provenance, target viewers (deaf, hard of hearing, hearing, age group), and the nature of on-screen-text (intra- vs. interlingual subtitles, but also scrolling vs. chunked subtitles). However, none of these guidelines on subtitle speed are based on sound empirical evidence, nor has the impact of different speeds on subtitle reading been investigated exhaustively (but cf. Szarkowska & Gerber-Morón, Reference Szarkowska and Gerber-Morón2018).

In addition to the fact that there is a large variability in terms of recommended subtitle speeds (which is often a recommended maximum speed), the actual subtitle speed in any given video often varies dramatically. In order to get a sense of the distribution of actual subtitle speeds (regardless of existing guidelines), we analyzed 23356 English same-language subtitles extracted from the top 11 “Blockbuster” movies on Netflix Australia on July 13, 2020.Footnote ² Although this is not a representative sample of movies, even on Netflix, it does give a sense of the variability of subtitle speed in mainstream subtitled products. The distribution of speeds is presented in Figure 1. What is interesting is that, in spite of the Netflix guidelines that English captions should be a maximum of 20 cps (Netflix, 2020), the average speed in this small corpus of subtitles was 12.6 cps, with a range of 0.8 cps to 50.4 cps. Importantly, a total of 15.2% of the subtitles were faster than 20 cps. This is not a quality judgment, but rather an indication of the variability in subtitle speed found in widely available films on a popular streaming platform, as well as the fact that this range is often obscured when average speeds are reported. It also confirms that the speed at which subtitles are delivered varies throughout audiovisual products due to the pace of its different parts, speech rates of individual speakers, or the strictness and consistency in following specific editing rules (cf. Fresno & Sepielak, Reference Fresno and Sepielak2020 for an overview).

Figure 1. Subtitle speed across a sample of 11 popular films on Netflix.

Panel A: Distribution of subtitle speed (in number of subtitles).

Panel B: Percentage of subtitles at different speed intervals.

Previous experimental studies on subtitle speed

Previous studies that examined how subtitle speed impacts processing and comprehension largely focused on global eye movements (i.e., averaged across videos without taking individual words into account) and used diverse post hoc measures of comprehension, subtitle, and scene recognition and viewers’ attitudes and preferences (cf., e.g., Koolstra, van der Voort, & d’Ydewalle, Reference Koolstra, Van Der Voort and d’Ydewalle1999; Romero-Fresco, Reference Romero-Fresco2019; Szarkowska et al., Reference Szarkowska, Krejtz, Pilipczuk, Dutka and Kruger2016; Szarkowska & Gerber-Morón, Reference Szarkowska and Gerber-Morón2018). Due to the lack of comparability between some experimental designs as well as the nature of authentic audiovisual materials used in those studies, the full picture of how subtitle speed affects multimodal processing is unavoidably distorted, making subtitle speed one of the most misconstrued topics in AVT research.

Word skipping and rereading

When reading, our eyes constantly alternate between fixations and saccades. Fixations are gaze points when the eyes stop scanning the scene, holding the central foveal vision in place so that the visual system can take in detailed information about what is being looked at. Saccades are rapid movements of the eyes from one fixation to the next (Rayner, Reference Rayner1998, Reference Rayner2009). Extensive eye-tracking research supports the idea that information intake occurs mostly while we fixate a specific point and not during the fast saccadic movement when we are functionally blind (Rayner, Smith, Malcolm, & Henderson, Reference Rayner, Smith, Malcolm and Henderson2009).

In spite of our perception that we can see our full field of vision in detail, only the center of vision, namely 2° of visual angle known as the fovea, provides us with enough visual acuity during fixations to identify the fine details of words and objects (Rayner & Morrison, Reference Rayner and Morrison1981). The region that extends 5° from the center of vision is known as the parafovea, which allows parafoveal processing such as parafoveal preview of words or objects around the fovea. In reading, there is only a limited region for effective visual processing, which is known as the perceptual span. Perceptual span extends asymmetrically from the center of vision for about 3-4 characters to the left and up to 14 characters to the right (in reading a language like English), although we can only identify words that fall within a space of around 7-8 characters to the right of a fixation (cf. Rayner, Reference Rayner1986, Reference Rayner1995). The rest of the visual field is known as peripheral vision and can only be used to track previously identified objects and identify sudden onsets or movement, such as a moving object on the video image while reading a subtitle, or the onset of a subtitle while looking at the image. In order to process the object or to start reading the subtitle, the viewer has to shift his or her gaze to bring it into the fovea.

In studies on the reading of static text, word skipping often refers to words not being fixated during first-pass reading and is found to increase with decreasing word length and increasing word frequency (cf. Angele, Laishley, Rayner & Liversedge, Reference Angele, Laishley, Rayner and Liversedge2014; Drieghe, Reference Drieghe2008; Drieghe, Rayner & Pollatsek, Reference Drieghe, Rayner and Pollatsek2005; Rayner, Slattery, Drieghe & Liversedge, Reference Rayner, Slattery, Drieghe and Liversedge2011). However, as pointed out by Reichle (Reference Reichle2021: 366), “most words have to be fixated to be identified” during reading because of the restrictions of perceptual span and visual acuity. In this study, we will therefore refer to word skipping as those words that were not fixated at least once. Reichle (Reference Reichle2021) also points to the fact that shorter words such as function words are more likely to be skipped. As documented by Rayner (Reference Rayner1998, Reference Rayner2009), between 70% and 80% of words are fixated at least once, which means that in the reading of static texts, between 20% and 30% of words are skipped. It stands to reason that the syntactic and discourse processing of subtitles would be impaired if a higher skipping ratio is observed, which would mean that a large portion of the words could not be identified and processed.

In addition to looking at word skipping during subtitle reading, we will also look at the extent to which words that are located at the end of subtitles are skipped. Should a larger proportion of words at the end of subtitles not be fixated, it would indicate that viewers were unable to read the subtitles to completion, which would then make it less likely that post-lexical integration would occur when linguistic representations of words are converted into propositional representations (c.f. Rayner, Sereno, Morris, Schmauder, & Clifton, Reference Rayner, Sereno, Morris, Schmauder and Clifton1989; Warren, White, & Reichle, Reference Warren, White and Reichle2009).

A key feature of reading is the fact that skilled readers routinely make regressions (eye movements in the opposite direction to the direction of reading, that is, saccades from left to right in the reading of a language like English) (Schotter, Tran, & Rayner, Reference Schotter, Tran and Rayner2014). According to Rayner, Pollatsek, Ashby and Clifton (Reference Rayner, Pollatsek, Ashby and Clifton2012), between 10% and 25% of eye movements are regressions, during which viewers revisit sections of the text either because of an oculomotor error, or, more interestingly, because they did not obtain sufficient information during first-pass reading and have to return to it to resolve an ambiguity, or correct for comprehension breakdown (cf. Eskenazi & Folk, Reference Eskenazi and Folk2017; Inhoff, Kim & Radach, Reference Inhoff, Kim and Radach2019; Schotter, Tran, & Rayner, Reference Schotter, Tran and Rayner2014). Inhoff et al. (Reference Inhoff, Kim and Radach2019) distinguish between “small” regressions (correcting for saccadic error when the fixation lands on the wrong word or skipped a word accidentally, and incomplete processing where the reader has to return to a partially processed word), and “large” regressions which are related to text comprehension (where segments of the text are reread for comprehension). A reduction in time for viewers to make regressions (as a result of faster subtitle speeds) may therefore interfere with comprehension (see Schotter, Tran & Rayner, Reference Schotter, Tran and Rayner2014; and Cook & Wei, Reference Cook and Wei2019; Inhoff et al., Reference Inhoff, Kim and Radach2019; Metzner, von der Malsurg, Vasishth, & Rösler, Reference Metzner, von der Malsburg, Vasishth and Rösler2017). In other words, and more specifically, given the increased difficulty of integrating the meaning of low-frequency words into the sentence context suggested in the literature (cf. White, Drieghe, Liversedge & Staub, Reference White, Drieghe, Liversedge and Staub2018), a reduction in regressions will most likely have a detrimental effect on sentence comprehension.

Hyönä, Lorch, and Rink (Reference Hyönä, Lorch, Rinck, Hyönä, Radach and Deubel2003, see also Hyönä, Lorch & Kaakinen, Reference Hyönä, Lorch and Kaakinen2002) distinguish between first-pass rereading time as the sum of all reinspective fixations during first-pass reading, and lookback fixation time (or second-pass fixation time) as the sum of all fixations during a second-pass reading of a text. In their view, reinspections are fixations on a word that has already been fixated and are initiated by a regression, but followed by either regressive or progressive fixations. The present study adopts this approach by investigating rereading of words in subtitles when a previously fixated word is refixated from any direction after the eyes moved out of the word for the first time.

In reading fast subtitles, readers may not have the time to make either lookback fixations or first-pass revisits to resolve syntactic or lexical issues. Studying rereading as evidenced by lookbacks or revisits to words could provide valuable information on the reading process in a multimodal context. More importantly, unlike in static reading, revisits to words (or rereading behavior) during subtitle reading could be initiated horizontally (from elsewhere within the subtitle) or vertically (from the video to the subtitle). These two types of rereading behavior most likely serve very different purposes, whereas rereading following horizontal eye movements likely reflect viewers’ engagement with linguistic processing (making sense of the text), rereading following vertical eye movements (closer to lookback fixations) initiated from within the video image probably serve the purpose of integrating the text with other visuals to build a situation model of the video (to make sense of the story) (see also Laeng & Teodorescu, Reference Laeng and Teodorescu2002).

As such, rereading is associated with global processing strategies (Hyönä, et al., Reference Hyönä, Lorch and Kaakinen2002) and regressions for text comprehension (Cook & Wei, Reference Cook and Wei2019; Inhoff, et al., Reference Inhoff, Kim and Radach2019). Since subtitles are typically single sentences, it is possible to relate rereading after horizontal eye movements to lexical processing and comprehension at sentence level, whereas rereading after vertical eye movements is most likely related to higher-level comprehension found in the reading of longer texts (see Cook & Wei, Reference Cook and Wei2019), and indeed reading of subtitles on video. A reduction in either of these types of rereading would therefore compromise, or at least detract from, both the local, linguistic processing of the subtitles, and the global processing or integration of the subtitles into the situation model of the video.

Reading in multimodal contexts

While a considerable amount has been learned about the mental processes engaged in the reading of static text where the reading pace is under the reader’s control (see Reichle, Reference Reichle2021), little is known about the cognitive processes of reading in dynamic and multimodal contexts as is the case when reading subtitles in video. When watching a subtitled film, the cognitive systems for reading have to be coordinated with those needed for the processing of other information (e.g., background video content or auditory input) within a limited period of time, at a pace dictated by the film and over which the viewer has no control (cf. Kruger & Steyn, Reference Kruger and Steyn2014).

A recent article by Liao, et al. (Reference Liao, Yu, Reichel and Kruger2021) presents a schematic diagram of the perceptual and cognitive processes supporting multimodal reading, such as reading subtitles in film (Figure 2).

Figure 2. A multimodal integrated language framework (adapted from Liao et al., Reference Liao, Yu, Reichel and Kruger2021).

This framework presents the different perceptual and cognitive processes that support reading in an integrated manner in a multimodal context. What the figure illustrates is that viewers’ comprehension of subtitled video does not rely solely on the subtitles or the image or the soundtrack, but that the objects identified in the image, as well as the spoken dialogue and the sounds contribute to the creation of a situation model that forms a context in which the subtitles are read. However, if a word is not identified, or is misunderstood, it could compromise the sentence processing, which would impact negatively on the text base and situation model, just as failure to identify an object could impact negatively on the situation model.

Although the multimodal integrated language framework provides a good account of how different sources of information interact and contribute to the comprehension of subtitled videos, it should be noted that the efficiency of subtitle processing is highly context-specific and derives from the very nature of audiovisual materials while also being moderated by viewer characteristics such as their ability to manage their attentional resources, their language proficiency, reading speed, prior knowledge, etc.

The coordination of eye movements and the integration of language in a multimodal framework as in Figure 2 is also subject to the availability of cognitive resources. Multimodal language integration is dependent on the accurate identification of words and objects. When any of the serial processes (such as word or object identification, or sentence processing) encounters a problem such as ambiguity or missing information, more cognitive resources have to be assigned to the processing of that item, with the result that other items that appear at the same time could easily be missed. In other words, when a viewer is focusing exclusively on one element, he or she becomes “blind” to other elements such as changes (also known as “change blindness”), or unexpected items (also known as “inattentional blindness”) (see Jensen, Yao, Street & Simons, Reference Jensen, Yao, Street and Simons2011). Romero-Fresco (Reference Romero-Fresco2019) uses the term “subtitling blindness” to refer to the fact that viewers could miss an important part of the image on screen because they were reading a subtitle.

Limitations in visual perception means that the effective processing of fast subtitles and synchronous video could very well be possible when all the systems can operate smoothly, but when things go wrong, the integration could be compromised. An important reason for this is that faster subtitles take important control away from the viewer since less time is available for text processing. In particular, due to the reduction in time available to read the subtitles, viewers may have less time to make regressions to low-frequency words or to correct for saccadic errors. Solid empirical evidence confirms the existence of systematic differences in the ease (or difficulty) of processing words in the text (see Rayner & Duffy, Reference Rayner and Duffy1986 for overview). In reading, shorter words, more frequent words, and more predictable words receive shorter fixations than longer, less frequent, or less predictable words (see Rayner, Ashby, Pollatsek, & Reichle, Reference Rayner, Ashby, Pollatsek and Reichle2004; Rayner & Duffy, Reference Rayner and Duffy1986). The existence of word frequency and word length effects point to the fact that linguistic complexity influences the reading process and that nodes of linguistic complexity attract increased processing, or, simply put, require more time to process. What this implies for subtitle reading is that any increase in linguistic complexity will have one of two results: either it will slow down the reading process with longer fixations and more regressions, with an increasing chance that sentence processing will not be complete before the subtitle disappears, or, if the reading does not slow down, these nodes of linguistic complexity will not receive sufficient processing. In the present study, we specifically test whether higher subtitle speeds will have an influence on the way viewers process linguistic complexity as evidenced by an increase in rereading of words following eye movements from elsewhere in the subtitle such as regressions, or the way viewers integrate the image with specific words in the subtitle evidenced by rereading of words following vertical eye movements from the image to the subtitle.

Research questions and hypotheses

This study seeks to determine whether an increase in subtitle speed impacts a) the rereading of words following horizontal regressions or revisits from elsewhere in the subtitle, likely as an indication of linguistic processing; b) the rereading of words following vertical eye movements from the image, likely as an indication of the integration of the words in the subtitle and the image to form a situation model; and c) the ability of viewers to identify words and read subtitles to completion (as evidenced by word skipping). Our hypotheses are that high subtitle speeds will result in fewer words being reread after both horizontal and vertical eye movements, resulting in more superficial linguistic processing (as found by Liao et al., Reference Liao, Yu, Reichel and Kruger2021), as well as impoverished situation models. We also hypothesize that an increase in subtitle speed will result in an increase in the number of words skipped both in the subtitles overall (marking a reduction in lexical processing and word identification), and at the end of subtitles (marking a reduction in post-lexical integration) compromising the viewer’s ability to integrate the information in the subtitles with the image.

Methodology

To address the research questions of the present study, we performed new analyses using the comprehension and eye-movement data from Liao et al.’s (Reference Liao, Yu, Reichel and Kruger2021) experiment. In Liao, et al.’s experiment, participants watched videos with subtitles at three speeds (12 cps, 20 cps, and 28 cps) with and without background video. As the present study focuses on the impact of subtitle speed on the integration between the subtitle and the video, we only report on the conditions with video presentation. While the analyses of word skipping and revisits are new, the analysis of comprehension is the same as that in Liao et al. (Reference Liao, Yu, Reichel and Kruger2021), but reported only for the video present condition.

Analyses

While Liao et al. (Reference Liao, Yu, Reichel and Kruger2021) reported a range of local measures (including mean fixation duration, mean saccade length, fixation count, and crossovers between subtitles and the image), as well as global measures (the effect of word frequency, word length and wrap-up using gaze duration, total times, and skipping probability of words in subtitle final positions), the current study reports only on word-based measures related to the rereading of words following vertical or horizontal eye movements, as well as to the skipping of words overall and at the end of subtitles (or incomplete reading).

We report the comprehension scores for each speed condition as an indication of the degree to which participants could recognize the information contained in the subtitles. The hypothesis is that comprehension will be impacted negatively by increasing speed, as the time to process the subtitle decreases.

In order to compare participants’ subtitle reading as a function of subtitle speed, we report word skipping and rereading:

1. (1) To measure the impact of speed on the ability of participants to identify words, we report the percentage of skipped words or the percentage of words that did not receive any fixations.

2. (2) To measure the impact of subtitle speed on the ability of participants to read subtitles to completion, we report the percentage of words skipped after right-most fixation. This is obtained from the number of words in each subtitle after the right-most fixation in the subtitle (i.e., words at the end of the subtitle that are not fixated), divided by the total number of words in that subtitle. Words after the rightmost fixation that were not fixated were coded as 1, and other words were coded as 0.

3. (3) To measure the impact of subtitle speed on linguistic processing, we analyze the percentage of words revisited following horizontal eye movements. Here we look at words that were fixated more than once (excluding refixations during first-pass reading of the word), following a horizontal eye movement. In other words, words fixated more than once were coded as 1, and the rest 0. This measure reflects the extent to which the progressive reading is interrupted by a return to a preceding word before progressive reading resumes.

4. (4) To measure the impact of subtitle speed on global processing and on the ability of viewers to integrate objects in the image with words in the subtitle through rereading, we analyze the percentage of words revisited following vertical eye movements. Here we also look at words that were fixated more than once (excluding refixations during first-pass reading of the word), but this time following a vertical eye movement initiated from within the video image, again coding refixated words as 1, and the rest as 0.

The comprehension and eye-movement data were analyzed using Generalized Linear Mixed Models (GLMMs) via the lme4 package (version 1.1-23) in R (version 3.6.3). For all the analyses, subtitle speed was treated as a fixed factor, with participant and word treated as random effects. Contrasts of the different speed conditions (as factor) were set up using the contr.sdif function to compare the means of each pair of consecutive levels (20-12 cps and 28-20 cps).

Only significant results are reported for brevity. Finally, the emmeans package (version 1.4.7) in R was used to compute the contrasts and extract the estimated means between any two conditions (e.g., 12 cps vs. 28 cps). When fitting a model, we started with a maximal random-effect structure, which was then progressively pruned following the Parsimonious Mixed Model approach (Bates, Kliegle, Vasishth, & Baayen, Reference Bates, Kliegl, Vasishth and Baayen2015). The following models were used for all analyses: DV ∼ Subtitle Speed + (1 | Participant) + (1 | Word). A summary of the models is provided in Appendix 2. Because the six video clips were all selected from the Planet Earth series and thus comparable, a single random effect variable was coded for each combination of video, subtitles, and words in our analyses (assuming no inherent video-subtitle hierarchy).