2. Eye-gaze in interaction

Some early seminal works focused on the role of eye-gaze in interactional settings, inspired by the work on paralinguistic (intonation, volume, pitch) and non-linguistic (gesture, gaze) signalling in conversation analysis. Kendon (Reference Kendon1967), one of the pioneers in multimodal analysis (and more particularly gesture studies), presented a first detailed empirical account of the direction of speakers’ and hearers’ gaze during a face-to-face conversation, based on snapshots of a video-recording (1 frame per 500 ms interval was used as a basis for the analysis) and corresponding transcription of speech. The analysis focused on the role of eye-gaze as an instrument of perception and processing on the one hand, and on its potential as a communicative signal on the other (e.g., for directing attention, managing the organization of the conversation, etc.). The pioneering work by Kendon fostered the interest of researchers across disciplines in the role of gaze behaviour in interactional discourse. In another landmark publication, Argyle and Cook (Reference Argyle and Cook1976) list a range of functions of gaze (cited in Raidt, Reference Raidt2008, p. 49), including the signalling of grammatical breaks, attention or disapproval, providing feedback (cf. also Bavelas Coates, & Johnson, Reference Bavelas, Coates and Johnson2002), etc. In more recent studies, some of these eye-gaze patterns have been studied in more detail and as part of a broader framework, as, e.g., the work on eye-gaze in conversational turn management and information structure by Cassell, Torres, and Prevost (Reference Cassell, Torres and Prevost1999), Jokinen (Reference Jokinen2010), and Novick, Hansen, and Ward (Reference Novick, Hansen, Ward, Bunnell and Idsardi1996).

Despite the detailed empirical analyses presented in the early work on eye-gaze in interaction, these studies typically had to deal with the technological–methodological shortcomings of video-recordings. Although gaze estimations on the basis of video data may be useful for a basic segmentation of the distribution of visual attention (e.g., looking at an interlocutor vs. looking away), they are notoriously coarse-grained and unreliable for more detailed analysis (Kendon, Reference Kendon2004; Streeck, Reference Streeck2008). For instance, video-based analysis does not provide useful information on short fixations (of 200 ms or less), saccades (i.e., fast movements of the eye), and visual scan paths. In order to be able to include this level of detail in the analysis, a different methodological paradigm is needed, viz. eye-tracking. Eye-tracking, or the measuring of gaze points and eye movements during ongoing behaviour, allows for a highly detailed analysis of gaze patterns (see Rayner, Reference Rayner1998, for an overview). With the development of unobtrusive eye-tracking equipment in the last decade, in the form of remote or head-mounted systems, it is possible to measure subjects’ gaze patterns while they are involved in interactive or collaborative tasks. This opens up a vast area of research in (cognitive) interaction studies, including the role of gaze as a directive instrument, the correlation between gaze and gesture, gaze as a disambiguation instrument, interactive alignment in various semiotic channels, etc. In what follows, we provide an overview of some recent work exploring eye-gaze in interaction, using eye-tracking technology.

One of the basic features of interaction is the joint focus of attention of co-participants in the process of establishing the coordinated action that is language use (Clark, Reference Clark1996). One correlate of this basic feature is gaze coordination, or the joint visual focus on relevant aspects of the context (e.g., referents that are the current topic of conversation), also referred to as shared gaze. In a series of experiments, Richardson and colleagues (Richardson, Dale, & Kirkham, Reference Richardson, Dale and Kirkham2007; Richardson, Dale, & Tomlinson, Reference Richardson, Dale and Tomlinson2009) measured the coupling of eye-movements between participants in mediated settings, i.e., with participants viewing a screen rather than a face-to-face setting. The results show a strong tendency towards joint visual attention, as well as an impact of shared background information (common ground) on this coupling. Brennan, Chen, Dickinson, Neider, and Zelinsky (Reference Brennan, Chen, Dickinson, Neider and Zelinsky2008) and Neider, Chen, Dickinson, Brennan, and Zelinsky (Reference Neider, Chen, Dickinson, Brennan and Zelinsky2010) used collaborative visual search tasks for remotely located pairs of people (both wearing head-mounted eye-trackers) to study the relative impact of shared gaze and speech on performance efficiency. The results show that the condition with shared gaze (with participants seeing the gaze cursor generated by the eye-tracker of the other) scored significantly better than (i) solitary search, (ii) a condition with only shared voice, and even (iii) the condition with both shared gaze and shared voice.

A final dimension that needs mentioning in this general overview, as it is of particular relevance to the present study, is the distribution of visual attention across the interactional space of the conversation. Co-participants do not only focus on specific referents that are the current topic of (verbal) communication, or on the speakers and/or addressees (see above). Rather, taking into account the active communicative role that gaze may play as a cue for referent assignment, turn management, etc., we obtain a complex interplay of gaze-as-cause and gaze-as-effect. One much-discussed feature is the so-called gaze cueing effect, which can be defined as the effect that cueing a target (e.g., by looking at it) has on the gaze behaviour of an addressee. Studies on the gaze cueing effect, which date back to early work by Posner, Snyder, and Davidson (Reference Posner, Snyder and Davidson1980), stress its role for joint attention in interaction (Emery, Reference Emery2000; Frischen, Bayliss, & Tipper, Reference Frischen, Bayliss and Tipper2007). Lachat, Conty, Hugueville, and George (Reference Lachat, Conty, Hugueville and George2012) are the first to test the gaze cueing effect in a spontaneous face-to-face setting (rather than in on-screen experiments), however, without the use of the eye-tracking paradigm. Gullberg and Holmqvist (Reference Gullberg and Holmqvist2006) and Gullberg and Kita (Reference Gullberg and Kita2009) focus on one specific case of gaze cueing, using head-mounted eye-trackers, viz. the effect of a speaker focusing on his/her own gesture on the addressee’s gaze behaviour. The studies reveal that a speaker’s gaze at own gestures is a powerful cue for addressees to leave the dominant fixation position (i.e., the face of the speaker) and give overt visual attention to the speaker’s gesture.

3. Interactive alignment: a multimodal approach

At various points in the previous section, it was stressed that efficient communication depends to a large extent on joint (visual) attention and coordination. In fact, coordinated actions in interaction have been observed for a variety of language and bodily parameters, which have been addressed in several disciplines and paradigms. Or, as Richardson et al. (2007, p. 407) aptly put it:

One central effect of coordinated action in dialogue is a high degree of convergence across speakers and their turns. The imitative nature of face-to-face communication has been dealt with in great detail in linguistics and conversation analysis (e.g., Bolinger, Reference Bolinger1961; Carter, Reference Carter1999; Halliday & Hasan, Reference Halliday and Hasan1976; Sacks et al., Reference Sacks, Schegloff and Jefferson1974; Tannen, Reference Tannen1987, Reference Tannen1989), and has been labelled alternatively as ‘accommodation’ (Giles et al., Reference Giles, Coupland, Coupland, Giles, Coupland and Coupland1992), ‘entrainment’ (Brennan & Clark, Reference Brennan and Clark1996), ‘resonance’ (Du Bois, Reference Du Bois2010), ‘convergence’ (Lewandowski, Reference Lewandowski2012), ‘interactive alignment’ (Pickering & Garrod, Reference Pickering and Garrod2004, Reference Pickering and Garrod2006), etc. Rather than presenting an overview of these different accounts (which would be well outside the scope of the present paper), we zoom in on the interactive alignment theory developed by Pickering and Garrod, as it has generated a substantial amount of theoretical and empirical work over the last decade. It focuses on the basic underlying mechanism driving the tendency towards convergence or accommodation, and argues that this is a largely automatic and unconscious process.

According to the interactive alignment theory, interlocutors are primed to use the linguistic input of immediately preceding utterances they have just processed: “We propose that this works via a priming mechanism, whereby encountering an utterance that activates a particular representation makes it more likely that the person will subsequently produce an utterance that uses that representation” (Pickering & Garrod, Reference Pickering and Garrod2004, p. 173). Alignment is defined as a state in which two or more dialogue partners have an identical (or at least highly similar) representation at a particular linguistic level. Alignment at different levels of linguistic representation enhances alignment of situation models, with similarly constructed situation models being a precondition for successful communication. The theory proposes a layered account of interconnected levels of representation (see Figure 1), with alignment at one level leading to alignment at other levels. The growing body of literature on interactive alignment has uncovered the cognitive reality of the phenomenon, most notably at the lexical and syntactic level (see, e.g., Garrod & Pickering, Reference Garrod and Pickering2009, for an overview of the literature). Some recent work has also focused on aspects of alignment at the level of phonetics (Lewandowski, Reference Lewandowski2012; Lewandowski & Schweitzer, Reference Lewandowski and Schweitzer2010; Szczepek Reed, Reference Szczepek Reed2010) and pragmatics (Roche, Dale, & Caucci, Reference Roche, Dale and Caucci2012).

Fig. 1. Interconnected levels of linguistic representation in the interactive alignment model of Pickering and Garrod (Reference Pickering and Garrod2004).

Although the interactive alignment model was construed as a mechanistic model of dialogue in psycholinguistics, and thus focuses primarily on linguistic levels of representation, the basic phenomenon can also be observed in other semiotic channels. Several studies have focused on the strong (and largely unconscious) tendency of dialogue partners towards convergence in non-linguistic behaviour, including gesture (Kimbara, Reference Kimbara2006), posture (Shockley, Richardson, & Dale, Reference Shockley, Richardson and Dale2009; Shockley, Santana, & Fowler, Reference Shockley, Santana and Fowler2003) and eye-gaze (Richardson & Dale, Reference Richardson and Dale2005, cf. references in Section 2). In fact, Garrod and Pickering (2009, p. 296) acknowledge that “interlocutors construct aligned non-linguistic representations. Just as linguistic alignment at one level can enhance alignment at other levels, so non-linguistic alignment can also enhance alignment.”

For the purpose of the present paper, a number of recent studies are of particular interest, as they are instructive for a multimodal approach that also takes into account the temporal dynamics of interactive alignment. Dale, Kirkham, and Richardson (Reference Dale, Kirkham and Richardson2011) studied eye-gaze patterns in a cooperative tangram task. Both participants were given the same set of puzzle pieces, but in a different order, and were asked to arrive at a matching order. A cross-recurrence analysis on the eye-movement data for both participants reveals growing synchronization over time: eye-gaze patterns become more coordinated. McNeill (Reference McNeill, Renals and Bengio2006) presents an exploratory study of the coordination of individual cognitive states in multi-party exchanges. He does so through the study of language, gesture, and gaze as packages of multimodal information (referred to as ‘hyperphrases’) that present a so-called ‘growth point’ or a speaker’s cognitive state, and which are crucial for achieving synchrony between speakers. McNeill explicitly presents his account as a multimodal extension of the interactive alignment model, albeit without buying into the ‘mechanistic’ account of a solely priming-based mechanism.

4. The present study: eye-gaze and multimodal alignment

The present study combines insights from recent work on eye-gaze in interaction (see Section 2) and interactive alignment (see Section 3) to explore the role of eye-gaze in the emergence of alignment at the gestural level. More specifically, we address the question whether gaze cueing by a speaker on his or her own gesture has an effect not only on the gaze behaviour of the interlocutor (Streeck, Reference Streeck1993), but also on the subsequent gesture production by the latter. Do interlocutors who directly focused on a speaker’s gesture exhibit a stronger tendency to use the same (or similar) gestures in subsequent turns than in those cases without direct fixation? We hypothesize that speakers’ and interlocutors’ attention to gesture not only has an effect on the visual attention of interlocutors and information uptake of gestural information (as shown by Gullberg & Kita, Reference Gullberg and Kita2009), but also on gestural alignment across speakers. Drawing on insights from the interactive alignment model, we present an experimental study of the relation between gaze cueing, visual processing, and aligned (gestural) representations.

By linking visual fixation on gestures (on the part of both speakers and addressees) to subsequent gesture production in an interactional setting, we add one important dimension to the study on eye-gaze and information uptake of gesture presented by Gullberg and Kita (Reference Gullberg and Kita2009). Whereas in their study, participants were shown video-recordings of naturally occurring gestures in narratives, after which the information uptake was measured using a drawing task after stimulus presentation, we explore the relationship between visual gesture processing and production in a dialogic setting, using a multimodal dialogue corpus that includes eye-tracking data for both dialogue partners (Section 5).

7. Discussion and conclusion

The main findings of this paper can be summarised in three points. First, SpeakerGaze alone (i.e., the speaker fixating his own gestures) does not affect gestural alignment. This finding ties in with what Wang and Hamilton (Reference Wang and Hamilton2014) reported in their reaction-time experiment: addressees looking at actors in a video were not faster in copying target gestures if those gestures were fixated by the actor, compared to when they were not fixated. Second, InterlocutorGaze (i.e., the interlocutor fixating the speaker’s gesture) does significantly co-occur with higher scores for gestural alignment, but it only does so in the SpeakerGaze– cases (i.e., when the speaker does not fixate his own gesture). The correlation between fixation behaviour and gesture behaviour is in line with Gullberg and Kita’s (2009) claim that “uptake from speaker-fixated trials with addressee fixation did not differ from speaker-fixated trials without addressee fixation” (p. 277). Third, and somewhat surprisingly, our results differ from Gullberg and Kita’s observation that “addressees were more likely to retain the directional information in gesture when speakers themselves had first fixated the gesture than when they had not” (p. 268). In our findings, there is no correlation between gaze cueing and gestural alignment.

The differences between the findings reported here and those of Gullberg and Kita (Reference Gullberg and Kita2009) might be due to a number of factors, of which we discuss the two most important ones here. First, there is a difference in conversation type: the results in this paper are drawn from eye-tracking participants in face-to-face conversations, whereas Gullberg and Kita start from eye-tracked participants watching videos. Moreover, the specific experimental task is different: targeted collaborative tasks here, versus a story-telling task in Gullberg and Kita. Second, although both studies use mobile eye-tracking to measure visual perception, they differ in how they link perception to the processing of perceived gestures. In other words, they differ in determining the dependent variable: gestural alignment in this paper and information uptake in Gullberg and Kita. The former measures the similarity between the gesture fixated and the subsequent gesture produced (using a five-point similarity scale); the latter measures the directional information retained from the fixated gestures (using drawings that participants made after they watched the target gestures on a video screen).

What appear to be contradictory results may, in fact, be indicative of the many functions of eye-gaze in communication and of the intricate relationship between gaze fixation and cognitive attention. Interlocutors in conversations can fixate their own or their partners’ gestures for many different reasons: disambiguating, gaze cueing, signalling uncertainty, deictic referencing, etc. The eye-tracking device is only able to measure visual fixations without, of course, differentiating between these different conversational functions. The issue with interpreting the data boils down to the following: with measuring gaze data we want to tap into low-level cognitive processes (i.e., the correlation between visual perception and gesture production); however, gaze behaviour is not only driven by low-level but also by high-level processes. Marked events in the ongoing interaction or very local, specific communicative goals will just as well drive the (aligned or not) gesture production. Although there might be an automatic, low-level coupling of perception (looking at gestures) and production (using those same gestures), this can at any time be disrupted by higher-level needs. Moreover, and linked to the issue of peripheral vision (see above), there is no one-to-one relation between fixation and attention. Interlocutors might fixate a gesture without cognitively processing it, and the other way around; they might have processed it without a fixation. These issues are important in linking gaze behaviour (processing) to gesture behaviour (production) in the framework of interactive alignment: we expect gesture production to be influenced by cognitive processing of preceding gestures. However, measuring cognitive processing is only estimated by using gaze fixation, which is a key but not the only factor.

The results presented in this paper are merely a first step towards a more systematic inquiry into the tight coupling of processing and production at the non-verbal level, with a view to developing a genuine multimodal account of interactive alignment. For the first basic empirical analyses discussed here, we necessarily ignored a whole range of parameters that may influence addressees’ (overt) attention to an interlocutor’s gesture or the likelihood of establishing aligned representations at the non-verbal level. McNeill (Reference McNeill, Renals and Bengio2006) and Gullberg and Kita (Reference Gullberg and Kita2009) provide an overview of such factors, including social status, interpersonal stance, speaker information structure, shared common ground, and the physical properties of the gesture. Apart from those factors, also the time difference between the fixation onset and gesture onset, the fixation duration, co-occurring verbal cues, the number of preceding gestures that were or were not fixated, etc. might be parameters with explanatory potential as well. Future work will need to look in more detail into the impact of each of these parameters to provide a more accurate, and fully interactionally grounded account of multimodal alignment.