Informal communication by texting proliferates worldwide in formats as diverse as cell phone calls, personal microblog posts, and workplace collaborations across geographically distributed networks. As more individuals interact by means of text-based technologies not only in personal but also in professional contexts, it becomes clear that technologically mediated human communication is worthy of systematic examination because of what it can reveal about the structure of informal communication and about what aspects of that structure can and cannot change depending on modality, the particular users, their communicative goals and their language proficiency. Speakers differ along many dimensions of communicative competence and their texting behavior, nonverbal as well as verbal, is likely to capture some of that variation. For example, both the presence of a smiley emoticon in general, as well as the presence or absence of a nose (:) vs.:-)) has been associated with attitudinal and affective differences among speakers (Schnoebelen, Reference Schnoebelen2012; Yus, Reference Yus2014).
Gesture with speech or sign creates a medium in which speakers communicate both nonverbally and verbally (Goldin-Meadow & Brentari, Reference Goldin-Meadow and Brentari2017). In a recent paper we argued that the systematic study of emoticon use in text communication evokes parallels with nonverbal gesture in face-to-face communication (Feldman, Aragon, Chen & Kroll, Reference Feldman, Aragon, Chen and Kroll2017). As the formats and contexts of communication expand, an analysis of informal text communication has the potential to provide new insights into the coordination between language and nonverbal communication and between language and cognition more generally.
Here, we focus on a potentially rich and under-investigated domain of nonverbal communication, emoticons in text communication among bilinguals. In their speech, bilinguals tend to incorporate nonverbal gesture into their communication more frequently than do monolinguals (Nicoladis, Pika & Marentette, Reference Nicoladis, Pika and Marentette2009; Pika, Nicoladis & Marentette, Reference Pika, Nicoladis and Marentette2006) and to produce gestures that are exaggerated relative to those (produced by the same person) in their L1 (Gullberg, Reference Gullberg2006). It has been suggested that gesturing partially offsets labored lexical retrieval and may be characteristic of lexically impoverished word production (for a review, see Alibali, Kita & Young, Reference Alibali, Kita and Young2000). Likewise for both L1 and L2 speakers, with controls for differences in baseline rate, the presence of emoticons and emojis in tweets is associated with reduced lexical variation when compared to tweets where they are absent and this too may be symptomatic of difficult lexical retrieval (Shaikh, Lalingkar, Barach & Feldman, Reference Shaikh, Lalingkar, Barach and Feldman2017; Shaikh, Lalingkar, Moscoso del Prado Martín & Feldman, Reference Shaikh, Lalingkar, Moscoso del Prado Martín and Feldman2017a). Hints of a compensatory relation between verbal and nonverbal dimensions of communication are particularly true of bilinguals with low L2 proficiency and are more salient in informal social settings. In this study we analyze emoticon use in text as monolingual and bilingual adult speakers communicate at work, in a closed group, over a period of four years.
Emoticons as nonverbal communication
Many claim that communicating in an L2 is less personally revealing, allowing a speaker to remain emotionally more distant than when communicating in a first language (Pavlenko, Reference Pavlenko2007). For example, at a physiological level in spoken presentations, taboo words elicit smaller changes from baseline in electrodermal measures in bilingual speakers in L2 than in L1 (Harris, Aycicegi & Gleason, Reference Harris, Aycicegi and Gleason2003). Attenuated emotionality in the L2 has been documented with hemodynamic response measures while reading text passages from Harry Potter. Specifically, only in the L1 did fMRI measures reveal differences due to positive or negative valence (see Hsu, Jacobs & Conrad, Reference Hsu, Jacobs and Conrad2015). In addition, L2 listeners are less accurate than L1 listeners in identifying emotions from prosody (Min & Shirmer, Reference Min and Schirmer2011). Differences in affective processing between the L1 and the L2 have been more difficult to demonstrate in tasks that require judging the valence of a word while ignoring emotional tone or in judging emotional tone while ignoring word valence (Eilola, Havelka & Sharma, Reference Eilola, Havelka and Sharma2007). Nonetheless, higher multilingual proficiency is associated with higher scores on general cognitive empathy (Dewaele & Wei, Reference Dewaele and Wei2012) and this could manifest itself in affective domains. Collectively, debate still ensues about the conditions under which one observes attenuated affective responding in a bilingual's second relative to a first language.
Emoticons are one means to convey emotion in text and yet emoticons have many functions, only one of which is to convey emotion (for others, see Feldman, Cho, Aragon & Kroll, Reference Feldman, Cho, Aragon, Kroll, Noelle, Dale, Warlaumont, Yoshimi, Matlock, Jennings and Maglio2015; Yus, Reference Yus2014). The multifunctionality of emoticon use among native and non-native speakers has been described for classroom settings (Vandergriff, Reference Vandergriff2014). Their function is likely to be at least as diverse in online communication. We describe emoticon usage in a bi-cultural collaborative chat log to demonstrate how big data generated in a natural setting can provide a new perspective on the long-standing question about parallels between nonverbal and verbal communication and their coordination.
Alignment between speakers in a conversation
A speaker's style of communication is subject to social and emotional as well as cognitive and linguistic constraints (e.g., Pardo, Jay, Hoshino, Hasbun, Sowemimo-Coker & Krauss, Reference Pardo, Jay, Hoshino, Hasbun, Sowemimo-Coker and Krauss2013). The effects of one person's speaking style on another during the course of a conversation have been variously termed alignment (Pickering & Garrod, Reference Pickering and Garrod2004), coordination (Clark, Reference Clark1996), linguistic style matching (Niederhoffer & Pennebaker, Reference Niederhoffer and Pennebaker2002), or accommodation (Shepard, Giles & Le Poire, Reference Shepard, Giles, Le Poire, Robinson and Giles2001). This pressure for complex behavioral matching is not restricted to a single level. In fact, increasing similarity between speaker and audience has been documented across syntactic (Branigan, Pickering & Cleland, Reference Branigan, Pickering and Cleland2000) and acoustic (Giles, Reference Giles1973) as well as lexical (Niederhoffer & Pennebaker, Reference Niederhoffer and Pennebaker2002; Wilkes-Gibbs & Clark, Reference Wilkes-Gibbs and Clark1992) and emotional (Moscoso del Prado Martín, Reference Moscoso del Prado Martín2015) levels. Convergence along these measures is influenced by social factors such as the relative dominance or perceived prestige of each speaker (Gregory, Dagan & Webster, Reference Gregory, Dagan and Webster1997; Gregory & Webster, Reference Gregory and Webster1996). Speakers in less dominant roles generally converge more toward the more dominant speaker's style (Gill, French, Gergle & Oberlander, Reference Gill, French, Gergle and Oberlander2008; Pardo et al., Reference Pardo, Jay, Hoshino, Hasbun, Sowemimo-Coker and Krauss2013). Finally, patterns of lexical convergence over time among active partners in a coalition can be revealing about a later outcome (viz., the composition of the final coalition) but are more difficult to document in the behavior of an extra-coalition person who is present but does not participate (Sagi & Diermeier, Reference Sagi and Diermeier2017).
For bilinguals, there is alignment both between speakers and across the two languages within a speaker. When bilinguals listen to other bilinguals with whom they code switch, they can exploit the presence of subtle acoustic cues to anticipate a switch of language (Fricke, Kroll & Dussias, Reference Fricke, Kroll and Dussias2016). They attend to the grammatical structure of the speech they hear even when their subsequent production is not in the same language (e.g., Hartsuiker, Pickering & Veltkamp, Reference Hartsuiker, Pickering and Veltkamp2004). Both phonetic cueing and cross-language syntactic priming suggest that bilinguals tune to each other so as to optimize communication. Many recent studies show that there are bidirectional cross-language influences across levels. These include the lexicon (e.g., Ameel, Storms, Malt & Sloman, Reference Ameel, Storms, Malt and Sloman2005), phonology (e.g., Chang, Reference Chang2013; de Leeuw, Schmid & Mennen, Reference de Leeuw, Schmid and Mennen2010) and grammar (e.g., Dussias & Sagarra, Reference Dussias and Sagarra2007). Critically for the focal issue in the present paper, these cross-language interactions are not constrained by language or its form. In the case of American Sign Language (ASL) and English, studies have shown that nonverbal facial expressions that have grammatical meaning in ASL but not in English, co-occur with the English of hearing bimodal bilinguals, even when they converse with other English speakers who do not use ASL (Pyers & Emmorey, Reference Pyers and Emmorey2008). When bilingual speakers align with one another, the process of cross-language influence is not limited by surface level perceptual representations that are common to both languages. Instead, the two languages appear to be permeable and open to one another.
New types of data allow researchers to ask novel questions about behavior (Goldstone & Lupyan, Reference Goldstone and Lupyan2016) and often require new analytic techniques (Moscoso del Prado Martín, Reference Moscoso del Prado Martín2011). Complex analyses can quantify conditional dependencies (e.g., behavior matching) between speakers. In addition to its theoretical value with respect to tracking similarities between speech and text in alignment and the coordination of verbal and nonverbal modes of communication, insights into cross-cultural team interactions have particular value in the virtual workplace. Successful communication and decision-making is at the core of productive remote collaboration. A key to success is apprehending the experiences and emotional cues of others (Gill et al., Reference Gill, French, Gergle and Oberlander2008). Even a cursory review of the literature on informal text-based communication (Brooks et al., Reference Brooks, Kuksenok, Torkildson, Perry, Robinson, Scott, Anicello and Aragon2013; Park, Barash, Fink & Cha, Reference Park, Barash, Fink and Cha2013) reveals a remarkable surge in the prevalence and variety of socio-emotional cues, including nonstandard spelling, repeated punctuation, and emoticons. These innovations can be characteristic of a particular subgroup and appear to enrich online text communication much the way facial cues, prosody and tone of voice, and body language augment the communication in face-to-face interactions (Derks, Bos & Von Grumbkow, Reference Derks, Bos and Von Grumbkow2008; Walther & D'Addario, Reference Walther and D'Addario2001). In the present study, we ask how bilingual and non-bilingual emoticon use differs and whether bilingual/non-bilingual speakers alter the variety or prevalence of emoticon use both over an extended time span and as the audience of native and nonnative speakers of English changes. We exploit the availability of a stable set of interlocutors who texted at work over a period of four years to conduct (repeated measures) analyses on their interactions. Finally, we compare patterns of convergence for emoticon and lexical diversity.
Methods
Participants
Thirty members of a cross-cultural team in an astrophysics collaboration served as participants. About half of the scientists worked at several different locations in the U.S. and the other half at three research institutes and universities in France. The primary lead scientist was an L1 speaker of English although several senior scientists were French speakers. All the French scientists also spoke English and often used it for professional communication. Typically, they also completed post-docs in English-speaking countries. Collaboration members used English in the chat whenever an English speaker was present. French speakers often defaulted to French when they were alone in the chat. In other words, American scientists were native English speakers, and French scientists used English as their second languageFootnote 1 .
Procedure
Collaborating scientists used AIM (AOL Instant Messenger) chat (augmented by a virtual assistant) and VNC (virtual network computing) as their primary means of communication during remote telescope observation from November 2004 to July 2008. Our analysis was confined to human-generated messages (290,306).
Corpus materials
Chat log sessions
Typically 5–6 people were present in the chat in any one session, although the range varied between 1 and 9. To investigate how participants reacted to their environment or audience, we grouped chat logs into chat log sessions and then defined language environments. A chat log session was defined as a continuous time interval with a constant set of participants. In other words, any participant's leaving or joining the chat defined the start of a new session. This criterion yielded 36280 sessions in total.
Language environments
We defined three language environments, unilingua l Footnote 2 , majority, and minority. The language environment of each session was classified depending on the first language of the speakers who were present in the session. In addition, the language environment of a session was defined separately for English and for French speakers. That is, a majority environment for English speakers was a session with more English than French speakers. Note that this same session was a minority environment for French speakers. A unilingual environment was a session with only English or only French speakers. If there were equal numbers of English speakers and French speakers, we classified it as a majority environment for English speakers and minority for French speakers. Of the more than 35,000 sessions, 43% were unilingual English, 24% were unilingual French, 6% were dominated by English speakers (French minority) and 20% were dominated by French speakers (English minority). The remaining 7 % of sessions included no speakers from our set of twenty, defined as speakers who appeared in all three types of environments.
Results
To detect emoticons in the dataset, we used a Perl script to run string matching with a comprehensive list of 2301 emoticons (Marshall, Reference Marshall2011) and manually filtered out those that were not real emoticons. Accordingly, 9282 emoticons were detected in our dataset. Before comparing emoticon use across unilingual, majority, and minority language environments (defined separately for English and French speakers), we removed participants who did not contribute to all three language environments. We also removed participants who used fewer than 23 emoticons, which corresponds to 1/10 of the mean number of emoticons per speaker [M = 226], so that we could examine how a speaker's emoticon use changes across different language environments without encountering a floor effect. These criteria allowed us to focus our investigation on the 10 native English speakers and 10 native French speakers who were most active. They produced a total of 263,098 messages and 9138 emoticons of 59 different types. All of the 10 English speakers were male, and one of the 10 French speakers was female.
Table 1 describes the mean number of messages and emoticons per speaker. Table 2 summarizes the frequency of each emoticon type.
Table 1. Number of emoticons (#Emos) and messages (#Msgs) by speaker
Table 2. Token count for each of 59 emoticon types in the full dataset.
We first compared the text and emoticon production of the two speaker groups in each language environment. Figure 1a shows that the English speakers tended to produce more pure text messages than the French speakers. Figure 1b shows that the French speakers tended to produce more emoticons than the English speakers. The latter is consistent with the observation that bilinguals tend to incorporate more gestures in their L2 than their L1. Comparing across language environments, L2 speakers altered their behavior more with respect to nonverbal aspects of communication whereas L1 speakers altered their behavior more with respect to number of messages, a global verbal measure.
Figure 1. (a) Mean (SE whisker) number of messages (b) Mean (SE whisker) number of emoticons.
In Figures 2(a) and 2(b) we summarize mean emoticon rate per message for French and for English speakers in each of the three language environments and % of available emoticons produced averaged across speakers. Proportions were computed relative to the number of distinct emoticons in the full dataset (n = 59). The results of a 2x3 ANOVA with speaker group (English/French) as a between-group variable and language environment (unilingual/majority/minority) as a within-group variable showed that the interaction between speaker group and language environment was significant [F(2, 36) = 4.064, p < .05]. A pairwise comparison for simple effects showed that the French speakers produced significantly more emoticons per message in both the majority and minority environments than in the unilingual environment [t(9) = −3.828, p < .02 and t(9) = −3.377, p < .02, respectively]. Significance levels are Bonferroni corrected here and throughout. Evidently French speakers varied more in their use of emoticons over language environments than the American speakers and they increased instances of emoticon production when they used the L2.
Figure 2. (a) Mean (SE whisker) number of emoticon (tokens) per message by language environment (b) Mean (SE whisker) number of different emoticons (types) tokens per message by language environment.
With respect to the diversity of emoticon use over language environments, results of a 2x3 ANOVA showed that the French speakers produced a significantly more diverse set of emoticons than the English speakers [F(1, 18) = 4.561, p < .05]. More important was that the interaction between speaker group and language environment was significant [F(2, 36) = 6.450, p < .01]. A pairwise comparison showed that French speakers relied on a more redundant set of emoticons when they were in a minority environment than in either a unilingual or a majority environment [t(9) = 4.364, p < .002 and t(9) = 3.841, p < .008, respectively]. By comparison, English speakers showed no difference between language environments. Here again, the results show that French speakers aligned their emoticon productions more when they constituted the linguistic minority within a language setting relative to when they were the majority or the sole demographic whereas English speakers did not show this alignment behavior. In summary, French speakers tended to have a larger emoticon vocabulary overall. In a monolingual setting French speakers used a more varied set of emoticons but used each less often. When they were surrounded by L1 English speakers, by contrast they used a smaller set of emoticons but used each more frequently. If emoticons functioned only as indicators of affect then finding more frequent, albeit less diverse productions, would fail to support claims for attenuated affective processing in the L2.
Emoticon entropy over time
The emoticon measures above capture only mean number of emoticons per language setting. Entropy measures are based on the distribution rather than the mean. That is, they reflect both the number of types (e.g., different emoticons) and the relative frequencies with which each is produced. Entropy increases as the number of types increases and when many types occur with equal relative frequency. Thus, it provides a measure of uncertainty such that the less differentiated the pattern of emoticon usage is (similar relative frequencies for different emoticons), the higher the entropy is (Shannon, Reference Shannon1948). Convergence among collaborators over time to a more restricted set of emoticons, with differences in their frequency of usage is characteristic of lower entropy.
When we calculated emoticon entropy in the chat, we used months as our unit of time and avoided incomplete months and months when the teams were not working. Accordingly, we filtered out the first month (November 2004), which started on November 27th, and also the months with fewer than 1500 messages in total (December 2005, January–March 2007, and January–March 2008). These were winter months when the searching task of the collaboration was shut down. In each time period we tracked the frequency with which each emoticon appeared and converted that to a relative frequency in order to compute entropy.
Figure 3a shows the separate computation of emoticon entropy for the two language groups and the caption describes how it was computed. From the figure we can see that in general, the French speakers had higher emoticon entropy than the English speakers, and that the difference between groups decreased over time.
Figure 3a. Emoticon Entropy by speaker group over time. Emoticon entropy is computed as:
$$\begin{equation*}
\begin{array}{l}
\hspace*{-10pt}{\rm{Emoticon\ Entropy}}\left( {{{\rm{t}}_{\rm{i}}}} \right){\rm{ = - }}\mathop \sum \limits_{\rm{k}}^{\left| {{\rm{D}}{{\rm{E}}_{{{\rm{t}}_{\rm{i}}}}}} \right|} {\rm{P}}\left( {{{\rm{E}}_{\rm{k}}}{\rm{,}}{{\rm{t}}_{\rm{i}}}} \right){\rm{ \times lo}}{{\rm{g}}_{\rm{2}}}\left( {{\rm{P}}\left( {{{\rm{E}}_{\rm{k}}}{\rm{,}}{{\rm{t}}_{\rm{i}}}} \right)} \right)\\
\hspace*{-10pt}{\rm{P}}\left( {{{\rm{E}}_{{\rm{k,\ }}}}{{\rm{t}}_{\rm{i}}}} \right){\rm{ = }}\frac{{\left| {{{\rm{E}}_{\rm{k}}}\left( {{{\rm{t}}_{\rm{i}}}} \right)} \right|}}{{\mathop \sum \nolimits_{\rm{j}}^{{\rm{|D}}{{\rm{E}}_{{{\rm{t}}_{\rm{i}}}}}{\rm{|}}} \left| {{{\rm{E}}_{\rm{j}}}\left( {{{\rm{t}}_{\rm{i}}}} \right)} \right|}}\end{array}
\end{equation*}$$
Ek: Emoticon k
|Ek(ti)|: The Number of Ek in ti
|DE ti |: The Number of Distinct Emoticons in ti
Entropy increases with number of emoticons and with relative frequencies that are closer in value to each other.
Figure 3b. Lexical Entropy by speaker group over time.
Figure 3b shows the separate computation of lexical diversity based on relative frequency of each word in each of the two language groups over time. From the figure we can see that diversity is high overall but the French speakers had lower English lexical diversity than the English speakers [t = 3.294, p < .002]. Slight systematic reductions over time were present only in the native French speakers [t = 2.175, p < .035]. Relative to English speakers, French speakers showed greater emoticon diversity but lower lexical diversity along with greater reductions over time both for emoticon diversity and for lexical diversity. In summary, both in the production of emoticons, where they were more proficient than English speakers, and in the production of English words where they were less proficient, native French speakers altered their communication style more than native English speakers.
Discussion
The present work examined emoticon use in a bilingual collaborative work chat log dataset that extended over four years and captures a new style of coordination among those who communicate online. Prominent in our findings are sustained differences between L2 and L1 speakers of English with respect to variation in their emoticon and lexical productions. Results are consistent with suggestions of a compensatory relation between verbal and nonverbal dimensions of communication that are exaggerated in an L2 relative to an L1 (Shaikh et al., Reference Shaikh, Lalingkar, Barach and Feldman2017). With respect to the nonverbal measure of communication, French L1–English L2 speakers tended to have command of more emoticons overall and they altered their behavior more depending on language environment than did the L1 speakers of English. In a monolingual setting, French speakers used a more varied set of emoticons and used each less frequently attesting to their higher emoticon proficiency. When the same French speakers were surrounded by L1 English speakers, they restricted themselves to a reduced emoticon set and used each more redundantly. No analogous emoticon pattern was evident for L1 speakers.
Over time, all speakers reduced the variability in their choice of emoticons. Like analyses on large corpora of spoken communication, entropy measures based on the data from online texting revealed alignment between interlocutors. With emoticons, we observed greater changes over time for L2 than for L1 speakers. Because the shift was to more frequent use of a smaller set of emoticons, we see little justification for interpreting this pattern as reflecting a general distancing from emotions per se. We consider two alternative interpretations for the reduction in emoticon diversity among L2 speakers. One possibility is that the proficiency of L2 speakers who started with higher rates of emoticon usage and enhanced fluency with emoticons could potentially be associated with greater changes in behavior. An account of L2 convergence based solely on proficiency seems inadequate, however, because the analysis of lexical diversity in L2 indicated lower proficiency than in L1 speakers and yet weak but systematic changes over time were present as well. In the setting we examined, more of the senior scientists were L1 speakers of English and many of the L2 speakers tended to be younger. Therefore another interpretation is that the L2 speakers occupied less dominant roles in the group so that social factors (Danescu-Niculescu-Mizil, Lee, Pang & Kleinberg, Reference Danescu-Niculescu-Mizil, Lee, Pang and Kleinberg2012 in van der Pol, Gieske & Fernández, Reference van der Pol, Gieske and Fernández2016) rather than proficiency directly could account for greater convergence. Complicating an account based on social factors, however, is the evidence that perceived prestige based on expertise across participants from different cultures can be more difficult to ascertain in online than in face-to-face contexts (Bazarova & Yuan, Reference Bazarova and Yuan2013). Also potentially relevant is that some variants of convergence are more difficult to document in the behavior of a person who functions outside of the predominant coalition (Sagi & Diermeier, Reference Sagi and Diermeier2017). Conceivably the reduced number of senior scientists may define them to be outside of the dominant coalition even though they are senior. In summary, we note that changes consistent with alignment were more prominent with emoticons, where L2 proficiency was higher than L1, but also present with lexical variation where it was lower. Thus, we believe that it is premature to dismiss a proficiency component to the pattern of alignment. At present, we can only assert the implausibility of a purely proficiency-based account of convergence. Therefore, we interpret the greater reduction in emoticon diversity over time in L2 than in L1, in conjunction with the weakly attenuated lexical diversity in L2 but its absence in L1, to be a manifestation of a general social pressure to align that may derive from factors like communicating in an L2 or lower social status due to age.
An appreciation of how communicative channels and devices influence remote communication is currently evolving. Here we have demonstrated one way in which technologically mediated human communication can provide new insights into informal communication by noting systematic changes in emoticon use that were greater for those communicating in an L2 than an L1 and greater with respect to emoticon than lexical variation. Obviously the potential for new modes of coordination among those who communicate remotely can be richer when contributions are temporally contiguous and encompass nonlexical (e.g., emoticon) as well as lexical dimensions. Likewise, coordination can increase in the context of sustained interaction where social roles become better defined over time and this encompasses online as well as face-to-face communication. Elsewhere we have reported tradeoffs between lexical and emoticon diversity in tweets, suggesting that dimensions of communication can be not only coordinated but also interdependent (Shaikh et al., Reference Shaikh, Lalingkar, Barach and Feldman2017).
Social media platforms like Twitter provide a source of data about language and communication whose quantity far exceeds what is possible in the laboratory and our understanding of the similarities and differences between online and real world communication is fast becoming a legitimate domain of investigation (Metaxas & Mustafaraj, Reference Metaxas and Mustafaraj2012; Bond, Fariss, Jones, Kramer, Marlow, Settle & Fowler, Reference Bond, Fariss, Jones, Kramer, Marlow, Settle and Fowler2012; Tagg, Lyons, Hu & Rock, Reference Tagg, Lyons, Hu and Rock2016). For topics such as who influences whom on line, much of the work focuses on nodes and edges in social networks, including communication patterns among users (Morone & Makse, Reference Morone and Makse2015; Kempe, Kleinberg & Tardos, Reference Kempe, Kleinberg and Tardos2015). Obviously the content of the communication including changes in style also play a crucial role. Psycholinguistic analyses of informal text communication have the potential to reveal new dimensions of coordination between speakers with respect to behaviors that include but are not limited to classical linguistic and nonverbal measures of communication. Like other types of productions, alignment and convergence across speakers with respect to verbal and nonverbal aspects of texting are likely to provide new insights into the principles that underlie language and cognition unconstrained by the laboratory.
