1.1. the gold standard in natural language processing: human coders

At first glance, using existing written text or speech transcripts for inferring properties of a person is a straightforward idea. Not only is language often used by psychologists to make inferences about properties of the human mind (Freud, Reference Freud1901; Rorschach, Reference Rorschach1964 [1921]; Murray, Reference Murray1943; Van Dijk & Kintsch, Reference Van Dijk, Kintsch and Dressier1977; Weber, Hsee, & Sokolowska, Reference Weber, Hsee and Sokolowska1998; Braun & Clarke, Reference Braun and Clarke2006), but it is also our primary mode of communication, and frequently our source of information about others. Based on what someone says, we make judgments about his or her personality, general knowledge, past, and, quite often, about the value of future interactions with the speaker. We are similarly good at interpreting written text: when we read a note, e-mail, letter, or article, we can often tell if the author was happy or sad, polite or rude, expert or novice, and sometimes we can even infer gender, religion, or political orientation. This ability has allowed social scientists to collect data using human coders as interpreters of spoken or written interviews. Using human coders exclusively, however, quickly becomes impractical as the amount of text increases. In today’s standards, with millions of new tweets, blog posts, comments, and reviews generated daily, traditional methods that rely on human coders can easily limit the scope of research projects.

If we need to deal with large volumes of text, automated text analysis quickly becomes the most plausible option. However, in doing so we face the problem of extracting meaning from text, and, while humans are strikingly good at this, computer algorithms find it particularly challenging. Despite decades of research on natural language processing by computer scientists, computational linguists, and cognitive psychologists, computers are still a long way away from matching human performance when it comes to identifying meaning. To illustrate the challenges that a computer program faces when trying to extract meaning, we will use a relatively simple example from the field of sentiment analysis. Imagine that we want to understand whether or not a person is happy with a particular camera. By using a parsing algorithm, it is not difficult to discover if the noun camera is linked to a value-laden adjective. Some reviewer might write: “This is an awful camera”, while another one can write “This is an awesome camera”. After consulting a dictionary to evaluate the meaning of words (e.g., Esuli & Sebastiani, Reference Esuli and Sebastiani2006), we can easily conclude that the second reviewer likes the camera while the first one does not. Unfortunately, easy sentences like the examples above are not that frequent, and in most cases the semantics of an adjective changes based on context. For example, in a blog post an expert might be comparing two cameras, and if she says: “The battery life of this Nikon is really long”, she probably has a positive attitude, but if she says: “The focusing time of the Pentax is really long”, the reviewer is probably expressing a negative attitude (after Liu, Reference Liu, Indurkhya and Damerau2010). The same word used for the same product might have very different meanings, depending on the particular feature being described.

Even though a fully functioning, automated extraction of meaning from text is not yet possible, researchers have made progress in using large-scale bodies of text as data sources. Avoiding the direct challenge that semantics presents to automated text analysis, most methods rely on the fact that computers can deal with large numbers of relatively simple features. Even if each feature captures a very small proportion of the meaning of a text, when many features are taken into account, the accuracy of predictions can become surprisingly high. For presentation purposes, we split the methods into three major groups, depending on the properties of the features they use for analysis. In the first group of methods, which we call ‘user-defined dictionaries’ (UDD), researchers generate the features themselves. In the second group, which we call ‘feature extraction’, researchers use computer algorithms to find the features that are the strongest predictors for some variables of interest. In the third group, which we call ‘word co-occurrence’, the focus is on the relationship between features. Since this three-prong distinction is for presentation purposes only, it leaves out a number of other methods, which we will briefly cover under ‘other methods’. Before we continue with the descriptions of the methods, it should be noted that this review is very broadly aimed at social scientists. Hence, we will minimize our discussion of the technical details and differences between the various methods and focus more on their applications.

1.2. user-defined dictionaries

Probably the most straightforward way to explore how language is linked to the properties of the speaker is to look for particular themes in his or her speech. Since many of the problems relevant to psychologists are often reflected in language, one can predefine sets of words associated with particular topics. If a researcher is interested in the general mood of a person, the focus can be placed on the emotional value of the words in text. For example, one can predefine a dictionary with negative words, such as sad, depressed, gloomy, pain, etc. Similarly, if a researcher cares about personality, the focus can be on adjectives describing a person, such as fun, cool, social, easygoing, etc. Then, the text is searched for the words from a particular dictionary, and the relative number of hits can be used as an indicator of the degree to which the text is related to a specific theoretical construct. This procedure is similar to content analysis of text using human coders, with the main difference being that in dictionary-based methods, the categories of interest are represented by single words, so a computer algorithm can automatically search through large bodies of text.

The most popular example of a dictionary-based approach in recent years^{Footnote 1} is the Linguistic Inquiry Word Count (LIWC^{Footnote 2}) developed by James Pennebaker and his collaborators (Pennebaker, Reference Pennebaker2011). LIWC has been extensively applied to various psychological domains (for a more detailed review, the reader should refer to Tausczik & Pennebaker, Reference Tausczik and Pennebaker2010). LIWC performs word counts and catalogs words into psychologically meaningful categories. The default LIWC2007 dictionary includes 76 different language categories containing 4,500 words and word stems. LIWC assigns each word to a specific linguistic category and reports the total number of words in each category normalized by the total number of words in the document. Some of these categories are related to specific contents, such as leisure, religion, money, or psychological processes. As a simple example, when people write about pleasant events, they are more likely to use words from the dictionary representing the positivity category (Kahn, Tobin, Massey, & Anderson, Reference Kahn, Tobin, Massey and Anderson2007). Similarly, depressed individuals score higher on words associated with negative emotions (Rude, Gortner, & Pennebaker, Reference Rude, Gortner and Pennebaker2004).^{Footnote 3} Some content-based categories have shown more surprising associations. For instance, when subjects are trying to write deceptive texts, they are more likely to use words from the motion category (Newman, Pennebaker, Berry, & Richards, Reference Newman, Pennebaker, Berry and Richards2003), and extroverted subjects are less likely to use causality related words (Pennebaker & King, Reference Pennebaker and King1999).

In addition to content-based categories, LIWC also analyzes some broader language categories, such as word count, long words, tense, and function words (articles, pronouns, conjunctions). Somewhat surprisingly, several of the most interesting findings come from these categories. For example, first person singular pronouns have been associated with negative experiences (Rude et al., Reference Rude, Gortner and Pennebaker2004). Suicidal poets are more likely to use first person singular pronouns than matched non-suicidal poets (Stirman & Pennebaker, Reference Stirman and Pennebaker2001). Similarly, depressed people are also more likely to use first person singular pronouns (Rude et al., Reference Rude, Gortner and Pennebaker2004), and the same is true for people in lower power positions (Kacewicz, Pennebaker, Davis, Jeon, & Graesser, Reference Kacewicz, Pennebaker, Davis, Jeon and Graesser2013). Further, individuals under stress are also more likely to use first person singulars; however, when a whole community copes with a tragedy, such as the terrorist attack on September 11, 2001, the usage of first person plural pronouns decreases (Cohn, Mehl, & Pennebaker, Reference Cohn, Mehl and Pennebaker2004). Other language features have also been linked to psychological variables: extroverted authors, for example, tend to write longer texts but prefer shorter words and less complex language (Pennebaker & King, Reference Pennebaker and King1999; Mehl, Gosling, & Pennebaker, Reference Mehl, Gosling and Pennebaker2006).

One of the main advantages of user-defined dictionaries is that researchers have the freedom to create sets of words that can target any theoretical construct of interest. While the categories of LIWC can be applied to multiple domains, and have the advantage of being empirically validated in many studies, sometimes scientists have to build their own dictionaries. For example, Graham, Haidt, and Nosek (Reference Graham, Haidt and Nosek2009) used a specialized dictionary as part of an extensive empirical test of the Moral Foundation Theory (Haidt & Joseph, Reference Haidt and Joseph2004; Graham et al., Reference Graham, Haidt, Koleva, Motyl, Iyer, Wojcik and Ditto2013). One implication of this theory is that liberals and conservatives differ in what they consider to be part of the moral domain. Graham et al. (Reference Graham, Haidt and Nosek2009) tested this prediction in three psychological studies, finding that liberals were concerned mainly with harm and fairness, while conservatives in addition were also concerned with loyalty to ingroup, authority, and purity. In their last study, the authors tested whether such differences can be measured in text corpora. They created a dictionary with words corresponding to each of the five moral foundations, and compared liberal and conservative sermons. The analysis of the relative frequencies of the words from the different subdictionaries largely replicated the questionnaire-based results: liberal sermons had a higher frequency of words related to harm and fairness, while conservative sermons were higher in words related to authority and purity.^{Footnote 4}

A recent domain where UDD has shown to be particularly useful is studying historical trends and cultural change. While historical analysis of text has been a common practice among psychologists interested in cultural change (e.g., Wolff, Medin, & Pankratz, Reference Wolff, Medin and Pankratz1999), the current availability of large-scale time-stamped text (Michel et al., Reference Michel, Shen, Aiden, Veres, Gray, Pickett, Hoiberg, Clancy, Norvig, Orwant, Pinker, Nowak and Lieberman-Aiden2011) has made such studies particularly detailed and easy to conduct. For example, cultural researchers often focus on East−West cross-cultural differences that can be traced back to ancient philosophical texts (Nisbett, Reference Nisbett2004). Such focus on static comparisons, however, ignores the possibility that some of the characteristics associated with Western cultures might be a recent development. One way to assess the degree to which values and attitudes have changed over time is to create specific UDDs and see the temporal pattern associated with particular words and expressions. Greenfield (Reference Greenfield2013) found that, over the last two centuries, words associated with individualism and independence have become more frequent. Similarly, Kesebir and Kesebir (Reference Kesebir and Kesebir2012) have found that words expressing concern about others and words indicating moral virtue have decreased in frequency over the last century. Similar pattern have been found by Twenge, Campbell, and Gentile (Reference Twenge, Campbell and Gentile2012) using a narrower time window and participant-generated dictionaries.

The UDD approach has proven applicable to a broad range of questions, including gender differences, personality, clinical diagnosis and treatment, morality, deception, motivation (e.g., Gill, Nowson, & Oberlander, Reference Gill, Nowson and Oberlander2009), knowledge assessment (Williams & Dmello, Reference Williams, D’Mello, Aleven, Kay and Mostow2010), and cultural epistemological orientations (Dehghani, Bang, Medin, Marin, Leddon, & Waxman, Reference Dehghani, Bang, Medin, Marin, Leddon and Waxman2013). Nevertheless, we need to mention some of the challenges that the method faces. First, in its basic form (as counts of words appearing in particular user-defined categories) it is blind to the context in which words appear. For example, if we have a dictionary of positive words, we will treat the sentences “I have never been happy in my life” and “I have never been this happy my life” very similarly, since they both include the word happy. Moreover, such dictionaries cannot easily capture sarcasm, metaphors, or idiomatic expressions. Consequently, while the statistical tests of studies using dictionaries are typically highly significant, the effect sizes are often quite small.^{Footnote 5} Yet the simplicity and theoretical flexibility of this method makes it a very useful tool for working with large bodies of texts, and its popularity will probably keep increasing.

1.3. feature extraction

While user-defined dictionaries have the advantage of high face validity, the small effect sizes associated with the different categories often make them less suitable for precise predictions, especially when dealing with large, noisy datasets. For example, researchers might be interested in gender differences in diurnal activity, and their data might consist of a large number of blog posts. Further, suppose that all blog posts are time stamped, but only for a small proportion is the gender known. How can they make use of the ones with missing data? As we saw in the previous section, they can rely on some LIWC categories, or they can construct their own list of features using their intuition or expertise, and then assign gender to the missing values depending on the overlap between their lists and given text. However, small effect sizes imply that they will not be much better than chance, which will result in lots of noise in their subsequent findings.

An alternative approach to UDD is to start with texts that differ in some dimension of interest, and then in a bottom-up manner find the features that maximize such differences. In theory, these features can have any property, but in most cases they are character n-grams, single words, short expressions, or tagged parts of speech. Typically, in algorithms used to build text classifiers, documents are represented as sets of features and then the algorithm searches for those features that are common in one type of document, but rare or absent in the other types.^{Footnote 6} In the example above, the texts whose gender is known can be used as input to the classification algorithm. The algorithm then tries to extract the features that are more likely to be present in texts written by females only (or by males only). Such algorithms are trained on a subset of the texts, and then the predictive validity of the extracted features is tested on the remainder of the texts (a common method in machine learning). Depending on the particular goals of the project, the training subset might be much smaller than the remainder, or it can be larger, but the procedure can be randomly repeated many times (known as cross-validation). The degree to which the extracted features from the training set correctly classify the remaining texts is taken as an indicator of its reliability/accuracy.^{Footnote 7}

Feature extraction methods have shown impressive accuracy in predicting a wide range of properties of the speaker. For example, Dave, Lawrence, and Pennock (Reference Dave, Lawrence and Pennock2003) used these methods to distinguish between positive and negative reviews with relatively high accuracy (88% for products and 82% for movies). Similar applications have also been able to identify the political party affiliations of US senators based on their speeches in the Senate with 92% accuracy (Diermeier, Godbout, Yu, & Kaufmann, Reference Diermeier, Godbout, Yu and Kaufmann2011), as well as political orientations of bloggers with 91.8% accuracy (Dehghani, Sagae, Sachdeva, & Gratch, Reference Dehghani, Sagae, Sachdeva and Gratch2014). Likewise, feature extraction algorithms have been found to perform well at attributing gender (Mukherjee & Liu, Reference Mukherjee and Liu2010), age and native language (Argamon, Koppel, Pennebaker, & Schler, Reference Argamon, Koppel, Pennebaker and Schler2009), personality dimensions (Oberlander & Nowson, Reference Oberlander and Nowson2006), sentiments (Dave et al., Reference Dave, Lawrence and Pennock2003), mental disorders (Strous, Koppel, Fine, Nachliel, Shaked, & Zivotofsky, Reference Strous, Koppel, Fine, Nachliel, Shaked and Zivotofsky2009), and identity of the author (Diederich, Kindermann, Leopold, & Paass, Reference Diederich, Kindermann, Leopold and Paass2003; Lewis, Kaufman, Gonzalez, Wimmer, & Christakis, Reference Lewis, Kaufman, Gonzalez, Wimmer and Christakis2008) with similarly high accuracy levels (see also Koppel, Scheler, & Argamon, Reference Koppel, Schler and Argamon2009, for a detailed treatment of the topic of author identification).

It is important for the reader to keep in mind that these impressive results are mostly due to the computational power of current algorithms and the availability and quality of the training sets. Since building the initial feature list is usually automated, these methods might consider a very large number of features. For example, when content words are used as features, the size of the list is often in the thousands, and lists of tens of thousands of features are not uncommon. The correspondence between a particular feature and variable of interest is not always transparent. For instance, even before we run any analysis, we might guess that the word homosexual in senators’ speeches will predict conservative ideology, while the word gay will predict liberal ideology. However, we could hardly guess in advance that catfish and grazing are also strong predictors for conservatism, while fishery and lakes predict liberal orientation (Diermeier et al., Reference Diermeier, Godbout, Yu and Kaufmann2011). Weaker predictors are frequently even less intuitive to comprehend, yet, when combined in large numbers, the predictive power of the model can be surprisingly high.

While machine learning algorithms used in feature extraction methods face many challenges, two of such shortcomings seem particularly important for the current review. One is practical and the other is theoretical. On the practical level, algorithms that are trained in one domain often perform much worse on other domains even when the variable of interest is the same. For example, Finn and Kushmerick (Reference Finn and Kushmerick2006) compared the performance of algorithms that predict the valence of movie and restaurant reviews, and found that even though the algorithms performed well in the domain they were trained in, they did poorly in predicting reviews from the unfamiliar domain. This means that algorithms might have to be retrained every time the topic of the document set is changed, which can be a significant limitation of their applicability.

The theoretical challenge for machine learning algorithms, from the perspective of social scientists, is that using thousands of features might lead to good practical results for text classification, but it might not be very informative for theoretical purposes. One way to overcome this problem is to analyze the results from a machine learning algorithm using more traditional forms of content analysis. One example comes from Diermeier et al. (Reference Diermeier, Godbout, Yu and Kaufmann2011), who extracted the words that maximized the difference between Republican and Democratic senators. The authors compared their results with common theories that the liberal−conservative divide in the Senate is mainly driven by economic concerns. Contrary to the economic divide hypotheses, the most predictive features were words related to cultural values and beliefs (e.g., abortion, same-sex marriage, stem cell research). Not only was their model useful for assigning political orientation to text entries, but they were also able to make inferences that argue for or against a particular hypothesis. This demonstrates that the integration of machine learning results with classical scientific expertise can lead to findings with significant theoretical implications.

To a large degree, psychologists have been reluctant to use feature extraction methods in their work, yet this reluctance might change in the near future. One potential domain where such methods can be quite helpful to researchers is in the analysis of open-ended questions. Sometimes such questions are part of the dependent variable of interest or part of larger interviews with multiple items. However, due to the time and costs of using human coders to analyze these types of question, often these answers do not get analyzed thoroughly, or the coders are instructed to look only for predefined keywords or expressions. Yet such predefined coding schemes might miss genuine language differences between the groups of participants, which could otherwise be captured by a supervised learning algorithm. Another issue with using human coders in this context is that sometimes manual coding schemes are developed simultaneously with the coding process, and as such they can maximize between-group differences by selecting the most distinctive features in the corpus. Such practice can lead to issues with reliability and generalizability, because we might not know if the same coding scheme will lead to the same results with a different sample of participants. The cross-validation methods used in supervised learning algorithms can help circumvent this problem by dividing the corpus into separate learning and testing parts and measuring the predictive accuracy of the extracted features. Last, in many cases open-ended questions are used as filler tasks, included in surveys to reduce carry-over and order effects or to cover the true purpose of the study. Typically such questions are not analyzed at all, yet they might be affected by the experimental manipulation, or they might interact with the target question. Supervised learning algorithms might be a quick and inexpensive way to test for the presence of such valuable information. For example, in an experimental study in social psychology, a researcher might choose to use a filler task to separate two measures of relevant dependent variables. In the filler task, participants might be asked to write a short essay on how they have spent the weekend, a topic unassociated with the main goal of the study. In most cases, such essays will not be analyzed, particularly in large sample experiments. Yet applying a supervised learning algorithm might reveal that the experimental and the control group used very different words and expressions in their essays, and these differences might be theoretically meaningful. Further, the prevalence of the features extracted by the algorithm might become a useful variable in mediation analysis. While such studies are yet to be seen, we believe that the potential of feature extracting methods for application in different types of psychological studies is very high.

1.4. word co-occurrences

The methods mentioned so far have all focused on analyzing documents mainly by looking at individual words. In the case of user-defined dictionaries, each word that is found in the dictionary contributes to the documents’ overall score in one (or more) dimensions. Supervised machine learning algorithms usually consider words (but also n-grams or syntactic structures) to be features that might be of use in determining the documents’ classification. However, words are rarely used in isolation. Words in a text often build on each other in order to convey a meaning that each word on its own cannot provide. In that sense, a text is more than the sum of the words it contains. One way to take this into account is to focus on which groups of words tend to occur together in particular contexts. For example, if the words lion, tiger, and zebra appear in text, it is more likely that such documents refer to animals, than a text containing the words stop, yield, and zebra. Capturing the relations between words is the goal of a family of methods that are referred to as Latent Semantic Analysis (LSA^{Footnote 8}).

At the core of the LSA family of methods is the assumption that words are not randomly distributed (Firth, Reference Firth1957). Consequently, it is possible to reduce the conceptual vocabulary to a smaller number of independent dimensions based on word co-occurrences. In traditional LSA, this set of dimensions forms a semantic space. The method of choice for constructing such a space is singular value decomposition, which is closely related to principal component analysis. The algorithm uses a matrix as an input to describe the frequency of word occurrences within documents. This matrix is then decomposed into matrices that describe both the documents and the words as vectors in a multidimensional space. Words that tend to appear in the same documents are closer in this space, and similarly, documents that use similar words are also close to each other. Since the space is shared, one can also compute the distance between a word and a document. For example, the word carburetor will be closer in semantic space to documents related to car maintenance than to documents about culinary arts. While this method has a strict mathematical basis, it is also flexible enough to be adjusted for different practical purposes. For example, one can choose the length of a document, where the document can be a single sentence or it can be a whole book. Similarly, one can choose the number of dimensions, with numbers between 100 and 400 typically showing the best trade-off between simplicity and informativeness (Dumais & Landauer, Reference Dumais and Landauer1997).

As a side benefit of this process, it should be noted that no human intervention is required to create a semantic space. That is, while the methods described earlier all required an external source of information (either a set of norms to assign words to categories or a set of preclassified documents to train the algorithm), LSA generates a space based solely on the content of the documents in the corpus. In a sense, the corpus itself forms the training set from which LSA learns. Choosing different training corpora, however, can result in different semantic spaces, since word frequencies, and word co-occurrences depend on genre, style, and topic. In this sense, how the reduced semantic space will be extracted is independent of the researcher, yet the training corpus needs to be carefully selected. For example, a training set from a biology textbook will position the term horse in a very different semantic space than a training set in history.

It is also important to note that the development of LSA methods, which have their roots in document retrieval, has been originally focused on what is shared across people rather than on what is different. Specifically, LSA was originally developed to address a problem in information retrieval, namely that the same information is often described in very different terms by different people. From this perspective, idiosyncrasies, both in documents and in search queries, were seen as an obstacle for computerized search rather than as useful information. Similarly, when the method was applied to cognitive modeling, the interest was still largely in the shared semantic representation (Landauer & Dumais, Reference Landauer and Dumais1997). Yet if we are interested in individual differences or in between-group differences, the method itself does not prevent us from asking the opposite question, namely, how authors differ between each other.

The most straightforward application of LSA methods for studying differences between authors is simply to check the degree to which authorship is a predictor of distance between texts in the semantic space. While there is evidence that authors tend to cluster together in the semantic space, the effects are not particularly strong and are sensitive to the number of dimensions chosen (Nakov, Reference Nakov2001). Such results should not be surprising, since LSA tends to cluster documents based on semantic similarity, so that documents from different authors on the same topic are more likely to be close in the semantic spaces than documents on different topics by the same author. Notice, however, that these results came mainly from analysis of texts by established writers and poets, and it is still an open question if broadening the sample of authors using on-line entries will lead to stronger within-author similarities.

Another way to use LSA methods for assessing properties of the author is to measure the semantic distance of a text to some target document. Such an approach has been particularly important for education researchers, with the hope of reducing manual labor in grading and providing quick on-line feedback for various types of classes. Typically, a set of pregraded texts are decomposed in a multidimensional space, after which the text to be graded is also decomposed in the same space. The proximity between the new text and the pregraded exemplars is used as a criterion, where the new grade is based on the grade of the closest exemplar. Alternatively, it is also possible to start only with a set of ‘ideal’ texts, without having lower-grade exemplars, and the grade to be assigned will depend only on the distance to the ‘ideal’. These methods have been applied to different domains of learning, often leading to impressive results. Foltz, Laham, and Landauer (Reference Foltz, Laham and Landauer1999) report correlations between human graders and LSA-based grade around .80, which was virtually the same as the correlation between two human raters.

Sometimes it is not enough to grade an essay, but it is also important to distinguish the particular ‘mental model’ that a student relies upon. In research on naive physics (McCloskey, Reference McCloskey, Gentner and Stevens1983; DiSessa, Reference DiSessa1993), for example, researchers found that while there is one normatively correct model, there are several categories of non-normative models, based on varieties of misconception. In terms of grading, different answers based on such non-normative models will all be far from the ‘ideal’, but an educator might further want to know which are the most widespread misconceptions so they can be addressed. A promising example in this regard comes from Dam and Kaufmann (Reference Dam and Kaufmann2008), who applied LSA to interview transcripts of seventh graders who were asked about the cause of seasons on the Earth. Previous research has isolated three main types of models: close/far distance to the Sun; facing / not facing the Sun; tilted axis of rotation of Earth (normatively correct). The researchers trained their algorithm on texts from geology and astronomy, after which they measured the similarity of sets of interviews to three comparison documents, each representing one of the three mental models. The authors’ LSA-based application reached 90% accuracy of classification when compared to human raters (see also Sherin, in press).

LSA-based methods can also be useful for studying change over time. One example comes from Campbell and Pennebaker (Reference Campbell and Pennebaker2003), who were interested in the relationship between writing and health. The authors used data from previous studies, where different groups of subjects had to write short essays on emotional topics for three days. The similarity between the essays from the same author was correlated with health outcomes. When conventional LSA, based on content words, was applied, the authors did not find any meaningful pattern. However, when they adjusted the LSA procedure to account for style, rather than content, the authors found that greater similarity between the essays written on different days was significantly correlated to subsequent medical visits. In other words, people who showed less diversity in their writing styles were more likely to have negative health outcomes. The results held for all three groups, with correlations between the essays’ similarity and doctor visits in the range of .34 to .51. Subsequent analysis showed that changes in the use of pronouns and particles were the strongest predictors. While the true mechanism behind such strong effects is not fully understood, one explanation suggested by the authors is that change of the context in which pronouns are used can reflect flexibility in perspective taking and thus re-evaluation of emotional experiences.

A closely related approach to LSA is a generative method called Latent Dirichlet Allocation (LDA; Blei, Ng, & Jordan, Reference Blei, Ng and Jordan2003). LDA^{Footnote 9} assumes that each document in the corpus under analysis is composed of a mixture of topics, and that each topic is a distribution over words. According to this view, documents are generated by repeatedly choosing a topic from a distribution over topics, then choosing a word from a distribution over words which correspond to the chosen topic. The goal of the method then is to find the most likely topic structure that generates the given collection of documents. Chen (Reference Chen2011) illustrates the intuition behind LDA using the following example: let us assume we have the following sentences in our corpus of analysis: “I like to eat broccoli and bananas. I ate a banana and spinach smoothie for breakfast. Chinchillas and kittens are cute. My sister adopted a kitten yesterday. Look at this cute hamster munching on a piece of broccoli” (Chen, Reference Chen2011). Given this corpus, and asked for two topics, LDA would try to find the set of two topics that cover the corpus. For instance, LDA could very well discover that the first two sentences are about one topic, sentences three and four about another topic, and the last sentence is a mixture of the first two topics. Given these topics, LDA would also tell us the distribution of the words that compose them. For example, the first topic could include words such as broccoli, bananas, spinach, and munching (with probabilities associated with each), and the second topic kittens, chinchillas, and hamster.^{Footnote 10}

Whereas the model underlying LSA is that the meaning of a word can be described as a position in a multidimensional semantic space, LDA makes a weaker assumption and focuses only on the statistical dependence among words. Consequently, the two methods choose very different tools – LSA relies on factorization methods while LDA is rooted in a statistical model of language. Nevertheless, both methods function similarly in general cases, such as those discussed above. However, LDA has an advantage when the researcher is interested in identifying the topics addressed by a corpus. While the dimensions of LSA are abstract and meaningless, the dimensions in LDA are the latent topics that emerge from the corpus. An interesting use of LDA includes the expressed agenda model (Grimmer, Reference Grimmer2010), which is used to examine authors’ priorities through the topics addressed in their language output. Another application comes from Yano, Cohen, and Smith (Reference Yano, Cohen and Smith2009), who modeled different characteristics of a collection of political blogs using topic modeling.

LDA can also be used in a supervised, or semi-supervised, manner. As an example of the former, Schwartz et al. (Reference Schwartz, Eichstaedt, Kern, Dziurzynski, Agrawal, Park and Lucas2013) used 2000 topics provided by previous LDA analysis, and computed the topic content of Twitter data by county. Using 10-fold cross-validation, they found that LDA topics predicted wellbeing by county beyond the variance accounted for by demographic variables. As an example of semi-supervised application, LDA with topic-in-set knowledge (Andrzejewski & Zhu, Reference Andrzejewski and Zhu2009) is used to seed small sets of words in a subset of the topics and thereby add a level of supervision to the process. This semi-supervised approach combines the advantages of unsupervised topic modeling using LDA with the ability to encourage the emergence of certain topics in the model through small sets of words selected from the outset as prior knowledge. However, instead of simply searching for the most probable set of latent topics, a subset of the topics can be initialized to contain specific words. For example, Dehghani et al. (Reference Dehghani, Sagae, Sachdeva and Gratch2014) used small sets of words selected from the Moral Foundations Dictionary (Graham et al., Reference Graham, Haidt and Nosek2009) as seeds to encourage the emergence of topics related to different moral concerns, and examined similarities and differences in how such concerns are expressed between liberals and conservatives.

1.5. other methods

The three-prong classification of methods that we have used so far is rather crude and schematic, and it inevitably leaves out many useful techniques. While we cannot cover all recent developments, we will mention three other types of method that might be of use for social scientists.