1. Introduction
Communication is ubiquitous in nature: every taxon that has been investigated displays some form of communication system (Kight et al. Reference Kight, McNamara, Stephens and Dall2013). However, linguistic communication (i.e., natural language) is (or is often taken to be) unique to humans. This raises the question: How did language evolve? That is, how did rich linguistic communication systems like the ones we see in humans evolve out of simpler nonlinguistic systems of communication? This is an inherently difficult question because of a lack of direct evidence—language does not fossilize, and we cannot observe the actual precursors of human language in, for example, extinct hominin ancestors.
Nonetheless, work on language origins has blossomed in recent decades. New data, increasingly sophisticated techniques and technologies, and productive interdisciplinary research have helped foster the development of subtle models of language evolution. This is achieved using a multicomponent approach to understand the mechanisms underlying language and how they might have evolved (Fitch Reference Fitch2017). For example, comparative methods in evolutionary biology start by breaking down a complex trait into multiple subcomponent mechanisms or features (Fitch Reference Fitch2017; Martinez Reference Martinez2018). We can then examine the presence or absence of traits, in phylogenetic terms, to infer facts about whether some particular trait common to several species is a homologue or an analogue.
Computer simulations further provide a concrete and explicit way to test hypotheses (Cangelosi and Parisi Reference Cangelosi and Parisi2002), furnishing a how-possibly explanation of the sort that is common in evolutionary biology (Resnik Reference Resnik1991). However, the plausibility of these results requires figuring empirical evidence from relevant fields—in the case of language origins, this includes evidence from biology, linguistics, animal communication, neuroscience, and more.
The most common feature of natural language that is appealed to as a gap-bridging explanatory target is compositionality (and related features like hierarchy and recursion). The idea is that if we could explain how compositional communication can evolve out of noncompositional communication, we would have taken great strides in explaining how language evolved. However, this is problematic insofar as (1) compositionality, in an evolutionary context, proffers asymmetric benefits for senders and receivers of signals, and researchers have not maintained adequate sensitivity to this role asymmetry (LaCroix Reference LaCroix2020a); (2) there is no empirical evidence for protocompositional communication as a precursor to natural language insofar as the oft-cited evidence is more likely analogous to human-level linguistic compositionality than homologous (LaCroix Reference LaCroix2019a); and (3) there is no gradualist explanation of compositionality, insofar as this is a binary property of language (Berwick and Chomsky Reference Berwick, Chomsky, Sciullo and Boeckx2011; LaCroix Reference LaCroix2020b).
In this article, I propose that reflexivity—the ability to use language to talk about language—provides an apt and plausible alternative explanatory target for language-origins research. I further explain how protoreflexivity, which depends on functional reference, gives rise to complex communication systems via modular composition. I argue that reflexivity does not succumb to the problems that compositionality faces since (1) role asymmetries are accounted for by the underlying mechanism of functional reference, (2) there exists empirical evidence of plausible precursors to reflexivity in nature, and (3) the precursors of reflexivity are graded. Finally, reflexivity allows for rich compositional structures that have been shown to give rise to genuinely compositional syntax.
2. Protoreflexivity, Functional Reference, and Their Evolutionary Precursors
Communication is a unique evolutionary system in the following sense. Once a group of individuals has learned some simple communication convention, those learned behaviors may be used to influence future communicative behavior, thereby affecting future communication conventions. This may give rise to a feedback loop, wherein more complex communication, in turn, is used to influence future communicative behaviors that are even more sophisticated.
When faced with a novel context, individuals can always learn a brand-new disposition from scratch. However, in some cases, it may be more advantageous or more efficient to use a preevolved disposition. When individuals take advantage of preevolved communicative dispositions to thereby influence future communication, this is a form of protoreflexivity. Such an ability is an evolutionary precursor to the reflexivity of natural languages, wherein one can use language to talk about language.
Protoreflexivity depends primarily on functional reference, which has been the subject of much empirical and theoretical work in animal communication (Sievers and Gruber Reference Sievers and Gruber2016). Functional reference is so-called because it is meant to evoke the idea of reference in language without being equivalent to reference in the way that words refer. So, the ability to refer functionally is an evolutionary precursor to the ability to refer linguistically. Signals are functionally referential if they are “elicited by a special class of stimuli and capable of causing behaviors adaptive to such stimuli in the absence of contextual cues” (Scarantino Reference Scarantino2013, 1006; see also Macedonia and Evans Reference Macedonia and Evans1993). They are therefore context specific for the signaler to produce and stimulus independent for the receiver to understand. This can be defined formally, as follows.
Definition 1 (Strong) Functional Reference. A token of type X functionally refers to a token of type Y just in case the following two criteria are jointly satisfied:
1. Production Criterion: Xs are reliably caused (only/mostly) by Ys;
2. Perception Criterion: Xs presentations reliably cause responses adaptive to Ys in the absence of Ys and any other contextual cues.
For example, vervet monkey (Chlorocebus pygerythrus) alarm calls are suggested to be functionally referential (Seyfarth, Cheney, and Marler Reference Seyfarth, Cheney and Marler1980). This is because the presence of an eagle (Y) reliably causes an eagle alarm call (X), satisfying the production criterion; furthermore, the presentation of an eagle alarm call (X) reliably causes recipients to hide in the bush (an adaptive response to the presence of an eagle, Y), satisfying the perception criterion. Playback experiments suggest that these responses occur in the absence of other contextual cues.
Female Diana monkeys (Cercopithecus diana) elicit alarm calls upon viewing a predator firsthand and respond to alarm calls of male Diana monkeys by repeating the call. Zuberbühler, Cheney, and Seyfarth (Reference Zuberbühler, Cheney and Seyfarth1999) perform playback experiments of various pairs of stimuli—a matching pair consists of an alarm call followed by the sound of the predator to which the call functionally refers; a mismatched pair consists of an alarm call followed by the sound of a predator to which the call does not functionally refer. In each case, pairs of stimuli are separated by 5 minutes of silence. In the experiment, the female monkeys displayed less concern upon hearing, for example, the characteristic shriek of an eagle 5 minutes after the eagle alarm call—the former conveys no new information. However, they showed significant concern upon hearing a characteristic leopard growl 5 minutes after hearing the eagle alarm call. The conclusion is that alarm calls do not just serve to trigger (behaviorally or deterministically) an evasive response: individuals have an ‘idea’—what Hurford (Reference Hurford2007) terms a ‘protoconcept’—of the relevant predator in mind for at least 5 minutes following the initial alarm call.
We might worry about the strength of definition 1 since, for example, aggression signals may functionally refer to future aggressive behavior, although it perhaps seems strange to say they are caused by it. We can weaken this by indexing to a context and replacing causation with correlation, as in definition 2 (Scarantino Reference Scarantino2013).
Definition 2: (Weak) Functional Reference. A token of type X in context C functionally refers to a token of type Y just in case the following two criteria are jointly satisfied:
1. Contextual Information Criterion: Xs in context C are correlated with Ys (weakly or strongly);
2. Contextual Perception Criterion: Xs presentations in context C reliably cause responses adaptive to Ys in the absence of Ys.
This definition is information-theoretic because X carries information about Y just in case Xs and Ys are correlated.Footnote 1 The intuition is that the signal and the functional referent must correlate enough to make responding to the signal in ways that are adaptive to the referent evolutionarily advantageous.
Functional reference, and therefore protoreflexivity, minimally requires several communicative precursors, including arbitrariness, specialization, semanticity, discreteness, and displacement (Hockett Reference Hockett1960). Arbitrariness requires that there is no ‘natural’ connection between a linguistic form and its meaning; this contrasts with iconic signals where there is a similarity between the form of a sign and its meaning (e.g., onomatopoeia in natural language). Specialization requires that the signal produced is intended for communication, and not because of another behavior; this contrasts with cues, which are a by-product of some other (noncommunicative) process—for example, the presence of CO2 transfers information about the location of a mammal, although exhalation of CO2 did not evolve for this purpose. Semanticity requires that there is a relationship between a signal and its meaning. However, these three features of communication are early evolving abilities that are common to mammals generally. Discreteness means that signals are perceived categorically, as opposed to continuously; this feature is present in primates generally. Finally, displacement is the ability to talk about things that are not present in the immediate environment.
Consider a situation in which individuals coordinate on a communication convention, like in a simple signaling game (Lewis Reference Lewis1969/2002; Skyrms Reference Skyrms2010b). In this case, the messages may functionally refer to the states of the world—as in the vervet monkey alarm call system. Now, suppose that this signaling situation occurs in a preevolved context. Suppose further that there is a novel context in which individuals must learn a new communication system. In some cases, the output of the novel signaling context may be an appropriate input for the preevolved signaling context (Barrett and Skyrms Reference Barrett and Skyrms2017; LaCroix Reference LaCroix2020c; see fig. 1).

Figure 1. Model of simple protoreflexivity.
However, signals functionally refer to states in the preevolved context, and the states are just the output of the communication system in the novel context, so messages come to functionally refer to the communication system itself, in a way that is protoreflexive: they functionally refer to a communication context as a whole, rather than linguistic symbols themselves. In such a situation, discrete, arbitrary, and meaningful signals, which are specialized for communicative purposes, come to (functionally) refer to something abstract, in a sense, and so displaced from the immediate environment.
How might such a property or ability evolve? This happens by way of modular composition and related processes. Various processes of this sort may include appropriation or template transfer, analogical reasoning, or genuine modular composition.
3. Modular Composition and Related Processes
3.1. Transfer (of) Learning
The simplest way of evolving new strategies from old strategies is appropriation. This process, minimally, requires the following steps. First, the agents must have evolved a disposition for a particular context. The agents then face a novel context, where the prior disposition just happens to be appropriate—although this may not be known at the outset. This novel context may be relevantly similar to, but nontrivially distinct from, the original context. Appropriation then consists in applying the prior strategy to the novel context. It may be that the agent happens, by chance, to try something preevolved when faced with a novel context. The appropriateness of the preevolved strategy may determine a sufficiently beneficial reward such that, when faced with this same context again, the agent learns quickly (even by simple reinforcement) to perform the old action. This simple form of appropriation is sometimes called transfer (of) learning (see, e.g., Ellis Reference Ellis1965; Pugh and Bergin Reference Pugh and Bergin2006; Hung Reference Hung2013).
This allows for flexibility of behavior in problem solving, via the ability to generalize learned rules to novel contexts. There is good evidence that many species of new- and old-world monkeys, as well as great apes, are capable of transfer; however, prosimians are not (Rumbaugh Reference Rumbaugh and Rosenblum1970, Reference Rumbaugh1971, Reference Rumbaugh1995; Rumbaugh and Pate Reference Rumbaugh, Pate, Roitblat, Bever and Terrace1984a, Reference Rumbaugh, Pate, Greenberg and Tobach1984b; Bonte, Kemp, and Fagot Reference Bonte, Kemp and Fagot2014). One example of transfer learning in nonhuman animals is an extension of classification tasks, involving ‘reversal learning’. Here, an animal is trained to associate a particular stimulus with a reward. Once the agent exhibits some degree of success, the relation between the stimulus and the reward reverses, so the agent must replace the prior association with the opposite association. If the animal can quickly reverse its associations, it is assumed that successful performance is based on a concept of oppositeness. Yet, if the new association takes as long or longer to be learned, no such application of conceptual understanding may be attributed to the agent.Footnote 2 Minimally, transfer learning requires only that an agent try prior strategies. Successful strategies may be learned via simple reinforcement, or they may be discovered via a more sophisticated trial and error. When salience is present (e.g., the physical properties of a new predator being saliently similar to an old predator) the new strategy may be implemented immediately; however, this is a more sophisticated version of transfer learning, which requires a concept of analogical similarity.
3.2. Analogical Reasoning
The most common way of testing analogical reasoning ability is with a set of analogy problems known as relational matching-to-sample tasks (see Skinner Reference Skinner1950; Blough Reference Blough1959; Ferster Reference Ferster1960). This experimental task involves showing the agent a sample set, which consists of two or more objects that are either identical or nonidentical. The agent is then shown two comparison sets, which contain novel objects—one of which involves identity, and the other of which involves nonidentity. To be successful, the agent must choose the comparison set that matches the sample set.
In this case, the analogy between various stimuli requires a concept of same versus different. As with transfer learning, there is some evidence that nonhuman animals can use analogical reasoning. Despite prior belief to the contrary (Thompson and Oden Reference Thompson and Oden2000), it has been shown experimentally that some apes (importantly, chimpanzees) can perform these tasks easily. Other apes and very few old-world monkeys can perform these tasks but only after extensive training. In each case, symbolic training results in better performance, implying a relationship between cognition and linguistic ability (see, e.g., Skinner Reference Skinner1950; Blough Reference Blough1959; Ferster Reference Ferster1960; Fagot, Wasserman, and Young Reference Fagot, Wasserman and Young2001; Wasserman, Young, and Fagot Reference Wasserman, Young and Fagot2001; Katz, Wright, and Bachevalier Reference Katz, Wright and Bachevalier2002; Flemming et al. Reference Flemming, Thompson, Beran and Washburn2011).
Noting and taking advantage of analogy is more cognitively complex than simple transfer. Increasing complexity again, we arrive at a full concept of modular composition.
3.3. Modular Composition
Finally, modular composition itself varies in complexity, but the most complex forms are supposed to be unique to humans and to depend on language. Spelke (Reference Spelke, Gentner and Goldin-Meadow2003) suggests that humans and other animals are endowed with early developing core systems of knowledge called ‘modules’. However, these core systems are limited in several ways. First, they are domain specific, since these modules represent only a subset of entities in the surroundings of the agent. Second, they are task specific, since they inform only a subset of the repertoire of the agent’s actions and cognitive processes. Third, they are (at least relatively) encapsulated, since there is a restriction on the flow of information into and out of a module. Finally, modules are (at least relatively) isolated from one another, since they do not readily combine (Spelke Reference Spelke, Gentner and Goldin-Meadow2003, 291; see also Fodor Reference Fodor1983, Reference Fodor2000; Sherry and Schacter Reference Sherry and Schacter1987; Sperber Reference Sperber, Hirschfeld and Gelman1994, Reference Sperber and Dupoux2002; Coltheart Reference Coltheart1999; Carruthers Reference Carruthers2002; Barrett and Kurzban Reference Barrett and Kurzban2006; Shettleworth Reference Shettleworth2012; Robbins Reference Robbins and Zalta2017).
Many core cognitive capacities that are available to (and were once thought to be unique to) humans are also available to nonhuman animals (Spelke Reference Spelke, Gentner and Goldin-Meadow2003).Footnote 3 Therefore, humans, but also nonhuman animals, have early developing core knowledge systems, which allow for a broad range of intelligent behavior and cognitive capacities, and, in many cases, these same core systems enable nonhuman animals to outperform human infants in similar tasks. Thus, core systems alone do not account for uniquely human cognitive capacities. Spelke (Reference Spelke, Gentner and Goldin-Meadow2003) suggests that human cognitive capacities depend on core knowledge systems, which are shared by other animals, and on a uniquely human combinatorial ability for conjoining these representations to create new systems of knowledge. Furthermore, she suggests that the latter capacity is made possible by natural language, which provides the medium for combining the representations delivered by core knowledge systems (305). Specifically, it is the compositional nature of natural language that gives rise to uniquely flexible human cognition, on her account.
The basic communicative abilities that give rise to human linguistic capacities are shared with many other species; however, the ability to produce and interpret recursive structures is uniquely human (Hauser, Chomsky, and Fitch Reference Hauser, Chomsky and Fitch2002). If we assume that the human capacity for language can be decomposed into a set of well-defined mechanisms that interact via interfaces, then we can begin to examine how such interfaces between individual components may ‘hook up’ in the first place. In essence, this is the concept of modular composition as it is described in Barrett and Skyrms (Reference Barrett and Skyrms2017). Modular composition ties together explanations of complexity in communicative, cognitive, and social structures.
4. Reflexivity as an Explanatory Target
Researchers typically propose evolutionary theories that explain how compositionality arose, moving from a one-word stage (simple signaling), to a two-word stage (combinatorial signaling), and eventually to (compositional) language (see, e.g., Bickerton Reference Bickerton1990; Jackendoff Reference Jackendoff1999; Progovac Reference Progovac2015). However, as was mentioned in the introduction, prioritizing linguistic compositionality as an explanatory target gives rise to significant theoretical and practical problems.
The novel approach to the evolution of language suggested here priorities reflexivity as an explanatory target. On this account, simple communicative capacities evolve alongside cognitive capacities. Signals may become functionally referential, referring to concrete objects in the world. Once individuals are able to make use of protoconcepts, they can refer to abstracta. Therefore, they can refer to communicative contexts, giving rise to protoreflexivity. This ability means that they can influence future communicative behavior via communication. Such capacities may evolve by modular composition and related processes. Furthermore, it has been demonstrated that reflexivity gives rise to functional composition (compositional syntax) as a by-product of these processes (LaCroix Reference LaCroix2019b).
Several recent works in the signaling game literature have demonstrated that modular compositional processes, like the ones described here, are more efficient and more effective for evolving or learning communication conventions than learning novel dispositions from scratch, often by orders of magnitude (Barrett and Skyrms Reference Barrett and Skyrms2017; LaCroix Reference LaCroix2019b, Reference LaCroix2020c; Barrett Reference Barrett2020; Barrett, Skyrms, and Cochran Reference Barrett, Skyrms and Cochran2020).
Furthermore, reflexivity does not succumb to the same problems that compositionality does, as an explanatory target. It was mentioned in the introduction that compositionality, as it is discussed in the literature, fails to maintain sensitivity to role asymmetries between producers and interpreters of signals (LaCroix Reference LaCroix2020a, Reference LaCroix2020b); however, for reflexivity, this role asymmetry is built-in via functional reference (definitions 1 and 2), which accounts for these differences by definition. Furthermore, there are no empirical precursors to compositionality (LaCroix Reference LaCroix2019a, Reference LaCroix2020b), whereas the processes by which reflexivity evolves are supported by significant empirical evidence. Finally, compositionality is a binary property of language (Berwick and Chomsky Reference Berwick, Chomsky, Sciullo and Boeckx2011), meaning that there is no gradualist explanation of the evolution of compositionality; in contrast, both reflexivity and the processes by which it might arise are graded notions. In nonreflexive functionally referential systems, signals refer to states; in protoreflexive functionally referential systems, signals refer to communicative contexts; and in reflexive language, words refer to linguistic entities. So, reflexivity is graded, but the processes by which it arises are also graded—appropriation is simpler than analogical reasoning, which is simpler than modular composition.
Finally, compositionality is focused too internally on language and syntax itself, so explanations do not (or at least need not) take account of related cognitive and social mechanisms that are important factors in the evolution of language. Yet, reflexivity does. Therefore, there are significant practical and theoretical reasons to replace compositionality with reflexivity as an explanatory target for language origins research.