2.1 Sensitive periods

Since Lenneberg (Reference Lenneberg1967) first used the term critical period in the context of human language development, a considerable amount of evidence has accumulated that shows a marked decline in the ultimate outcome (not the speed) of language acquisition as age of onset varies from early childhood to late adolescence. This decline has been documented in numerous studies, for both L1 and L2 development, for both spoken and signed languages, and for phonology as well as morphology and syntax (for overviews, see DeKeyser, Reference DeKeyser, Gass and Mackey2012; Hyltenstam & Abrahamsson, Reference Hyltenstam, Abrahamsson, Doughty and Long2003).

Numerous questions remain, however, at least where the L2 is concerned. The most debated one is whether the age effects observed are truly maturational or due to confounds with other variables (e.g., DeKeyser & Larson-Hall, Reference DeKeyser, Larson-Hall, Kroll and de Groot2005; Long, Reference Long2005). Most commonly mentioned in the discussion of potential confounds are the extent of L1 entrenchment (e.g., MacWhinney, Reference MacWhinney, Han and Odlin2006), the quantity and quality of input and practice in L2 (e.g., Jia & Aaronson, Reference Jia and Aaronson2003; Jia, Aaronson & Wu, Reference Jia, Aaronson and Wu2002), the extent to which the learner is motivated to sound like a native speaker (e.g., Bley-Vroman, Reference Bley-Vroman, Rutherford and Sharwood Smith1988), and the extent to which formal education took place in the L2 (e.g., Hakuta, Bialystok & Wiley, Reference Hakuta, Bialystok and Wiley2003). An equally important question concerns the nature of inter-individual variation (e.g., whether high levels of some forms of aptitude mitigate the effect of age of onset; Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2008; DeKeyser, Alfi-Shabtay & Ravid, Reference DeKeyser, Alfi-Shabtay and Ravid2010). Finally, there is the question of intra-individual variation depending on the aspects of grammar or pronunciation concerned. In the area of grammar, syntax may be less sensitive to age effects than morphology (Johnson & Newport, Reference Johnson and Newport1989), regulars less than irregulars (see Hudson Kam & Newport, Reference Hudson Kam and Newport2005, Reference Hudson Kam and Newport2009), and salient structures less than non-salient ones (DeKeyser, Reference DeKeyser2000). Even for a given structure, age effects may be detected with ERP without showing up in the behavioral data (e.g., for subject–verb agreement in Chen, Shu, Liu, Zhao & Li, Reference Chen, Shu, Liu, Zhao and Li2007). In the area of pronunciation, phonetic detail such as precise voice onset time (Abrahamsson & Hyltenstam, Reference Abrahamsson and Hyltenstam2009) seems particularly problematic for older learners. Some phonetic cues to phonemic status may be easier to pick up than others (vowel duration being easier than closure duration; Baker, Reference Baker2010); some suprasegmentals such as stress timing may be less sensitive to age than others (Trofimovich & Baker, Reference Trofimovich and Baker2006); and age may even affect different kinds of stress placement differently, the effect being strongest for stress determined by syllable structure (Guion, Harada & Clark, Reference Guion, Harada and Clark2004). In sign language, handshape may be more resistant to age effects than location or movement (Morford & Carlson, Reference Morford and Carlson2011).

Those researchers who suspect that sensitive periods are maturational in nature have couched their causal explanations in both neurological and psychological terms. Neurological explanations have evolved over time from hemispheric specialization (e.g., Lenneberg, Reference Lenneberg1967) to myelination (e.g., Long, Reference Long1990) to varying rates of neurogenesis, synaptogenesis, or synaptic pruning (e.g., Uylings, Reference Uylings2006). These explanations have focused on the brain as a whole, while others more on specific areas such as the prefrontal cortex (e.g., Petanjek, Judas, Kostović & Uylings, Reference Petanjek, Judaš, Kostović and Uylings2008); the amygdala (e.g., Pulvermüller & Schumann, Reference Pulvermüller and Schumann1994); or the hippocampus, medial temporal lobe, and the basal ganglia (e.g., Ullman, Reference Ullman2004). Psychological explanations, rather surprisingly, came onto the scene later and have included growth of working memory capacity (the less-is-more hypothesis, e.g., Newport, Reference Newport1990), increased susceptibility to proactive interference (e.g., Iverson, Kuhl, Akahane-Yamada, Diesch, Tohkura, Kettermann & Siebert, Reference Iverson, Kuhl, Akahane-Yamada, Diesch, Tohkura, Kettermann and Siebert2003), and gradual shifts from predominantly procedural/implicit to predominantly declarative/explicit processes (e.g., DeKeyser, Reference DeKeyser2000; Paradis, Reference Paradis2009; Ullman, Reference Ullman2004). Ultimately, of course, full explanatory adequacy will only be reached if psychological mechanisms can be tied to concurrent neurological developments that together explain the specific learning differences observed.

Empirical research on these issues is usually difficult for many reasons, in large part because the natural confounds of many of the variables involved cannot be experimentally disentangled in research on human learners. Perhaps the only notable exception is in the study of age of acquisition effects in sign language research (Mayberry, Lock & Kazmi, Reference Mayberry, Lock and Kazmi2002), which is discussed in detail in the supplementary material Section S.1.

2.2 Grammatical gender

The linguistic phenomenon that our models will learn about is grammatical gender, which refers to an arbitrary classification of nouns, often marked by phonological, morphological, and/or semantic properties. Several studies suggest that grammatical gender is subject to sensitive periods. Studies with adult L2 learners of languages like French (Guillelmon & Grosjean, Reference Guillelmon and Grosjean2001), Spanish (Lew-Williams & Fernald, Reference Lew-Williams and Fernald2010) and German (Scherag et al., Reference Scherag, Demuth, Rösler, Neville and Röder2004) have shown that non-native adults are slower than L1 speakers at processing nouns, and that their processing does not seem to benefit from patterns in gender agreement that are present in the language. Even childhood learners who begin acquiring French in an immersion program at age six do not achieve native-like gender agreement (Harley, Reference Harley1979; Lapkin & Swain, Reference Lapkin and Swain1977), indicating that acquisition of this grammatical component is subject to early age effects. For a more detailed background of the acquisition of grammatical gender, see Section S.2 in the supplementary material.

Gender systems vary in complexity; many Indo-European languages, such as French and German, divide nouns into only two or three gender classes, whereas Bantu languages employ extensive gender systems with up to twenty gender classes (Corbett, Reference Corbett1991). The degree to which grammatical gender is marked throughout a sentence also varies widely. In English, for example, gender is only marked on pronominals with animate reference, whereas gender in the Bantu language Swazi may be marked on adjectives, verbs, adverbs, numerals, and conjunctions.

The languages examined in the current study, French and Spanish, both assign masculine and feminine gender to all nouns; however, subtle differences between the gender classification systems exist. In French, a noun's final phoneme provides cues to gender, though the predictive value of the final phoneme is not always reliable. For example, according to Surridge (Reference Surridge1993, Reference Surridge1995), only one “feminine” ([z]), and eight “masculine” ([], [], [ã], [ø], [o], [ʒ], [m], [ɛ]) final phonemes indicate gender with more than 90% accuracy; eight “masculine” ([f], [u], [a], [ʁ], [g], [y], [k], [b]) and nine “feminine” ([i], [], [n], [v], [j], [ʃ], [d], [s], [ɲ]) final phonemes indicate gender with 60–89% accuracy; and four final phonemes ([l], [m], [p], [t]) are considered ambiguous and do not provide any indication of the noun's gender. In addition, not everyone agrees with the phonemes’ predictability values. For example, Lyster (Reference Lyster2006) carried out a final phoneme predictive value analysis based on a corpus different from that of Surridge, and while the results are largely similar, some differences exist. Furthermore, the effect of a noun's phonological ending may be overridden by the noun's morphological ending (Surridge, Reference Surridge1989). Under this hierarchy, a word ending in the typically masculine final phoneme [ʁ] will be feminine when encompassed by the typically feminine morphological suffix -ure, as in coiffure “hairstyle”. Overall, the French gender system is governed by patterns, but it is a complex system with many exceptions.

The Spanish gender system is less complex and more reliable than that of French. According to Teschner and Russell (Reference Teschner and Russell1984), the majority of Spanish nouns’ final phonemes are predictive of gender. Specifically, 90% of nouns ending in the phonemes [a] and [d] are feminine, and 89% of nouns ending in [e], [l], [o], and [ɾ] – which account for the majority of nouns – and also [i], [m], [t], [u], [x], [y], [b], [c], [tʃ] are masculine. Only three final phonemes, [n], [θ], and [s], are considered ambiguous in that they do not predict one gender over another. Morphological gender regularities in Spanish also exist, though they do not override final phonemes, as seen in French. Teschner and Russell identify seven morphological endings that are typically feminine (-ción, -gión, -nión, -sión, -tión, -xión, and -ez) and four morphological endings that are typically masculine (-ón, -az, -oz, and -uz). Note that these morphological endings encompass two of the ambiguous final phonemes, [n] and [θ], but not phonemes that are predictive of masculine or feminine. Finally, in both languages, animate nouns referring to humans assume semantic gender, so that the words for “man” and “woman” are masculine and feminine, respectively.

Both French and Spanish mark gender on determiners, pronouns, and adjectives. Examples of determiner and adjective markings are shown in sentences 1 and 2.

1. (1) “The little book is white.”

2. French: Le petit livre (masc) est blanc.

3. Spanish: El libro (masc) pequeño es blanco.

4. (2) “The little table is white.”

5. French: La petite table (fem) est blanche.

6. Spanish: La mesa (fem) pequeña es blanca.

French adjectives may end in almost any phoneme, with the feminine adjective typically marked by an additional and often unpredictable suffix. For example, the adjective blanc [blã] “white, masc” becomes blanche [blãʃ] in its feminine form, and petit [pəti] “small, masc” becomes petite [pətit]. A number of adjectives have the same phonological form for both masculine and feminine, even when the orthographic form differs. For example, the adjective “difficult” has only one orthographic (difficile) and phonological [difisil] form, and the adjective “expensive”, while represented by two orthographic forms (cher, masc; chère, fem), are both pronounced [ʃɛʁ].

Spanish adjective formation, on the other hand, is more predictable. The majority of adjectives are marked by an -o ending for masculine, and an -a ending for feminine, as in blanco/blanca “white”. As in French, not all adjectives have distinct orthographic and phonological masculine and feminine forms. Adjectives ending in -e, -ista, or a consonant, generally maintain the same form in both masculine and feminine, as in verde “green”, idealista “idealist”, and difícil “difficult”. However, exceptions exist and certain types of adjectives ending in a consonant, such as those referring to nationalities, have a feminine form marked by an -a ending, as in español/española “Spanish” and alemán/alemana “German”. Other exceptions include adjectives ending in -ín, -ón, -or, such as juguetón/juguetona “playful” and hablador/habladora “talkative”.

Despite the differences described above, the French and Spanish gender systems are similar in that both classify nouns into masculine and feminine based on phonological regularities, and gender is marked throughout a sentence on determiners, adjectives, and pronouns.

2.3 Memory development and language learning

Our neural network models undergo memory development, in the form of changes in both working memory capacity and long-term memory capacity, in order to examine the effects of maturation on sensitive period effects. At the most simple description, working memory (used interchangeably here with short-term memory) allows pieces of information to be held in the mind for brief periods of time in the absence of the input that caused them. In reality, working memory is most likely composed of a complex interaction of factors, such as attention (Conway, Cowan & Bunting, Reference Conway, Cowan and Bunting2001; Engle, Reference Engle2002; Kane & Engle, Reference Kane and Engle2003), inhibition or filtering mechanisms (Vogel, McCollough & Machizawa, Reference Vogel, McCollough and Machizawa2005), rehearsability (Baddeley, Reference Baddeley2003; Gathercole & Baddeley, Reference Gathercole and Baddeley1993; Wilson & Emmorey, Reference Wilson and Emmorey1997), and “chunking” strategies (Miller, Reference Miller1956). Thus, although working memory is probably not a unitary construct, the core ability to store and integrate multiple items is critical to many aspects of cognitive functioning, including language processing. Working memory capacity refers to the number of items that can be stored and manipulated for a task. In general, higher capacities are associated with better cognitive function (Baddeley, Reference Baddeley2003; Duncan, Seitz, Kolodny, Bor, Herzog & Ahmed, Reference Duncan, Seitz, Kolodny, Bor, Herzog and Ahmed2000) since lower capacities impose greater informational bottlenecks on processing. In development, working memory capacity grows rapidly from early childhood into adolescence, showing up to a three-fold increase (see Gathercole, Reference Gathercole1999). This presents a paradox for language acquisition since higher cognitive function associated with higher memory capacity seems to be inversely correlated with overall language learning ability.

However, this is only a paradox if only the end state of development is considered. In reality, the maturation of working memory as well as language learning occur through time. One possibility is that limited cognitive ability, in particular a small memory capacity, is crucial to early stages of language acquisition, and that memory growth supports full language acquisition. Newport's (Reference Newport1988, Reference Newport1990) less-is-more hypothesis draws upon data from cases where age of acquisition is not confounded with L1 entrenchment: the large proportion of deaf individuals who are not exposed to an accessible form of language early in life. During the language acquisition process and at final language attainment, these late learners have distinct profiles from early learners. As seen among hearing children during early stages of acquisition and word production, young signers (who have been exposed to American Sign Language since birth) morphologically simplify complex signs. This stage is considered to be important for morphological analysis of words and signs. Late learners do not make these types of errors or simplifications, rather processing the forms as “unanalyzed wholes” (Newport, Reference Newport1990). As adults, these late learners use these complex forms in both ungrammatical and grammatical contexts, suggesting that they have not successfully learned their internal morphology. Early learners, in contrast, progressively develop the complex forms and do not make these types of mistakes as adults.

If the development of working memory is indeed inextricably linked with language acquisition abilities, there are two possible explanations for this relationship. The first is addressed by the less-is-more hypothesis, where the crucial factor is starting with a smaller working memory capacity (Newport, Reference Newport1990). The rationale is that when a learning system is incapable of processing and holding in memory larger chunks of input, it is forced to analyze the input at lower level of complexity, picking out the highest-level and most prominent patterns while possibly abstracting away much of the detail. Another potential explanation comes from computational modeling, where it has also been demonstrated that controlling the size of the input, perhaps by providing smaller inputs at the beginning of training, contributes to better learning (Elman, Reference Elman1993). Both models have been experimentally tested in adults, where smaller natural working memory or smaller inputs were associated with better detection of correlations between two binary variables (Kareev, Lieberman & Lev, Reference Kareev, Lieberman and Lev1997).

Most studies that directly investigate the relationship between working memory capacity and language learning in children suggest that the development of phonological short-term memory in particular is critical to word learning (Avons, Wragg, Cupplesa & Lovegrove, Reference Avons, Wragg, Cupplesa and Lovegrove1998; Baddeley, Gathercole & Papagno, Reference Baddeley, Gathercole and Papagno1998; Gathercole & Pickering, Reference Gathercole and Pickering2000). Higher spans in phonological short-term memory are linked with larger vocabulary sizes and better performance at learning new words. These working memory capacities are often measured by performance on non-word repetition tasks. However, these correlations leave the causal relationships inconclusive. The ability to temporarily store phonological traces of new utterances may be an important precursor to storing that item in long-term memory. On the other hand, it has been suggested that vocabulary growth leads to a better ability to analyze the representations into phonological segments, which in turn leads to more robust representations of new words (Metsala, Reference Metsala1999). What these two ideas agree on is the importance of the development of decomposed, sublexical representations – such as phonemes – for language learning. Newport (Reference Newport1990) has made a similar argument about morphology. Drawing upon accompanying behavioral evidence that longer words are learned later in development than shorter words even when frequencies of these words are matched, Brown and Hulme (Reference Brown, Hulme and Gathercole1996) demonstrate a computational model in which shorter words are maintained in short-term memory for longer given a limited short-term memory, facilitating encoding in long-term memory. A consequence of forming representations for smaller input first may be a better recognition of incremental patterns throughout learning.

2.4 Connectionist modeling

As discussed in Section 1 above, it is often difficult to experimentally separate the various possible causes of age effects when performing empirical research on human subjects. Computational modeling has a key advantage in its ability to independently manipulate a number of variables and to observe their main effects and interactions. Early attempts at computational modeling of linguistic sensitive periods (Goldowsky & Newport, Reference Goldowsky, Newport and Clark1993) show support for the less-is-more hypothesis in that a smaller working memory was shown to be better for the learning of some grammatical patterns, and this conclusion was supported by later studies, computational and otherwise (Cochran, McDonald & Parault, Reference Cochran, McDonald and Parault1999; Kareev et al., Reference Kareev, Lieberman and Lev1997; Kersten & Earles, Reference Kersten and Earles2001). Neurocomputational modeling studies (reviewed in Hernandez & Li, Reference Hernandez and Li2007) favor explanations of age-related performance deficits in terms of changes in neural plasticity due the normal accumulation of experience. This idea, that the learning process itself could cause the observed sensitive period effects, is supported by many other modeling studies (reviewed in e.g., Thomas & Johnson, Reference Thomas and Johnson2008) and has been called the “paradox of success” since learning one task to proficiency can harm the learning of other tasks (Seidenberg & Zevin, Reference Seidenberg, Zevin, Munakata and Johnson2006). Sensitive period effects can be produced via the learning process itself in a number of ways, including entrenchment, where early experience leaves the learning system in a state not readily compatible with a new learning task; competition for resources between different tasks to be learned; and catastrophic interference, where a new learning task may impact performance on a previously learned task that is not actively maintained.

Previous neural network models that have dealt with aspects of memory development have used varying approaches to limiting working memory. Elman (Reference Elman1993) trained Simple Recurrent Networks (SRNs) on a complex subset of English. This type of network uses recurrent connections to allow the network to access its own previous states, creating an analog of working memory. Elman found that these networks had better eventual performance when this working memory was initially limited to a discrete window of a few steps and gradually increased, consistent with the less-is-more hypothesis. While others have failed to find a difference between developing and mature networks on similar tasks (e.g., Rohde & Plaut, Reference Rohde and Plaut1999), Elman's study shows one way in which working memory capacity can be modeled in a neural network. As we will explain, our model uses a different approach, directly limiting the capacity of, or physical access to, previous states instead of limiting the network's temporal window of access to these states. Our approach is, in a sense, similar to that of the DevLex models of word and meaning acquisition (Li, Farkas & MacWhinney, Reference Li, Farkas and MacWhinney2004; Li, Zhao & MacWhinney, Reference Li, Zhao and Mac Whinney2007), which utilize growing self-organizing maps to represent semantics and phonology. These maps grow by adding new units to accommodate storage of new lexical and semantic representations; as such, the growth involved more closely resembles long-term memory growth. Our model, in contrast, grows by adding new units that form the substrate for working memory.

There have also been a few notable neural network models that touch on the topic of grammatical gender. MacWhinney, Leinbach, Taraban and McDonald (Reference MacWhinney, Leinbach, Taraban and McDonald1989) presented two neural network models of the acquisition of gender, case, and number in German. Both of these models learned to predict the article associated with a given noun, one using hand-coded semantic, phonological, morphological, and case cues, and the other using only observable data in the form of a complete phonological representation of the input noun along with some semantic and case cues. Both models succeeded at learning the nouns they were trained on, and also generalized very well to new nouns. The second model, without the hand-coded cues, outperformed the first. Unfortunately, the static phonological representations in this model only allow it to be applied to words of two syllables or fewer; our model employs temporal phonological representations that allow any word to be encoded. Additionally, Sokolik and Smith (Reference Sokolik and Smith1992) trained a feed-forward neural network to identify a corpus of French nouns as either masculine or feminine. Their study, however, has been widely criticized (Carroll, Reference Carroll1995; Matthews, Reference Matthews1999) for, among other things, using orthographic input, giving explicit gender feedback, and building in language-specific knowledge about gender classes. We believe that our approach adequately addresses these and other concerns, resulting in a model that only utilizes the information available to language learners.

4.1 Gender assignment

In our first set of experiments, we investigate how well neural networks can learn to perform a gender assignment task using realistic sources of information. These networks take single nouns as input and use that information to predict which determiners can appear with that noun. Since nouns commonly occur with determiners in our target languages, French and Spanish, both the input and the output data are readily available to any learner by simply listening to everyday speech. After training, we determine the network's assignment of gender to individual nouns by presenting those nouns as input and observing the network's predictions for determiner pairing. The gender of the most strongly predicted determiner is taken to be the network's gender assignment for the input noun.

Our approach is similar to that taken by the third model from MacWhinney et al. (Reference MacWhinney, Leinbach, Taraban and McDonald1989) in that our model uses the complete phonological form of a noun to predict the article to be used with that noun. We diverge from this earlier model in a few important ways. First, we eschew semantic features to investigate what can be learned from phonology alone. Even though phonology is predictive for the majority of words in our target languages, this choice deprives our model of information that learners are known to use (see Section 2.2 above). Second, we present the input noun as a temporal sequence of phonemes instead of a single phonological pattern, the latter of which will always have trouble representing long words or those that do not conform to the prespecified representational form. In addition, our approach corrects the most severe issues with the model of gender assignment by Sokolik and Smith (Reference Sokolik and Smith1992). Where their approach was criticized (Carroll, Reference Carroll1995; Matthews, Reference Matthews1999) for using orthographic input, we use phonemic input instead. Where their network came a priori equipped with knowledge of the genders of the training language – and indeed the knowledge that grammatical gender exists at all – our model has no such built-in knowledge. Finally, where their model required explicit feedback about the genders of individual words, our model relies instead upon the co-occurrence of gendered articles with nouns in order to deduce gender assignments. As a result of these differences, our model is more closely aligned with the real-world circumstances of human language learning in most contexts.

An input noun is presented to the network as a temporal sequence of phonemes. Each such phoneme is represented as a set of binary auditory features, with the activations of the network's input layer adjusted to reflect the feature set of each phoneme in turn. We use this representation because such features are universal in the sense that various configurations of these features can represent virtually any phoneme. As such, units representing these features could potentially be a built-in component of the brain of a language learner, or could be learned. That said, we only included enough features here to distinguish all phonemes in our target languages. The full set of phonemes and features are detailed in Table 1. After processing an entire sequence of phonemes representing the input noun, the network activates units in its output layer that correspond to determiners that it predicts to be compatible with the input noun. The network learns to perform this behavior by observing determiner–noun pairings and adjusting its connection weights accordingly.

Table 1. Binary feature representations of phonemes.

The left half of Figure 3 shows the general architecture of the networks we train to perform this gender assignment task. The networks have an input layer of units corresponding to the on and off states of the features that make up the input phonemes. Units in the input layer project to units in a single hidden layer of memory cells. The intrinsic self-recurrence of the memory cells forms the substrate for working memory in the network. Finally, the hidden layer projects forward to the output layer which consists of nine units representing the definite and indefinite singular determiners of our target languages: le, la, l’, un, and une in French, and el, la, un, and una in Spanish. We do not posit that units representing these words could be built into the brains of language learners, nor that the words are represented in single units. However, since these determiners form a small closed class of words, we feel it is not too large a leap to presume that the learner represents these frequent determiners as distinct entities before much gender learning takes place. Our single-unit representation for each determiner is the simplest possible in this context, though other representations would likely work as well.

Figure 3. On the left is shown the architecture of networks used in the gender assignment task. The assignment network takes features of auditory phonemes as input, passes them through a hidden layer of self-recurrent memory cells, and maps a sequence of such inputs onto an output layer of units representing determiners in two languages. On the right is the architecture of networks used in the gender agreement task. The agreement network also takes auditory phoneme features as input, but passes them through a series of two hidden layers of memory cells. After processing each input phoneme, the network uses its two output layers to predict the next two phonemes that it will be given as input. Though no recurrent connections are depicted at this level for either network, each individual memory cell is self-recurrent, remembering its activation from the previous step.

For this set of experiments, we used the 600 French words from the Sokolik and Smith (Reference Sokolik and Smith1992) paper as the input data for our model, and a set of 600 equivalent words from Spanish. For each trial during training, we first select a language and then select a noun at random from our corpus. We pair the noun with either a gender-matched definite or indefinite determiner from the appropriate language to form a simple noun phrase. The noun is given as input to the network, which then predicts applicable determiners and adjusts its weights in such a way that, in the future, it will be more likely to predict the determiner that actually co-occurred with the input noun. A network is considered to have assigned the correct gender for an input noun if an article of the appropriate gender is most active after presentation.

To determine a baseline level of performance on the gender assignment task, we trained networks on either French or Spanish only and recorded their performance. The results are shown in the left half of Figure 4. As one would expect, networks trained on French alone scored well in excess of 90% after training, while scoring at chance on Spanish; similarly, Spanish-trained networks performed well on their native language and at chance on French. It is worth noting that Spanish performance was consistently a few percentage points better than French performance, likely due to the phonemic cues to gender assignment in Spanish being simpler and more reliable than those in French. Performance was consistent across the four development conditions, suggesting that, alone, network development has little impact on outcomes for the gender assignment task, at least in the first language.

Figure 4. Results for monolingual networks, of all four developmental varieties, on the gender assignment and agreement tasks. We trained 30 separate networks in each developmental condition for each language. Each network was trained for two million trials in one language, and then evaluated on both languages.

With a baseline level of performance established for networks that are “native” to either French or Spanish, we next investigated the performance of bilingual networks under a number of different learning conditions designed to assess the role of L1 entrenchment. Each condition varies the length of time t which the network spends learning the task on L1 alone before L2 is introduced (Zhao & Li, Reference Zhao and Li2010). We describe the conditions in terms of two periods, the first of which consists of training only in L1 for t trials, where t varies widely across conditions. This is immediately followed by the second period, in which L1 and L2 trials are mixed with equal probability. The duration of the second period is always two million trials in an effort to ensure that the networks have time to reach peak performance on both languages. While this second mixed training period will undoubtedly create competition and interference between the two languages, the amount of interference should be the same in each condition because the size and mix proportion of the second training period are the same across conditions. In contrast, the amount of L1-only training prior to the introduction of L2 is varied across conditions, meaning that networks that start with different values of t will end up in different states – reflecting differing levels of entrenchment – when L2 training begins.

A network whose training regimen has t = 0 is a native bilingual in the sense that L1 and L2 are presented at precisely the same time, and in the same proportions. Thus, such a network should exhibit no L1 entrenchment. Networks trained with higher values of t, having had a longer time with exposure only to L1, should exhibit more entrenchment. Given this, the prevailing ideas about L1 entrenchment offer a number of predictions about the final, peak L1 and L2 performance of the networks:

1. 1. Networks’ final L1 performance should not decrease as t increases.

2. 2. Networks trained with t = 0, as native bilinguals, should not exhibit impairment in either language with respect to the other.

3. 3. Networks should show increasing degradation of final L2 performance as t increases, at least until the networks have mastered L1 to a point at which the effect of entrenchment saturates.

These predictions can be investigated by plotting the final L1 and L2 performance of fully trained networks on the gender assignment task versus the value of t with which they were trained. We trained 30 separate networks for each of 15 values of t as well as for each of the four maturation conditions and each of two languages; thus a total of 3,600 networks were trained to produce the following figures. For conditions in which the network matures during training, each of these networks begins training in its most immature state and develops over the course of the first 400,000 trials, at which point it reaches maturity – i.e. architectural parity with the networks in the no growth condition. Thus, some networks in the connection growth and unit growth conditions (i.e. those with t = 0) are first exposed to L2 in their most immature state, while others (i.e. t = 400,000 and above) are not exposed to L2 until after reaching maturity.

After both training periods were complete, we recorded the fraction of inputs to which each network assigned the correct gender, for both languages, and plotted them in the left half of Figure 5. These graphs depict the final performance of the networks on the y-axis versus the value of t – i.e. the duration of the L1-only training period and thus the delay before L2 onset relative to L1 – on the x-axis. Thus, the expected L1 entrenchment increases from negligible to maximal as we move from left to right in each figure; another way of saying this is that the networks towards the left of the x-axis are closer to true bilinguals whereas the networks closer to the right edge are late L2 learners. The y-axis values always depict final performance after the conclusion of training. These graphs show fitted curves for each of the different network maturation conditions, and for each such curve, the shaded area behind it represents the 95% confidence interval.

Figure 5. Results for bilingual networks on the gender assignment and agreement tasks. Each network was assigned an L1 and trained initially on only that language before the other language was introduced. The x-axis varies the time t each network spent with L1 in isolation before L2 was introduced. Note that t = 0 corresponds to a native bilingual network for which neither language is ever prioritized, while larger values of t correspond to increased time spent with L1 alone, and thus presumably increased levels of L1 entrenchment. The y-axis shows the final performance on the language task after training was complete. We trained 30 separate networks in each combination of developmental condition, native language, and t-value (shown as ticks on x-axis). The networks were trained for t trials of L1 alone before a period of 2 million trials of L1 and L2 in equal proportion. The lines depicted for each combination of development condition and task are smoothed-average curves shown over the sampled values of t, which are indicated by the vertical lines in each pane.

To examine the first prediction above, we first look at the performance of the various networks in their native language. Table 2 shows the results of a statistical analysis on the performance results, comparing the means for each condition at the first and last t-values using a two-proportion z-test. In addition to statistical significance, the table also provides an indication of the magnitude of the performance change. This answers the question, for each condition, of whether performance is statistically flat, increasing, or decreasing as t values increase. The table also shows codes for the magnitude of the significant changes in performance. The prediction of non-decreasing performance with increasing t appears to be largely borne out. When native language and task match, performance is flat or occasionally weakly increasing as t values rise. Figure 5 shows us that, as with the monolingual networks, bilingual networks have a slightly harder time learning French than Spanish as an L1. Differences in Spanish performance between the different maturational variants of Spanish-native networks were minimal, while the French-native networks that grew their working memory capacity during training showed a slight disadvantage. However, the expected general pattern of flat or improving performance with increasing t held for all conditions.

Table 2. Significance and magnitude of performance change as t increases.

Significance: *** for p < .001, ** for p < .01, * for p < .05

Magnitude: ***** for m > 12%, **** for m > 9%, *** for m > 6%, ** for m > 3%, * for m > 1%

We can investigate the second prediction above by examining each network's performance on its second language. We do this by comparing second-language performance of networks with t = 0 on the x-axis to the native-language performance. We see that across all maturational conditions, the true bilingual networks (those with t = 0) perform well when compared to the native networks in both languages, lending support to the second prediction above.

Moving on to the third prediction, we can clearly see a t-related performance deficit in the no growth condition for L2 French; increasing Spanish exposure before French is introduced causes the final French performance of the network to decrease at a rate that is at first rapid but eventually slows for larger delays. The maturational properties in play for the connection growth and unit growth conditions, however, appear to have helped these networks compensate for the expected declines in French performance due to Spanish entrenchment. Networks in the unit replacement condition tended to perform at levels comparable to the no growth networks, suggesting that introduction of new working-memory resources without starting small may not be sufficient to gain a significant reprieve from the deleterious effects of increasing L1 entrenchment. In the case where French was the L1 and Spanish the L2, no appreciable t-related performance decreases were observed. We expect that this is due again to the relative ease of the task for Spanish as compared to French.

At least in the case of French as an L2, the data shown in Figure 5 and Table 2 seems to support both the predictions of performance declining due to increased entrenchment and of maturation during learning helping to overcome these difficulties. Next we trained networks on the more difficult task of gender agreement, the results of which are reported in the next section.

4.2 Gender agreement

Our second set of experiments explores how neural networks perform on a gender agreement task. During a trial, networks in these experiments receive a noun phrase (e.g., el mecanismo interno “the internal mechanism” in Spanish) presented as an unsegmented sequence of phonemes (e.g., [elmekanismointeɾno]) as input. The network's job at every point in this phoneme sequence is to predict the next few phonemes that it will hear. As such, the network uses a phonemic representation of everyday speech as both the input and the training signal. After training, the network's gender agreement performance is evaluated using noun phrases of the form determiner–noun–adjective – common constructions in our target languages of Spanish and French. To determine gender agreement, we give the network the determiner and noun as input, followed by the portion of the adjective that is gender-neutral, and ask the network to predict the correct ending for the adjective. If the network predicts the gender-appropriate ending more strongly than the gender-inappropriate ending, we consider the network's answer to be correct.

The noun phrases we used as training data for the gender agreement task were extracted from the French and Spanish versions of Wikipedia (2011). We downloaded archives containing the complete text of each version of Wikipedia and applied part-of-speech tags to each word using TreeTagger (Schmid, Reference Schmid1994). We then extracted all noun phrases of the forms determiner–noun, determiner–noun–adjective, and the less frequent determiner–adjective–noun, where the determiner is one from Table 3. From this list of noun phrases we removed any phrases containing words that were not in our language dictionaries – Lexique 3 for French (New, Reference New2006) and CUMBRE for Spanish (CUMBRE, n.d.). Finally, we extracted the most frequent 100,000 noun phrases for each language. These phrases comprise the training data. On each training trial, we chose a phrase probabilistically, based on the phrases’ corpus frequencies; we used this phrase as the input – and training signal – for the network on that trial.

Table 3. Determiners used by the gender agreement model.

The right side of Figure 3 presents the architecture of the networks trained on the gender agreement task. The input layer is the same as it was for the gender assignment experiments, with each input unit corresponding to a binary auditory feature of a phoneme. These networks, however, have two hidden layers of memory cells instead of one. This is because the gender agreement task involves two separate levels of segmentation of the input. To perform the task effectively, we expect that any learner needs to divide the phoneme sequence first into morphemes and words and, at a higher level, into noun phrases in which gender agreement must be maintained. Previous experiments with these types of networks on language tasks (Monner & Reggia, Reference Monner and Reggia2012) have shown a network with two hidden layers to be more effective in this case than networks with a single hidden layer.

The network's output layers are each identical to the input layer because the network is predicting upcoming phonemes. There are two such output layers because the network must predict not only the next phoneme that will occur in the input, but the phoneme after that as well. We require the network to make predictions of two future phonemes because some of the gendered adjective endings that we would like to predict consist of two phonemes. For example, the French adjective for “particular” is particulier [paʁtikylje] in the masculine and particulière [paʁtikyljɛʁ] in the feminine; we can see in the phonetic spellings that the gendered endings of these adjectives differ across two phonemes, with [-e] ending the masculine form and [-ɛʁ] ending the feminine form. Since we can only show the network the gender-neutral portion of the phoneme sequence (i.e. [paʁtikylj-]) without giving away the gendered form intended by the speaker, we must have the network predict two subsequent phonemes (either of which may be null if subsequent phonemes do not exist) in order to capture gendered endings with two phonemes such as [-ɛʁ].

When evaluating performance on the gender agreement task after training, we use only phrases of the determiner–noun–adjective form because it is the only form that is adjective-final. Our testing paradigm requires an adjective-final form because the network must predict the gender-appropriate ending of the last word, and only adjectives generally have two distinct gendered endings. Gender-neutral adjectives, and adjectives where the two gendered forms are orthographically distinct but phonetically identical (e.g., in French, the masculine architectural and the feminine architecturale are both pronounced [aʁʃitɛktyʁal]), are present during gender agreement training but ignored during the performance evaluation.

To determine a baseline level of performance on the gender agreement task, we trained sets of networks on either French or Spanish only and recorded their performance. The results are shown in the right half of Figure 4. As was the case with the gender assignment task from the previous section, we find here that networks trained on French do well on French and perform at chance on Spanish. Networks trained on Spanish perform as expected on that language and do significantly worse on French.

We use the same experimental setup as in the gender assignment task to investigate the effects of L1 entrenchment alone (i.e. the no growth condition) and together with network maturation (i.e. the unit growth, unit replacement, and connection growth conditions) in the gender agreement task. As before, training consists of two periods, the first consisting of t trials in which inputs come exclusively from the designated L1, and the second consisting of two million trials where inputs may be drawn from either language. We trained 30 networks in each maturation condition and for each value of t, the duration of the initial L1-only training period. The results are shown in the right half of Figure 5 above.

We can examine the networks’ performance on their first languages, broken out by language and maturation condition as before, by looking at the bottom half of Table 2. As expected, we do not see decreasing performance with increasing t in any of the conditions where the task and native language match.

Next we examine the final performance scores on L2 for networks in each condition of the gender agreement task as a function of t on the x-axis. The results for both languages here are similar to what we observed in the gender assignment task for the case of L2 French. The mature networks in the no growth condition show a marked susceptibility to L1 entrenchment, with L2 performance decreasing by as much as 17% as t is increased, delaying the onset of L2 relative to L1. However, the networks in the unit growth condition were largely able to mitigate this performance decrease by introducing new units and connections during learning. Performance of networks in the connection growth condition fall between these two. The addition of new connections to the networks appears to successfully stave off entrenchment effects when the level of entrenchment is small, but for values of t > 200,000 the entrenchment effects again start to become apparent. This tells us that addition of new units and new connections both help to counteract deficits due to entrenchment. Viewed from the cognitive perspective, growth in long-term memory capacity – in the connection growth condition – during training helped to mitigate the effects of L1 entrenchment, as did growth in working memory capacity, as evidenced by the superior performance of the unit growth condition over the connection growth condition for higher values of t. However, as shown by the unit replacement networks again tending to track the performance of the no growth networks, the addition of fresh neural resources is not all that is required to reap a performance benefit. Instead, it seems that starting small, either in terms of working memory capacity or long-term memory capacity, or both, is an essential factor that, combined with growth of neural resources, leads to the performance increase.