Meaningful interactions between linguistic units occur on many timescales. After listening to 10 minutes of a typical English narrative, a listener will have heard ~1,000 words composing ~100 sentences grouped into ~25 paragraphs. When the 1,001st word in the narrative arrives, it enters a rich syntactic and semantic context that spans multiple timescales and levels of abstraction. Christiansen & Chater (C&C) rightfully emphasize the constraints imposed by the rapidity of language input. Here we highlight the importance of a related class of constraints: those imposed by the need to integrate incoming information with prior information over multiple timescales.
C&C motivate the “Now-or-Never bottleneck” with the observations that “memory is fleeting” and “new material rapidly obliterates previous material” (abstract). These statements tend to hold true in low-level auditory masking (Elliott Reference Elliott1962) and in short-term memory experiments involving unrelated auditory items (Warren et al. Reference Warren, Obusek, Farmer and Warren1969). However, in real-life language processing, memory cannot all be fleeting. This is because new stimuli have a prior relationship to, and must be actively integrated with, the stimuli that were just encountered. Thus, in real-life contexts, previous material exerts a powerful influence on the processing of new material.
Consider the difference between hearing the sequence of words “friend-mulch-key” and hearing the sequence of words “friend-ship-pact.” In the first sequence, the representation of the word friend is degraded by interference with mulch and key. In the second sequence, by contrast, the word friend interacts meaningfully with ship and pact. This simple example reflects a general and ubiquitous phenomenon in real-life language: New material does not necessarily obliterate previous material. Instead, past and present information interact to produce understanding, and the memory of past events continually shapes the present (Nieuwland & Van Berkum Reference Nieuwland and Van Berkum2006).
It seems the processing bottleneck that C&C describe applies best to early processing areas (e.g., primary sensory cortex), where sensory traces may have a very short lifetime (<200 ms). At higher levels of the language hierarchy, however, neural circuits must retain a longer history of past input to enable the integration of information over time. Temporal integration is necessary for higher-order regions to support the understanding of a new word in relation to a prior sentence or a new sentence in relation to the larger discourse. We have found that temporal integration occurs over longer timescales in higher-order regions (Hasson et al. Reference Hasson, Yang, Vallines, Heeger and Rubin2008; Lerner et al. Reference Lerner, Honey, Silbert and Hasson2011), and that the intrinsic neural dynamics become slower across consecutive stages of the cortical hierarchy (Honey et al. Reference Honey, Thesen, Donner, Silbert, Carlson, Devinsky, Doyle, Rubin, Heeger and Hasson2012; Stephens et al. Reference Stephens, Honey and Hasson2013). Thus, the temporal bottleneck appears to gradually widen across the consecutive stages of the language processing hierarchy, as increasingly abstract linguistic structures are processed over longer timescales.
Influenced by the ideas of Macdonald and Christiansen (1996), as well as Fuster (Reference Fuster1997), concerning the memory that is intrinsic to ongoing information processing, and supported by recent single-unit, electrocorticography, and functional imaging data, we have developed a brain-based framework for such a functional organization (Hasson et al. Reference Hasson, Chen and Honey2015). In this framework, (a) virtually all cortical circuits can accumulate information over time, and (b) the timescale of accumulation varies hierarchically, from early sensory areas with short processing timescales (tens to hundreds of milliseconds) to higher-order areas with long processing timescales (many seconds to minutes). In this hierarchical systems perspective, memory is not restricted to a few localized stores and it is not transient; instead memory is intrinsic to information processing that unfolds throughout the brain on timescales from milliseconds to minutes. We have suggested the term “process memory” to refer to active traces of past information that are used by a local neural circuit to process incoming information in the present moment; this is in distinction to the more traditional notion of “working memory,” which is a more functionally encapsulated memory store.
Process memory may support the Chunk-and-Pass mechanism that C&C propose for organizing inter-regional information flow. As they note: “incremental processing in comprehension and production takes place in parallel across multiple levels of linguistic representation, each with a characteristic temporal window” (sect. 3.2, para. 5). In our view, the Now-or-Never bottleneck can be made compatible with contextual language processing by allowing the “Now” (i.e., the local circuit memory of prior events) to have a variable duration. For example, the “Now” could be understood to have a short (e.g., milliseconds) timescale in sensory areas, where representations are fleeting, and then to gradually expand in duration in higher-order areas, where chunking is required over longer (e.g., seconds) and longer (e.g., minutes) windows of input. Thus, the “Now” may be understood as a time window around the present moment, in which information can be integrated, and the duration of the “Now” may lengthen as one moves from sensory areas toward higher-order language circuits.
In summary, we share the vision of C&C in which language function arises from a chain of transformations across a hierarchy of circuits, and that language learning is a kind of “learning to process.” At the same time, we suggest that this hierarchical processing framework could be refined to account for the process memory that is intrinsic to language processing and is needed for comprehending incoming input within multiple timescales of prior context.
Meaningful interactions between linguistic units occur on many timescales. After listening to 10 minutes of a typical English narrative, a listener will have heard ~1,000 words composing ~100 sentences grouped into ~25 paragraphs. When the 1,001st word in the narrative arrives, it enters a rich syntactic and semantic context that spans multiple timescales and levels of abstraction. Christiansen & Chater (C&C) rightfully emphasize the constraints imposed by the rapidity of language input. Here we highlight the importance of a related class of constraints: those imposed by the need to integrate incoming information with prior information over multiple timescales.
C&C motivate the “Now-or-Never bottleneck” with the observations that “memory is fleeting” and “new material rapidly obliterates previous material” (abstract). These statements tend to hold true in low-level auditory masking (Elliott Reference Elliott1962) and in short-term memory experiments involving unrelated auditory items (Warren et al. Reference Warren, Obusek, Farmer and Warren1969). However, in real-life language processing, memory cannot all be fleeting. This is because new stimuli have a prior relationship to, and must be actively integrated with, the stimuli that were just encountered. Thus, in real-life contexts, previous material exerts a powerful influence on the processing of new material.
Consider the difference between hearing the sequence of words “friend-mulch-key” and hearing the sequence of words “friend-ship-pact.” In the first sequence, the representation of the word friend is degraded by interference with mulch and key. In the second sequence, by contrast, the word friend interacts meaningfully with ship and pact. This simple example reflects a general and ubiquitous phenomenon in real-life language: New material does not necessarily obliterate previous material. Instead, past and present information interact to produce understanding, and the memory of past events continually shapes the present (Nieuwland & Van Berkum Reference Nieuwland and Van Berkum2006).
It seems the processing bottleneck that C&C describe applies best to early processing areas (e.g., primary sensory cortex), where sensory traces may have a very short lifetime (<200 ms). At higher levels of the language hierarchy, however, neural circuits must retain a longer history of past input to enable the integration of information over time. Temporal integration is necessary for higher-order regions to support the understanding of a new word in relation to a prior sentence or a new sentence in relation to the larger discourse. We have found that temporal integration occurs over longer timescales in higher-order regions (Hasson et al. Reference Hasson, Yang, Vallines, Heeger and Rubin2008; Lerner et al. Reference Lerner, Honey, Silbert and Hasson2011), and that the intrinsic neural dynamics become slower across consecutive stages of the cortical hierarchy (Honey et al. Reference Honey, Thesen, Donner, Silbert, Carlson, Devinsky, Doyle, Rubin, Heeger and Hasson2012; Stephens et al. Reference Stephens, Honey and Hasson2013). Thus, the temporal bottleneck appears to gradually widen across the consecutive stages of the language processing hierarchy, as increasingly abstract linguistic structures are processed over longer timescales.
Influenced by the ideas of Macdonald and Christiansen (1996), as well as Fuster (Reference Fuster1997), concerning the memory that is intrinsic to ongoing information processing, and supported by recent single-unit, electrocorticography, and functional imaging data, we have developed a brain-based framework for such a functional organization (Hasson et al. Reference Hasson, Chen and Honey2015). In this framework, (a) virtually all cortical circuits can accumulate information over time, and (b) the timescale of accumulation varies hierarchically, from early sensory areas with short processing timescales (tens to hundreds of milliseconds) to higher-order areas with long processing timescales (many seconds to minutes). In this hierarchical systems perspective, memory is not restricted to a few localized stores and it is not transient; instead memory is intrinsic to information processing that unfolds throughout the brain on timescales from milliseconds to minutes. We have suggested the term “process memory” to refer to active traces of past information that are used by a local neural circuit to process incoming information in the present moment; this is in distinction to the more traditional notion of “working memory,” which is a more functionally encapsulated memory store.
Process memory may support the Chunk-and-Pass mechanism that C&C propose for organizing inter-regional information flow. As they note: “incremental processing in comprehension and production takes place in parallel across multiple levels of linguistic representation, each with a characteristic temporal window” (sect. 3.2, para. 5). In our view, the Now-or-Never bottleneck can be made compatible with contextual language processing by allowing the “Now” (i.e., the local circuit memory of prior events) to have a variable duration. For example, the “Now” could be understood to have a short (e.g., milliseconds) timescale in sensory areas, where representations are fleeting, and then to gradually expand in duration in higher-order areas, where chunking is required over longer (e.g., seconds) and longer (e.g., minutes) windows of input. Thus, the “Now” may be understood as a time window around the present moment, in which information can be integrated, and the duration of the “Now” may lengthen as one moves from sensory areas toward higher-order language circuits.
In summary, we share the vision of C&C in which language function arises from a chain of transformations across a hierarchy of circuits, and that language learning is a kind of “learning to process.” At the same time, we suggest that this hierarchical processing framework could be refined to account for the process memory that is intrinsic to language processing and is needed for comprehending incoming input within multiple timescales of prior context.