Artificial intelligence has recently been seen as successful in a number of domains, such a playing chess or Go, recognising handwritten characters, or describing visual scenes in natural language. Lake et al. discuss these kinds of breakthroughs as a big step for artificial intelligence, but raise the question how we can build machines that learn like people? We can find an indication in a survey of mind perception (Gray et al. Reference Gray, Gray and Wegner2007), which is the “amount of mind” people are willing to attribute to others. Participants judged machines to be high on agency but low on experience. We attribute this to the fact that computers are trained on individual tasks, often involving a single modality such as vision or speech, or a single context such as classifying traffic signs, as opposed to interpreting spoken and gestured utterances. In contrast, for people, the “world” essentially appears as a multimodal stream of stimuli, which unfold over time. Therefore, we suggest that the next paradigm shift in intelligent machines will have to include processing the “world” through lifelong and crossmodal learning. This is important because people develop problem-solving capabilities, including language processing, over their life span and via interaction with the environment and other people (Elman Reference Elman1993, Christiansen and Chater Reference Christiansen and Chater2016). In addition, the learning is embodied, as developing infants have a body-rational view of the world, but also seem to apply general problem-solving strategies to a wide range of quite different tasks (Cangelosi and Schlesinger Reference Cangelosi and Schlesinger2015).
Hence, we argue that the proposed principles or “start-up software” are coupled tightly with general learning mechanisms in the brain. We argue that these conditions inherently enable the development of distributed representations of knowledge. For example, in our research, we found that architectural mechanisms, like different timings in the information processing in the cortex, foster compositionality that in turn enables both the development of more complex body actions and the development of language competence from primitives (Heinrich Reference Heinrich2016). These kinds of distributed representations are coherent with the cognitive science on embodied cognition. Lakoff and Johnson (Reference Lakoff and Johnson2003), for example, argue that people describe personal relationships in terms of the physical sensation of temperature. The transfer from one domain to the other is plausible, as an embrace or handshake between friends or family members, for example, will cause a warm sensation for the participants. These kinds of temperature exchanging actions are supposed to be signs of people's positive feelings towards each other (Hall Reference Hall1966). The connection between temperature sensation and social relatedness is argued to reflect neural “bindings” (Gallese and Lakoff Reference Gallese and Lakoff2005). The domain knowledge that is used later in life can be derived from the primitives that are encountered early in childhood, for example, in interactions between infants and parents, and is referred to as intermodal synchrony (Rohlfing and Nomikou Reference Rohlfing and Nomikou2014). As a further example, our own research shows that learning, which is based on crossmodal integration, like the integration of real sensory perception on low and on intermediate levels (as suggested for the superior colliculus in the brain), can enable both super-additivity and dominance of certain modalities based on the tasks (Bauer et al. Reference Bauer, Dávila-Chacón and Wermter2015).
In developing machines, approaches such as transfer learning and zero-shot learning are receiving increasing attention, but are often restricted to transfers from domain to domain or from modality to modality. In the domain case, this can take the form of a horizontal transfer, in which a concept in one domain is learned and then transferred to another domain within the same modality. For example, it is possible to learn about affect in speech and to transfer that model of affect to music (Coutinho et al. Reference Coutinho, Deng and Schuller2014). In the modality case, one can vertically transfer concepts from one modality to another. This could be a learning process in which language knowledge is transferred to the visual domain (Laptev Reference Laptev, Marszalek, Schmid and Rozenfeld2008; Donahue Reference Donahue, Hendricks, Guadarrama, Rohrbach, Venugopalan, Saenko and Darrell2015). However, based on the previous crossmodal integration in people, we must look into combinations of both, such that transferring between domains is not merely switching between two modalities, but integrating into both. Therefore, machines must exploit the representations that form when integrating multiple modalities that are richer than the sum of the parts. Recent initial examples include (1) understanding continuous counting expressed in spoken numbers from learned spatial differences in gestural motor grounding (Ruciński Reference Ruciński2014) and (2) classifying affective states' audiovisual emotion expressions via music, speech, facial expressions, and motion (Barros and Wermter Reference Barros and Wermter2016).
Freeing learning from modalities and domains in favour of distributed representations, and reusing learned representations in the next individual learning tasks, will enable a larger view of learning to learn. Having underlying hybrid neural embodied architectures (Wermter et al. Reference Wermter, Palm, Weber, Elshaw, Wermter, Palm and Elshaw2005) will support horizontal and vertical transfer and integration. This is the “true experience” machines need to learn and think like people. All in all, Lake et al. stress the important point of grounding learning in general principles of embodied perception and social cognition. Yet, we suggest it is still necessary to go a step further and consider lifelong learning, which includes developmental learning, focused on embodiment as applied in developmental robotics and neurorobotics, and crossmodal learning, which facilitates the integration of multiple senses.
Artificial intelligence has recently been seen as successful in a number of domains, such a playing chess or Go, recognising handwritten characters, or describing visual scenes in natural language. Lake et al. discuss these kinds of breakthroughs as a big step for artificial intelligence, but raise the question how we can build machines that learn like people? We can find an indication in a survey of mind perception (Gray et al. Reference Gray, Gray and Wegner2007), which is the “amount of mind” people are willing to attribute to others. Participants judged machines to be high on agency but low on experience. We attribute this to the fact that computers are trained on individual tasks, often involving a single modality such as vision or speech, or a single context such as classifying traffic signs, as opposed to interpreting spoken and gestured utterances. In contrast, for people, the “world” essentially appears as a multimodal stream of stimuli, which unfold over time. Therefore, we suggest that the next paradigm shift in intelligent machines will have to include processing the “world” through lifelong and crossmodal learning. This is important because people develop problem-solving capabilities, including language processing, over their life span and via interaction with the environment and other people (Elman Reference Elman1993, Christiansen and Chater Reference Christiansen and Chater2016). In addition, the learning is embodied, as developing infants have a body-rational view of the world, but also seem to apply general problem-solving strategies to a wide range of quite different tasks (Cangelosi and Schlesinger Reference Cangelosi and Schlesinger2015).
Hence, we argue that the proposed principles or “start-up software” are coupled tightly with general learning mechanisms in the brain. We argue that these conditions inherently enable the development of distributed representations of knowledge. For example, in our research, we found that architectural mechanisms, like different timings in the information processing in the cortex, foster compositionality that in turn enables both the development of more complex body actions and the development of language competence from primitives (Heinrich Reference Heinrich2016). These kinds of distributed representations are coherent with the cognitive science on embodied cognition. Lakoff and Johnson (Reference Lakoff and Johnson2003), for example, argue that people describe personal relationships in terms of the physical sensation of temperature. The transfer from one domain to the other is plausible, as an embrace or handshake between friends or family members, for example, will cause a warm sensation for the participants. These kinds of temperature exchanging actions are supposed to be signs of people's positive feelings towards each other (Hall Reference Hall1966). The connection between temperature sensation and social relatedness is argued to reflect neural “bindings” (Gallese and Lakoff Reference Gallese and Lakoff2005). The domain knowledge that is used later in life can be derived from the primitives that are encountered early in childhood, for example, in interactions between infants and parents, and is referred to as intermodal synchrony (Rohlfing and Nomikou Reference Rohlfing and Nomikou2014). As a further example, our own research shows that learning, which is based on crossmodal integration, like the integration of real sensory perception on low and on intermediate levels (as suggested for the superior colliculus in the brain), can enable both super-additivity and dominance of certain modalities based on the tasks (Bauer et al. Reference Bauer, Dávila-Chacón and Wermter2015).
In developing machines, approaches such as transfer learning and zero-shot learning are receiving increasing attention, but are often restricted to transfers from domain to domain or from modality to modality. In the domain case, this can take the form of a horizontal transfer, in which a concept in one domain is learned and then transferred to another domain within the same modality. For example, it is possible to learn about affect in speech and to transfer that model of affect to music (Coutinho et al. Reference Coutinho, Deng and Schuller2014). In the modality case, one can vertically transfer concepts from one modality to another. This could be a learning process in which language knowledge is transferred to the visual domain (Laptev Reference Laptev, Marszalek, Schmid and Rozenfeld2008; Donahue Reference Donahue, Hendricks, Guadarrama, Rohrbach, Venugopalan, Saenko and Darrell2015). However, based on the previous crossmodal integration in people, we must look into combinations of both, such that transferring between domains is not merely switching between two modalities, but integrating into both. Therefore, machines must exploit the representations that form when integrating multiple modalities that are richer than the sum of the parts. Recent initial examples include (1) understanding continuous counting expressed in spoken numbers from learned spatial differences in gestural motor grounding (Ruciński Reference Ruciński2014) and (2) classifying affective states' audiovisual emotion expressions via music, speech, facial expressions, and motion (Barros and Wermter Reference Barros and Wermter2016).
Freeing learning from modalities and domains in favour of distributed representations, and reusing learned representations in the next individual learning tasks, will enable a larger view of learning to learn. Having underlying hybrid neural embodied architectures (Wermter et al. Reference Wermter, Palm, Weber, Elshaw, Wermter, Palm and Elshaw2005) will support horizontal and vertical transfer and integration. This is the “true experience” machines need to learn and think like people. All in all, Lake et al. stress the important point of grounding learning in general principles of embodied perception and social cognition. Yet, we suggest it is still necessary to go a step further and consider lifelong learning, which includes developmental learning, focused on embodiment as applied in developmental robotics and neurorobotics, and crossmodal learning, which facilitates the integration of multiple senses.