R1. Summary

We were pleased to see so many thoughtful commentaries and critiques in response to our target article. The project of “building machines that learn and think like people” will require input and insight from a broad range of disciplines, and it was encouraging that we received responses from experts in artificial intelligence (AI), machine learning, cognitive psychology, cognitive development, social psychology, philosophy, robotics, and neuroscience. As to be expected, there were many differences in perspective and approach, but before turning to those disagreements we think it is worth starting with several main points of agreement.

First, we were encouraged to see broad enthusiasm for the general enterprise and the opportunities it would bring. Like us, many researchers have been inspired by recent AI advances to seek a better computational understanding of human intelligence, and see this project's potential for driving new breakthroughs in building more human-like intelligence in machines. There were notable exceptions: A few respondents focused more on the potential risks and harms of this effort, or questioned its whole foundations or motivations. We return to these issues at the end of this response.

Most commenters also agreed that despite rapid progress in AI technologies over the last few years, machine systems are still not close to achieving human-like learning and thought. It is not merely a matter of scaling up current systems with more processors and bigger data sets. Fundamental ingredients of human cognition are missing, and fundamental innovations must be made to incorporate these ingredients into any kind of general-purpose, human-like AI.

Our target article articulated one vision for making progress toward this goal. We argued that human-like intelligence will come from machines that build models of the world – models that support explanation and understanding, prediction and planning, and flexible generalization for an open-ended array of new tasks – rather than machines that merely perform pattern recognition to optimize performance in a previously specified task or set of tasks.

We outlined a set of key cognitive ingredients that could support this approach, which are missing from many current AI systems (especially those based on deep learning), but could add great value: the “developmental start-up software” of intuitive physics and intuitive psychology, and mechanisms for rapid model learning based on the principles of compositionality, causality, and learning-to-learn (along with complementary mechanisms for efficient inference and planning with these models). We were gratified to read that many commentators found these suggested cognitive ingredients useful: “We agree … on their list of ‘key ingredients’ for building human-like intelligence” (Botvinick, Barrett, Battaglia, de Freitas, Kumaran, Leibo, Lillicrap, Modayil, Mohamed, Rabinowitz, Rezende, Santoro, Schaul, Summerfield, Wayne, Weber, Wierstra, Legg, & Hassabis [Botvinick et al.], abstract): “We entirely agree with the central thrust of the article” (Davis & Marcus, para. 1): “Causality, compositionality, and learning-to-learn … are central for human learning” (Tessler, Goodman, & Frank [Tessler et al.], para. 1): “Their ideas of ‘start-up software’ and tools for rapid model learning … help pin-point the sources of general, flexible intelligence” (Dennet & Lambert, para. 1).

This is not to say that there was universal agreement about our suggested ingredients. Our list was carefully chosen but not meant to be complete, and many commentators offered additional suggestions: emotion (Clark; Güss & Dörner), embodiment and action (Baldassarre, Santucci, Cartoni, & Caligiore; [Baldassarre et al.]; MacLennan; Marin & Mostafaoui; Oudeyer; Wermter, Griffiths, & Heinrich [Wermter et al.]), social and cultural learning (Clegg & Corriveau; Dennett & Lambert; Tessler et al.; Marin & Mostafaoui), and open-ended learning through intrinsic motivation (Baldassarre et al.; Güss & Dörner; Oudeyer; Wermter et al.). We appreciate these suggested additions, which help paint a richer and more complete picture of the mind and the ingredients of human intelligence. We discuss prospects for incorporating them into human-like AI systems in Section 5.

The main dimensions of disagreement in the commentaries revolved around how to implement our suggested ingredients in building AI: To what extent should they be explicitly built in, versus expected to emerge? What is their real content? How integrated or fragmented is the mind's internal structure? And what form do they take? How are these capacities represented in the mind or instantiated in the brain, and what kinds of algorithms or data structures should we be looking to in building an AI system?

Perhaps, unsurprisingly, these dimensions tended to align with classic debates in cognitive science and AI, and we found ourselves being critiqued from all sides. The first dimension is essentially the nature versus nurture debate (Section 2), and we were charged with advocating both for too much nature (Botvinick et al.;Clegg & Corriveau;Cooper) and too little (Spelke & Blass). The second dimension relates to whether human mental models are better characterized in terms of coherent theories versus theory fragments (Section 3): We were criticized for positing theory-forming systems that were too strong (Chater & Oaksford;Davis & Marcus;Livesey, Goldwater, & Colagiuri [Livesey et al.]), but also too weak (Dennett & Lambert). The third dimension concerns symbolic versus sub-symbolic representations (Section 4): To some commenters our proposal felt too allied with symbolic cognitive architectures (Çağlar & Hanson;Hansen, Lampinen, Suri, & McClelland [Hansen et al.];MacLennan). To others, we did not embrace symbols deeply enough (Forbus & Gentner).

Some who saw our article through the lens of these classic debates, experienced a troubling sense of déjà vu. It was “Back to the Future: The Return of Cognitive Functionalism” for Çağlar & Hanson. For Cooper, it appeared “that cognitive science has advanced little in the last 30 years with respect to the underlying debates.” We felt differently. We took this broad spectrum of reactions from commentators (who also, by and large, felt they agreed with our main points), as a sign that our field collectively might be looking to break out from these debates – to move in new directions that are not so easily classified as just more of the same. It is understandable that many commentators would see our argument through the lens of these well-known and entrenched lines of argument, perhaps because we, as individuals, have contributed to them in previous publications. However, we wrote this target article, in part, because we felt it was vital to redefine this decades-long discussion in light of the recent progress in AI and machine learning.

Recent AI successes, on the one hand, make us optimistic about the project of building machines that learn and think like people. Working toward this goal seems much more plausible to many people than it did just a few years ago. At the same time, recent AI successes, when viewed from the perspective of a cognitive scientist, also highlight the gaps between machine and human intelligence. Our target article begins from this contrast: Whereas the driving force behind most of today's machine learning systems is sophisticated pattern recognition, scaled up to increasingly large and diverse data sets, the most impressive feats of human learning are better understood in terms of model building, often with much more limited data. We take the goal of building machines that can build models of the world as richly, as flexibly, and as quickly as humans can, as a worthy target for the next phase of AI research. Our target article lays out some of the key ingredients of human cognition that could serve as a basis for making progress toward that goal.

We explicitly tried to avoid framing these suggestions in terms of classic lines of argument that neural network researchers and other cognitive scientists have engaged in, to encourage more building and less arguing. With regards to nature versus nurture (sect. 2 of this article), we tried our best to describe these ingredients in a way that was “agnostic with regards to [their] origins” (target article, sect. 4, para. 2), but instead focused on their engineering value. We made this choice, not because we do not have views on the matter, but because we see the role of the ingredients as more important than their origins, for the next phase of AI research and the dialog between scientists and engineers. Whether learned, innate, or enriched, the fact that these ingredients are active so early in development, is a signal of their importance. They are present long before a person learns a new handwritten letter in the Character Challenge, or learns to play a new video game in the Frostbite Challenge (target article, sects. 3.1 and 3.2). AI systems could similarly benefit from utilizing these ingredients. With regards to symbolic versus sub-symbolic modeling (sect. 4 of this article), we think the ingredients could take either form, and they could potentially be added to symbolic architectures, sub-symbolic architectures, or hybrid architectures that transcend the dichotomy. Similarly, the model-building activities we describe could potentially be implemented in a diverse range of architectures, including deep learning. Regardless of implementation, demonstrations such as the Characters and Frostbite challenges show that people can rapidly build models of the world, and then flexibly reconfigure these models for new tasks without having to retrain. We see this as an ambitious target for AI that can be pursued in a variety of ways, and will have many practical applications (target article, sect. 6.2).

The rest of our response is organized as follows: The next three sections cover in detail the main dimensions of debate regarding the origin and structure of our ingredients: nature versus nurture (sect. 2), coherent theories versus theory fragments (sect. 3), and symbolic versus sub-symbolic representations (sect. 4). Additional ingredients suggested by the commentators are covered in Section 5. We discuss insights from neuroscience and the brain in Section 6. We end by discussing the societal risks and benefits of building machines that learn and think like people, in the light of the ethical issues raised by some commentators (sect. 7).

R2. Nature versus nurture

As mentioned, our target article did not intend to take a strong stance on “nature versus nurture” or “designing versus learning” for how our proposed ingredients should come to be incorporated into more human-like AI systems. We believe this question is important, but we placed our focus elsewhere in the target article. The main thesis is that a set of ingredients – each with deep roots in cognitive science – would be powerful additions to AI systems in whichever way a researcher chooses to include them. Whether the ingredients are learned, built in, or enriched through learning, we see them as a primary goal to strive for when building the next generation of AI systems. There are multiple possible paths for developing AI systems with these ingredients, and we expect individual researchers will vary in the paths they choose for pursuing these goals.

Understandably, many of the commentators linked their views on the biological origin of our cognitive principles to their strategy for developing AI systems with these principles. In contrast to the target article and its agnostic stance, some commentators took a stronger nativist stance, arguing that aspects of intuitive physics, intuitive psychology, and causality are innate, and it would be valuable to develop AI systems that “begin with human core knowledge” (Spelke & Blass, para. 4). Other commentators took a stronger nurture stance, arguing that the goal should be to learn these core ingredients rather than build systems that start with them (Botvinick et al.; Cooper). Relatedly, many commentators pointed out additional nurture-based factors that are important for human-like learning, such as social and cultural forms of learning (Clegg & Corriveau; Dennet & Lambert; Marin & Mostafaoui; Tessler et al.). In the section that follows, we respond to the different suggestions regarding the origin of the key ingredients, leaving the discussion of additional ingredients, such as social learning, for Section 5.

The response from researchers at Google DeepMind (Botvinick et al.) is of particular interest because our target article draws on aspects of their recent work. We offered their work as examples of recent accomplishments in AI (e.g., Graves et al. Reference Graves, Wayne, Reynolds, Harley, Danihelka, Grabska-Barwińska, Colmenarejo, Grefenstette, Ramalho, Agapiou, Badia, Hermann, Zwols, Ostrovski, Cain, King, Summerfield, Blunsom, Kayukcuoglu and Hassabis2016; Mnih et al. Reference Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik, Antonoglous, King, Kumaran, Wierstra and Hassabis2015; Silver et al. Reference Silver, Huang, Maddison, Guez, Sifre, Driessche, Schrittwieser, Antonoglou, Panneershelvam, Lanctot, Dieleman, Grewe, Nham, Kalchbrenner, Sutskever, Lillicrap, Leach, Kavukcuoglu, Graepel and Hassabis2016). At the same time, we highlighted ways that their systems do not learn or think like people (e.g., the Frostbite Challenge), but could potentially be improved by aiming for this target and by incorporating additional cognitive ingredients. Botvinick et al.'s response suggests that there are substantial areas of agreement. In particular, they see the five principles as “a powerful set of target goals for AI research” (para. 1), suggesting similar visions of what future accomplishments in AI will look like, and what the required building blocks are for getting there.

Botvinick et al. strongly emphasized an additional principle: Machines should learn for themselves with minimal hand engineering from their human designers. We agree this is a valuable principle to guide researchers seeking to build learning-based general AI systems, as DeepMind aims to. To the extent that this principle is related to our principle of “learning-to-learn,” we also endorse it in building machines that learn and think like people. Children are born capable of learning for themselves everything they will ultimately learn, without the need for an engineer to tweak their representations or algorithms along the way. However, it is not clear that the goals of building general AI systems and building machines that learn like people always converge, and the best design approach might be correspondingly different. Human beings (and other animals) may be born genetically programmed with mechanisms that effectively amount to highly engineered cognitive representations or algorithms – mechanisms that enable their subsequent learning and learning-to-learn abilities. Some AI designers may want to emulate this approach, whereas others may not.

The differences between our views may also reflect a difference in how we prioritize a set of shared principles and how much power we attribute to learning-to-learn mechanisms. Botvinick et al. suggest – but do not state explicitly – that they prioritize learning with minimal engineering above the other principles (and, thus, maximize the role of learning-to-learn). Under this strategy, the goal is to develop systems with our other key ingredients (compositionality, causality, intuitive physics, and intuitive psychology), insofar as they can be learned from scratch without engineering them. In the short term, this approach rests heavily on the power of learning-to-learn mechanisms to construct these other aspects of an intelligent system. In cases where this strategy is not feasible, Botvinick et al. state their approach also licenses them to build in ingredients too, but (we assume) with a strong preference for learning the ingredients wherever possible.

Although these distinctions may seem subtle, they can have important consequences for research strategy and outcome. Compare DeepMind's work on the Deep Q-Network (Mnih et al. Reference Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik, Antonoglous, King, Kumaran, Wierstra and Hassabis2015) to the theory learning approach our target article advocated for tackling the Frostbite Challenge, or their work on one-shot learning in deep neural networks (Rezende et al. Reference Rezende, Mohamed, Danihelka, Gregor and Wierstra2016; Santoro et al. Reference Santoro, Bartunov, Botvinick, Wierstra and Lillicrap2016; Vinyals et al. Reference Vinyals, Blundell, Lillicrap, Wierstra, Lee, Sugiyama, Luxburg, Guyon and Garnett2016) and our work on Bayesian Program Learning (Lake et al. Reference Lake, Salakhutdinov and Tenenbaum2015a). DeepMind's approaches to these problems clearly learn with less initial structure than we advocate for, and also clearly have yet to approach the speed, flexibility, and richness of human learning, even in these constrained domains.

We sympathize with DeepMind's goals and believe their approach should be pursued vigorously, along with related suggestions by Cooper and Hansen et al. However, we are not sure how realistic it is to pursue all of our key cognitive ingredients as emergent phenomena (see related discussion in sect. 5 of the target article), using the learning-to-learn mechanisms currently on offer in the deep learning landscape. Genuine intuitive physics, intuitive psychology, and compositionality, are unlikely to emerge from gradient-based learning in a relatively generic neural network. Instead, a far more expensive evolutionary-style search over discrete architectural variants may be required (e.g., Real et al. Reference Real, Moore, Selle, Saxena, Suematsu, Le and Kurakin2017; Stanley & Miikkulainen Reference Stanley and Miikkulainen2002). This approach may be characterized as “building machines that evolve to learn and think like people,” in that such an extensive search would presumably include aspects of both phylogeny and ontogeny. As discussed in Section 4.1 of the target article, children have a foundational understanding of physics (objects, substances, and their dynamics) and psychology (agents and their goals) early in development. Whether innate, enriched, or rapidly learned, it seems unlikely that these ingredients arise purely in ontogeny from an extensive structural search over a large space of cognitive architectures, with no initial bias toward building these kinds of structures. In contrast, our preferred approach is to explore both powerful learning algorithms and starting ingredients together.

Over the last decade, this approach has led us to the key ingredients that are the topic of the target article (e.g., Baker et al. Reference Baker, Saxe and Tenenbaum2009; Reference Baker, Jara-Ettinger, Saxe and Tenenbaum2017; Battaglia et al. Reference Battaglia, Hamrick and Tenenbaum2013; Goodman et al. Reference Goodman, Tenenbaum, Feldman and Griffiths2008; Kemp et al. Reference Kemp, Perfors and Tenenbaum2007; Lake et al. Reference Lake, Salakhutdinov and Tenenbaum2015a; Ullman et al. Reference Ullman, Baker, Macindoe, Evans, Goodman, Tenenbaum, Bengio, Schuumans, Lafferty, Williams and Culotta2009); we did not start with these principles as dogma. After discovering which representations, learning algorithms, and inference mechanisms appear especially powerful in combination with each other, it is easier to investigate their origins and generalize them so they apply more broadly. Examples of this strategy from our work include the grammar-based framework for discovering structural forms in data (Kemp & Tenenbaum Reference Kemp and Tenenbaum2008), and a more emergent approach for implicitly learning some of the same forms (Lake et al. Reference Lake, Lawrence and Tenenbaum2016), as well as models of causal reasoning and learning built on the theory of causal Bayesian networks (Goodman et al. Reference Goodman, Ullman and Tenenbaum2011; Griffiths & Tenenbaum Reference Griffiths and Tenenbaum2005, Reference Griffiths and Tenenbaum2009). This strategy has allowed us to initially consider a wider spectrum of models, without a priori rejecting those that do not learn everything from scratch. Once an ingredient is established as important, it provides important guidance for additional research on how it might be learned.

We have pursued this strategy primarily through structured probabilistic modeling, but we believe it can be fruitfully pursued using neural networks as well. As Botvinick et al. point out, this strategy would not feel out of place in contemporary deep learning research. Convolutional neural networks build in a form of translation invariance that proved to be highly useful for object recognition (Krizhevsky et al. Reference Krizhevsky, Sutskever, Hinton, Pereira, Burges, Bottou and Weinberger2012; LeCun et al. Reference LeCun, Boser, Denker, Henderson, Howard, Hubbard and Jackel1989), and more recent work has explored building various forms of compositionality into neural networks (e.g., Eslami et al. Reference Eslami, Heess, Weber, Tassa, Kavukcuoglu, Hinton, Lee, Sugiyama, Luxburg, Guyon and Garnett2016; Reed & de Freitas Reference Reed and de Freitas2016). Increasingly, we are seeing more examples of integrating neural networks with lower-level building blocks from classic psychology and computer science (see sect. 6 of target article): selective attention (Bahdanau et al. Reference Bahdanau, Cho and Bengio2015; Mnih et al. Reference Mnih, Heess, Graves, Kavukcuoglu, Ghahramani, Welling, Cortes, Lawrence and Weinberger2014; Xu et al. Reference Xu, Ba, Kiros, Cho, Courville, Salakhutdinov, Zemel and Bengio2015), augmented working memory (Graves et al. Reference Graves, Wayne and Danihelka2014; Grefenstette et al. Reference Grefenstette, Hermann, Suleyman, Blunsom, Cortes, Lawrence, Lee, Sugiyama and Garnett2015; Sukhbaatar et al. Reference Sukhbaatar, Szlam, Weston, Fergus, Cortes, Lawrence, Lee, Sugiyama and Garnett2015; Weston et al. Reference Weston, Chopra and Bordes2015b), and experience replay (McClelland et al. Reference McClelland, McNaughton and O'Reilly1995; Mnih et al. Reference Mnih, Kavukcuoglu, Silver, Rusu, Veness, Bellemare, Graves, Riedmiller, Fidjeland, Ostrovski, Petersen, Beattie, Sadik, Antonoglous, King, Kumaran, Wierstra and Hassabis2015). AlphaGo has an explicit model of the game of Go and builds in a wide range of high level and game-specific features, including how many stones a move captures, how many turns since a move was played, the number of liberties, and whether a ladder will be successful or not (Silver et al. Reference Silver, Huang, Maddison, Guez, Sifre, Driessche, Schrittwieser, Antonoglou, Panneershelvam, Lanctot, Dieleman, Grewe, Nham, Kalchbrenner, Sutskever, Lillicrap, Leach, Kavukcuoglu, Graepel and Hassabis2016). If researchers are willing to include these types of representations and ingredients, we hope they will also consider our higher level cognitive ingredients.

It is easy to miss fruitful alternative representations by considering only models with minimal assumptions, especially in cases where the principles and representations have strong empirical backing (as is the case with our suggested principles). In fact, Botvinick et al. acknowledge that intuitive physics and psychology may be exceptions to their general philosophy, and could be usefully built in, given their breadth of empirical support. We were gratified to see this, and we hope it is clear to them and like-minded AI researchers that our recommendations are to consider building in only a relatively small set of core ingredients that have this level of support and scope. Moreover, a purely tabula rasa strategy can lead to models that require unrealistic amounts of training experience, and then struggle to generalize flexibly to new tasks without retraining. We believe that has been the case so far for deep learning approaches to the Characters and Frostbite challenges.

R3. Coherent theories versus theory fragments

Beyond the question of where our core ingredients come from, there is the question of their content and structure. In our article, we argued for theory-like systems of knowledge and causally structured representations, in particular (but not limited to) early-emerging intuitive physics and intuitive psychology. This view builds on extensive empirical research showing how young infants organize the world according to general principles that allow them to generalize across varied scenarios (Spelke Reference Spelke, Gentner and Goldin-Meadow2003; Spelke & Kinzler Reference Spelke and Kinzler2007), and on theoretical and empirical research applied to children and adults that sees human knowledge in different domains as explained by theory-like structures (Carey Reference Carey2009; Gopnik et al. Reference Gopnik, Glymour, Sobel, Schulz, Kushnir and Danks2004; Murphy & Medin Reference Murphy and Medin1985; Schulz Reference Schulz2012b; Wellman & Gelman Reference Wellman and Gelman1992; Reference Wellman, Gelman, Damon and Damon1998).

Commentators were split over how rich and how theory-like (or causal) these representations really are in the human mind and what that implies for building human-like AI. Dennett & Lambert see our view of theories as too limited – useful for describing cognitive processes shared with animals, but falling short of many distinctively human ways of learning and thought. On the other hand, several commentators saw our proposal as too rich for much of human knowledge. Chater & Oaksford argue by analogy to case-law, that “knowledge has the form of a loosely inter-linked history of reusable fragments” (para. 6) rather than a coherent framework. They stress that mental models are often shallow, and Livesey et al. add that people's causal models are not only shallow, but also often wrong, and resistant to change (such as the belief that vaccines cause autism). Davis & Marcus similarly suggest that the models we propose for the core ingredients are too causal, too complete, and too narrow to capture all of cognitive reasoning: Telling cats from dogs does not require understanding their underlying biological generative process; telling that a tower will fall does not require specifying in detail all of the forces and masses at play along the trajectory; and telling that someone is going to call someone does not require understanding whom they are calling or why.

In our target article, although we emphasized a view of cognition as model building, we also argued that pattern recognition can be valuable and even essential – in particular, for enabling efficient inference, prediction, and learning in rich causal theories. We suggested that different behaviors might be best explained by one, or the other, or both. For example, identifying the presence in a scene of an object that we call a “fridge” may indeed be driven by pattern recognition. But representing that object as a heavy, rigid, inanimate entity, and the corresponding predictions and plans that representation allows, is likely driven by more general abstract knowledge about physics and objects, whose core elements are not tied down to extensive patterns of experience with particular categories of objects. We could just as well form this representation upon our first encounter with a fridge, without knowing what it is called or knowing anything about the category of artifacts that it is one instance of.

On the issue of rich versus shallow representations, whereas intuitive theories of physics and psychology may be rich in the range of generalizable inferences they support, these and other intuitive theories are shallow in another sense; they are far more shallow than the type of formal theories scientists aim to develop, at the level of base reality and mechanism. From the point of view of a physicist, a game engine representation of a tower of blocks falling down is definitely not, as Davis & Marcus describe it, a “physically precise description of the situation” (para. 4). A game engine representation is a simplification of the physical world; it does not go down to the molecular or atomic level, and it does not give predictions at the level of a nanosecond. It represents objects with simplified bounding shapes, and it can give coarse predictions for coarse time-steps. Also, although real physics engines are useful analogies for a mental representation, they are not one and the same, and finding the level of granularity of the mental physics engine (if it exists) is an empirical question. To the point about intuitive psychology, theories that support reasoning about agents and goals do not need to specify all of the moving mental or neural parts involved in planning, to make useful predictions and explanations about what an agent might do in a given situation.

Returning to the need for multiple types of models, and to the example of the fridge, Chater & Oaksford point to a significant type of reasoning not captured by either recognizing an image of a fridge or reasoning about its physical behavior as a heavy, inert object. Rather, they consider the shallow and sketchy understanding of how a fridge stays cold. Chater & Oaksford use such examples to reason that, in general, reasoning is done by reference to examplars. They place stored, fragmented exemplars in the stead of wide-scope and deep theories. However, we suggest that even the shallow understanding of the operation of a fridge may best be phrased in the language of a causal, generative model, albeit a shallow or incomplete one. That is, even in cases in which we make use of previously stored examples, these examples are probably best represented by a causal structure, rather than by external or superficial features. To use Chater & Oaksford's analogy, deciding which precedent holds in a new case relies on the nature of the offense and the constraining circumstances, not the surname of the plaintiff. In the same way that two letters are considered similar not because of a pixel-difference measure, but because of the similar strokes that created them, exemplar-based reasoning would rely on the structural similarity of causal models of a new example and stored fragments (Medin & Ortony Reference Medin, Ortony, Vosniadou and Ortony1989). An interesting hypothesis is that shallow causal models or mini-theories could be filling their gaps with more general, data-driven statistical machinery, such as a causal model with some of its latent variables generated by neural networks. Another possibility is that some mini-causal theories are generated ad hoc and on the fly (Schulz, Reference Schulz2012a), and so it should not be surprising that they are sometimes ill-specified and come into conflict with one another.

Unlike more general and early developing core theories such as intuitive physics and intuitive psychology, these mini-theory fragments may rely on later-developing language faculties (Carey Reference Carey2009). More generally, the early forms of core theories such as intuitive physics and psychology, may be, as Dennett & Lambert put it, “[bootstrapped] into reflective comprehension” (para. 3). Similar points have been made in the past by Carey (Reference Carey2009) and Spelke (Spelke Reference Spelke, Gentner and Goldin-Meadow2003; Spelke & Kinzler Reference Spelke and Kinzler2007), among others, regarding the role of later-developing language in using domain-specific and core knowledge concepts to extend intuitive theories as well as to build formal theories. The principles of core knowledge by themselves, are not meant to fully capture the formal or qualitative physical understanding of electricity, heat, light, and sound (distinguishing it from from the qualitative reasoning that Forbus & Gentner discuss). But, if later-developing aspects of physical understanding are built on these early foundations, that may be one source of the ontological confusion and messiness that permeates our later intuitive theories as well as people's attempts to understand formal theories in intuitive terms (Livesey et al.). Electrons are not little colliding ping-pong balls; enzymes are not trying to achieve an aspiration. But our parsing of the world into regular-bounded objects and intentional agents produces these category errors, because of the core role objects and agents play in cognition.

R4. Symbolic versus sub-symbolic representations

Beyond the richness and depth of our intuitive theories, the nature of representation was hotly contested in other ways, for the purposes of both cognitive modeling and developing more human-like AI. A salient division in the commentaries was between advocates of “symbolic” versus “sub-symbolic” representations, or relatedly, those who viewed our work through the lens of “explicit” versus “implicit” representations or “rules” versus “associations.” Several commentators thought our proposal relied too much on symbolic representations, especially because sub-symbolic distributed representations have helped facilitate much recent progress in machine learning (Çağlar & Hanson; Hansen et al.; MacLennan). Other commentators argued that human intelligence rests on more powerful forms of symbolic representation and reasoning than our article emphasized, such as abstract relational representations and analogical comparison (Forbus & Gentner).

This is a deeply entrenched debate in cognitive science and AI – one that some of us have directly debated in past articles (along with some of the commentators [e.g., Griffiths et al. Reference Griffiths, Chater, Kemp, Perfors and Tenenbaum2010; McClelland et al. Reference McClelland, Botvinick, Noelle, Plaut, Rogers, Seidenberg and Smith2010]), and we are not surprised to see it resurfacing here. Although we believe that this is still an interesting debate, we also see that recent work in AI and computational modeling of human cognition has begun to move beyond it, in ways that could be valuable.

To this end, we suggested that pattern recognition versus model building – and the ability to rapidly acquire new models and then reconfigure these models for new tasks without having to retrain – is a useful way to view the wide gap between human and machine intelligence. Implementing AI systems with our key ingredients would be a promising route for beginning to bridge this gap. Although our proposal is not entirely orthogonal to the symbolic versus sub-symbolic debate, we do see it as importantly different. Genuine model-building capabilities could be implemented in fully symbolic architectures or in a range of architectures that combine minimal symbolic components (e.g., objects, relations, agents, goals) with compositionality and sub-symbolic representation.

These ingredients could also be implemented in an architecture that does not appear to have symbols in any conventional sense – one that advocates of sub-symbolic approaches might even call non-symbolic – although we expect that advocates of symbolic approaches would point to computational states, which are effectively functioning as symbols. We do not claim to be breaking any new ground with these possibilities; the theoretical landscape has been well explored in philosophy of mind. We merely want to point out that our set of key ingredients is not something that should trouble people who feel that symbols are problematic. On the contrary, we hope this path can help bridge the gap between those who see symbols as essential, and those who find them mysterious or elusive.

Of our suggested ingredients, compositionality is arguably the most closely associated with strongly symbolic architectures. In relation to the above points, it is especially instructive to discuss how close this association has to be, and how much compositionality could be achievable within approaches to building intelligent machines that might not traditionally be seen as symbolic.

Hansen et al. argue that there are inherent limitations to “symbolic compositionality” that deep neural networks help overcome. Although we have found traditional symbolic forms of compositionality to be useful in our work, especially in interaction with other key cognitive ingredients such as causality and learning-to-learn (e.g., Goodman et al. Reference Goodman, Ullman and Tenenbaum2011; Reference Goodman, Tenenbaum, Gerstenberg, Margolis and Laurence2015; Lake et al. Reference Lake, Salakhutdinov and Tenenbaum2015a), there may be other forms of compositionality that are useful for learning and thinking like humans, and easier to incorporate into neural networks. For example, neural networks designed to understand scenes with multiple objects (see also Fig. 6 of our target article), or to generate globally coherent text (such as a recipe), have found simple forms of compositionality to be extremely useful (e.g., Eslami et al. Reference Eslami, Heess, Weber, Tassa, Kavukcuoglu, Hinton, Lee, Sugiyama, Luxburg, Guyon and Garnett2016; Kiddon et al. Reference Kiddon, Zettlemoyer and Choi2016). In particular, “objects” are minimal symbols that can support powerfully compositional model building, even if implemented in an architecture that would otherwise be characterized as sub-symbolic (e.g., Eslami et al. Reference Eslami, Heess, Weber, Tassa, Kavukcuoglu, Hinton, Lee, Sugiyama, Luxburg, Guyon and Garnett2016; Raposo et al. Reference Raposo, Santoro, Barrett, Pascanu, Lillicrap and Battaglia2017). The notion of a physical object – a chunk of solid matter that moves as a whole, moves smoothly through space and time without teleporting, disappearing, or passing through other solid objects – emerges very early in development (Carey Reference Carey2009). It is arguably the central representational construct of human beings' earliest intuitive physics, one of the first symbolic concepts in any domain that infants have access to, and likely shared with many other animal species in some form (see target article, sect. 4.1.1). Hence, the “object” concept is one of the best candidates for engineering AI to start with, and a promising target for advocates of sub-symbolic approaches who might want to incorporate useful but minimal forms of symbols and compositionality into their systems.

Deep learning research is also beginning to explore more general forms of compositionality, often by utilizing hybrid symbolic and sub-symbolic representations. Differentiable neural computers (DNCs) are designed to process symbolic structures such as graphs, and they use a mixture of sub-symbolic neural network-style computation and symbolic program traces to reason with these representations (Graves et al. Reference Graves, Wayne, Reynolds, Harley, Danihelka, Grabska-Barwińska, Colmenarejo, Grefenstette, Ramalho, Agapiou, Badia, Hermann, Zwols, Ostrovski, Cain, King, Summerfield, Blunsom, Kayukcuoglu and Hassabis2016). Neural programmer-interpreters (NPIs) begin with symbolic program primitives embedded in their architecture, and they learn to control the flow of higher-level symbolic programs that are constructed from these primitives (Reed & de Freitas Reference Reed and de Freitas2016). Interestingly, the learned controller is a sub-symbolic neural network, but it is trained with symbolic supervision. These systems are very far from achieving the powerful forms of model building that we see in human intelligence, and it is likely that more fundamental breakthroughs will be needed. Still, we are greatly encouraged to see neural network researchers who are not ideologically opposed to the role of symbols and compositionality in the mind and, indeed, are actively looking for ways to incorporate these notions into their paradigm.

In sum, by viewing the impressive achievements of human learning as model building rather than pattern recognition, we hope to emphasize a new distinction, different from classic debates of symbolic versus sub-symbolic computation, rules versus associations, or explicit versus implicit reasoning. We would like to focus on people's capacity for learning flexible models of the world as a target for AI research – one that might be reached successfully through a variety of representational paradigms if they incorporate the right ingredients. We are pleased that the commentators seem to broadly support “model building” and our key ingredients as important goals for AI research. This suggests a path for moving forward together.

R5. Additional ingredients

Many commentators agreed that although our key ingredients were important, we neglected another obvious, crucial component of human-like intelligence. There was less agreement on which component we had neglected. Overlooked components included emotion (Güss & Dörner; Clark); embodiment and action (Baldassarre et al.; MacLennan; Marin & Mostafaoui; Oudeyer; Wermter et al.); learning from others through social and cultural interaction (Clegg & Corriveau; Dennett & Lambert; Marin & Mostafaoui; Tessler et al.); open-ended learning combined with the ability to set one's own goal (Baldassarre et al.; Oudeyer; Wermter et al.); architectural diversity (Buscema & Sacco); dynamic network communication (Graham); and the ability to get a joke (Moerman).

Clearly, our recipe for building machines that learn and think like people was not complete. We agree that each of these capacities should figure in any complete scientific understanding of human cognition, and will likely be important for building artificial human-like cognition. There are likely other missing components as well. However, the question for us as researchers interested in the reverse engineering of cognition is: Where to start?

We focused on ingredients that were largely missing from today's deep learning AI systems, ones that were clearly crucial and present early in human development, and with large expected payoffs in terms of core AI problems. Importantly, for us, we also wanted to draw focus to ingredients that to our mind can be implemented in the relatively short term, given a concentrated effort. Our challenges are not meant to be AI-complete, but ones that can potentially be met in the next few years. For many of the suggestions the commentators made, it is hard (for us, at least) to know where to begin concrete implementation.

We do not mean that there have not been engineering advances and theoretical proposals for many of these suggestions. The commentators have certainly made progress on them, and we and our colleagues have also made theoretical and engineering contributions to some. But to do full justice to all of these missing components – from emotion to sociocultural learning to embodiment – there are many gaps that we do not know how to fill yet. Our aim was to set big goal posts on the immediate horizon, and we admit that there are others beyond. With these implementation gaps in mind, we have several things to say about each of these missing components.

R6. Insights from neuroscience and the brain

Our article did not emphasize neuroscience as a source of constraint on AI, not because we think it is irrelevant (quite the contrary), but because we felt that it was necessary to first crystallize the core ingredients of human intelligence at a computational level before trying to figure out how they are implemented in physical hardware. In this sense, we are advocating a mostly top-down route through the famous Marr levels of analysis, much as Marr himself did. This was unconvincing to some commentators (Baldassarre et al.; George; Kriegeskorte & Mok; Marblestone, Wayne, & Kording). Surely it is necessary to consider neurobiological constraints from the start, if one wishes to build human-like intelligence?

We agree that it would be foolish to argue for cognitive processes that are in direct disagreement with known neurobiology. However, we do not believe that neurobiology in its current state provides many strong constraints of this sort. For example, George suggests that lateral connections in visual cortex indicate that the internal model used by the brain enforces contour continuity. This seems plausible, but it is not the whole story. We see the world in three dimensions, and there is considerable evidence from psychophysics that we expect the surfaces of objects to be continuous in three dimensions, even if such continuity violates two-dimensional contour continuity (Nakayama et al. Reference Nakayama, Shimojo and Silverman1989). Thus, the situation is more like the opposite of what George argues: a challenge for neuroscience is to explain how neurons in visual cortex enforce the three-dimensional continuity constraints we know exist from psychophysical research.

Kriegeskorte & Mok point to higher-level vision as a place where neural constraints have been valuable. They write that core object recognition has been “conquered” by brain-inspired neural networks. We agree that there has been remarkable progress on basic object recognition tasks, but there is still a lot more to understand scientifically and to achieve on the engineering front, even in visual object perception. Take, for example, the problem of occlusion. Because most neural network models of object recognition have no explicit representation of objects arranged in depth, they are forced to process occlusion as a kind of noise. Again, psychophysical evidence argues strongly against this: When objects pass behind an occluding surface, we do not see them as disappearing or becoming corrupted by a massive amount of noise (Kellman & Spelke Reference Kellman and Spelke1983). A challenge for neuroscience is to explain how neurons in the ventral visual stream build a 3D representation of scenes that can appropriately handle occlusion. The analogous challenge exists in AI for brain-inspired artificial neural networks.

Further challenges, just in the domain of object perception, include perceiving multiple objects in a scene at once; perceiving the fine-grained shape and surface properties of novel objects for which one does not have a class label; and learning new object classes from just one or a few examples, and then generalizing to new instances. In emphasizing the constraints biology places on cognition, it is sometimes underappreciated to what extent cognition places strong constraints on biology.

R7. Coda: Ethics, responsibility, and opportunities

Given recent progress, AI is now widely recognized as a source of transformative technologies, with the potential to impact science, medicine, business, home life, civic life, and society, in ways that improve the human condition. There is also real potential for more negative impacts, including dangerous side effects or misuse. Recognizing both the positive and negative potential has spurred a welcome discussion of ethical issues and responsibility in AI research. Along these lines, a few commentators questioned the moral and ethical aspects of the very idea of building machines that learn and think like people. Moerman argues that the project is both unachievable and undesirable and, instead, advocates for building useful, yet inherently limited “single-purpose” machines. As he puts it (para. 2), “There are 7 billion humans on earth already. Why do we need fake humans when we have so many real ones?” Dennett & Lambert worry that machines may become intelligent enough to be given control of many vital tasks, before they become intelligent or human-like enough to be considered responsible for the consequences of their behavior.

We believe that trying to build more human-like intelligence in machines could have tremendous benefits. Many of these benefits will come from progress in AI more broadly – progress that we believe would be accelerated by the project described in our target article. There are also risks, but we believe these risks are not, for the foreseeable future, existential risks to humanity, or uniquely new kinds of risks that will sneak up on us suddenly. For anyone worried that AI research may be making too much progress too quickly, we would remind them that the best machine-learning systems are still very far from achieving human-like learning and thought, in all of the ways we discussed in the target article. Superintelligent AIs are even further away, so far that we believe it is hard to plan for them, except in the most general sense. Without new insights, ingredients, and ideas – well beyond those we have written about – we think that the loftiest goals for AI will be difficult to reach. Nonetheless, we see the current debate on AI ethics as responsible and healthy, and we take Dennett & Lambert's suggestion regarding AI co-pilots in that spirit.

Moerman's commentary fits well with many of these points: Simply scaling up current methods is unlikely to achieve anything like human intelligence. However, he takes the project of building more human-like learning machines to its logical extreme – building a doppelgänger machine that can mimic all aspects of being human, including incidental ones. Beyond rapid model building and flexible generalization, and even after adding the additional abilities suggested by the other commentators (sect. 5), Moerman's doppelgänger machine would still need the capability to get a joke, get a Ph.D., fall in love, get married, get divorced, get remarried, prefer Bourbon to Scotch (or vice versa), and so on. We agree that it is difficult to imagine machines will do all of these things any time soon. Nonetheless, the current AI landscape would benefit from more human-like learning – with its speed, flexibility, and richness – far before machines attempt to tackle many of the abilities that Moerman discusses. We think that this type of progress, even if only incremental, would still have far-reaching, practical applications (target article, sect. 6.2), and broader benefits for society.

Apart from advances in AI more generally, advances in human-like AI would bring additional unique benefits. Several commentators remarked on this. Spelke & Blass point out that a better understanding of our own minds will enable new kinds of machines that “can foster our thinking and learning” (para. 5). In addition, Patrzyk, Link, & Marewski expound on the benefits of “explainable AI,” such that algorithms can generate human-readable explanations of their output, limitations, and potential failures (Doshi-Velez & Kim Reference Doshi-Velez and Kim2017). People often learn by constructing explanations (Lombrozo Reference Lombrozo2016, relating to our “model building”), and a human-like machine learner would seek to do so too. Moreover, as it pertains to human-machine interaction (e.g., Dennett & Lambert), it is far easier to communicate with machines that generate human-understandable explanations than with opaque machines that cannot explain their decisions.

In sum, building machines that learn and think like people is an ambitious project, with great potential for positive impact: through more powerful AI systems, a deeper understanding of our own minds, new technologies for easing and enhancing human cognition, and explainable AI for easier communication with the technologies of the future. As AI systems become more fully autonomous and agentive, building machines that learn and think like people will be the best route to building machines that treat people the way people want and expect to be treated by others: with a sense of fairness, trust, kindness, considerateness, and intelligence.