The network approach to psychopathology began a decade ago with a simple hypothesis: symptoms may cohere as syndromes because of causal relations among the symptoms themselves (Borsboom, Reference Borsboom2008; Cramer, Waldorp, van der Maas, & Borsboom, Reference Cramer, Waldorp, van der Maas and Borsboom2010a). From this perspective, symptoms are not passive indicators of a latent ‘common cause’; they are agents in a causal system (Borsboom, Reference Borsboom2008; Kendler, Reference Kendler2016; Kendler, Zachar, & Craver, Reference Kendler, Zachar and Craver2011). This hypothesis has proven generative, stimulating a growing body of theoretical, methodological, and empirical work predicated on the idea that mental disorders can be characterized as complex systems in which symptoms play an active causal role (Fried et al., Reference Fried, van Borkulo, Cramer, Boschloo, Schoevers and Borsboom2017; McNally, Reference McNally2016). In this paper, we provide an overview of the first decade of this ‘network approach’ to psychopathology and consider directions forward for this research.
Literature review. We began by identifying articles within this literature. A description of our literature search and a PRISMA diagram appear in online Supplementary Materials. Briefly, we identified articles using keyword searches and citation records in ISI Web of Science, PsycINFO, and PubMed and reference lists from prior reviews. This search produced 1656 unique records, 924 of which were screened out, most commonly because they pertained to another domain of network analysis (e.g. protein interaction networks). The remaining 732 articles were assessed for eligibility. An article was eligible if it addressed a psychiatric phenomenon and incorporated the perspective of the network approach. To facilitate this determination, we first identified articles as being principally theoretical, methodological, or empirical. We deemed 363 articles eligible, including 98 theoretical, 61 methodological, and 204 empirical articles.
Notably, this literature has grown rapidly in recent years, with 90% of articles published in the last 5 years and 60% published in the last 2 years alone (see Fig. 1). Much of this growth is driven by empirical articles, especially those using ‘network psychometrics’ (Epskamp, Maris, Waldorp, & Borsboom, Reference Epskamp, Maris, Waldorp, Borsboom, Irwing, Hughes and Booth2016): a methodological approach developed within this literature. Indeed, the network approach has become largely synonymous with this type of empirical contribution. However, considerable work has been carried out beyond these empirical studies. To provide an overview of the full breadth of this work, we independently review theoretical, methodological, and empirical contributions. To our knowledge, this is the first systematic review of the full network approach literature (for a review of the empirical literature, see Contreras, Nieto, Valiente, Espinosa, & Vazquez, Reference Contreras, Nieto, Valiente, Espinosa and Vazquez2019).
Fig. 1. An overview of the network approach literature. Examining the cumulative number of empirical, methodological, and theoretical papers published in the network literature from 2008 to 2018.
Network theory
Among theory articles (n = 97), we identified five areas of work that have substantially contributed to the network conceptualization of psychopathology.
Philosophy of psychiatry
Parallel to the earliest conceptualization of mental disorders as symptom networks (Borsboom, Reference Borsboom2008; Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a), another group of theorists independently proposed that mental disorders can be conceptualized as ‘mechanistic property clusters’ (MPCs): kinds unified not by a shared underlying essence, but by causal relations among the features of the disorder (Kendler, Reference Kendler2016; Kendler et al., Reference Kendler, Zachar and Craver2011; Zachar, Reference Zachar2015). Symptom networks can be understood as a particular kind of MPC: one which emphasizes symptoms as features of the disorder. Yet, the origins of these concepts are quite distinct. Whereas symptom networks are rooted in psychometric theory (Borsboom, Reference Borsboom2008), MPC theorists draw their ideas from the philosophy of biology (Boyd, Reference Boyd1991, Reference Boyd1999). In doing so, they provide a philosophical foundation for network theory, undergirding it in two ways. First, as argued by Held (Reference Held2017), it clarifies what kind of things mental disorders are from this perspective: clusters of features sufficiently unified by the causal relations among them that they support induction, explanation, and prediction. Second, it positions network theory within the broader history of psychiatry, contrasting it especially with monocausal and essentialist frameworks that have long dominated psychiatric research. As argued by Radden (Reference Radden2018), the bacterial model of disease has been a powerful and entrenched metaphor in psychiatry, shaping the theories, methods, interpretations, and expectations of the field. A major contribution of network theory, Radden suggests, is that it provides a new metaphor for thinking about mental disorders.
Consistent with this position, much of the network theory literature can be seen as an effort to grapple with what mental disorders are, how they arise, and how they are treated when viewing them through this new lens. Theorists have used this framework to examine psychiatric comorbidity (Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a; Eaton, Reference Eaton2015; van Loo & Romeijn, Reference van Loo and Romeijn2015; Yordanova, Kolev, Kirov, & Rothenberger, Reference Yordanova, Kolev, Kirov and Rothenberger2010), sudden shifts in the onset or remission of symptoms (Hofmann, Curtiss, & McNally, Reference Hofmann, Curtiss and McNally2016; van de Leemput et al., Reference van de Leemput, Wichers, Cramer, Borsboom, Tuerlinckx, Kuppens and Scheffer2014), developmental psychopathology (Wass & Karmiloff-Smith, Reference Wass and Karmiloff-Smith2010), biological psychiatry (Walter, Reference Walter2013), psychiatric diagnosis (Maung, Reference Maung2016; Tsou, Reference Tsou2016; van Os, Delespaul, Wigman, Myin-Germeys, & Wichers, Reference van Os, Delespaul, Wigman, Myin-Germeys and Wichers2013a, Reference van Os, Delespaul, Wigman, Myin-Germeys and Wichers2013b), the ‘p factor’ (a general psychometric factor of psychopathology analogous to the general factor of intelligence; Caspi & Moffitt, Reference Caspi and Moffitt2018; van Bork, Epskamp, Rhemtulla, Borsboom, & van der Maas, Reference van Bork, Epskamp, Rhemtulla, Borsboom and van der Maas2017), and the equifinality and multi-finality of mental disorders (a disorder reached by many causal factors and multiple disorders reached by the same causal factor, respectively; Borsboom, Reference Borsboom2017; McGorry, Hartmann, Spooner, & Nelson, Reference McGorry, Hartmann, Spooner and Nelson2018). Theorists have also used this lens to examine specific disorders, including depression (de Jonge, Wardenaar, & Wichers, Reference de Jonge, Wardenaar and Wichers2015; Hayes, Yasinski, Ben Barnes, & Bockting, Reference Hayes, Yasinski, Ben Barnes and Bockting2015; Wittenborn, Rahmandad, Rick, & Hosseinichimeh, Reference Wittenborn, Rahmandad, Rick and Hosseinichimeh2016), post-traumatic stress disorder (Armour, Fried, & Olff, Reference Armour, Fried and Olff2017; McNally, Reference McNally2012, Reference McNally2017), anxiety disorders (Heeren & McNally, Reference Heeren and McNally2016), obsessive compulsive disorder (van den Hout, Reference van den Hout2014), eating disorders (Smith et al., Reference Smith, Crosby, Wonderlich, Forbush, Mason and Moessner2018), autism (Anderson, Reference Anderson2015; Verhoeff, Reference Verhoeff2013), psychosis (Bentall, Reference Bentall2014; Isvoranu, Borsboom, van Os, & Guloksuz, Reference Isvoranu, Borsboom, van Os and Guloksuz2016; Looijestijn, Blom, Aleman, Hoek, & Goekoop, Reference Looijestijn, Blom, Aleman, Hoek and Goekoop2015), psychopathy (Brzović, Jurjako, & Šustar, Reference Brzović, Jurjako and Šustar2017), sleep disorders (Blake, Trinder, & Allen, Reference Blake, Trinder and Allen2018; Marques & Azevedo, Reference Marques and Azevedo2018), and suicidality (de Beurs, Reference de Beurs2017).
Network science
Making use of the network metaphor, early theoretical work drew heavily from network science (Barabasi, Reference Barabasi2012), focusing on the network's structure (i.e. the web of relations among symptoms) and its effect on the network's state (i.e. the activation of symptoms). Theorists posited that when causal relations among symptoms are strong, the onset of one symptom will lead to the onset of others (‘causality hypothesis,’ Borsboom, Reference Borsboom2008; Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a). Strongly inter-connected symptom networks are thus vulnerable to a ‘contagion’ effect of spreading activation through the network (‘connectivity hypothesis’). Computational models have supported this notion (Cramer et al., Reference Cramer, van Borkulo, Giltay, van der Maas, Kendler, Scheffer and Borsboom2016), showing that, in highly connected networks, modest activation of select symptoms initiated by an ‘external stressor’ can trigger a cascade of activation (for a discussion on how external factors relate to the symptom network, see Borsboom, Reference Borsboom2017; Fried & Cramer, Reference Fried and Cramer2017). Moreover, when symptom activation becomes widespread, it will persist even after the initiating stressor is removed.
From a network perspective, mental disorder is characterized not only by the state of the network (i.e. elevated symptom activation), but also by the structure of the network: in particular, a strongly connected network in which inter-symptom relationships are sufficient to maintain elevated symptom activation over time (see Fig. 2; Borsboom, Reference Borsboom2017). In other words, mental disorder is characterized by a state of harmful equilibrium (for thought provoking and sometimes critical discussions of this characterization, see Friston, Redish, and Gordon, Reference Friston, Redish and Gordon2017; Guyon, Falissard, and Kop, Reference Guyon, Falissard and Kop2017; Porter, Reference Porter2015; Vosgerau & Soom, Reference Vosgerau and Soom2018; Young, Reference Young2015). Interestingly, computational modeling studies suggest that the boundary between health and disorder will vary as a function of network structure (Borsboom et al., Reference Borsboom, Rhemtulla, Cramer, van der Maas, Scheffer and Dolan2016), yielding a novel perspective on whether disorders are continuous or discrete phenomena (Haslam, Holland, & Kuppens, Reference Haslam, Holland and Kuppens2012; Waller & Meehl, Reference Waller and Meehl1998). In weakly connected networks, activation varies dimensionally. However, strongly connected networks produce sharper boundaries, as any activation within the system rapidly cascades into a state of psychopathology.
Fig. 2. Network structure, network state, and the definition of mental disorder. Network structure (high v. low connectivity) and network state (low v. high symptom activation) can be used to form concrete definitions of mental health and mental disorder (Borsboom, Reference Borsboom2017; Cramer, Waldorp, van der Maas, & Borsboom, Reference Cramer, Waldorp, van der Maas and Borsboom2010b). A weakly connected network with low symptom activation is in a state of mental health (top left panel). If elevated symptom activation arises (e.g. due to the effects of time-limited external stressor) the system will no longer be in a state of mental health, but will also not be in a state of mental disorder as symptoms will diminish once the external stressor is removed (bottom left panel). In contrast, a strongly connected network is vulnerable to the persistence of symptom activation even absent the effects of an external stressor. It is this stable state of elevated symptom activation that we refer to as a mental disorder (bottom right panel). Finally, strongly connected networks with minimal symptom activation are perhaps not in a state of mental disorder, but are in a state of vulnerability to the onset or recurrence of a disorder (top right panel). Such a system may thus represent a ‘silent disorder’ where symptoms are not currently manifest, but the risk for such symptoms remains high (Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010b).
Early work on network structure also contributed to a shifting perspective on individual symptoms (Fried, Reference Fried2015; Fried et al., Reference Fried, Boschloo, van Borkulo, Schoevers, Romeijn, Wichers and Borsboom2015). From a network perspective, symptoms are not interchangeable indicators. They are agents in a causal network whose role depends upon their position in that network (Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a). Theorists posited that highly ‘central’ symptoms (e.g. those with stronger inter-symptom connections) have greater potential to spread symptom activation throughout the network than do symptoms on the periphery (‘centrality hypothesis’; Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a). Such symptoms were thus posited to feature in disorder onset and remission (Borsboom & Cramer, Reference Borsboom and Cramer2013; Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a; McNally et al., Reference McNally, Robinaugh, Wu, Wang, Deserno and Borsboom2015). Further, because some symptoms occur in multiple disorders, symptom activation can spread between syndromes, with symptoms bridging these syndromes playing a critical role in psychiatric comorbidity (‘comorbidity hypothesis,’ Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a).
Affect dynamics and momentary experience
As network theory developed, some researchers argued that we must shift our focus not only from disorders to symptoms, but further to the level of momentary experiences (van Os et al., Reference van Os, Delespaul, Wigman, Myin-Germeys and Wichers2013a, Reference van Os, Delespaul, Wigman, Myin-Germeys and Wichers2013b; Wichers, Reference Wichers2014; Wichers, Wigman, & Myin-Germeys, Reference Wichers, Wigman and Myin-Germeys2015). These researchers note that symptoms are aggregates of moment-to-moment experiences. It is these moment-to-moment ‘micro-processes,’ they argue, that constitute the true building blocks of psychopathology (see Fig. 3; cf. Wichers, Reference Wichers2014, p. 1351). This perspective shifts our focus to a more granular level of experience and highlights the importance of understanding the ‘chronometry’ of experiences, symptoms, and disorders (Treadway & Leonard, Reference Treadway and Leonard2016).
Fig. 3. Illustrating symptom networks and momentary experiences with the example of panic disorder. Panic disorder comprises three core symptoms (recurrent panic attacks, persistent concern or worry about those attacks, and avoidance behavior) that play out on a time scale of days, weeks, or even months (e.g. to meet criteria for panic disorder, these symptoms must persist for at least 1 month). Panic attacks, in turn, comprise momentary experiences of arousal-related bodily sensations, a sense of impending threat (e.g. fear of having a heart attack), and an urge to escape from or mitigate that threat. These momentary experiences play out over the course of seconds of minutes (e.g. to meet criteria for a panic attack, these experiences must surge to a peak within 10 min). Cognitive behavioral theories posit that causal relations at both of these time scales play critical roles in panic attacks (Clark, Reference Clark1986) and panic disorder (Goldstein & Chambless, Reference Goldstein and Chambless1978). A critical challenge for the network approach literature will be determining precisely how processes at these different time scales interact with one another (cf. Wichers, Reference Wichers2014).
Cognitive behavioral theory
From its earliest stages, network theory has drawn on cognitive behavioral models of psychopathology when discussing plausible causal relations among symptoms (e.g. ‘vicious cycle’ theories of panic disorder; Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a). Indeed, the cognitive behavioral literature offers a long history of theorizing about causal relations among symptoms, empirical research investigating those relationships, and treatment research suggesting that symptom-level interventions are effective treatments (e.g. see van den Hout, Reference van den Hout2014). Not surprisingly then, some cognitive behavioral theorists have embraced network theory and proposed integrating it with cognitive behavioral models (Hoffart & Johnson, Reference Hoffart and Johnson2017; Hofmann, Reference Hofmann2014; Hofmann et al., Reference Hofmann, Curtiss and McNally2016; McNally, Reference McNally2016; van den Hout, Reference van den Hout2014).
Systems science
Most recently, theorists have drawn from areas of systems science beyond network science alone, including dynamical systems theory, catastrophe theory, and cybernetics (Hayes et al., Reference Hayes, Yasinski, Ben Barnes and Bockting2015; Nelson, McGorry, Wichers, Wigman, & Hartmann, Reference Nelson, McGorry, Wichers, Wigman and Hartmann2017; Tretter & Loffler-Stastka, Reference Tretter and Loffler-Stastka2018; Yordanova et al., Reference Yordanova, Kolev, Kirov and Rothenberger2010). These fields offer new tools for psychiatric research. For example, the feedback loops emphasized in cognitive behavioral theories of psychopathology have been rigorously investigated in cybernetics and dynamical systems theory (Scheffer, Reference Scheffer2009). Illustrating the value of this work, Wittenborn et al. (Reference Wittenborn, Rahmandad, Rick and Hosseinichimeh2016) used the ‘causal loop diagrams’ of dynamical systems theory to generate a conceptual model of depression as a dynamical system and dynamical systems concepts such as alternative stable states and critical slowing have already begun to be applied to psychopathology (Hayes et al., Reference Hayes, Yasinski, Ben Barnes and Bockting2015; Hofmann & Curtiss, Reference Hofmann and Curtiss2018; Nelson et al., Reference Nelson, McGorry, Wichers, Wigman and Hartmann2017; Rikkert et al., Reference Rikkert, Marcel, Dakos, Buchman, Boer, Glass and Scheffer2016). Indeed, there is research examining mental disorders as dynamical systems that predates (e.g. Granic, Reference Granic2005; Hayes & Strauss, Reference Hayes and Strauss1998; Schiepek, Reference Schiepek2003) and, in some cases, has continued to develop independently of (e.g. Pincus & Metten, Reference Pincus and Metten2010) the network approach literature. This work anticipated many of the ideas that would later develop within the context of the network approach (e.g. ‘ideographical system modeling,’ Schiepek, Reference Schiepek2003) and is thus an invaluable resource for those studying mental disorders as complex systems.
Critical analysis and future directions
With these diverse scientific and philosophical contributions, considerable progress has been made in developing a general network theory of mental disorders (Borsboom, Reference Borsboom2017). However, formal theories that specify precisely how any specific disorder operates as a causal system are still scarce. We regard the development of such theories to be a critical next step for network theory. Computational models will play a critical role if the field is to achieve this aim (Huys, Maia, & Frank, Reference Huys, Maia and Frank2016; Robinaugh et al., Reference Robinaugh, Haslbeck, Waldorp, Kossakowski, Fried, Millner and Borsboom2019; Tryon, Reference Tryon2018).
One key advantage of such models is that they render all aspects of the theory explicit and available for evaluation (Epstein, Reference Epstein2008). Computational models of symptom networks thus provide insight into the assumptions made in network theory, as they are the most explicit representation in the literature of how symptom networks are posited to operate. Unfortunately, currently operational models (Borsboom et al., Reference Borsboom, Rhemtulla, Cramer, van der Maas, Scheffer and Dolan2016; Cramer et al., Reference Cramer, van Borkulo, Giltay, van der Maas, Kendler, Scheffer and Borsboom2016) reveal a restrictive set of assumptions: they typically assume binary symptoms that operate on the same time scale and interact with instantaneous and positive effects (i.e. symptoms exacerbate, rather than inhibit one another). Further, interactions between symptoms are pairwise and symmetric (i.e. X causes Y to the same extent that Y causes X).
Models adhering to these assumptions [Binary Instantaneous Positive Pairwise and Symmetric (BIPPS) models] are unrealistic for most mental disorders. Many symptoms are dimensional. Some inter-symptom effects occur over minutes (fear → avoidance) whereas others occur over days (insomnia → fatigue), weeks (appetite loss → weight loss), or years (childhood sexual abuse → adulthood social disconnection). Negative inter-symptom effects are plausible, as are higher order interactions (e.g. sleep moderating the effect of trauma memories on emotional reactivity), and many inter-symptom relations are asymmetric (e.g. fear elicits avoidance, but avoidance dampens fear). Violations of these assumptions will almost certainly impact core network theory hypotheses. For example, whether ‘central symptoms’ are indeed especially important almost certainly depends on the specifics of the network structure and inter-symptom interactions, making it unclear when this hypothesis should be expected to hold (cf. Bringmann et al., Reference Bringmann, Elmber, Epskamp, Kraus, Schoch, Wichers and Snippe2019; Dablander & Hinne, Reference Dablander and Hinne2018; Hallquist, Wright, & Molenaar, Reference Hallquist, Wright and Molenaar2019). Theorists may rightly object that their own theory of symptom networks does not require adherence to these assumptions. However, to the extent that such theories remain verbal in character, their assumptions remain unclear and unavailable for careful evaluation.
It will thus be important to develop computational models in which the assumptions embodied in the model better align with how specific disorders are thought to operate (e.g. see Hosseinichimeh, Wittenborn, Rick, Jalali, & Rahmandad, Reference Hosseinichimeh, Wittenborn, Rick, Jalali and Rahmandad2018). This work can (indeed, must) be directly informed by each of the areas of work that have already contributed to network theory. For example, cognitive behavioral models of psychopathology bring a wealth of theory and empirical research about inter-symptom causal relations (cf. van den Hout, Reference van den Hout2014); the literature on affective dynamics can inform our understanding of the timescales on which processes operate and how moment-to-moment experiences become symptoms and, in turn, disorders (cf. Wichers, Reference Wichers2014); and the dynamical systems literature can provide a host of tools for modeling and evaluating complex systems (cf. Wittenborn et al., Reference Wittenborn, Rahmandad, Rick and Hosseinichimeh2016; Yang et al., Reference Yang, Ram, Gest, Lydon-Staley, Conroy, Pincus and Molenaar2018).
Methodology
When the network approach was first proposed, there was no established method for assessing symptom network structure. The methodology developed to meet this need in the 61 methodological articles we reviewed has two primary components: (a) estimating network structure and (b) assessing network characteristics.
Network structure
Early efforts defined network structure using symptom co-occurrence or correlation (Cramer, Borsboom, Aggen, & Kendler, Reference Cramer, Borsboom, Aggen and Kendler2012; Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a). Aiming to better identify the direct inter-symptom relations posited in network theory, subsequent efforts have focused on conditional dependence relationships with a penalty on regression parameters to obtain a sparse network of dependence relationships known as a pairwise Markov random field (PMRF; see Fig. 4). This approach was first implemented by van Borkulo et al. (Reference van Borkulo, Borsboom, Epskamp, Blanken, Boschloo, Schoevers and Waldorp2014) and subsequently extended to multivariate normal data (Epskamp, Borsboom, & Fried, Reference Epskamp, Borsboom and Fried2018a), mixed data (Haslbeck & Lourens, Reference Haslbeck and Lourens2016), and latent variables (Epskamp, Rhemtulla, & Borsboom, Reference Epskamp, Rhemtulla and Borsboom2017b). Researchers have made these methods freely available (Epskamp, Cramer, Waldorp, Schmittmann, & Borsboom, Reference Epskamp, Cramer, Waldorp, Schmittmann and Borsboom2012; van Borkulo et al., Reference van Borkulo, Borsboom, Epskamp, Blanken, Boschloo, Schoevers and Waldorp2014), provided tutorials on their use (Borsboom & Cramer, Reference Borsboom and Cramer2013; Costantini et al., Reference Costantini, Epskamp, Borsboom, Perugini, Mottus, Waldorp and Cramer2015; Epskamp & Fried, Reference Epskamp and Fried2018; Epskamp, Kruis, & Marsman, Reference Epskamp, Kruis and Marsman2017a; Jones, Mair, & McNally, Reference Jones, Mair and McNally2018), and extensively discussed their properties (Kruis & Maris, Reference Kruis and Maris2016; Marsman et al., Reference Marsman, Borsboom, Kruis, Epskamp, van Bork, Waldorp and Maris2018).
Fig. 4. Network estimation methods commonly utilized in empirical network studies. This figure depicts methods commonly used in estimating network structure from cross-sectional (panels a and b) and time-series data (panels c and d). Panels a and c depict the cumulative number of articles applying a given estimation method for cross-sectional and time-series data, respectively. Panels b and d depict the proportion of articles in a given year that utilized these estimation methods. For the purposes of this summary, we considered any network based on multiple time points to be based on ‘time-series’ data, thus incorporating change score networks into this category. Note that the earliest cross-sectional (Cramer et al., Reference Cramer, Waldorp, van der Maas and Borsboom2010a) and time-series (Bringmann et al., Reference Bringmann, Vissers, Wichers, Geschwind, Kuppens, Peeters and Tuerlinckx2013) networks were regarded as theoretical and methodological contributions, respectively, given their substantial contributions in these domains, and thus are not included in this report. PMRF = pairwise Markov random field; SEM = structural equation modeling; GIMME = group iterative multiple model estimation.
Conditional dependence networks can be constructed from either cross-sectional or intra-individual time series data. Cross-sectional networks have been criticized on two fronts. First, researchers have expressed concerns about their replicability (Fried & Cramer, Reference Fried and Cramer2017; Fried, Epskamp, Nesse, Tuerlinckx, & Borsboom, Reference Fried, Epskamp, Nesse, Tuerlinckx and Borsboom2016), with some arguing that these methods are inherently unstable (for an extended discussion, see Borsboom, Robinaugh, The Psychosystems Group, Rhemtulla, & Cramer, Reference Borsboom, Robinaugh, Rhemtulla and Cramer2018; Borsboom et al., Reference Borsboom, Fried, Epskamp, Waldorp, van Borkulo, van der Maas and Cramer2017; Forbes, Wright, Markon, & Krueger, Reference Forbes, Wright, Markon and Krueger2017a, Reference Forbes, Wright, Markon and Krueger2017b). Some have further argued that instability is exacerbated by the use of single-item assessments and small samples (DeYoung & Krueger, Reference DeYoung and Krueger2018). To promote robustness, methods have been developed for evaluating the precision and stability of estimated network parameters (Epskamp et al., Reference Epskamp, Borsboom and Fried2018a). These methods have been rapidly adopted in the empirical network literature (see Fig. 4). Nonetheless, replicability remains an important subject of ongoing debate (e.g. Forbes, Wright, Markon, & Krueger, Reference Forbes, Wright, Markon and Krueger2019; Jones, Williams, & McNally, Reference Jones, Williams and McNally2019).
Researchers have also argued that cross-sectional findings cannot demonstrate causality and cannot be assumed to generalize to the level of most interest in network theory: the level of the individual (e.g. Bos & Wanders, Reference Bos and Wanders2016; Bringmann & Eronen, Reference Bringmann and Eronen2018; Forbes et al., Reference Forbes, Wright, Markon and Krueger2017a; Tzur-Bitan, Meiran, & Shahar, Reference Tzur-Bitan, Meiran and Shahar2010). These researchers suggest that conditional dependence structure should instead be based on within-person variation, assessing network relations ‘where (i.e. within-person) and when (i.e. in real time)’ they are posited to operate (Hamaker & Wichers, Reference Hamaker and Wichers2017). Researchers have thus developed (Beltz & Gates, Reference Beltz and Gates2017; Bringmann, Ferrer, Hamaker, Borsboom, & Tuerlinckx, Reference Bringmann, Ferrer, Hamaker, Borsboom and Tuerlinckx2018; Bringmann et al., Reference Bringmann, Vissers, Wichers, Geschwind, Kuppens, Peeters and Tuerlinckx2013; Bulteel, Tuerlinckx, Brose, & Ceulemans, Reference Bulteel, Tuerlinckx, Brose and Ceulemans2018; Epskamp et al., Reference Epskamp, van Borkulo, van der Veen, Servaas, Isvoranu, Riese and Cramer2018c) and evaluated (Bulteel, Tuerlinckx, Brose, & Ceulemans, Reference Bulteel, Tuerlinckx, Brose and Ceulemans2016b; de Haan-Rietdijk, Voelkle, Keijsers, & Hamaker, Reference de Haan-Rietdijk, Voelkle, Keijsers and Hamaker2017; Kuiper & Ryan, Reference Kuiper and Ryan2018; Schuurman, Ferrer, de Boer-Sonnenschein, & Hamaker, Reference Schuurman, Ferrer, de Boer-Sonnenschein and Hamaker2016; Schuurman, Houtveen, & Hamaker, Reference Schuurman, Houtveen and Hamaker2015) methods for estimating within-subject network structure. These methods, especially vector autoregressive models, are growing in popularity and promise to substantially inform our understanding of the relationships among symptoms (see Fig. 4).
Alternative methods
Although most studies estimate network structure using statistical associations, additional methods have been used (e.g. Borsboom, Cramer, Schmittmann, Epskamp, & Waldorp, Reference Borsboom, Cramer, Schmittmann, Epskamp and Waldorp2011; Tio, Epskamp, Noordhof, & Borsboom, Reference Tio, Epskamp, Noordhof and Borsboom2016). In one method of note, researchers query patients or clinicians about the relations among symptoms (Frewen, Schmittmann, Bringmann, & Borsboom, Reference Frewen, Schmittmann, Bringmann and Borsboom2013; Ruzzano, Borsboom, & Geurts, Reference Ruzzano, Borsboom and Geurts2015). Despite their promise, these perceived causal relation networks remain underutilized.
Network characteristics
Researchers have adopted or developed methods for examining local and global characteristics of symptom networks, including assessments of node centrality (Epskamp et al., Reference Epskamp, Cramer, Waldorp, Schmittmann and Borsboom2012), node predictability (Haslbeck & Waldorp, Reference Haslbeck and Waldorp2018), node clustering (Costantini & Perugini, Reference Costantini and Perugini2014), community structure (Blanken et al., Reference Blanken, Deserno, Dalege, Borsboom, Blanken, Kerkhof and Cramer2018; Golino & Epskamp, Reference Golino and Epskamp2017; Zhao, Yang, Liu, & Chen, Reference Zhao, Yang, Liu and Chen2017), and similarity of intra-individual network structure (Bulteel, Tuerlinckx, Brose, & Ceulemans, Reference Bulteel, Tuerlinckx, Brose and Ceulemans2016a). These characteristics are often a primary focus of network analysis studies.
Critical analysis and future directions
The network psychometric toolbox is rapidly evolving. Researchers are developing new methods for comparing (van Borkulo et al., Reference van Borkulo, Boschloo, Kossakowski, Tio, Schoevers, Borsboom and Waldorp2017; Williams, Rast, & Mulder, Reference Williams, Rast and Mulder2019) and estimating network structure, including methods that integrate latent variable and network models (Chen, Li, Liu, & Ying, Reference Chen, Li, Liu and Ying2018; Christensen, Kenett, Aste, Silvia, & Kwapil, Reference Christensen, Kenett, Aste, Silvia and Kwapil2018; Epskamp, Reference Epskamp2019; Williams & Rast, Reference Williams and Rast2018). Researchers have also called for increased adoption of tools from other domains of systems science (Nelson et al., Reference Nelson, McGorry, Wichers, Wigman and Hartmann2017), such as the use of autocorrelations to signal shifts into a state of psychopathology (van de Leemput et al., Reference van de Leemput, Wichers, Cramer, Borsboom, Tuerlinckx, Kuppens and Scheffer2014). Here, we highlight three additional areas in need of development.
Most importantly, the relationship between data generating causal systems and empirical networks estimated from data requires further evaluation (see Fig. 5). Network psychometrics recover the ‘true’ causal system when that system is itself a PMRF (Epskamp et al., Reference Epskamp, Fried, van Borkulo, Robinaugh, Marsman, Dalege and Cramer2018b), as would be the case for a causal system meeting the BIPPS assumptions. We also know what empirical PMRF we should expect for one specific type of causal system (i.e. directed acyclic graphs; Pearl, Reference Pearl2009). However, neither of these systems is theoretically plausible for most mental disorders, and we have minimal insight into the relationship between causal systems and network structures estimated using network psychometrics in more realistic frameworks. This is true for both cross-sectional networks and within-subject networks. Clarifying this relationship is fundamental to our ability to draw inferences from these analyses and, therefore, a critical direction for future research.
Fig. 5. An overview of network methodology, with a focus on the relationship between causal systems, data, and the empirical networks most commonly used in the network approach literature (PMRFs). In many areas of network science, both the elements of the network and the connections among them can be directly observed (e.g. train stations and the tracks that connect them). In psychiatry, symptoms can be assessed, but the relationships among them must be inferred. Network psychometrics aims to infer those relationships using statistical associations. The method by which this is done depends on the data collected (for a discussion of Cattell's data cube and its relation to specific analyses, see Wardenaar & de Jonge, Reference Wardenaar and de Jonge2013). For cross-sectional data, a single network is estimated based on the covariation of symptoms between-persons at that point in time. For n = 1 time-series data, networks are estimated based on the covariation of symptoms over time within one individual, and can be used to inform contemporaneous and temporal (lagged) associations among symptoms. In time series data in larger samples, networks can be estimated using both within- and between-person information. Importantly, the network structure derived from between-person analyses and within-person analyses are unlikely to be equivalent and, for many plausible causal systems, it remains unclear how the structure derived from either analysis corresponds to the ‘true structure' of the causal system. The relationships among between-person networks, within-person networks, and the ‘true structure’ of different types of causal systems are critical directions for future research.
Second, methodologies for aggregating findings across this growing literature are needed, including methods for combining networks derived from similar analyses and methods for integrating findings across different analyses. Because no single methodology has been shown to directly recover the network structure of mental disorders, both cross-sectional networks and within-subjects networks provide valuable but incomplete information about the relationships among symptoms. Accordingly, a genuine understanding of causal systems will almost certainly require an aggregation of evidence across multiple studies and methodological approaches.
Finally, the field must develop a methodology for network data collection. The majority of empirical network studies have used data that were not collected for the purpose of network analysis (Guloksuz, Pries, & van Os, Reference Guloksuz, Pries and van Os2017). The selection and measurement of network components is fundamentally important to the estimation (Hallquist et al., Reference Hallquist, Wright and Molenaar2019) and interpretation (Forbes et al., Reference Forbes, Wright, Markon and Krueger2017a) of network structure. Even seemingly minor measurement decisions can impact the results of these analyses (Hoffman, Steinley, Trull, & Sher, Reference Hoffman, Steinley, Trull and Sher2018). Accordingly, psychometric strategies focused on the optimal assessment of components of psychopathology networks are needed.
Empirical studies
Among 204 empirical articles in the reviewed literature, 174 used some form of network analysis. Among these, 170 used network psychometrics to estimate network structure, including 141 articles that examined cross-sectional data in 176 samples (mean N = 2169; median N = 508) and 32 articles that examined time-series data in 44 samples (mean N = 185; median N = 76). This approach has been applied to a wide range of disorders, but especially to depression (69 articles) and posttraumatic stress disorder (PTSD; 31 articles). This empirical work provides rich and nuanced information about inter-symptom relationships for these disorders. A complete review of these disorder-specific findings is beyond the scope of this review. In our online Supplementary Materials, we list empirical publications organized by disorder and we encourage readers to use this resource to read further. Here, we adopt a broader focus and consider findings across disorders for the two most commonly examined network characteristics: network connectivity and node centrality (see Fig. 6).
Fig. 6. Network characteristics commonly examined in empirical network studies. This figure depicts characteristics commonly examined in empirical network studies utilizing cross-sectional (panels a and b) and time-series data (panels c and d). Panels a and c depict the cumulative number of articles reporting a given characteristic for cross-sectional and time-series data, respectively. Panels b and d depict the proportion of articles in a given year that examined those characteristics. In both cross-sectional and time-series networks, node centrality and network connectivity were the most examined network characteristics.
Network connectivity
Most fundamentally, empirical network studies have found that symptoms are highly interconnected. There is not only a positive manifold (i.e. all variables are positively correlated), but a conditional positive manifold for most disorders. That is, with some exceptions, even after controlling for shared variance among symptoms, these symptoms tend to be positively interconnected. This high connectivity is not surprising, but bears noting as it suggests meaningful clustering of symptoms in the syndromes we identify as mental disorders. Connectivity tends to be consistent across time (e.g. Curtiss, Ito, Takebayashi, & Hofmann, Reference Curtiss, Ito, Takebayashi and Hofmann2018; Rouquette et al., Reference Rouquette, Pingault, Fried, Orri, Falissard, Kossakowski and Borsboom2018; von Stockert, Fried, Armour, & Pietrzak, Reference von Stockert, Fried, Armour and Pietrzak2018) and demographic groups, especially gender and age (e.g. Belvederi Murri, Amore, Respino, & Alexopoulos, Reference Belvederi Murri, Amore, Respino and Alexopoulos2018; Fonseca-Pedrero et al., Reference Fonseca-Pedrero, Ortuno, Debbane, Chan, Cicero, Zhang and Fried2018; Russell, Neill, Carrion, & Weems, Reference Russell, Neill, Carrion and Weems2017); though differences have been observed between countries (Fonseca-Pedrero et al. Reference Fonseca-Pedrero, Ortuno, Debbane, Chan, Cicero, Zhang and Fried2018; Wusten et al. Reference Wusten, Schlier, Jaya, Genetic, Fonseca-Pedrero, Peters and Lincoln2018).
Researchers have investigated the hypothesis that greater connectivity (aka network density or global strength) confers risk for psychopathology using cross-sectional analyses in several ways. First, researchers have compared connectivity between groups based on clinical severity. These studies typically found greater connectivity in those with greater severity (e.g. Heeren & McNally, Reference Heeren and McNally2018; Santos, Fried, Asafu-Adjei, & Ruiz, Reference Santos, Fried, Asafu-Adjei and Ruiz2017; van Rooijen et al., Reference van Rooijen, Isvoranu, Kruijt, van Borkulo, Meijer and Wigman2018), though one reported no difference (Levinson et al., Reference Levinson, Brosof, Vanzhula, Christian, Jones, Rodebaugh and Fernandez2018a) and another found the opposite pattern (Southward & Cheavens, Reference Southward and Cheavens2018). However, these findings should be interpreted with caution as grouping subjects on severity affects the estimation of the network (De Ron, Fried, & Epskamp, Reference De Ron, Fried and Epskamp2019). Second, researchers have retrospectively examined baseline network connectivity in those whose disorder subsequently remitted v. persisted, reasoning that greater connectivity should lead to disorder persistence. van Borkulo et al. (Reference van Borkulo, Boschloo, Borsboom, Penninx, Waldorp and Schoevers2015) indeed found greater baseline connectivity among depression symptoms in those with persistent v. remitted depression. In two subsequent studies, one similarly reported greater baseline connectivity in those with persistent depression, but could not reject the null hypothesis that networks were equal at the population level (Schweren, van Borkulo, Fried, & Goodyer, Reference Schweren, van Borkulo, Fried and Goodyer2018) and another reported greater connectivity in treatment non-responders v. responders for psychotic disorder symptoms, but no statistical tests were performed (Esfahlani, Sayama, Visser, & Strauss, Reference Esfahlani, Sayama, Visser and Strauss2017). Accordingly, there is at best modest support for the notion that connectivity is associated with disorder persistence. Third, researchers have examined whether those at risk for psychopathology (e.g. genetic risk; van Loo et al., Reference van Loo, Van Borkulo, Peterson, Fried, Aggen, Borsboom and Kendler2018) exhibit elevated network connectivity. No such elevations have been observed. Fourth, researchers have examined whether connectivity reduces over the course of treatment, hypothesizing that treatments may have their effect by reducing connectivity. Most such studies reported no change in connectivity (e.g. Levine & Leucht, Reference Levine and Leucht2016; Lydon-Staley, Schnoll, Hitsman, & Bassett, Reference Lydon-Staley, Schnoll, Hitsman and Bassett2018; Schuler, Wittmann, Faller, & Schultz, Reference Schuler, Wittmann, Faller and Schultz2018) and two reported an increase in connectivity over time (Beard et al., Reference Beard, Millner, Forgeard, Fried, Hsu, Treadway and Bjorgvinsson2016; Bos et al., Reference Bos, Fried, Hollon, Bringmann, Dimidjian, DeRubeis and Bockting2018). Notably, the only study to find significant connectivity change naturally over time similarly found increased connectivity among PTSD symptoms from 24-h to 12-months post-trauma, a timeframe in which there is significant reduction in symptom severity (Bryant et al., Reference Bryant, Creamer, O'Donnell, Forbes, McFarlane, Silove and Hadzi-Pavlovic2017).
Importantly, these cross-sectional studies are predicated on the assumption that greater connectivity at the group level indicates greater connectivity in the individuals who compose the group (Bos & Wanders, Reference Bos and Wanders2016; van Borkulo, Borsboom, & Schoevers, Reference van Borkulo, Borsboom and Schoevers2016). To our knowledge, there is no evidence to support or disconfirm this assumption, making it unclear whether these findings are an adequate test of the connectivity hypothesis. Several studies have avoided this limitation using intra-individual network analyses. These studies have found greater connectivity among negative mood states in those with depression (Pe et al., Reference Pe, Kircanski, Thompson, Bringmann, Tuerlinckx, Mestdagh and Gotlib2015), elevated neuroticism (Bringmann et al., Reference Bringmann, Pe, Vissers, Ceulemans, Borsboom, Vanpaemel and Kuppens2016), and genetic liability for psychopathology (Hasmi et al., Reference Hasmi, Drukker, Guloksuz, Menne-Lothmann, Decoster, van Winkel and van Os2017). However, de Vos et al. (Reference de Vos, Wardenaar, Bos, Wit, Bouwmans and de Jonge2017) found that whether mood state network connectivity was indeed stronger among depressed patients varied considerably depending on methodological choices. In addition, in a network encompassing a broader range of components (e.g. cognitions, emotions, psychotic experiences), Klippel et al. (Reference Klippel, Viechtbauer, Reininghaus, Wigman, van Borkulo, MERGE and Wichers2018) observed differences in the number but not strength of connections among psychotic patients, first-degree relatives, and healthy controls. Together, these studies utilizing time-series data provide qualified support for the notion that connectivity of negative mood state networks is associated with psychopathology, but minimal evidence that broader networks of momentary experiences exhibit such associations.
Node centrality
Empirical research has focused heavily on examining individual symptoms using node centrality. Among centrality indices, the most popular and robust is node strength (i.e. summed absolute strength of a node's direct links). At least two broad conclusions can be drawn from this work. First, there is no evidence that the symptoms identified in the DSM play a privileged role in these networks. Fried et al. (Reference Fried, Epskamp, Nesse, Tuerlinckx and Borsboom2016) and Kendler, Aggen, Flint, Borsboom, and Fried (Reference Kendler, Aggen, Flint, Borsboom and Fried2018) found that there was, on average, no difference between DSM and non-DSM symptoms of depression. Indeed, non-DSM symptoms often exhibit elevated centrality (e.g. feeling disliked in depression and fear of weight gain in bulimia nervosa; Levinson, Vanzhula, Brosof, & Forbush, Reference Levinson, Vanzhula, Brosof and Forbush2018b; Santos et al., Reference Santos, Fried, Asafu-Adjei and Ruiz2017) and some DSM nodes are weakly connected to the network (e.g. traumatic amnesia in PTSD networks; Fried et al., Reference Fried, Eidhof, Palic, Costantini, Huisman-van Dijk, Bockting and Karstoft2018). Relatedly, there is mixed evidence as to whether those symptoms the DSM identifies as being especially important (e.g. depressed mood and anhedonia) are more central than other DSM symptoms. For example, the cardinal symptoms of depression are indeed often highly central, but perhaps the most consistently central depression symptom is low energy/fatigue (cf. Contreras et al., Reference Contreras, Nieto, Valiente, Espinosa and Vazquez2019). Accordingly, while connectivity analyses suggest that the DSM accurately identifies symptoms that cohere as syndromes, centrality analyses suggest it has not captured all symptoms or even uniquely important symptoms within these syndromes.
Second, there is modest evidence to support the notion that centrality indices identify important symptoms. Central symptoms have been found to be more predictive of subsequent diagnosis than are peripheral symptoms in both depression and PTSD (Boschloo, van Borkulo, Borsboom, & Schoevers, Reference Boschloo, van Borkulo, Borsboom and Schoevers2016; Haag, Robinaugh, Ehlers, & Kleim, Reference Haag, Robinaugh, Ehlers and Kleim2017), and a symptom's centrality is positively correlated with the strength of association between change in the symptom and change in the remainder of the network (Robinaugh, Millner, & McNally, Reference Robinaugh, Millner and McNally2016; Rodebaugh et al., Reference Rodebaugh, Tonge, Piccirillo, Fried, Horenstein, Morrison and Heimberg2018). However, while these findings are consistent with the centrality hypothesis, they are also consistent with a common cause framework where symptoms arise from a single underlying dimension. In that case, symptoms with higher strength centrality would be identified as more reliable indicator variables and the the same pattern of findings would be expected, as more reliable indicators are more predictive (Lord & Novick, Reference Lord and Novick1968; Muchinsky, Reference Muchinsky1996). Accordingly, these findings are consistent with, but not especially strong evidence for, the centrality hypothesis.
Critical analysis and future directions
We have three broad critiques of the empirical literature. First, these studies have relied heavily on data collected for purposes other than network analysis. This raises concern about whether individual items used in these analyses indeed represent discrete components of the syndrome (Fried & Cramer, Reference Fried and Cramer2017; Hallquist et al., Reference Hallquist, Wright and Molenaar2019; Levinson et al., Reference Levinson, Brosof, Vanzhula, Christian, Jones, Rodebaugh and Fernandez2018a). It has also contributed to substantial differences between studies in the components included in the network. For example, among 18 studies estimating the depression symptom network in isolation, researchers used 12 different pre-existing scales, with the number of symptoms ranging from 9 to 28 (see online Supplementary Materials). This problem is not unique to the network approach (Fried, Reference Fried2017), but is especially problematic in this context given the emphasis on individual symptoms. It will thus be important to design studies expressly for the purpose of investigating mental disorders as complex systems. Second, to our knowledge no studies within this literature have experimentally manipulated individual symptoms in order to evaluate their impact on other symptoms. Even in the absence of network analysis, such studies are fundamentally important to our understanding of how symptoms relate to one another (Blanken et al., Reference Blanken, Van Der Zweerde, Van Straten, Van Someren, Borsboom and Lancee2019; Marsman et al., Reference Marsman, Borsboom, Kruis, Epskamp, van Bork, Waldorp and Maris2018). Finally, we believe that researchers (including authors of this article), must be more cautious when deriving hypotheses from network theory to be tested in empirical research. The behavior of complex systems is notoriously difficult to predict and the process of deriving hypotheses about the data models these systems should produce is likely prone to error. For instance, it is uncertain whether stronger inter-symptom causal relations will indeed produce greater connectivity in conditional dependence networks, as intuition would suggest. As detailed in the next section, we believe computational models are needed to more accurately derive the predictions made by network theory.
An agenda for future research
The central task for the next decade of the network approach is to build on the theoretical, methodological, and empirical foundations we have described here and advance our understanding of how specific mental disorders operate as causal systems. We believe that the path to this aim is clarified by viewing the network approach as an abductive program of research: one rooted in the twin aims of identifying robust empirical phenomena and developing formal theories that explain those phenomena (Haig, Reference Haig2005, Reference Haig2008).
Most empirical network studies can be understood as carrying out the initial stages of phenomena detection: using exploratory network analyses to uncover data patterns (e.g. a conditional positive manifold, community structure, or centrality ordering). Indeed, the most widely used methodological tools developed within this literature are especially valuable for this exploratory work, as they allow researchers to visualize and quantify complex dependencies in the data. Over the next decade, it will be necessary for empirical researchers to evaluate which among these exploratory findings are replicable and generalizable phenomena. Carrying out our recommendations for methodological development will support this effort, providing tools for aggregating findings across studies and for conducting confirmatory network analyses. It will be important for empirical researchers to use such methods in studies designed specifically for assessing symptom networks, ideally with large samples and time-series data that can support both between-subject and within-subject analyses (cf. Wichers, Wigman, Bringmann, & De Jonge, Reference Wichers, Wigman, Bringmann and De Jonge2017). These efforts will allow researchers to evaluate whether observed data patterns are sufficiently robust that they can inform and constrain the development of formal theories.
The development of network theory over the past decade has provided a theoretical framework that accounts for a fundamental psychiatric phenomenon: the tendency for symptoms to cohere as syndromes. Over the next decade, we must use this framework to develop computational models that posit precisely how specific mental disorders are thought to operate. Such models will explicate theory assumptions and provide a tool for evaluating what theories can and cannot explain, thus guiding their ongoing development. Moreover, these models will facilitate the recommendations for methodological and empirical work we have made in this review. For methodologists, computational models will provide theoretically plausible causal systems from which to generate data, equipping them to investigate the relation between network psychometric models and the causal systems we expect to see in psychiatric research. For empirical researchers, computational models can guide study design, clarifying the components to be investigated and the timeframe in which they should be assessed. Moreover, model simulations can more precisely derive theory predictions, thereby strengthening empirical tests of network theory hypotheses. In turn, the phenomena detected through empirical research can inform the advancement of network theories, clarifying the features of psychopathology for which they must account. Through this ongoing exchange between theoretical development and empirical research, formal theories can be evaluated and advanced, permitting genuine advances in our understanding of how mental disorders operate as causal systems.
Conclusion
The first decade of the network approach has been a period of considerable growth. An inter-disciplinary group of researchers developed the hypothesis that there are important causal relationships among symptoms into a theory regarding the nature of psychopathology, a nascent methodology for investigating symptom networks, and a growing body of empirical research. To build upon this work, we believe that theorists, methodologists, and empirical researchers must collaboratively work toward two overarching aims: (a) establishing robust empirical phenomena and (b) developing formal theories that can explain those phenomena. Progress toward these aims will be critical if the network approach is to be a cumulative program of research over the next decade: one that does not merely produce a growing number of empirical analyses, but also represents the accumulation of knowledge about psychopathology.
Supplementary material
The supplementary material for this article can be found at https://doi.org/10.1017/S0033291719003404.
Acknowledgements
This work was supported by a National Institute of Mental Health Career Development Award (1K23MH113805-01A1) awarded to D. Robinaugh, ERC Consolidator Grant no. 647209 awarded to D. Borsboom, and NWO Research Talent Grant no. 406-18-532 awarded to R. Hoekstra. We thank Alex Millner and Eiko Fried for their thoughtful feedback on an earlier draft of this manuscript.
