As developmental researchers, we agree with Yarkoni's assertion that descriptive research offers a potential solution to the generalizability crisis. Careful descriptions of behavior are foundational to psychological science, and especially critical for developmental science where theoretical progress relies on behavior due to children's limited verbal and motor skills. Many scientific fields have rich histories of descriptive work that drive theory building – Galileo's observations of celestial objects, Ramón y Cajal's depictions of neuron structures, and Golgi's cell visualizations. We argue that such descriptions (when done well) are more enduring and valuable than theories based on behaviorally impoverished data. We offer four suggestions to those who want to “take descriptive research more seriously” with examples from developmental science.
Prioritize context over control
Many researchers assume that to understand a psychological phenomenon, they must first distill it into its simplest form. After the fundamentals are established, the idea is that researchers will gradually add in layers of complexity until behavior in the lab resembles natural behavior. However, by prioritizing control over context, researchers may unwittingly sacrifice critical aspects of the original phenomena and risk reifying abstractions that do not generalize beyond a simplified setting. For example, decades of research on the development of walking focused on periodic gait – infants’ ability to walk in a straight line over flat ground at a constant speed (see Adolph & Robinson, Reference Adolph, Robinson and Zelazo2013 for review). Although this simplification enabled researchers to carefully measure infant walking skill, infants rarely walk that way. Instead, at every point in development, infants take omnidirectional steps along curved paths in short activity bursts (Lee, Cole, Golenia, & Adolph, Reference Lee, Cole, Golenia and Adolph2018). Work with simulated robots highlights the consequences of “controlling for” these critical aspects of real-world walking. Compared to robots that learned to walk with less variable paths, robots trained with more infant-like, variable paths displayed more functional walking (Ossmy et al., Reference Ossmy, Hoch, MacAlpine, Hasan, Stone and Adolph2018). Thus, prioritizing context over control can help researchers capture the aspects of phenomena that are necessary for generalization.
Let naturalistic observations guide and contextualize structured observations
Researchers’ decisions about which methods to use can powerfully shape study outcomes. This is particularly true for infants whose behavior is easily influenced by the environment – who knew, for example, that superfluous sounds can increase infants’ attention in looking-time studies? (Spelke, Reference Spelke, Gottlieb and Krasnegor1985). Indeed, there is power in methods. Caregivers talk far more to their infants during structured play with standardized toys than during daily routines in the home (Tamis-LeMonda, Kuchirko, Luo, & Escobar, Reference Tamis-LeMonda, Kuchirko, Luo and Escobar2017). And caregivers’ speech is constant during structured play, whereas it ebbs and flows during natural activity. Thus, it is critical for researchers to consider the “facts on the ground” from naturalistic observations as they design, interpret, and generalize data from artificially constructed experimental situations. At minimum, researchers should take care to interpret data from structured tasks as reflecting what infants can do – but not necessarily what actually happens in real-world settings.
Rigorously and transparently operationalize behaviors of interest
Researchers should operationalize descriptions of behavior to be robust, straightforward, and transparent. Operational definitions can be tricky. Psychologists typically study higher-order (latent) constructs and may be tempted to quantify constructs by taking a “you-know-it-when-you-see-it” approach, rating the phenomena on an ordinal scale, using yes/no codes, and so on. But gestalt approaches require extensive training to identify constructs reliably and leave future researchers with little information about what participants actually did. Instead, researchers should quantify the actual behaviors. To illustrate, a series of studies documented infants’ perception of affordances – whether infants perceive drop-offs and slopes as safe or risky (Adolph & Hoch, Reference Adolph and Hoch2019). Perception of affordances is a higher-level construct that could be scored as yes or no, but researchers measured it with directly observable behaviors such as whether infants attempted to cross, hesitated at the edge, explored the precipice by looking or touching, and displayed negative facial expressions. Such an approach generates a rich description of what happened, including behaviors that may be surprising when considering the abstract construct (e.g., infants rarely display negative emotions when avoiding a risky precipice). Importantly, behavioral descriptions retain their value and will be interpretable to future scientists, whereas higher-level constructs survive only as long as those constructs retain favor.
Attend to individual data
Inter- and intra-individual variability are endemic in development and highly illustrative: Over development, variability can increase or decrease, and the structure of variability can change (Adolph, Cole, & Vereijken, Reference Adolph, Cole, Vereijken, Diehl, Hooker and Sliwinski2015). Thus, ignoring variability can obscure the true nature of phenomena and render generalizations uninformative. Variability is more than measurement error or noise. Rather, understanding each individual's behavior yields better insight into the true nature of the phenomena and can inform mechanisms of change – the nature of the behavior is different if the pattern holds for 95% versus 55% of the sample (Vereijken, Reference Vereijken2010). We propose that prior to hypothesis testing with inferential statistics, researchers interrogate each participant's data to assure themselves that group-level effects are truly representative. They should use descriptive statistics and simple visualizations to understand the raw data before engaging in complex analyses. Further, to assure readers that results are truly representative, plots should show how individual data comprise group differences (e.g., Fig. 1).
Figure 1. Depictions of individual data that comprise differences between groups or conditions. (A) Infants cover more ground in a toy-filled room than in an empty room. Each plot shows one infant's locomotor path through the toy-filled room (purple) and the empty room (gold) ordered from most to least area covered in m2. (B) Infants (square symbols) move more than their mothers (triangular symbols) during free play. Gray bars connect each dyad. (C) Infants spontaneously explore objects more frequently while standing (red circles) than while walking (blue symbols) during free play. Each pair of symbols shows one infant's data. Inset shows differences across the group. Infants’ propensity to explore objects did not differ by infant age (left panel) or walking experience (right panel).
Notably, momentum is building in developmental science for large-scale collaborative data collection initiatives, with potential to produce highly generalizable descriptive datasets. Indeed, the Play & Learning Across a Year (PLAY) project leverages 70 labs across North America – with expertise in locomotion, object interaction, emotion, language, gender, environment, and more – to design a common protocol to collect and code videos of 1000+ mothers and infants during natural activity in the home. The data are then shared, so each expert can generate descriptions of behavior that address their own research interests.
As Yarkoni attests, psychology historically focused on testing theories that often fail to generalize to real-world settings. Looking forward, we contend that psychological science should focus on careful, rich descriptions of behavior. Although our suggestions for conducting generalizable descriptive research stem from developmental science, we believe these principles apply broadly across psychological science.
As developmental researchers, we agree with Yarkoni's assertion that descriptive research offers a potential solution to the generalizability crisis. Careful descriptions of behavior are foundational to psychological science, and especially critical for developmental science where theoretical progress relies on behavior due to children's limited verbal and motor skills. Many scientific fields have rich histories of descriptive work that drive theory building – Galileo's observations of celestial objects, Ramón y Cajal's depictions of neuron structures, and Golgi's cell visualizations. We argue that such descriptions (when done well) are more enduring and valuable than theories based on behaviorally impoverished data. We offer four suggestions to those who want to “take descriptive research more seriously” with examples from developmental science.
Prioritize context over control
Many researchers assume that to understand a psychological phenomenon, they must first distill it into its simplest form. After the fundamentals are established, the idea is that researchers will gradually add in layers of complexity until behavior in the lab resembles natural behavior. However, by prioritizing control over context, researchers may unwittingly sacrifice critical aspects of the original phenomena and risk reifying abstractions that do not generalize beyond a simplified setting. For example, decades of research on the development of walking focused on periodic gait – infants’ ability to walk in a straight line over flat ground at a constant speed (see Adolph & Robinson, Reference Adolph, Robinson and Zelazo2013 for review). Although this simplification enabled researchers to carefully measure infant walking skill, infants rarely walk that way. Instead, at every point in development, infants take omnidirectional steps along curved paths in short activity bursts (Lee, Cole, Golenia, & Adolph, Reference Lee, Cole, Golenia and Adolph2018). Work with simulated robots highlights the consequences of “controlling for” these critical aspects of real-world walking. Compared to robots that learned to walk with less variable paths, robots trained with more infant-like, variable paths displayed more functional walking (Ossmy et al., Reference Ossmy, Hoch, MacAlpine, Hasan, Stone and Adolph2018). Thus, prioritizing context over control can help researchers capture the aspects of phenomena that are necessary for generalization.
Let naturalistic observations guide and contextualize structured observations
Researchers’ decisions about which methods to use can powerfully shape study outcomes. This is particularly true for infants whose behavior is easily influenced by the environment – who knew, for example, that superfluous sounds can increase infants’ attention in looking-time studies? (Spelke, Reference Spelke, Gottlieb and Krasnegor1985). Indeed, there is power in methods. Caregivers talk far more to their infants during structured play with standardized toys than during daily routines in the home (Tamis-LeMonda, Kuchirko, Luo, & Escobar, Reference Tamis-LeMonda, Kuchirko, Luo and Escobar2017). And caregivers’ speech is constant during structured play, whereas it ebbs and flows during natural activity. Thus, it is critical for researchers to consider the “facts on the ground” from naturalistic observations as they design, interpret, and generalize data from artificially constructed experimental situations. At minimum, researchers should take care to interpret data from structured tasks as reflecting what infants can do – but not necessarily what actually happens in real-world settings.
Rigorously and transparently operationalize behaviors of interest
Researchers should operationalize descriptions of behavior to be robust, straightforward, and transparent. Operational definitions can be tricky. Psychologists typically study higher-order (latent) constructs and may be tempted to quantify constructs by taking a “you-know-it-when-you-see-it” approach, rating the phenomena on an ordinal scale, using yes/no codes, and so on. But gestalt approaches require extensive training to identify constructs reliably and leave future researchers with little information about what participants actually did. Instead, researchers should quantify the actual behaviors. To illustrate, a series of studies documented infants’ perception of affordances – whether infants perceive drop-offs and slopes as safe or risky (Adolph & Hoch, Reference Adolph and Hoch2019). Perception of affordances is a higher-level construct that could be scored as yes or no, but researchers measured it with directly observable behaviors such as whether infants attempted to cross, hesitated at the edge, explored the precipice by looking or touching, and displayed negative facial expressions. Such an approach generates a rich description of what happened, including behaviors that may be surprising when considering the abstract construct (e.g., infants rarely display negative emotions when avoiding a risky precipice). Importantly, behavioral descriptions retain their value and will be interpretable to future scientists, whereas higher-level constructs survive only as long as those constructs retain favor.
Attend to individual data
Inter- and intra-individual variability are endemic in development and highly illustrative: Over development, variability can increase or decrease, and the structure of variability can change (Adolph, Cole, & Vereijken, Reference Adolph, Cole, Vereijken, Diehl, Hooker and Sliwinski2015). Thus, ignoring variability can obscure the true nature of phenomena and render generalizations uninformative. Variability is more than measurement error or noise. Rather, understanding each individual's behavior yields better insight into the true nature of the phenomena and can inform mechanisms of change – the nature of the behavior is different if the pattern holds for 95% versus 55% of the sample (Vereijken, Reference Vereijken2010). We propose that prior to hypothesis testing with inferential statistics, researchers interrogate each participant's data to assure themselves that group-level effects are truly representative. They should use descriptive statistics and simple visualizations to understand the raw data before engaging in complex analyses. Further, to assure readers that results are truly representative, plots should show how individual data comprise group differences (e.g., Fig. 1).
Figure 1. Depictions of individual data that comprise differences between groups or conditions. (A) Infants cover more ground in a toy-filled room than in an empty room. Each plot shows one infant's locomotor path through the toy-filled room (purple) and the empty room (gold) ordered from most to least area covered in m2. (B) Infants (square symbols) move more than their mothers (triangular symbols) during free play. Gray bars connect each dyad. (C) Infants spontaneously explore objects more frequently while standing (red circles) than while walking (blue symbols) during free play. Each pair of symbols shows one infant's data. Inset shows differences across the group. Infants’ propensity to explore objects did not differ by infant age (left panel) or walking experience (right panel).
Notably, momentum is building in developmental science for large-scale collaborative data collection initiatives, with potential to produce highly generalizable descriptive datasets. Indeed, the Play & Learning Across a Year (PLAY) project leverages 70 labs across North America – with expertise in locomotion, object interaction, emotion, language, gender, environment, and more – to design a common protocol to collect and code videos of 1000+ mothers and infants during natural activity in the home. The data are then shared, so each expert can generate descriptions of behavior that address their own research interests.
As Yarkoni attests, psychology historically focused on testing theories that often fail to generalize to real-world settings. Looking forward, we contend that psychological science should focus on careful, rich descriptions of behavior. Although our suggestions for conducting generalizable descriptive research stem from developmental science, we believe these principles apply broadly across psychological science.
Financial support
Work on this article was supported by NICHD F32 DC017903 to Kelsey West and by NICHD R01 HD-094830, NICHD R01 HD-033486, NICHD R01 HD086034, and DARPA N66001-19-2-4035 to Karen Adolph.
Conflict of interest
None.