Thanks in part to major international efforts like the United Nations Millennium Development Goals, over 90% of children worldwide now have access to primary education (United Nations Development Programme [UNDP], 2014). However, commensurate gains have yet to reach children in conflict-affected countries (CACs).Footnote 1 While children in CACs make up 22% of primary school-aged children worldwide, they account for fully 50% of primary school-aged children without access to education (Education for All, 2013). Moreover, children in conflict-affected areas who are in school are not learning. For example, our own research in the Democratic Republic of the Congo (DRC) indicates that 91% of primary school children in Grades 2–4 could not correctly respond to even one reading comprehension question of the Early Grade Reading Assessment (EGRA), a test designed specifically for use in low- and middle-income countries (Torrente, Aber, Shivshanker, Annan, & Bundervoet, 2013).
Emergent theory and research indicate that equitable access to quality education can mitigate some of the most severe consequences of conflict for children and potentially help break the intergenerational transmission of poverty and violence through the effective provision of safe and supportive spaces that promote children's academic and socioemotional development (Betancourt & Williams, Reference Betancourt and Williams2008; Burde, Guven, Kelcey, Lahmann, & Al-Abbadi, Reference Burde, Guven, Kelcey, Lahmann and Al-Abbadi2015; Winthrop & Kirk, Reference Winthrop and Kirk2008). To date, however, practitioners have been limited in their ability to develop and implement such programs due in part to the “stunning lack of evidence” as to what works to promote children's learning and development in the context of conflict and crisis (Jordans, Pigott, & Tol, Reference Jordans, Pigott and Tol2016; Masten & Narayen, Reference Masten and Narayan2012). We were able to identify fewer than 20 evaluations of programs to promote children's academic learning or socioemotional development in CACs that allow for causal inference through the use of a randomly assigned control group (Burde, Guven, Kelcey, Lahmann, & Al-Abbadi, Reference Burde, Guven, Kelcey, Lahmann and Al-Abbadi2015). In addition, review articles across different disciplines have criticized “the complete lack of treatment mechanisms research” (Jordans, Tol, Komproe, & de Jong, 2008, p. 10) to date on interventions for children in CACs (US Agency for International Development, 2013). Such research, a variant of mediational mechanisms research in developmental science, is critically important in order promote stakeholders’ ability to strengthen and replicate the mechanisms of the intervention that do work, and in turn develop the effective and scalable interventions urgently needed for children in CACs (Yoshikawa & Roy, Reference Yoshikwa and Roy2014).
In this paper, we draw on principles from developmental psychopathology and school ecology to contribute to a nascent evidence base on how to promote the quality of education and children's learning in CACs. We employ data from the first year of the International Rescue Committee's (IRC's) Learning to Read in a Healing Classroom (LRHC) intervention—the only large-scale cluster randomized trial of a universal school-based socioemotional learning program in a CAC to date—in the DRC, in order to provide initial evidence for a dynamic, multilevel theory of change in which proximal changes in the quality of school and classroom ecologies lead to distal changes in multiple domains of children's learning and development, including academic skills (literacy and numeracy) and socioemotional functioning (mental health problems and victimization; see Figure 1). Specifically, building on our prior work quantifying the impact of 1 year of exposure to LRHC on teacher motivation (Wolf, Aber, & Torrente, Reference Wolf, Aber and Torrente2016), school and classroom social and pedagogical interactions between students and teachers (Torrente et al., Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015), student socioemotional outcomes (Torrente et al., Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015), and student academic outcomes (Aber et al., Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolf2016), we employ multilevel structural equation modeling techniques to examine whether LRHC caused changes in school and classroom processes that are associated with students’ academic and socioemotional outcomes after 1 year of implementation.
Figure 1. Hypothesized Learning to Read in a Healing Classroom evaluation theory of change.
We first discuss education and education initiatives in CACs in order both to contextualize the current study and to underscore the importance of quality education in CACs. We then summarize the LRHC intervention and program impacts to date, with a focus on describing the multiple-stakeholder process of applying developmental and social–ecological theory in order to hypothesize a multilevel theory of change by which an existing program may impact children's outcomes. Finally, we briefly review evidence from high-income countries in support of the hypothesized pathways in our theory of change, followed by our research questions.
Education in CACs
Current state of access to and quality of education in CACs
Children living in CACs are the most educationally disadvantaged in the world, as defined by both access to and quality of education (UNDP, 2014). The barriers to education access in CACs are not only economic—children living in CACs are three times more likely to be out of school than children in low-income countries—but also a compound failure of poor or unstable governance, security threats, and lack of management and service delivery systems (International Network for Education in Emergencies, 2014). Such disruptions in educational access due to violent conflict have been shown to have significant detrimental effects on future development, in particular on long-term educational attainment, health, and earnings (Akbulut-Yuksel, Reference Akbulut-Yuksel2009; Justino, Reference Justino2010).
Increasing the probability of long-term negative consequences, as access to education falters, so too does quality of education and children's learning outcomes. Armed conflict diverts funds away from quality improvement initiatives; increases teacher, administrator, and service provider absenteeism and availability; and creates psychological distress and fear (International Network for Education in Emergencies, 2014). While microlevel data on school quality and learning outcomes in CACs is extremely scarce (Montjourides, Reference Montjourides2013), our own research suggests that children in CACs are not learning. In addition to the above cited statistics from the DRC, on average 70% of Syrian refugee children ages 5–14 enrolled in community-based education in informal tented settlements in Lebanon could not at baseline read a letter of Arabic, the language of instruction in Syria (IRC, 2014).
The role of education in CACs
Only in the past two decades have humanitarian and development organizations, policymakers, and funders begun to recognize the importance of protecting and promoting education as part of humanitarian responses to conflict situations (for a discussion, see Burde et al., Reference Burde, Guven, Kelcey, Lahmann and Al-Abbadi2015). While nongovernmental organizations now provide education programming in over 80 CACs around the world, several challenges still disrupt the timely and comprehensive provision of education services to children in CACs, including lack of funding (Save the Children, 2015), the perception that education is an economic development, not humanitarian, activity (Winthrop, Ndaruhutse, Dolan, & Adams, Reference Winthrop, Ndaruhutse, Dolan and Adams2010), and the perception that providing education programs is too risky in conflict-affected settings (UNESCO, 2011). Yet evidence from both developmental science and political science suggests that education supports can play a transformative role for children and nations affected by armed conflict.
As discussed by Cummings et al. in the editorial for this Special Section, developmental science research indicates that direct and indirect exposure to armed conflict and/or political violence may pose enduring threats to children's physical, academic, and socioemotional development (Cummings, Merrilees, Taylor, & Mondi, Reference Cummings, Merrilees, Taylor and Mondi2016; Devakumar, Birch, Osrin, & Sondorp, Reference Devakumar, Birch, Osrin and Sondorp2014; United Nations Children's Fund [UNICEF], 2009a). Yet research in developmental psychopathology and related disciplines also indicates that children can demonstrate remarkable resilience to such adversities through a complex confluence of developmental timing (e.g., age and stage differences) and protective and promotive factors at myriad ecological levels: from the neurobiological to the individual to the family, school, and community (Cicchetti, Reference Cicchetti2016; Masten & Narayen, Reference Masten and Cicchetti2012).
Safe and supportive quality school settings may be particularly important protective factors to develop in CACs given that such settings are both amenable to change and have the potential to reach a large population of children across multiple developmental stages. School experiences can bring a degree of normalcy and predictability to children's lives, partially mitigate the negative effects that exposure to armed conflict and accompanying life stressors have on their mental health, and promote the multiple skills necessary for future productivity and well-being (Betancourt & Williams, Reference Betancourt and Williams2008; Burde et al., 2016; de Jong, Reference de Jong2010; Mosselson, Wheaton, & Frisoli, Reference Mosselson, Wheaton and Frisoli2009; UNICEF, 2009a; Winthrop & Kirk, Reference Winthrop and Kirk2008). Moreover, the absence of quality school settings may actually risk the development of the prejudices and intergroup tensions that can lead to future violence (King, Reference King2013; Thyne, Reference Thyne2006; Walter, Reference Walter2004) This suggest a dual role for education in CACs: whereas equitable access to and quality of education may promote positive development for children and sustainable futures for nations, the lack of access to and quality of education supports can increase the risk for future individual and national difficulties.
Strategies to promote the quality of education in CACs
Recognizing this duality, national and international organizations have worked over the past decade to overcome barriers to education in CACs by incorporating education programming and practices into humanitarian responses (Burde et al., Reference Burde, Guven, Kelcey, Lahmann and Al-Abbadi2015). Given that the focus of this paper is an intervention to improve the quality of school settings, we review here the evidence base on programs designed to be incorporated into existing schools (either formal or nonformal community-based systems) to promote children's learning and development (for the reader interested in programs to improve access to education in CACs, see Betancourt et al., Reference Betancourt, Simmons, Borisova, Brewer, Iweala and De La Soudière2008; Burde & Linden, Reference Burde and Linden2013).Footnote 2
Programs to promote the quality of education in CACs are typically designed either to address delays and insults to children's academic and socioemotional development that occurred as a result of exposure to conflict or to promote children's resilience and normative academic and socioemotional development. A diverse set of strategies targeting different elements of school settings are used to accomplish these goals, the most common of which are (a) teacher training programs, to support teachers in using instructional and classroom practices that promote students’ academic well-being (e.g., Davidson & Hobbs, Reference Davidson and Hobbs2013); (b) accelerated learning or “catch up” programs, to prepare students who have missed 2 or more years of education to enter schools at developmentally age-appropriate levels (Betancourt et al., Reference Betancourt, Simmons, Borisova, Brewer, Iweala and De La Soudière2008; Dryden-Peterson, Reference Dryden-Peterson2011); (c) peace education programs, to increase contact and understanding between different ethnic/racial groups (e.g., Bar-Tal & Rosen, Reference Bar Tal and Rosen2009); and (d) psychosocial and mental health programs, to treat children who are at risk of or display elevated symptoms of depression, anxiety, and posttraumatic stress disorder (e.g., Jordans et al., Reference Jordans, Tol, Komproe and de Jong2009). Despite evidence from developmental psychopathology of the potential for cross-domain and developmental cascade effects (Masten & Cicchetti, Reference Masten and Cicchetti2010), these programs most commonly aim to change only one domain of children's functioning (i.e., either academic skills or socioemotional well-being) at a time.
While the four types of programs described above primarily have been evaluated in CACs using qualitative research designs, rigorous causal evidence is emerging for three types of programs (teacher training, peace education, and mental health initiatives) based on recent efficacy trials. (For teacher training programs in Kenya, Mali, the Phillipines, and Uganda, see McEwan, Reference McEwan2013; for peace education, see Burde et al., 2016; and for psychosocial and mental health initiatives, see Jordans et al., Reference Jordans, Tol, Komproe and de Jong2009; Tol et al., Reference Tol, Barbui, Galappatti, Silove, Betancourt, Souza and Van Ommeren2011.) Although individual interventions within each category show promise, it is difficult to draw any definitive conclusions about broad program effectiveness due to a large degree of heterogeneity in intervention impacts. This likely results from factors such as variations in intervention content, delivery, and implementation; limitations in study conceptualization or design (e.g., lack of theory of change or small sample sizes); inconsistent, differential, or unreliable/invalid measurement of outcomes; differences in social and political context (e.g., low-income countries vs. middle-income countries); and timing of intervention and evaluation (e.g., during conflict vs. postconflict).
Moreover, we were only able to identify one study in a CAC that hypothesized and tested possible mechanisms or mediating processes by which the intervention likely effected change in children's learning and development outcomes (Tol et al., Reference Tol, Komproe, Jordans, Gross, Susanty, Macy and de Jong2010). Such tests are key for making sense of null findings, as well as for contributing to our understanding of children's development. If an intervention changes a hypothesized mediating process but not a distal child outcome, we can conclude that either (a) not enough time has passed to see change in the child outcome or (b) change in the mediating process is not associated with change in the child outcome as predicted and may not be a useful target for intervention. If an intervention does not change the hypothesized mediating process but this process is associated with change in the child outcome, we can conclude that either (a) the mediating process is not a malleable target for that particular intervention or (b) the intervention did not work as expected and requires adjustment. In turn, such pieces of information are critical for enabling researchers and practitioners to better design and evaluate interventions.
In the remainder of this paper, we provide evidence for an intervention that bridges some of the above-described limitations in design and evaluation. The LRHC intervention integrates strategies from the three major types of quality improvement programs in CACs: it uses a teacher training program (Strategy a) to promote multiple domains of children's functioning, including academic skills (Strategy b), victimization (Strategy c), and mental health (Strategy d). The evaluation is guided by the application of both developmental psychopathology and social–ecological perspectives (Cummings et al., 2015), thereby enabling reliable measurement and tests of the intermediary processes by which and contexts in which the intervention works, as we describe below.
LRHC in the DRC: Theory of Change and Impacts
Theory of change and supporting evidence
The LRHC intervention is a universal school-based program developed by the IRC that uses a teacher professional development system to improve primary school-aged children's academic skills and socioemotional development in CACs (see Content section for more information). When the IRC approached researchers at New York University about evaluating LRHC in the DRC, they were motivated by a desire to generate rigorous evidence as to what might happen in schools and to children as a result of the LHRC program, versions of which they are using currently in more than 12 countries. Stakeholders from the IRC and New York University thus collaborated on developing a multilevel program theory of change (see Figure 1) hypothesizing the mechanisms by which LRHC operated to change children's outcomes. The research team then used the theory of change to guide the design and conduct of the evaluation (for other examples of such a process, see Cappella, Masseti, & Yampolskey, Reference Cappella, Massetti, Yampolsky and Dinella2009; and Jones, Brown, & Aber, Reference Jones, Brown and Aber2011). To develop this theory of change, we applied two meta-theories from developmental science—bioecological models of human development and developmental psychopathology, in particular developmental cascades theory—as well as examined corresponding evidence from education research in high-income countries.
First, bioecological models of human development propose that children's development (a) unfolds within a nested set of ecologies, and (b) is primarily driven by the transactions and proximal social processes that occur within children's daily ecologies or microsystems, which prominently involve classrooms and schools during middle childhood (Bronfenbrenner & Morris, Reference Bronfenbrenner, Morris and Lerner2006; Tseng & Seidman, Reference Tseng and Seidman2007). Thus, although the LRHC intervention targeted teacher resources and practices, bioecological theory suggests that what is important is the extent to which these resources and practices lead to the supportive transactional social and pedagogical processes within children's classrooms and schools that underlie children's development and learning (see Figure 1, Path a → Paths e and f, Path b → Paths g and h). Second, developmental psychopathology (in general) and developmental cascades theory (in particular) indicate that domains of children's functioning exert both bidirectional and progressive influences on each other (Masten et al., Reference Masten, Roisman, Long, Burt, Obradović, Riley and Tellegen2005). Accordingly, we hypothesized that the LRHC intervention, which targeted teachers’ inclusion of social and emotional learning principles into reading instruction, would impact both children's academic skills and their socioemotional well-being (Paths c & d). Moreover, it is likely that improving functioning in one domain (e.g., socioemotional well-being) may lead to changes in other domains (academic skills; Path i). Third, we expected that changes resulting from LRHC would be progressive, with impacts first seen in processes most proximal to the intervention (e.g., improvements in classroom and school ecologies) and later observed in more distal outcomes (e.g., academic and socioemotional outcomes).
Correlational, longitudinal, and experimental evidence from high-income countries supports the pathways in our theory of change (for overviews on evidence supporting specific mediating and outcome constructs, see Aber et al., Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolfin press; and Torrente et al., Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015, Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aberin press). First, a recent meta-analysis of over 200 school-based social–emotional learning (SEL) programs in the United States and other high-income countries shows that such programs are a viable and effective approach to improving both academic and socioemotional outcomes (Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, Reference Durlak, Weissberg, Dymnicki, Taylor and Schellinger2011; see Figure 1, Paths c and d). Experimental evidence from the United States also indicates that teacher professional development programs with a socioemotional focus improve classroom quality (Allen et al., Reference Allen, Gregory, Mikami, Lun, Hamre and Pianta2013; see Figure 1, Paths a and b). For example, an experimental evaluation of the 4Rs intervention, an SEL literacy curricular and teacher training program implemented in 18 New York City primary schools, detected improvements in classroom quality (as measured by behavioral observations of classroom emotional support, instructional support, and management) on a magnitude of 0.66 SD after 1 year (Brown, Jones, LaRusso, & Aber, Reference Brown, Jones, LaRusso and Aber2010).
In turn, classrooms characterized by trust, warmth, and a low level of conflict between teachers and students are associated with higher academic engagement and achievement, concurrently and over the long term (Cameron, Connor, Morrison, & Jewkes, Reference Cameron, Connor, Morrison and Jewkes2008; Hughes, Luo, Kowk, & Loyd, Reference Hughes, Luo, Kwok and Loyd2008; Ponitz, Rimm-Kaufman, Grimm, & Curby, Reference Ponitz, Rimm-Kaufman, Grimm and Curby2009; see Figure 1, Path a → Path e). Schools and classrooms that have predictable routines and behavioral expectations, and where students enjoy cooperative relationships with their teachers and peers, predict higher levels of motivation for learning and academic attainment (Cameron et al., Reference Cameron, Connor, Morrison and Jewkes2008; Merritt, Wanless, Rimm-Kaufman, Cameron, & Peugh, Reference Merritt, Wanless, Rimm-Kaufman, Cameron and Peugh2012; Ponitz et al., Reference Ponitz, Rimm-Kaufman, Grimm and Curby2009; Reyes, Brackett, Rivers, Whie, & Salovey, Reference Reyes, Brackett, Rivers, White and Salovey2012; Wilson, Pianta, & Stuhlman Reference Wilson, Pianta and Stuhlman2007; see Figure 1, Path b → Path g). Moreover, emotionally supportive classrooms have been linked to a range of positive mental health outcomes, including improved social competence, life satisfaction, and behavioral self-control, as well as reduced depression, anxiety, and aggression (Merrit et al., Reference Merritt, Wanless, Rimm-Kaufman, Cameron and Peugh2012; Suldo et al., Reference Suldo, Friedrich, White, Farmer, Minch and Michalowski2009; Wilson et al., Reference Wilson, Pianta and Stuhlman2007; see Figure 1, Path a → Path f, Path b → h). While support for these hypothesized pathways in LRHC's theory of change was available from high-income countries, such pathways had never been tested in low-income and CACs.
LRHC Year 1 impacts and research question
Aber et al. (Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolfin press) and Torrente et al. (Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015) leverage the random assignment of LRHC to clusters of schools in order to estimate the causal impact of LRHC on children's perceptions of school and classroom pedagogical and social processes and children's academic and socioemotional outcomes. The results after 1 year of implementation indicate that LRHC caused significant improvements in children's reading and math scores (Path c; Aber et al.), but had no impact on their mental health symptoms or their experiences of victimization (Path d, Torrente et al.). LRHC also changed children's perceptions of their school ecologies, improving how caring and supportive they felt their teachers and schools to be (Path a), but decreasing how predictable and cooperative they perceived their learning environments to be (Path b; Torrente et al.).
In this paper, we move beyond asking whether LRHC directly impacted school-, classroom-, and child-level processes and outcomes to examine, in accordance with our theory of change, how LRHC may have indirectly impacted children's outcomes. That is, prior analyses considered only whether LRHC impacted one outcome at a time. However, our theory of change hypothesizes that LRHC operated within an ecological system in which changes in multiple school and classroom mediating processes occurred simultaneously, with potentially cascading and differential effects on children's outcomes. Thus, in this paper we empirically model our theory of change in order to address our primary research question: are LRHC-induced changes in school and classroom pedagogical and social processes associated with children's academic and socioemotional outcomes after 1 year of intervention implementation? While this analysis does not establish that the intervention-induced changes in school and classroom processes in turn caused changes in children's outcomes, it does provide much needed support for the plausibility of these pathways through which an effort to improve the quality of education in a CAC may have improved children's learning and development.
Methods
Context
The DRC, the second largest country on the African continent, ranks next to last in the world on the Human Development Index, an indicator of well-being that combines measures of life expectancy, educational attainment, and income (UNDP, 2014).Footnote 3 The dramatically low levels of social and economic development are in part due to decades of ongoing political instability and violence. Between 1998 and 2003, the DRC was at the center of Africa's World War, a conflict involving nine African countries and 20 armed groups that resulted in an estimated 5.4 million deaths. Despite a series of peace agreements, eastern regions of the Congo where this intervention and evaluation took place, including Katanga, North Kivu, and South Kivu, are still volatile and subject to ongoing political violence and armed conflict (BBC, 2015).
Content
In this context, the US Agency for International Development, in partnership with the IRC and in collaboration with the DRC's Ministry of Primary, Secondary, and Professional Education, mounted a systematic initiative known as Opportunities for Equitable Access to Quality Basic Education between 2011 and 2016.Footnote 4 This broader initiative included the implementation and evaluation in the first year (2011–2012) of the LRHC intervention. As described fully in Torrente et al. (Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015), the LRHC teacher professional development system has two components: (a) integrated teacher resource materials, and (b) collaborative school-based Teacher Learning Circles.Footnote 5
Research design
Data used in this study are from the first and second waves of a multiyear evaluation of LRHC in four subdivisions of the Katanga province in eastern DRC. The evaluation employed a cluster-randomized trial with a wait-list control design, whereby clusters of two to six schools, rather than individual schools, students, or teachers, were the unit of randomization. This paper compares clusters that were randomly assigned to receive the program in 2011–2012 to clusters that had not yet received the program (i.e., clusters randomly assigned to begin receiving the program in 2012–2013 or 2013–2014).
School sample
A total of 153 schools in four educational subdivisions in Katanga were targeted to participate in the program. These schools were organized in 40 school clusters of 2 to 6 schools, based on geographical proximity. The evaluation randomly selected a sample of 64 schools out of 153 to participate in data collection. Given unequal cluster sizes, 1 school was selected from clusters that contained 3 schools or fewer, and 2 schools were selected from clusters containing more than 3 schools. All the schools agreed to participate in the evaluation. However, 1 was excluded from the analysis due to a research management error that led to unreliable data for that school. Thus, the effective sample for this study includes 63 schools nested in 39 clusters. Twenty clusters (j = 33 schools) were assigned to receiving the program in 2011, and 19 clusters (j = 30 schools) were assigned to 1- and 2-year wait-list control conditions. In 2011 (baseline year), sample schools had an average of 389.75 students (SD = 234.97; min. = 82, max. = 1,130) and 8.03 classrooms (SD = 2.98; min. = 5, max. = 16). The majority of schools were Protestant (34.4%) or Catholic (31.3%); other religious affiliations included Orthodox (9.4%), Kimbanguiste (3.1%), and Muslim (1.6%). On average, 71.58% of the teachers were male, as were nearly all school principals (93.7%).
Student sample
Students in the second, third, and fourth grades were randomly selected by field research staff from school rosters to participate in the evaluation. In the second year, a minimum of 30 and a maximum of 81 students per school were assessed in the 63 sample schools. Data from this study uses the samples reported previously elsewhere, resulting in data from a total of 4,142 students (Aber et al., Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolfin press; Torrente et al., Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015). All students were assigned to complete the school quality and SEL measures; however, in order to reduce participant burden, students in each school were randomly assigned to complete either the Early Grade Math Assessment (EGMA) or the EGRA. Of those students, we could accurately match EGMA and school quality/SEL data for 1,728 students and EGRA and school quality/SEL data for 1,650 students. An additional 18.4% (n = 764) of students had data for school quality/SEL, EGMA, or EGRA. To determine if this data was differentially missing by treatment condition, we fit a set of multilevel logistic regressions in which missingness was predicted by treatment status. We found no evidence to indicate differential missingness across treatment conditions, suggesting that missingness does not pose a threat to the study's internal validity. As such, and given that the majority of missing data was planned and missing at random, all analyses herein adjust for missing data using full information maximum likelihood estimation. This statistical approach makes use of all available data points, even for cases with missing responses, by utilizing missing data patterns as well as by using all of the information from the data points available during parameter estimation (Muthén, Kaplan, & Hollis, Reference Muthén, Kaplan and Hollis1987). Descriptive data for the students in the sample is provided in Table 1.
Table 1. Descriptive statistics and bivariate correlations for child- and school-level variables
*p < .10.
Measures
We note that the mediating and outcome variables described below were both assessed at the same time, after the first year of program implementation.Footnote 6 However, all models adjust for baseline school-level measures of the mediating and outcome variables; these baseline measures were collected prerandomization.
Mediating processes: Children's perceptions of school and classroom ecologies
Supportive schools and teachers
Students’ perceptions of support were assessed using 17 items from two previously validated measures, which asked students about how welcome, included, intellectually engaged, and emotionally supported they felt at school (UNICEF, 2009b). The remaining 3 items came from the Relationship With Teacher Questionnaire (Blankemeyer, Flannery, & Vazsonyi, Reference Blankemeyer, Flannery and Vazsonyi2002). For all items, children indicated how true or untrue the items were using a 4-point Likert scale ranging from 0 (completely false) to 3 (completely true). A single score was obtained by averaging all of the items (α = 0.91).
Predictable and cooperative learning environments
Students’ perceptions of predictability and cooperation were measured with 10 items developed by the authors. The items assessed children's knowledge of their school routines, the extent to which teachers encouraged cooperation, and whether peers were supportive and shared activities and materials with each other. Students used a 4-point Likert scale ranging from 0 (never) to 3 (always). All items were averaged to create a single score (α = 0.86).
Student outcomes: academic skills
Factor analysis and scoring techniques were employed to estimate student proficiency on reading and math using the EGMA and EGRA subtests, respectively (Halpin, Torrente, & Aber, Reference Halpin, Torrente and Aber2016).
Reading skills
Students’ reading skills were assessed using the EGRA (RTI International, 2009a), which has been used to date in 31 different countries including the DRC. The EGRA takes about 15 min per child to administer and consists of up to nine orally administered subtests: vocabulary, initial sound identification, knowledge of graphemes, familiar word reading, invented word reading, oral passage reading fluency, reading comprehension, listening comprehension, and writing of a complete sentence. Factor analyses yielded one factor for students’ reading skills, which we refer to as overall reading proficiency or literacy.
Math skills
Students’ math skills were assessed using the EGMA (RTI International, 2009b). The EGMA also takes about 15 min per child to administer and consists of up to nine orally administered subtests: number identification, quantity discrimination, missing number/pattern completion, word problems, addition/subtraction/multiplication problems, shape identification/geometry, and shape naming/geometry. Factor analyses yielded two distinct but correlated factors: one factor for addition/subtraction and one factor for geometry.
Student outcomes: Socioemotional well-being
Victimization
Five items adapted from the Aggression, Victimization, and Social Skills Scale (Orpinas & Frankowski, Reference Orpinas and Frankowski2001) were used to measure the degree to which students experienced relational and physical victimization. Children answered using a 4-point Likert scale ranging from 0 (never) to 3 (numerous times). All items were averaged to form a single score (α = 0.83).
Mental health problems
Students’ mental health problems were measured with 12 items selected, adapted, and translated from three subscales of the Strengths and Difficulties Questionnaire (Goodman, Reference Goodman1997): conduct problems, hyperactivity, and emotional symptoms. Children rated the frequency of these occurrences on a 4-point Likert scale ranging from 0 (never) to 3 (numerous times). All items were averaged to form a single score (α = 0.84).
Covariates
We measured several other variables that were included both as covariates and as potential moderators of impact. Child variables included gender (0 = girl, 1 = boy), grade (second to fourth), and language. A language minority indicator was computed based on whether or not the child's primary language differed from the majority language spoken at that child's school (0 = not a language minority, 1 = language minority). In most schools, the majority language was Swahili (90%); the other majority languages were Kibemba (8%) and Kisanga (2%). As noted, school-level measures included the baseline mean scores for the student outcomes (e.g., baseline victimization mean for each school) and the mediator variables. (e.g., baseline supportiveness mean for each school) described above. (See Table 1 for descriptive statistics of school-level data.) In order to account for variation in conditions and outcomes across regions, and in the size of the unit of randomization (clusters of schools), cluster-level covariates included four dummy indicators for the four Katanga subdivisions and a dummy indicator of cluster size (0 = cluster with one school sampled, 1 = cluster with two schools sampled)
Analysis strategy
In order to account for the nested nature of the data (students within schools within clusters), we used multilevel structural equation modeling (MSEM). The main advantage of MSEM is that it allows for tests of mediation involving variables at different levels of a multilevel framework by partitioning the variance into between-school and within-school levels (see Preacher, Zyphur, & Zhang, Reference Preacher, Zyphur and Zhang2010). Although recent advances have extended this framework to three-level designs (Muthén & Muthén, Reference Muthén and Muthén2014; Preacher, Reference Preacher2011), the approach we take here focuses on schools at Level 2 and uses cluster-robust standard errors to control for the effects of administrative school clusters at Level 3. This approach was taken because there were relatively few administrative clusters and also because 15 clusters (38%) contain only 1 school, both of which represent serious challenges in terms of statistical power.
Using this approach, our analysis focused on whether the causal effect of LRHC treatment (a Level 2 variable) on student outcomes (i.e., geometry, addition/subtraction, reading, mental health, and victimization; Level 1 variables) was plausibly mediated by student perceptions of their school ecologies as supportive and predictable (Level 1 variables). In this “2-1-1 design,” the indirect effect of the Level 2 treatment variable occurs at the between-group level—even though both student mediators and student outcomes are assessed at Level 1—given that the only kind of effect the treatment can exert within an MSEM framework is at the between-group level (Preacher et al., Reference Preacher, Zyphur and Zhang2010). As such, the question of interest is whether, and to what degree, school-level variability in students’ perceptions of their school ecologies accounts for the effect of LHRC on school-level student academic and social–emotional outcomes. While the MSEM framework does not permit inferences at the individual level in a 2-1-1 design, it improves upon multilevel modeling approaches to assessing mediation (e.g., Krull & MacKinnon, Reference Krull and MacKinnon2001) that cannot disentangle between and within effects without bias.
It is important to note that the resulting indirect effects quantified by the use of MSEM cannot be interpreted causally: this analytic strategy does not test whether LRHC caused changes in school-level perceptions of school and classroom processes that in turn caused changes in students’ outcomes. Such an approach, known as causal mediation, requires the assumption of sequential ignorability, or that assignment to the treatment and then the mediator are random conditional on pretreatment covariates (Imai, Keele, & Tingley, Reference Imai, Keele and Tingley2010). While in a randomized control trial the treatment is randomly assigned and thus ignorable, the mediator is not randomly assigned; in this case, it is possible but not empirically testable that there might be unobserved variables confounding the relationship between the mediator and the outcome. While researchers have developed methods to test the plausibility of the sequential ignorability assumption in single-level mediation models, unfortunately such analyses for multiple mediators in a multilevel framework remain uncharted territory. As such, we cannot causally interpret the indirect effects reported in this paper due to untestable assumptions that cannot be thoroughly assessed in the present context.
Despite these limitations of current statistical methodology, the present analysis does allow us to assess whether LRHC caused changes in school-level perceptions of the school and classroom that are in turn associated with school-level changes in students’ outcomes. As such associations are a precondition for causal mediation, we refer to evidence of such pathways as “plausibly causal.” To assess these pathways, we used MPlus v7.3 (Muthén & Muthén, Reference Muthén and Muthén2014) to fit five models that included direct and indirect paths from the treatment to student outcomes via students’ perceptions of their school ecologies as supportive and predictable. These models adjusted for student, school, and cluster covariates, and included correlated error terms between all covariates and between students’ perceptions of their school ecologies. Given that the models are saturated (i.e., they perfectly reproduce the variances and covariances of the observed data) goodness of fit indices do not provide a test of the plausibility of the model against the data (P. F. Halpin, personal communication, March 8, 2015). Instead, we tested the indirect effects using a product of the coefficients approach (MacKinnon, Lockwood, Hoffman, West, & Sheets, Reference MacKinnon, Lockwood, Hoffman, West and Sheets2002; Sobel, Reference Sobel1982) included in the MODEL CONSTRAINT command. Given that current implementations of MSEM do not permit bootstrapped confidence intervals to be used when testing mediation effects (Shrout & Bolger, Reference Shrout and Bolger2002), we followed the recommended procedure of using robust standard errors. We also tested for full versus partial mediation by constraining the direct path between treatment and outcome to zero and examining the change in model fit using adjusted procedures for Satorra–Bentler scaled chi-squares (Muthén & Muthén, Reference Muthén and Muthén2014).
Results
The psychometric and descriptive properties of the variables included in this study are presented in Torrente et al. (Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015), Aber et al. (Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolfin press), and Halpin et al. (Reference Halpin, Torrente and Aber2016) and meet the assumptions required to test our model using MSEM. We present the results of the MSEM analysis by outcome domain.
Academic skills
Math
We describe results separately for the impacts on geometry and on addition/subtraction. As illustrated in Figure 2, in this analysis we found support for both the direct and indirect effect of LRHC on children's geometry scores at the school level. When adjusting for all variables in the model, LRHC significantly improved children's geometry scores between schools (Path c). LRHC also caused significant increases in school-level perceptions of caring and supportiveness (Path a). In turn, schools perceived as more supportive and caring were associated with higher school-level geometry scores (Path e), a marginally significant indirect effect (Path ae = 0.06, SE = 0.04, p = .08). A significant decrement of fit resulted when we constrained the direct pathway between LRHC and school geometry scores to zero (Δχ2 = –121, df = 1, p = .000), suggesting that school-level perceptions of caring and supportiveness partially, but not fully, mediate the association between LRHC and school-level geometry scores. Turning to the bottom part of the model, LRHC had a nonsignificant but negative association with school-level perceptions of predictability and cooperation (Path b). In turn, schools perceived as predictable and cooperative had significantly higher school-level geometry scores (Path h), although this indirect effect was not significant (Path bh = –0.03, SE = 0.02, p = .16).
Figure 2. Results of multilevel structural equation model testing the indirect effect of Learning to Read in a Healing Classroom on school-level geometry scores through changes in school-level perceptions of school ecologies. The model only displays the results of between-group analysis because we only interpret results at the between-group level (results of within-group analysis are available upon request). Unstandardized values are reported and standard errors are in parentheses. *p < .05, **p < .01. Correlated error values among covariates and mediating variables are not shown. This model adjusts for the following covariates at the within level: child gender, grade, and language; it adjusts for the following covariates at the between level: baseline school-level supportiveness, baseline school-level predictability, baseline school-level geometry, subdivision, and number of schools per cluster.
In this model that adjusted for all covariates and mediating processes, LRHC also caused a marginally significant improved addition/subtraction scores at the school level, as shown in Table 2 (Path c). Then, a marginally significant positive indirect effect of LRHC was detected via school-level perceptions of caring and supportiveness (Path ae), such that LRHC caused significant increases in school-level perceptions of caring and supportiveness (Path a), which in turn were associated with higher school-level addition/subtraction scores (Path e). Further tests indicated that school-level perceptions of caring and supportiveness partially, but not fully, mediated the association between LRHC and school-level addition/subtraction scores (Δχ2 = –110, df = 1, p = .000).
Table 2. Results from multilevel structural equation modeling models
Note: All paths are delineated in Figure 1. NA refers to paths that were not included in the model given that only one outcome was included in each model. LRHC, Learning to Read in a Healing Classroom.
Literacy
Using this modeling strategy, we did not find support for the direct or indirect effects of LRHC on children's literacy scores at the school-level (see Discussion). As described above and shown in Table 2, LRHC did significantly improve school-level perceptions of caring and supportiveness (Path a). However, in this model we did not detect a significant association between school-level perceptions of caring and supportiveness and school-level literacy scores (Path e). Then, as described before, LRHC had a nonsignificant but negative association with school-level perceptions of predictability and cooperation (Path b). In this model, there was no significant association between schools perceived as predictable and cooperative and school-level literacy scores (Path g).
Socioemotional well-being
Victimization
As illustrated in Figure 3, in this model LRHC did not significantly change children's victimization at the school level (Path d). We did find evidence, however, that suggests the nonsignificant direct effect is due to opposing mediation processes, in which two indirect effects work in the opposite direction (Kenny, Reference Kenny2014). LRHC significantly improved school-level perceptions of caring and supportiveness (Path a), which in turn were associated with lower school-level victimization scores (Path f), a significant and negative indirect effect (Path af = –0.06, SE = 0.03, p = .04). Further testing indicated that school-level caring and supportive schools partially mediated the impact of LRHC on victimization (Δχ2 = –46.16, df = 1, p = .000). Then, LRHC resulted in nonsignificant decreases in school-level perceptions of predictability and cooperation (Path b), but greater perceptions of predictability and cooperation at the school level significantly predicted less victimization at the school level (Path h). This indirect effect was nonsignificant but positive.
Figure 3. Results of multilevel structural equation model testing the indirect effect of Learning to Read in a Healing Classroom on school-level victimization through changes in school-level perceptions of school ecologies. The model only displays results of between-group analysis (results of within-group analysis are available upon request). Unstandardized values are reported and standard errors are displayed in parentheses. *p < .05. Correlated error values among covariates and mediating variables are not shown but also are available upon request. This model adjusts for the following covariates at the within level: child gender, grade, and language; it adjusts for the following covariates at the between level: baseline school-level supportiveness, baseline school-level predictability, baseline school-level geometry, subdivision, and number of schools per cluster.
Mental health
A pattern and direction of effects similar to the pattern for victimization was observed for mental health. No direct effect of LRHC on children's mental health problems at the school level was detected (Path d). However, a significant and negative indirect effect (Path af) was found via school-level perceptions of caring and supportiveness, such that these perceptions were associated with decreased mental health symptomology at the school level (Path f), while a nonsignificant but positive indirect effect was found via school-level perceptions of predictability and cooperation (Path bh).
Discussion
In this paper we applied a conceptual framework derived from developmental psychopathology and social ecology theories to identify and test the mechanisms by which a universal school-based program aimed at promoting children's academic and socioemotional outcomes plausibly worked to improve children's learning and development in a CAC. We have previously reported LRHC program impacts that tested the direct effect of LRHC on children's individual-level outcomes. This paper does not replicate but rather extends and challenges the previous analyses by providing evidence for how the school-level processes and outcomes hypothesized to be impacted by LRHC operate together as a system across schools in the Katanga province of eastern DRC. In keeping with the concept of dynamic transactions and developmental cascades across individuals and their environments, we stated at the outset that such an approach is key for designing better interventions. This is because it allows stakeholders to strengthen and replicate the mechanisms of the intervention that do work, and to adjust the mechanisms that don't work. Such an approach also helps advance theory in developmental psychopathology and social ecology by identifying key processes that may mediate changes in social ecologies and their development. Therefore, what does this analysis tell us about mechanisms that may be particularly important for both for improving the quality of education and learning outcomes in CACs and for theory?
The results provide evidence first and foremost that improving the caring and supportiveness of school ecologies may be a viable and promising target for school-based efforts to improve learning in CACs. First, even when taking into account other processes operating within schools due to the LRHC intervention, LRHC increased children's perceptions of their schools and teachers as caring and supportive at the school level. In turn, schools perceived as more caring and supportive had higher school geometry and addition/subtraction scores, less school victimization, and fewer mental health problems. This provides causal evidence that perceptions of school ecologies as caring and supportive are malleable to intervention efforts, and promising evidence that care and support at the school level is important for multiple domains of children's development in CACs. While such links have been strongly and experimentally established in high-income countries (e.g., Jones, Brown, & Aber, Reference Jones, Brown and Aber2016; Raver et al., Reference Raver, Jones, Li-Grining, Zhai, Bub and Pressler2011; Wilson et al., Reference Wilson, Pianta and Stuhlman2007), this provides the first such evidence for this pathway in school-based intervention research in low-income or CACs.
Second, our results suggest that maintaining or improving the predictability and cooperativeness of school ecologies may be difficult in countries affected by political violence but that it nonetheless may be particularly important for certain domains of children's learning and development. Our original impact analyses indicated that LRHC significantly decreased children's individual-level perceptions of their schools as predictable and cooperative after the first year of implementation. However, these analyses did not control for the covariation in students’ perceptions of their schools as safe and supportive. The analyses herein indicate that LRHC was negatively but not significantly associated with changes in school-level predictability and cooperation, a difference due to the MSEM modeling strategy that simultaneously adjusts for multiple mediating processes. As reported in Torrente et al. (Reference Torrente, Johnston, Starkey, Seidman, Shivshanker, Annan and Aber2015), enormous field challenges (common to working in CACs) resulted in delays in finalizing the curricular components of the interventions and in implementing the Teacher Learning Circles at the planned frequency or intensity. These delays and changes in implementation may have limited the ability of teachers to create a predictable and cooperative environment in schools. However, our analysis provides evidence that creating such an environment may be critical for children's development even adjusting for other concurrent school processes: schools that were perceived as more predictable and cooperative had significantly higher school math scores and lower levels of victimization. Identifying what works to reduce victimization is particularly important in CACs given the role ethnic and religious victimization can play in potentiating armed conflict (King, Reference King2013).
Third, these results suggest that understanding how to improve children's literacy scores appears to require attention to family processes and to classroom/school processes beyond the supportiveness and predictability of school ecologies. While previous analyses indicated that LRHC significantly improved children's literacy scores (Aber et al., Reference Aber, Torrente, Starkey, Johnston, Seidman, Halpin and Wolfin press), our analysis suggests that it did not necessarily do so by improving school-level supportiveness and predictability. Given that LRHC is a teacher professional development program, it is possible that the intervention improved children's literacy scores by improving teachers’ own reading ability or their ability to teach literacy skills, two constructs that are not captured in our measures of the classroom ecology. In addition, LRHC was evaluated within the context of a broader community program that engaged community mobilizers to create stronger education networks and norms between parents, educators, and students. Thus it is also plausible that children exposed to LRHC were better able to use such family and community supports in order to scaffold literacy development. As highlighted by theories in developmental psychopathology and social ecology as well as by decades of education research globally, this suggests that myriad processes transact within and across multiple settings (families, schools, and communities) to shape children's learning (e.g., Cicchetti & Toth, Reference Cicchetti and Toth2009; Costa & Carnoy, Reference Costa and Carnoy2015; Dearing, Kreider, Simpkins, & Weiss, Reference Dearing, Kreider, Simpkins and Weiss2006; Leve & Cicchetti, Reference Leve and Cicchetti2016; Wasik & Hindman, Reference Wasik and Hindman2011). Where possible, future evaluations of academic and socioemotional learning programs in CACs should attempt to assess such processes across settings to better understand and therefore target the factors that promote both types of learning in CACs.
We caution that the analyses conducted herein do not provide evidence of causal mediation: we cannot say that the changes in children's perceptions of their school ecologies caused by LRHC then caused changes in children's outcomes. While establishing the causal processes by which interventions work is critical for “applied” developmental psychopathology and for confidently building effective programs and policies, doing so requires the use of techniques that often have untenable or untestable assumptions. For example, establishing causal mediation using instrumental variables methods assumes the exclusion principle, or that the effect of the treatment on children's outcomes must only operate through one and only one mediator. In our theory of change, and consistent with basic theories in developmental psychopathology and social ecology, we explicitly posit that LRHC may operate through multiple mechanisms. Theories of multiple causal mediating processes are as yet untestable hypotheses in a causal framework due to limitations in methodology. Thus, the limitations to our analysis in terms of causality are also gains in terms of providing plausible support for the dynamic ecological systems by which LRHC changes children's learning and development. In addition, in the original study design, we intended to follow children over time. This proved to be impossible due to high levels of student mobility, lack of an official education management information system with unique child identifiers, and difficulties in using students’ names as reliable identifiers. Therefore, instead of adjusting for individual children's baseline scores on mediating and outcome variables, we aggregated baseline data for the mediators and outcomes to the school level, and include those as covariates to increase the precision of our results.
Fourth and finally, the results of this study suggest the value of bringing developmental psychopathological and social–ecological perspectives to the design and rigorous evaluation of school-based interventions for children in CACs. Advances in both theory and research in the developmental and intervention sciences are now being drawn upon to better understand how to promote the development of children in countries and communities affected by political violence and armed conflict. Sometimes, protracted conflicts like the African World War in the Great Lakes region of the DRC or, more recently, the Syrian war and refugee crisis, feel overwhelming and nearly hopeless. For war-weary citizens, policymakers, and donors, efforts to protect children from conflict or enhance their development even in the face of conflict seem futile. Developmentally and social–ecologically informed interventions and evaluations like the one presented here can be an antidote to policy nihilism, to the belief that nothing can work. True, the results reported in this paper indicate modestly positive impacts of a strategy like LRHC. However, it does offer an existence proof that interventions can make a difference and that it is possible to conduct high-quality developmental and intervention science in some of the most difficult corners of the world.
Target article
Promoting children's learning and development in conflict-affected countries: Testing change process in the Democratic Republic of the Congo
Related commentaries (1)
Commentary on Aber et al.