We construct a framework for meta-analysis and other multi-level data structures that codifies the sources of heterogeneity between studies or settings in treatment effects and examines their implications for analyses. The key idea is to consider, for each of the treatments under investigation, the subject’s potential outcome in each study or setting were he to receive that treatment. We consider four sources of heterogeneity: (1) response inconsistency, whereby a subject’s response to a given treatment would vary across different studies or settings, (2) the grouping of nonequivalent treatments, where two or more treatments are grouped and treated as a single treatment under the incorrect assumption that a subject’s responses to the different treatments would be identical, (3) nonignorable treatment assignment, and (4) response-related variability in the composition of subjects in different studies or settings. We then examine how these sources affect heterogeneity/homogeneity of conditional and unconditional treatment effects. To illustrate the utility of our approach, we re-analyze individual participant data from 29 randomized placebo-controlled studies on the cardiovascular risk of Vioxx, a Cox-2 selective nonsteroidal anti-inflammatory drug approved by the FDA in 1999 for the management of pain and withdrawn from the market in 2004.

When the process of publication favors studies with small p-values, and hence large effect estimates, combined estimates from many studies may be biased. This paper describes a model for estimation of effect size when there is selection based on one-tailed p-values. The model employs the method of maximum likelihood in the context of a mixed (fixed and random) effects general linear model for effect sizes. It offers a test for the presence of publication bias, and corrected estimates of the parameters of the linear model for effect magnitude. The model is illustrated using a well-known data set on the benefits of psychotherapy.

In the past two decades, the synergistic relationship among task-based language teaching (TBLT), instructed second language acquisition, and computer-assisted language learning has gained increasing interest.Technology-mediated TBLT combines these three research domains by integrating the use of technology with task-based approaches for second language (L2) learning purposes. Since the emergence of this framework, empirical studies have increasingly explored the incorporation of tasks with technology-mediated settings for L2 learning and teaching purposes. To understand the methodological characteristics of technology-mediated TBLT research to date, we conducted a systematic search and reviewed 254 technology-mediated TBLT studies published between 2000 and 2022 in peer-reviewed journals and book chapters. These studies were coded for methodological features, research foci, and types of technology. We further examined the role of technologies in task performance to identify their effectiveness in creating authentic tasks. The findings revealed that technology-mediated TBLT research investigated a rather limited scope of contexts, learner groups, and linguistic features, with little attention paid to evaluating the quality of task outcomes. The types of technology used were skewed toward computer-mediated communication. The results also showed that studies examined various interactional features, and the majority reported both quantitative and qualitative data. Furthermore, technologies were integrated into task design to create meaningful language use contexts. Based on these findings, we share suggestions for future technology-mediated TBLT research.

This study traces the evolution of computer-assisted language learning (CALL) by investigating published research articles (RAs) in four major CALL journals: ReCALL, Computer Assisted Language Learning, Language Learning & Technology, and CALICO Journal. All 2,397 RAs published over four decades (1983–2019) were included in the pool of data, and the Google Scholar citation metric was adopted to assess the impact of the papers. By selecting the top 15% of widely cited papers from each individual year, we minimized the time bias between years, enabling a balanced narration of the history of CALL through a representative dataset of 426 high-impact RAs. To identify the evolution of research trends, the contexts, methodologies, theoretical underpinnings and research foci of all 426 RAs were investigated using NVivo 12 and AntConc. The analysis of the data yielded encouraging results such as the upward trend in the number of publications and the international reach of CALL in the last two decades, the physical or virtual presence of language learners with diverse language profiles, and the growing tendency to triangulate methodology for increased complexity. However, long-standing issues such as the heavy reliance on traditional research contexts, poor reporting practices of basic demographic information, the large number of atheoretical papers and the concentration on a limited number of research foci continue to pose challenges in CALL research. Based on the findings, the paper suggests solutions for the controversies and addresses key issues for future research in CALL.

Meta-analysis overcomes a number of the limitations of traditional literature reviews (Norris & Ortega, 2006). Consequently, the use of meta-analysis as a synthetic technique has been applied across a range of scientific disciplines in recent decades. This paper seeks to formally introduce the potential of meta-analysis to the field of bilingualism. In doing so, we first describe a number of advantages to the meta-analytic approach such as greater systematicity, objectivity, and transparency relative to narrative reviews. We also outline the major stages in conducting a meta-analysis, highlighting critical considerations encountered at each stage. These include (a) domain definition, (b) coding scheme development and implementation, (c) analysis, and (d) interpretation. The focus, however, is on providing a conceptual introduction rather than a full-length tutorial. Meta-analyses in bilingualism and nearby fields are referred to throughout in order to illustrate the points being made.

The ever growing interest in the development of foreign or second (L2) oral proficiency in a computer-mediated communication (CMC) classroom has resulted in a large body of studies looking at both the direct and indirect effects of CMC interventions on the acquisition of oral competences. The present study employed a quantitative meta-analytic approach to investigate such effects by synthesizing (quasi)experimental studies that provide empirical quantitative data for effect size calculation. A literature search located 25 relevant studies for the final analysis. Each study was independently coded for learner, design and publication characteristics. The averaged effect size was estimated from the included studies. The results of the meta-analysis reveal that communication mediated by computer/technologies produced a moderate positive effect on L2 learners’ oral proficiency compared to face-to-face (F2F) communication or no interaction. Furthermore, CMC has roughly similar effect on pronunciation, lexical and syntactic level of oral production; however, it might have a negative impact on fluency and accuracy. This meta-analysis also found that the effect of CMC on oral proficiency depends on several methodological factors such as task type, outcome measurement, treatment length, and assessment task. Major findings of the current meta-analysis include: (1) studies relying on elicited data are superior to those utilizing naturalistic data; (2) reading aloud seems to be the task that could elicit the best oral performance from students; (3) surprisingly, CMC appeared to be harmful for accuracy and fluency; (4) studies that employed decision-making generated the largest effect size, followed by studies that used more than one task type; (5) among the four tasks, jigsaw actually generated a negative effect on oral performance; and (6) as the most popular task employed by primary researchers, opinion-exchange studies produced the smallest effect size. These findings need to be interpreted as exploratory rather than confirmatory since each of them became less trustworthy after taking into consideration numerous other factors such as CMC task and the particular CMC tool used, etc. Future research suggestions are provided and the limitations of this meta-analysis are addressed.

With the aim of summarizing years of research comparing pedagogies for second/foreign language teaching supported with computer technology and pedagogy not-supported by computer technology, a meta-analysis was conducted of empirical research investigating language outcomes. Thirty-seven studies yielding 52 effect sizes were included, following a search of literature from 1970 to 2006 and screening of studies based on stated criteria. The differences in research designs required subdivision of studies, but overall results favored the technology-supported pedagogy, with a small, but positive and statistically significant effect size. Second/foreign language instruction supported by computer technology was found to be at least as effective as instruction without technology, and in studies using rigorous research designs the CALL groups outperformed the non-CALL groups. The analyses of instructional conditions, characteristics of participants, and conditions of the research design did not provide reliable results because of the small number of effect sizes representing each group. The meta-analysis results provide an empirically-based response to the questions of whether or not technology-supported pedagogies enhance language learning, and the process of conducting the meta-analysis pointed to areas in research methodology that would benefit from attention in future research.