2. Learning analytics applied to LMOOCs: engagement, motivation, and retention

The low completion rates of MOOCs (Jordan, Reference Jordan2015) have triggered a number of investigations into various aspects that may affect student engagement and retention. Scholars have emphasized that retention in MOOCs should be explored in relation to factors such as learning material, instructional design, and pedagogical practices, and also individual reasons that may impact student engagement (Henderikx, Kreijns & Kalz, Reference Henderikx, Kreijns, Kalz, Pammer-Schindler, Pérez-Sanagustín, Drachsler, Elferink and Scheffel2018; Hew, Reference Hew2016; Hone & El Said, Reference Hone and El Said2016; Koller, Ng & Chen, Reference Koller, Ng and Chen2013; Shapiro et al., Reference Shapiro, Lee, Wyman Roth, Li, Çetinkaya-Rundel and Canelas2017). The use of learning analytics provides a valuable means to examine learner-produced data at the time of their engagement with a course and enables a better understanding of the effectiveness of course materials (Martín-Monje et al., Reference Martín-Monje, Castrillo and Mañana-Rodríguez2018; Thomas & Gelan, Reference Thomas and Gelan2018). This is the focus of this study. The data from 400 learners in IOL 2 were collected by tracking users’ tangible engagement with the course content. The same learners then reported on their experiences in a survey on the effects of content-specific and individual factors on why they did not complete the course.

LMOOCs, many of which are still in their early stages of development (Bárcena & Martín-Monje, Reference Bárcena, Martín-Monje, Martín-Monje and Bárcena2014; Colpaert, Reference Colpaert, Martín-Monje and Bárcena2014; Sokolik, Reference Sokolik, Martín-Monje and Bárcena2014), have been described as “dedicated Web-based online courses for second languages with unrestricted access and potentially unlimited participation” (Bárcena & Martín-Monje, Reference Bárcena, Martín-Monje, Martín-Monje and Bárcena2014: 1). Concerns have arisen about whether language learners are provided with engaging forms of design and pedagogy to facilitate the process of language learning in such environments (Colpaert, Reference Colpaert, Martín-Monje and Bárcena2014; de Larreta-Azelain, Reference de Larreta-Azelain, Martín-Monje and Bárcena2014; Sokolik, Reference Sokolik, Martín-Monje and Bárcena2014). Colpaert (Reference Colpaert, Martín-Monje and Bárcena2014) argues that the design of (L)MOOCs mainly reflects the design of the tool applied, “assuming some globally applicable learning and teaching model” (p. 167), and criticizes a lack of adaptation to the relevant subject matter and a specific targeted user group with different needs. These are issues that could explain the high dropout rates in such courses.

In this vein, de Larreta-Azelain (Reference de Larreta-Azelain, Martín-Monje and Bárcena2014) and Sokolik (Reference Sokolik, Martín-Monje and Bárcena2014) stress that LMOOCs require a platform that is specifically aimed at the complex undertaking of teaching and learning a language. De Larreta-Azelain (Reference de Larreta-Azelain, Martín-Monje and Bárcena2014) emphasizes the significance of creating a structure where technology and pedagogy go hand in hand, considering different learning styles, and is based on the Common European Framework of Reference for Languages (CEFR) (Council of Europe, 2018). The value of CALL for the language learner has been pointed out in this context, as it has the potential of an effective intertwining of technology and pedagogy (Chun, Reference Chun2012, Reference Chun2016; Colpaert, Reference Colpaert, Martín-Monje and Bárcena2014). CALL researchers call for more evidence on how technologically enhanced devices and activities may support the language learning process and for more knowledge based on research on CALL’s theories, methods, and models (Chun, Reference Chun2012, Reference Chun2016; Colpaert, Reference Colpaert2018). More evidence on learners’ perspectives in LMOOCs is also needed and whether course design factors and pedagogy influence retention (Bárcena & Martín-Monje, Reference Bárcena, Martín-Monje, Martín-Monje and Bárcena2014; El Said, Reference El Said2017; Hew, Reference Hew2016; Hone & El Said, Reference Hone and El Said2016).

Factors related to MOOC course design have been identified as critical factors for retention (El Said, Reference El Said2017; Hone & El Said, Reference Hone and El Said2016) and diverse course elements and strategies have been proposed as significant engagement features. Hubbard (Reference Hubbard2013) highlights the need for content curation for autonomous learners in open access learning where materials are selected and organized relevant to a topic and the appropriate language level. Dörnyei, Muir and Ibrahim (Reference Dörnyei, Muir, Ibrahim, Lasagabaster, Doiz and Sierra2014) call for well-organized and structured course content for learners consisting of organized objectives and expectations that are clearly presented so that learners may understand what is expected and thus promote engagement (El Said, Reference El Said2017). De Freitas et al. (Reference de Freitas, Morgan and Gibson2015), Doiz, Lasagabaster and Sierra (Reference Doiz, Lasagabaster, Sierra, Lasagabaster, Doiz and Sierra2014), Hew (Reference Hew2016), and Teixeira and Mota (Reference Teixeira, Mota, Martín-Monje and Bárcena2014) point out the need for a more scaffolded approach and more structured support to increase retention. Furthermore, evidence suggests that appealing learning materials, including multimedia presentation of materials, serve as important stimuli for learners (El Said, Reference El Said2017; Friðriksdóttir, Reference Friðriksdóttir2019; Hew, Reference Hew2016). It has thus been highlighted that situational interest in a topic may be a strong motivational factor in MOOCs and that course activities and content are key elements to trigger and maintain students’ attention (de Barba, Kennedy & Ainley, Reference de Barba, Kennedy and Ainley2016; Friðriksdóttir, Reference Friðriksdóttir2019; Hone & El Said, Reference Hone and El Said2016).

Although motivating content is important for retention, individual factors also play a role. Students participate in MOOCs with multiple motives and goals that also need to be considered in the discussion on retention (de Barba et al., Reference de Barba, Kennedy and Ainley2016; Friðriksdóttir, Reference Friðriksdóttir2019; Sokolik, Reference Sokolik, Martín-Monje and Bárcena2014). Evidence shows that many learners join a MOOC with the intention to complete it. However, many of those fail to do so (Friðriksdóttir, Reference Friðriksdóttir2019; Reich, Reference Reich2014). Previous research has identified disengaging features of MOOCs that relate to issues with course design, low course interactivity, lack of support, and outside elements such as time constraints (de Freitas et al., Reference de Freitas, Morgan and Gibson2015; El Said, Reference El Said2017; Henderikx et al., Reference Henderikx, Kreijns, Kalz, Pammer-Schindler, Pérez-Sanagustín, Drachsler, Elferink and Scheffel2018; Shapiro et al., Reference Shapiro, Lee, Wyman Roth, Li, Çetinkaya-Rundel and Canelas2017). Greater understanding, however, is needed about why committed learners disengage before completion (El Said, Reference El Said2017; Reich, Reference Reich2014).

While course content has been found to be a critical factor in MOOC retention (El Said, Reference El Said2017; Hone & El Said, Reference Hone and El Said2016), and specific LMOOC features have been proposed as important factors in engaging learners (Bárcena & Martín-Monje, Reference Bárcena, Martín-Monje, Martín-Monje and Bárcena2014; de Larreta-Azelain, Reference de Larreta-Azelain, Martín-Monje and Bárcena2014; Sokolik, Reference Sokolik, Martín-Monje and Bárcena2014; Teixeira & Mota, Reference Teixeira, Mota, Martín-Monje and Bárcena2014), this study provides empirical evidence not only of learners’ experiences with content-specific factors and pedagogical practices in IOL 2 but also of the influence of these factors on actual retention. Additionally, this study addresses another gap in the literature, namely the lack of previous research examining committed learners’ views about why they disengage before completing (El Said, Reference El Said2017; Reich, Reference Reich2014). It thus attempts to reveal learners’ views on why they did not complete IOL 2 as they intended. The next section introduces the study’s research context.

3. Context of the study

The IOL program is provided in an independent design platform and contains features specifically designed for language learning and teaching of highly inflected languages such as Icelandic. The courses are asynchronous, fully online, and interactive, as well as curated and sequential. The seven IOL courses are skill based (A1–C1) and linked to the CEFR levels, and allow for open and free access and enrollment. They include support materials such as electronic dictionaries and a database of Icelandic inflections. The courses are defined here as language MOOCs. The development of the program began at UI in 2001. The first courses, IOL 1 and IOL 2, were launched in 2004 and 2005 and the latest in 2013.Footnote ¹ IOL’s tracking system was implemented in 2006. The courseware was designed in collaboration with second language acquisition (SLA) experts, Icelandic linguists, and experienced teachers of Icelandic as an L2, designers, and software engineers who were also experts in language technology and linguistics. IOL is an SLA theory-based CALL program and comprises specific engagement strategies intended to support learners in the learning process. The theoretical foundations and instructional methodology of the program as well as the content-specific factors that are the focus of the study are outlined as follows.

The instructional approach in IOL mirrors the intention to facilitate a gradual promotion of language learning through appealing course content with an effort to engage learners and keep them enrolled. The theoretical criteria that guided the development of the course content included scaffolding of the structure (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004; Bruner, Reference Bruner1974; Lantolf, Reference Lantolf2000) and organization of the content (Aebersold & Field, Reference Aebersold and Field1997; Arnbjörnsdóttir, Reference Arnbjörnsdóttir, Þráinsdóttir and Sigurðsson2008) in order to gradually build proficiency with sequential and gradual presentation of input and scaffolded and visual presentation of grammar (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004). Efforts to implement solutions to the presentation of inflectional morphology in a meaningful context are based on Chapelle’s (Reference Chapelle1998, Reference Chapelle2003) model of “relevant SLA hypothesis” and Schmidt’s (Reference Schmidt1992) ideas of attention and noticing (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004). As Icelandic has a complex morphological structure, an effort is made at lower levels in IOL to facilitate L2 learners’ comprehension of the variety of ways in which a nominal can change forms and to support learners in constructing and negotiating meaning.Footnote ² These include input enhancement through focus on form (Doughty & Williams, Reference Doughty and Williams1998) where the focus is on grammar in the input and each of the different foci is assigned a different color (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004). A bi-leveled presentation of the grammatical information in IOL is scaffolded to provide learners with gradual and sequential support, taking into account that not all learners may be grammatically inclined, and to avoid overwhelming the lower-level learner with information that may not be useful for some (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004). Chapelle’s (Reference Chapelle1998, Reference Chapelle2003) idea of choosing relevant SLA theories in reconciling CALL and the multitude of different approaches to SLA theory and pedagogy mirrors the working criteria adopted by the developers of IOL (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004). Each of these approaches guided the development of IOL with the intention of facilitating and promoting language learning and engagement. This study focuses on six content factors in IOL 2, presented as follows, and investigates their potential effect on student engagement and retention. These factors refer to design elements concerning structure and organization of the course and pedagogical principles.

Curated and sequenced course structure and clear and salient learning objectives. Consistent with IOL’s pedagogical approach of guiding the learners through a sequenced and structured learning path, the course material is organized and segmented into five thematic sections each with three lessons or parts (discussed as follows). Learners are initially introduced to explicit section-specific learning objectives regarding vocabulary, grammar, or language usage in each section.

Gradual and scaffolded presentation of input. Based on the sequenced course structure and organization of the course material, the topic is introduced in three steps throughout each section in an attempt to present the input gradually and provide scaffolding of information.Footnote ³ Part I introduces what the learner is going to focus on and comprises visual input that includes vocabulary, grammar, and language usage. Part II contains texts, written and oral, where the vocabulary and grammar introduced in Part I are presented in context, followed by comprehension exercises. Part III focuses on output and provides opportunities for learners to practice what they have focused on in the two previous parts.

Continuing storylines. As a context for language input and practice, activities in the course center around storylines. They are supported by illustrations that are meant to appeal to a wide range of interests to provide a stimulating and engaging content. Each section focuses on a specific story, which is based on authentic texts or conversations that are adapted to learners’ levels and the learning objectives. For example, they follow a protagonist, an exchange student at UI, throughout a section as he goes about his academic and social life.

Form-focused and scaffolded presentation of grammar. In an effort to draw the learners’ attention to the target grammar in a meaningful context and to explain linguistic features, the forms are highlighted in the text and grammatical information is delivered in a scaffolded presentation. First, to point out the grammar in focus, the learners are provided with the option of moving their mouse cursor over the text, which then displays the grammar items highlighted in the text with different colors depending on the input. Then, a one-liner (referred to as “grammar help” in the survey), which explains briefly the core of the particular grammar item, appears underneath. Finally, if the learners want to know more about this grammar, they can go deeper into the grammar by clicking on the one-liner, which directs them to a larger grammar resource (“read more” in the survey).

Variety in types of learning objects. Learning objects were specifically pre-programmed to serve IOL’s pedagogical approach of providing learners with varied and engaging learning material and learning opportunities, resulting in a wide range of learning objects throughout the course that are derived from approximately 40 different software design patterns. This involves different types of media and includes short videos as a source of authentic language material, soundtracks, visual and interactive exercises that introduce grammar and vocabulary, and various tasks for language usage. Thus the course as a whole was carefully planned beforehand to present different and interchanging characteristics of learning objects to account for different learning styles. Additionally, certain types of learning objects were deliberately chosen to approach different inputs and to serve the target language from a linguistic point of view. For example, animation was considered well suited to demonstrate how and why declensional endings change in Icelandic. Audio files and automated feedback are provided for nearly every exercise in the course.

4. The study

This study is based on tracking and survey data from students (n = 400) in IOL 2 in order to examine the effect of content and external factors on retention. Two research questions were addressed:

1. 1. Do learners in IOL 2 consider the following factors pertaining to the structure and organization of the course and the design and pedagogical principles important for their motivation to carry on in the course? If so, are they more likely to complete than those who consider them unimportant?

   1. (a) Curated and sequenced course structure

   2. (b) Clear and salient learning objectives

   3. (c) Gradual and scaffolded presentation of input

   4. (d) Continuing storylines

   5. (e) Form-focused and scaffolded presentation of grammar

   6. (f) Variety in types of learning objects

2. 2. Why do learners who intend to complete IOL 2 not complete it? What is their primary reason for leaving?

5. Results and discussion

Each research question will be addressed in turn. The first is whether learners in IOL 2 considered the content-specific factors important for their motivation and whether they influenced retention. Second, discussed in section 5.2, is why learners who intended to complete the course did not succeed in doing so.

5.1 Content-specific factors in terms of learners’ motivation and retention

To address the first research question of whether or not students in IOL 2 considered the content factors important for their motivation to carry on in the course, the survey responses (n = 400) were tabulated in regard to the six content-specific factors. As Table 1 shows, the sample contains respondents who thought of these factors as significant motivators (marked “strongly agree”/“agree” or “definitely”/“probably” in relation to relevant statements), against respondents who did not consider them important for motivation (marked “neither agree nor disagree”/“disagree”/“strongly disagree” or “possibly”/“probably not”/“definitely not”).Footnote ⁴ First, the two factors concerning the structure and organization of the course content are considered and then the factors regarding the pedagogical principles.

Table 1. Students’ reports on the importance of content-specific factors in IOL 2 for their motivation

Considering first the factor of curated and sequenced course structure in Table 1, the findings show that 73.8% (n = 295) of the participants responded that this factor was important for their motivation. Related to this factor, the survey asked two additional questions of whether participants found the website to be easy or difficult to navigate and whether they accessed the material in the suggested order. The data revealed that 80% (n = 320) found the navigation easy and that 88% (n = 350) navigated through the course as suggested. The results on the second factor of clear and salient learning objectives show that 55% (n = 220) considered it to be important for their motivation. In a related question that asked whether the presentation of learning objectives helped to keep them focused, the data showed that 60% (n = 240) considered such presentation of objectives helpful. In another related sub-question on whether participants would have preferred to set their own learning goals, only 22.8% (n = 91) agreed.

In Table 1, the third factor of gradual and scaffolded presentation of input stands out as it is considered to be important for motivation in 84.8% (n = 339) of the cases. The fourth factor, variety in types of learning objects, turns out to be the second-most important factor with a response rate of 82.2% (n = 329). Four sub-questions on the learning objects were then addressed: (a) whether the learners perceived that the diverse learning objects were fun to work with, (b) whether they were clearly presented such that the users understood what to do, (c) whether the learners accessed the attached audio files, and (d) whether they asked for feedback on their learning objects. The results show that 88.8% (n = 355) stated that the variable exercises made the course fun and that 81.2% (n = 325) claimed that they were clearly presented, while 78.8% (n = 315) stated that they accessed the audio files frequently, and 77.5% (n = 310) asked frequently for feedback. For the fifth factor in Table 1, continuing storylines, the findings show that 69.8% (n = 279) of the participants responded that this factor was important for their motivation, and when asked in a sub-question whether using the storylines made the course fun, 72% (n = 288) agreed.

Finally, regarding learners’ experiences of the form-focused and scaffolded presentation of grammar factor, 66.2% (n = 265) considered this kind of grammar presentation helpful in keeping them motivated. In relation to this factor, the survey asked whether they used the grammar help provided or went to the second step in the grammar resource. The data show that 50.8% (n = 203) stated that they frequently used the grammar help and that 63.2% (n = 253) sought more information in the grammar resource.

In summary, the data reveal that most of the users consider all these factors to be important motivators, indicating that the instructional methodology and engagement strategies employed in IOL 2 are useful in encouraging learners. These findings relate to previous studies highlighting the value of MOOC structural design that a curated and salient structure of course content, where learning goals are spelled out in advance, may be of benefit to motivate learners and keep them focused (de Larreta-Azelain, Reference de Larreta-Azelain, Martín-Monje and Bárcena2014; Dörnyei et al., Reference Dörnyei, Muir, Ibrahim, Lasagabaster, Doiz and Sierra2014; El Said, Reference El Said2017; Hubbard, Reference Hubbard2013). Furthermore, the data show that gradual and scaffolded presentation of input is highly appreciated by the learners, which contributes to earlier arguments of the necessity of providing scaffolding of course content and activities in such environments (de Larreta-Azelain, Reference de Larreta-Azelain, Martín-Monje and Bárcena2014; Doiz et al., Reference Doiz, Lasagabaster, Sierra, Lasagabaster, Doiz and Sierra2014; Hew, Reference Hew2016; Teixeira & Mota, Reference Teixeira, Mota, Martín-Monje and Bárcena2014). The results also support previous evidence that an appealing and entertaining learning context is an important factor in evoking and maintaining learners’ motivation (de Barba et al., Reference de Barba, Kennedy and Ainley2016; Friðriksdóttir, Reference Friðriksdóttir2019; Hew, Reference Hew2016). Moreover, the findings imply that technologically enhanced devices may encourage the LMOOC learner (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004; Chun, Reference Chun2012; Colpaert, Reference Colpaert2010, Reference Colpaert, Martín-Monje and Bárcena2014).

To address the question of whether learners who valued these content-specific factors as important motivators are more likely to complete than those who disagreed, survey data (n = 400) were tabulated with regard to the factors and were then measured against tracking data (Table 2).

Table 2. Retention of learners who considered the content-specific factors in IOL 2 important for their motivation as opposed to those who did not

Focusing first on the factor of curated and sequenced course structure, the outcome shows that 45.8% (n = 135) of those who found this factor to be important for their motivation completed the course, whereas only 38.1% (n = 40) of those who did not agree completed. Concerning the second factor of clear and salient learning objectives, the findings show that 42.7% (n = 94) of those who considered this factor important for their motivation completed the course, in comparison to 45% (n = 81) of those who disagreed. For factor three, gradual and scaffolded presentation of input, 46% (n = 156) of those who identified this factor as important completed the course, as compared to only 31.1% (n = 19) of those who disagreed. Regarding the fourth factor, variety in types of learning objects, the results show that 45% (n = 148) of those who reported this factor to be important for their motivation completed, while only 38% (n = 27) of the other group completed. For the fifth factor of form-focused and scaffolded presentation of grammar, the study found that 43.4% (n = 115) of those who considered it important for their motivation completed, while 44.4% (n = 60) of those who disagreed completed. Finally, when observing the impact of the factor of continuing storylines, 43% (n = 120) of those who valued it important for their motivation completed, while 45.5% (n = 55) of the other group completed.

The results in Table 2 indicate that there is a positive association between three of the factors and course completion; that is, the factors of curated and sequenced course structure, gradual and scaffolded presentation of input, and variety in types of learning objects. When measured against the tracking data, participants who considered these factors important for their motivation were more likely to complete the course than those who did not consider them important. A chi-square test of homogeneity indicates statistically significant differences in completion rates (p = 0.0439) between those who experienced the factor of gradual and scaffolded presentation of input to be important for their motivation and those who did not. However, no statistically significant differences were found in completion rates between the two comparison groups regarding the factor of curated and sequenced course structure (p = 0.2129) or the factor of variety in types of learning objects (p = 0.3474). The other factors in Table 2, clear and salient learning objectives, form-focused and scaffolded presentation of grammar, and continuing storylines, were not found to have a positive impact on course completion in the study. This result suggests that course developers should pay more attention to these factors. The findings contribute to the arguments that course content needs to be curated for the online language learner (Hubbard, Reference Hubbard2013) and that a scaffolded approach and structured support (Arnbjörnsdóttir, Reference Arnbjörnsdóttir and Henrichsen2004; de Larreta-Azelain, Reference de Larreta-Azelain, Martín-Monje and Bárcena2014; Sokolik, Reference Sokolik, Martín-Monje and Bárcena2014; Teixeira & Mota, Reference Teixeira, Mota, Martín-Monje and Bárcena2014) as well as varied course content (de Freitas et al., Reference de Freitas, Morgan and Gibson2015; Doiz et al., Reference Doiz, Lasagabaster, Sierra, Lasagabaster, Doiz and Sierra2014) may be critical to engage and retain the LMOOC learner.

To conclude the findings regarding the first research question, most of the participants, or between 55% to 85% of them, found value in all the content-specific factors for their motivation to continue the course, in particular factors related to gradual and scaffolded presentation of input and the variety in types of learning objects. As the major goal of the IOL course was to engage learners and support their learning process, the majority of the survey respondents demonstrated that the course was successful. When measured against the tracking data, those who considered three out of the six factors to be engaging elements were in greater likelihood of completing than those who did not, where the factor of gradual and scaffolded presentation of input was found to have a statistically significant association with course completion. Overall, the study identified various elements of LMOOC course design and CALL practices that seem to promote learner engagement and, in some cases, retention.

6. Conclusion

The primary objective of this research was to identify important determinants of student engagement and retention in LMOOCs. For this, the study investigated self-reports from 400 learners in IOL 2 on their perspectives of specific content factors in correlation with retention and on possible external factors that may influence retention. The study found that most of the participants considered the course structure, organization, and the instructional design to be important factors for their motivation, and that some of these factors have favorable effects on retention. The data indicate that curated and structured language learning with gradual and scaffolded presentation of input and richly varied course content and activities is appreciated by the beginner language learner and serves as a significant motivator for engagement. Based on the informants’ responses, another major conclusion was that individual and outside factors, such as insufficient time to focus on their studies, may explain why learners who intend to complete LMOOCs leave before completing. The qualitative data analysis reflects the diversity of LMOOC learners who attend with various motives, objectives, and needs that must be considered in the discussion on student retention.

The findings are a component of a larger study on student retention and their experiences when using LMOOCs. This investigation provides new self-reported data on the influence of content-related elements on engagement and retention in correlation with tracking data, along with disengaging factors in a beginner course, but there are some limitations that must be mentioned. The current research sample reflects a limited population of enrollees in IOL 2, or 15.4%, in the period under investigation. The low response rate may be due to the fact that the enrollees may not have been involved with the course for some time and that some may have been unable to participate because the survey was in English. As the study used a post-course survey, this may also have affected the low response rate. Future studies should include questionnaires at different levels of the course to promote survey participation. It should also be mentioned that the selection of the study’s sample may have affected the results since the answers to questions of experience and meaning relate to users’ social affiliations and various age groups where instructional needs may differ. Supplementary studies are essential to investigate the benefits of content factors and pedagogical design for advanced learners and different age groups. This study provides a basis for further investigation of content factors in LMOOCs and engagement strategies in CALL, since the new IOL tracker gives the opportunity to measure the effect of different types of tasks on student engagement in more detail, such as what types learners mostly engage with. Although this study provides valuable insight into critical factors of attrition by using an open-ended questionnaire, there are some limitations that must be acknowledged. Future studies should draw upon multiple methods and sources of qualitative data, such as interviews, to provide a more detailed picture of the setting and strengthen the validity of interpretations in this study. For future research, it would be possible to replicate this study in different settings, as the IOL program has been reproduced and made available for other languages.

This study has contributed wide-ranging data from different sources on the perspectives of LMOOC learners in the context of their learning material and illuminates how numerous course-related and external aspects may affect retention. It provides a new framework on how to promote student engagement and suggests specific strategies involving thoroughly structured and organized content, scaffolding of input and instruction, and a diverse learning context for educators and LMOOC developers to use. The study also offers a new perspective on how to measure completion by including users who cover course content almost to the end. One of the main implications of the study is the suggestion that more use should be made of tracking devices built into LMOOCs to obtain a nuanced picture of learners’ actual involvement throughout a course and to provide a better understanding of curriculum impact. Finally, because this study and its previous phase capture the learners’ thoughts on why they did or did not complete a course, the research contributes a broader perspective on critical factors in LMOOC retention.

In conclusion, the findings emphasize the importance of considering engaging content for learners in the growing LMOOC field while acknowledging that non-course-related factors may hinder them from participating. While the current study investigated the impact of content factors individually, future research could also conduct a regression analysis to describe the relationships between these variables and retention to make predictions about significant elements of LMOOC retention.