Retention in MOOCS: some key factors

Deserción en los MOOC: algunos factores clave

Deserção em MOOCS: alguns fatores-chave

Abstract

The dropping out of students in the Massive Open Online Courses (Mooc) has been the subject of debate and concern on the part of educational researchers and practitioners during the last decade. Considering its growth as an educational trend and the emerging research generated on this topic, a systematic review of literature on 131 studies was conducted about attrition in both cMooc and xMoocs. The results highlight the role of collaboration, the sense of community, the need for certification and standardization as the main factors that affect attrition in Moocs.

Mooc; Online Courses; Open Education; Dropout; Virtual Education; Certification

Resumen

La deserción de los estudiantes en los Cursos Masivos Abiertos y en línea (Mooc) ha venido siendo tema de debate y preocupación por parte de educadores e investigadores durante la última década. Teniendo en cuenta su crecimiento como tendencia educativa y la emergente investigación generada en esta materia, se adelantó una revisión sistemática de literatura sobre 131 estudios realizados acerca de la deserción tanto en los cMooc como en los xMooc, cuyos resultados resaltan el papel de la colaboración, el sentido de comunidad, la necesidad de certificación y la estandarización como principales factores que afectan la deserción en los Mooc.

Mooc; Cursos en Línea; Educación Abierta; Abandono Escolar; Educación Virtual; Certificación

Resumo

O abandono dos alunos nos Cursos Online Abertos Massivos (Mooc) tem sido objeto de debate e preocupação por parte de pesquisadores e profissionais da área educacional durante a última década. Considerando seu crescimento como uma tendência educacional e as pesquisas emergentes geradas sobre esse tema, foi realizada uma revisão sistemática da literatura em 131 estudos sobre o atrito nos cMooc e xMoocs. Os resultados destacam o papel da colaboração, o senso de comunidade, a necessidade de certificação e padronização como os principais fatores que afetam o atrito nos Moocs.

Mooc; Cursos Online; Educação Aberta; Cair Fora; Educação Virtual; Certificação

1 Introduction

There are very few educational phenomena associated with the use of Information and Communication Technologies (ICT) that in the last decade have caused such a stir and generated so many expectations in the academic community, especially in Higher Education, such as Massive Open Online Courses (Mooc). In this regard, the data extracted from GoogleTrends shown in Figure 1 , confirm not only the increase but the persistence of interest in this topic compared to other structural issues in the context of ICT educational use, such as e-learning and mobile learning.

Figure 1
Comparison of Mooc - e-learning and m-learning searches on GoogleTrends

Although there is no unified conceptualization of Moocs in the literature, Anderson (2013) and Yousef et al. (2014) refer to them from the four constituent elements of their acronym:

First of all, they are Massive courses: While the concept of ‘massive course’ implies a ‘large number of students, it is difficult to define what is massive in terms of an absolute value or an order of specific magnitude. What ultimately determines the massive nature in this context is the scalability of its implementation, that is, its technical capacity to expand student reception without affecting student performance or the quality of its deployment.

On the other hand, the courses must be Open: most Moocs allow the free participation of anyone anywhere, without academic access prerequisites or other limitations, except the technical need to own a computer and the ability to connect to the Internet with sufficient bandwidth. In this regard, part of the current discussion on this issue lies in the scope of the opening in the Moocs, especially if its openness implies or does not go beyond gratuitousness towards its contents being open, that is, modifiable or susceptible if remixed, reused or shared ( CHIAPPE; ADAME, 2017 ).

In addition to openness, courses should be available Online: in essence, a Mooc is part of computer-mediated distance Education ( ANDERSON, 2013 ); however, there are Moocs that are part of hybrid or blended training spaces that mix online and classroom instruction.

Finally, these are Courses: unlike other educational resources available on the Internet, Moocs are designed to operate as a complete curricular element that seeks to achieve a defined set of learning outcomes within the framework of a social process with clear formative intentions ( DIAS; PINTO, 2019 ). As a course, its instructional design may include and articulate open educational resources, network management tools (social networking), assessments and learning analytics tools.

Although Moocs have been developing for a very short period, it is already possible to find some typologies as an attempt to understand their complex diversity. Regarding the above, Cha and So (2020) propose the “Mooc 3.0 or hMooc” as a type of hybrid course integrated into traditional programs or courses that offer academic credits, also called as “Flipped Mooc” by Klemke, Eradsze and Antonaci (2018). Additional to this, Osuna-Acedo, Marta-Lazo and Frau-Meigs (2018) describe the sMooc as “Social Moocs” which are collaborative, ubiquitous and seamless courses and the tMooc or “Transfer Moocs” related to authentic task-based learning.

In a less formal way, Clarke (2013) presents a group of Moocs based on some particular characteristics, among which the madeMoocs, the synchMoocs, the asynchMoocs, the adaptiveMoocs, the groupMoocs and the miniMoocs stand out.

Other Mooc variants are proposed by Sánchez-Gordon and Luján-Mora (2015) (SPOC or Small Private Online Course) and by Altinpulluk and Kesim (2016) as COOC (Classically Offered Online Classes) or gMoocs (Game-based Massive Open Online Course).

However, the typology that has more sustenance and meaning in the specialized literature refers to its connectivist origin. From this distinction, there are differences between cMoocs and xMoocs.

George Siemens´ perspective regarding what is called as “networked learning” is described in his works about Connectivism (SIEMENS, 2006, 2012a,b) which is the main conceptual framework of Moocs following the principles of “openness” and connections as main drivers of learning. This perspective has been strengthen with the works of Downes (2010) and Wiley (2010) , among others.

To identify the cMooc or “connectivist Mooc”, it is based on the characterization made by Bates (2015 , p. 176), which establishes as the first premise that “knowledge is developed through connections and discussions between participants through social media, understood in a broad sense, since there is no standard technological platform for cMoocs.” The cMooc mobilization vectors are the interests and contributions of the participants, usually having no formal evaluation; and an “instructor” formally identified.

Regarding the essential design principles, cMoocs are based on 1) the learner’s autonomy: the content and skills to be learned are chosen by the participants, whose learning process is personal, without formal curriculum; 2) diversity: the tools, content and participants are diverse, even in terms of level of knowledge; 3) interactivity: learning is cooperative through communication between participants, from which knowledge emerges; 4) the opening: access, content, activities and evaluation are open (WANG; ANDERSON; CHEN, 2018).

Regarding practices, cMoocs are characterized by 1) the indistinct and interconnected use of various social media, provided that they allow participants to share their contributions to the course; in this way, any means can be incorporated into the process; 2) The content is chosen, contributed and distributed by the participants; 3) the communication is not centralized, but is distributed: all communication interactions are open but they are part of a “self-organized network with many subcomponents” ( BATES, 2015 , p. 160); y 4) there is no formal evaluation: each participant recognizes and decides if learning has been appropriate ( MOHAMED; HAMMOND, 2018 ).

Unlike the “connective” approach of cMoocs, the xMoocs or “extension Moocs” use pedagogical models focused on the transmission of information, with contents that can be considered of high quality. The assessment and other interactions are automated, and the instructors communicate primarily through broad-spread messages. In general, they have the following common design features: 1) they are implemented on specialized platforms; 2) they are based on master lectures recorded on video; 3) assessments and tasks are automated and are automatically graded; 4) they include supporting materials; 5) they offer discussion and comment spaces; 6) there is little moderation of the discussions, so the participants depend on the responses of other students; 7) they are oriented to offer certification; 8) they generate a large amount of usable data to make learning analytics ( RHOADS et al., 2015 ).

It is necessary to make an important clarification regarding the pedagogical foundations of the xMoocs, since there is no consensus on that. On one side, Yousef et al. (2014) assign a wide range of possible learning theories as the foundation of extension Moocs – that is, “all who are not cMoocs” –: behaviorism, cognitivism and constructivism (social). For his part, Rodríguez (2013) seems to make a theoretical syncretism by stating that xMoocs fall into the category of “cognitive-behavioral pedagogy” or the theory of “learning for mastery” or Mastery Learning. Furthermore, De Waard (2015) , argued that xMoocs are based on transformative learning, a theory more in line with their androgynous character. Of all this variety of pedagogical postulates, the practical reality is that “non-interactive information and instruction are delivered in a similar way to that of classical distance Education” ( YIN, 2016 , p. 21).

1.1 Student retention and the Mooc

Given the rapid growth of the participation of Moocs in different educational levels, there is great interest on the part of educators and educational researchers to understand the causes and possibilities of managing dropout, which has been considered one of their greatest weaknesses ( VITIELLO et al., 2017 ).

Although of great interest, the magnitude of the research on this specific issue, compared to the general research on Moocs, is clearly emerging as shown in Figure 2 .

Figure 2
Research on Mooc and retention in journals indexed in Scopus

For the purposes of understanding and recognizing attrition in Moocs, whether cMooc or xMooc, 4 basic aspects are considered: obtaining of certificates, eligibility for obtaining certificates, having completed the course and having approved the course ( DAVIS et al., 2017 ).

However, addressing this issue involves considering a conceptual problem regarding comparisons between cMoocs and xMoocs: of the aspects presented above, obtaining certificates and eligibility to obtain certificates do not necessarily apply to cMoocs. However, given the low levels of student retention in the Moocs mentioned above, it was considered pertinent to discern whether there is any difference in these levels between the cMoocs and the xMoocs, a matter that to date is not easy to establish. For this purpose, it is necessary to consider the need to consolidate the lessons learned from various investigations that have addressed dropout in Moocs from different perspectives ( ALMEDA et al., 2018 ; GARDNER; BROOKS, 2018 ; VITIELLO et al., 2017 ) but also that they have not been able to specify this phenomenon based on the differences between cMoocs and xMoocs, beyond what they can report from a specific case such as the one described in Dawson et al. (2015) . Such an approach would make it possible to generate practical knowledge for those who design this type of learning experience, for which reason it has been proposed to carry out a systematic review of literature on studies conducted on attrition in Moocs in order to identify possible differences in this matter between two most recognized types of Mooc.

2 Method

The literature review was carried out following the phases reported by Barn, Barat and Clark (2017), whose detail is shown in Figure 3 .

Figure 3
Phases of the review method

From this reference, three major phases were established to conduct the review, with their corresponding processes, which are described below:

2.1 Planning

The planning phase began with the process of determining the purpose of the review, which focused on identifying possible differences in desertion between the two most recognized types of Moocs. For this purpose, two guiding questions were asked for the review: Are there significant differences in terms of attrition between cMoocs and xMoocs?, and if so, what do these differences consist of?

To address such questions, the following search descriptor string was formulated: (“retention” OR “completion” OR “dropout” OR “attrition”) AND Mooc, which was applied in two databases of refereed and indexed journals: Scopus and Scientific Electronic Library Online (SciELO), whereby broad coverage was achieved in both English and Spanish. For the Spanish search in SciELO, the descriptor “deserción” was added to the search string.

2.2 Conducting

The second phase began with the application of the string of descriptors in the databases, for which a first general set of documents was generated with the results of Scopus (n = 1,879) and SciELO (n = 24). In order to identify the approximate magnitude of an initial sample, the representative sample calculation was applied (n = 320) with 95% of reliability and 5% error.

With the purpose of delimiting the results of the search to articles with research results, a first filtering was performed by type of document, limiting the selection to “articles and articles in press” which generated a second set of documents (n = 776).

Considering the date on which the first Mooc was generated, a time window was established for the search between 2009 and 2018, with which the set of documents was reduced to 773 items. Based on this set of documents, a proportional sample (n = 131) was generated based on the initial set and the first representative sample. Table 1 shows the number of documents reviewed per year proportionally in both databases.

Table 1
Articles reviewed per year

Year	#Scopus	#SciELO	Year	#Scopus	#SciELO
2018	24	2	2013	8	0
2017	27	4	2012	1	0
2016	25	3	2011	1	0
2015	20	1	2010	1	0
2014	13	0	2019	1	0

Source: Own elaboration based on Scopus and SciELO data (2019)

At the end of this phase, three subsequent processes were carried out: (1) Abstracting, that is, the reading of the title and summary of all the items in the final set of documents, through which the thematic relevance of the articles and their correspondence with articles that present research results, (2) the extraction of data from the in-depth reading of each selected article and finally, (3) the analysis and interpretation of the extracted data, through processes of frequency analysis and categorization qualitative. To increase the reliability of this phase, these processes were carried out by two observers, who confirmed or denied the registration of data for each article selected by the other observer. The consistency level of this process was measured through the application of a Cohen Kappa coefficient (K = 0.508), which according to Liao, Hunt and Chen (2010) corresponds to an adequate level of inter-rater reliability.

2.3 Synthesis

The final phase of the review consisted of compiling and synthesizing what was found from the previous steps in a review article following the IMRaD structure.

3 Results

3.1 Bibliometric results

The final set of reviewed articles was distributed heterogeneously in 32 high-impact journals in the “Social Sciences” and “Computer Sciences” categories in SJR. Table 2 shows the Top 10 of the journals with the most reviewed articles of journals indexed in Scopus. The quality of these sources is referenced based on their impact factor and location in the SJR quartiles.

Table 2
Top 10 journals indexed in Scopus with reviewed articles

Journal	% sample	SJR	Q-SJR
International Review of Research in Open and Distance Learning	8.39%	1.26	Q1
Computers and Education	4.77%	2.63	Q1
Computers in Human Behavior	2.45%	1.55	Q1
Distance Education	2.32%	0.70	Q1
British Journal of Educational Technology	1.94%	1.34	Q1
Internet and Higher Education	1.55%	3.35	Q1
Journal of Computer Assisted Learning	1.55%	1.40	Q1
Educational Technology and Society	1.16%	1.09	Q1
Educational Technology Research and Development	1.16%	1.22	Q1
Eurasia Journal of Mathematics Science and Technology Education	1.16%	0.42	Q2

Source: Own elaboration based on Scopus data (2019)

3.2 Factors of student dropout in Moocs

It has been mentioned previously that the measurement of attrition is presented from four base factors. In the studies reviewed, the most common successful completion factor (75.23%) is that of certificate issuance ( BRUNTON et al., 2017 ; GREENE; OSWALD; POMERANTZ, 2015), followed with 8.26% for the eligibility to obtain certificates ( DESPUJOL et al., 2017 ; WARRIEM; MURTHY; IYER, 2016), then with 6.42% for “having completed the course” ( DHORNE et al., 2017 ; LUIK et al., 2018) and finally with 4.12%, “having passed the course” (CONIJN; VAN DEN BEEMT; CUIJPERS, 2018; ENGLE; MANKOFF; CARBREY, 2015).

However, and to give a broader look at the phenomenon of student dropout within Moocs, it is relevant to cite the study of Zheng et al. (2015 , p. 8), in which the motivations of Mooc students to withdraw from these courses are analyzed and in which the following is highlighted:

We identified eight factors associated with low retention rates in Moocs: high workload, difficult content in the course, lack of time, lack of pressure, lack of community awareness, social influence, delay in the beginning of the course and learning on demand.

The foregoing was contrasted by the authors with previous research such as those by Adamopoulos (2013) and Vihavainen, Luukkainen and Kurhila (2012), who found agreement between their findings and those of the aforementioned teams regarding the first two factors mentioned, that is, the difficulty of the course and the workload as factors that negatively affect the retention of students in the course. It is worth noting that in these studies there are both positive and negative correlations between the different variables analyzed and the probability of successfully completing the course, that is, completing all of its contents and tasks (ALRAIMI; ZO; CIGANEK, 2015). The variables that have the highest positive correlation in this study with the successful completion are perceptions about the teacher (p = 0.391), about the tasks (p = 0.398) and about the material (p = 0.134). Interestingly, the presence of discussion forums - a collaborative tool that should counteract the “lack of community awareness” factor - has a negative correlation (p = -0.717) with the probability of successful completion ( ADAMOPOULOS, 2013 , p. 11).

A result of the comparison between the mentioned studies lies in the consistency between the negative effect of the lack of pressure in the course and the negative correlation (p = -0.793) that is found for the courses that are developed at the student’s pace ( self-paced ). Although the constructs are different, it is possible to include the time limitation to develop the activity of the course as an element of pressure for the student. The above does not present a real implication for differentiating xMoocs from cMoocs in terms of desertion, since both types of Moocs are considered self-paced, although cMoocs would tend to be much more flexible in time management and therefore more prone to the existence of this risk circumstance ( HONE; EL SAID, 2016 ).

On the other hand, discrepancies were found between these findings and those presented by Milligan, Littlejohn and Margaryan (2013, p. 155-156) as for other factors, since they account for three main factors that affect participation, but not retention: the student’s confidence, his previous experiences with cMoocs and his motivation. However, since the study of Zheng et al. (2015 , p. 1883) focusing on xMoocs, it can be said that this comparison between the results of both studies would reveal indications of a possible difference in the levels or retention rates between the two types of Moocs, as long as it was possible to associate the levels of participation with those of retention, as the aforementioned authors do seem to indicate in their analysis of the “lack of community awareness” factor.

Another of the dropout factors that does not appear directly related in the previous studies is the level of difficulty of the evaluation. On the one hand, the surveys carried out by Zheng et al. (2015) present as a negative factor the difficulty in the course contents, but, on the other hand, de Freitas, Morgan and Gibson (2015) show that increasing the level of difficulty in the evaluation seems to generate a quality signal of the course that increases retention. The above seems not to be an indicator of difference in attrition between cMoocs and xMoocs, since there is no defined and differentiated typology depending on the mechanisms, instruments or evaluation styles between them. However, xMoocs typically present, depending on the functionalities of the platforms on which they are implemented, more uniform and standardized forms of evaluation, almost always through questionnaires. On the other hand, in the cMooc it is possible to observe, in principle, the presence of mechanisms of self-evaluation and peer co-evaluation ( ESTÉVEZ-AYRES et al., 2013 ), which although does not have a correlation with the level of difficulty, if it is established as a feedback channel and access to support, affects dropout levels ( DAVIS et al., 2017 ).

Another of the most prominent factors among those mentioned by Zheng et al. (2015 , p. 1891) is that of “learning on demand”. The authors clarify that it is a pattern of student behavior consisting of the modularized use of the Mooc as a learning resource, that is, once the student has acquired the knowledge that he wishes to acquire, or that he achieves the learning objective that requires, abandons the course without any mediation. The studies emphasize that it is necessary to understand these objectives in advance to clearly establish a definition of “completed course”. From this perspective, it is considered to count as a “possibility of retention” to the students who present this kind of behavior, keeping them as a separate category from those who present “retention problems”, that is to say those who leave without fulfilling their objectives. These results are consistent with the approach of Gasevic et al. (2014) , who highlight the difficulty of determining the intentions the student has in advance or the reasons for changing those participant intentions when registering in a Mooc. In this regard, it is more feasible that this factor is presented in cMoocs, given the flexibility of an open scheme that is based not on free access (more typical of xMoocs) but on other attributes of “openness” such as sharing and reuse, which is more typical of cMoocs. In that order of ideas, it is necessary to recognize that the xMoocs are deployed in more linear modular structures and with prerequisites, so, if the module of interest is at the end, the student must make the complete journey through the entire curriculum of the course.

3.3 Student retention rates

The effectiveness of Moocs has been repeatedly questioned and investigated since the percentages of successful completion are quite low, ranging between 5% and 15% (FIDALGO-BLANCO; SEIN-ECHALUCE; GARCÍA-PEÑALVO, 2016).

The analysis of the set of studies recovered shows that 83% of the articles analyzed correspond to the so-called extension Moocs or xMoocs, and 17% to the connectivist or cMooc type Moocs, which were systematically analyzed to obtain the retention percentage from students. Figure 4 shows in comparison the dispersion in terms of retention rates of xMoocs and cMoocs.

Figure 4
Student retention rates on cMooc and xMooc

A first result derived from the analysis of retention rates indicates that xMoocs have a lower retention rate (average of 4.5%). These Moocs are based on the use of specialized distribution platforms and emphasize the mastery of content. Also, they are centralized on a website and use automated qualification tools to support hundreds of thousands of students. The initial study on the xMoocs made by Breslow et al. (2013 , p. 21) indicated that students who worked with someone else achieved a predictive score three points higher than those who worked alone. The above is consistent with the retention rate of cMoocs (average of 15.86%), whose character is “more collaborative”, which points to the findings of Fidalgo-Blanco et al. (2016 , p. 23) who suggest that attrition decreases as the level of cooperation increases. Also the analysis of Gasevic et al. (2014 , parag. 54) reflects the idea that social aspects of learning are evidenced as important in the educational research proposals of the Mooc Research Initiative, and point towards “the educational benefits of socialization”.

The very definition of the concept of “successful completion” seems to be operatively linked to the coverage of activities during the course that allow obtaining a certification ( JORDAN, 2015 ); that is, for some authors, the effectiveness of Moocs is associated with getting the participant to complete a predetermined series of activities over a pre-established period of time for the course. In this regard, it was evidenced in the literature that there is a decreasing pattern of participation and retention of participants over time for both cMoocs and xMoocs ( HILL, 2013 ; HO et al., 2014 ; WANG; ANDERSON; CHEN, 2018), which seems to be congruent with similar patterns found in other forms of distance Education prior to the consolidation of Moocs ( PARK; CHOI, 2009 ; ZHANG et al., 2016 ).

4 Discussion

We propose that the concepts of successful completion and quality of a Mooc should be evaluated from a different perspective than traditional online courses (ADMIRAAL; HUISMAN; PILLI, 2015). While the very definition of Mooc as a course implies a series of learning objectives (PADILLA RODRIGUEZ; ARMELLINI; CACERES VILLALBA, 2016), imposing the university coverage model of a “complete series” of such objectives throughout a single course and sequentially is harming the potential of Moocs. The cMoocs, having an open structure by definition, and clearly affirming the complete autonomy of the apprentice, are protected against accusations of not being effective when their participants do not complete the sequence of learning objectives; however, they are accused of being harder to follow, even of being unstructured or confusing ( JASNANI, 2013 ; SOKOLOVSKAYA, 2015 ). However, the crucial point to attend is the proposal of a new structure, not referred to the pedagogical character – be it connective or purely instructional – but in a much simpler way to cover learning objectives.

From the data obtained in the literature review, in particular regarding the congruence in the figures of academic production over the period 2012-2018, a downward stabilization seems to be evidenced, which could imply that it is beginning to accept the phenomenon of attrition as an inherent characteristic of this type of courses. Although the literature evidences permanent efforts to reduce attrition rates in Moocs, some preconditions such as the student’s previous intention and the on-demand learning factor presented by Zheng et al. (2015) suggest that it may be ineffective in cost/benefit to work against what may well be an inherent characteristic of the environment. Additionally, refining Moocs to increase retention probably implies sacrificing at least a part of their openness, for example, when designed with the needs of a particular population in mind, such as variations such as sMoocs or “small” Moocs ( YOUSEF et al., 2014 ).

Regarding the efforts being made to reduce the dropout rate in Moocs, it is worth highlighting the many proposals that are evidenced in the literature ( CROSSLIN et al., 2018 ; ROSÉ; FERSCHKE, 2016 ). In general, it is about combining the characteristics of the xMoocs and the cMoocs in a single offer that allows the participant to choose the learning path. Although these kinds of solutions are clearly motivated by the desire to maintain the effectiveness of the courses offered openly and massively online, when trying to reduce attrition by incorporating collaborative elements there is a double risk. On the one hand, the correlation between the existence of collaborative elements and the attrition rate is not always negative; that is, in some cases it was identified that these elements go against the desired effect ( ADAMOPOULOS, 2013 ). On the other hand, when looking for this type of unification, it is inevitable to reduce the diversity of the offer, which results in a detriment to the overall quality of the ecosystem.

On the other hand, in order to deal with the phenomenon properly, it is convenient to standardize it, but in doing so, its pedagogical possibilities become impoverished and crystallized, as has generally happened with the school system in many countries. To break this vicious circle, some authors allow us to glimpse an exit through another of the elements that a future conceptual framework on Moocs requires: that of quality (FREITAS; MORGAN; GIBSON, 2015; JANSEN; ROSEWELL; KEAR, 2017). A basic definition of quality in Moocs would be to determine whether or not they met the learning objectives that their students or participants set themselves; or more simply, a definition of minimum quality implies subjecting the courses under scrutiny to the criterion of verification of whether or not their participants learned anything effectively. Under this criterion, a different concept of retention could be realized, responding to concerns about the distinction between retention problem and possibility of retention. That is, the quality and retention constructs would be unified, since the participant would be considered as “retention potential”, comparable to a “successful termination” of the Mooc if he can show that he learned something. This dichotomous basis – learned or did not learn – allows to set different “quality levels” according to the percentage of the participating population that meets the criteria. Although this theoretical approach requires elaboration that exceeds the scope of the present study, it can become the basis of future research in the field of Moocs.

References

• ADAMOPOULOS, P. What makes a great MOOC? An interdisciplinary analysis of student retention in online courses. In: INTERNATIONAL CONFERENCE ON INFORMATION SYSTEMS (ICIS 2013): RESHAPING SOCIETY THROUGH INFORMATION SYSTEMS DESIGN, 2013, Milan. Proceedings [...] Milan: AIS, 2013. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-84897795912&partnerID=40&md5=0851525eea67bd826c83e58beff68500 Access: 2019 Oct. 15.
  » https://www.scopus.com/inward/record.uri?eid=2-s2.0-84897795912&partnerID=40&md5=0851525eea67bd826c83e58beff68500
• ADMIRAAL, W.; HUISMAN, B.; PILLI, O. Assessment in massive open online courses. Electronic Journal of E-learning , London, v. 13, n. 4, p. 207-216, Apr. 2015.
• ALMEDA, M. V. et al. Comparing the factors that predict completion and grades among for-credit and open/MOOC students in online learning. Online Learning , Newburyport, v. 22, n. 1, p. 1-18, Mar. 2018. http://dx.doi.org/10.24059/olj.v22i1.1060
  » http://dx.doi.org/10.24059/olj.v22i1.1060
• ALRAIMI, K. M.; ZO, H.; CIGANEK, A. P. Understanding the MOOCs continuance: the role of openness and reputation. Computers & Education , [s. l.], v. 80, p. 28-38, Jan. 2015. https://doi.org/10.1016/j.compedu.2014.08.006
  » https://doi.org/10.1016/j.compedu.2014.08.006
• ALTINPULLUK, H.; KESIM, M. The evolution of MOOCs and a clarification of terminology through literature review. In: EDEN EUROPEAN DISTANCE AND E-LEARNING NETWORK 2016 ANNUAL CONFERENCE, 2016, Budapest, Hungary. Proceedings [...] Budapest: EDEN, 2016. Available from: https://goo.gl/wMjdf8 Access: 2019 Nov. 11.
  » https://goo.gl/wMjdf8
• ANDERSON, T. Promise and/or peril: MOOCs and open and distance education. Edmonton: Athabasca University, 2013. Available from: http://www.ethicalforum.be/sites/default/files/MOOCsPromisePeril.pdf Acess: 2019 Oct. 14
  » http://www.ethicalforum.be/sites/default/files/MOOCsPromisePeril.pdf
• BARN, B.; BARAT, S.; CLARK, T. Conducting systematic literature reviews and systematic mapping studies. In: INNOVATIONS IN SOFTWARE ENGINEERING CONFERENCE, 10., 2017, Jaipur. Proceedings […] Jaipur: ACM Press, 2017. Available from: http://dl.acm.org/citation.cfm?doid=3021460.3021489 Access: 2018 Oct. 14.
  » http://dl.acm.org/citation.cfm?doid=3021460.3021489
• BATES, A. W. Teaching in a digital age: guidelines for designing teaching and learning for a digital age. British Columbia: BC Open Texbooks, 2015.
• BRESLOW, L. et al. Studying learning in the worldwide classroom research into edX’s first MOOC. Research & Practice in Assessment , Farmvile, v. 8, p. 13-25, Summer 2013.
• BRUNTON, J. et al. Giving flexible learners a head start on higher education: designing and implementing a pre-induction socialisation MOOC. In: DELGADO KLOOS, C. et al. (eds.). Digital education: out to the world and back to the campus. Cham: Springer, 2017. p. 10-19. European MOOCs Stakeholders Summit, EMOOCs 2017, Madrid. Proceedings […].
• CHA, H.; SO, H.-J. Integration of formal, non-formal and informal learning through MOOCs. In: BURGOS, D. (ed.). Radical solutions and open science: lecture notes in educational technology. Singapore: Springer Singapore, 2020. p. 135-158.
• CHIAPPE, A.; ADAME, S. I. Open educational practices: a learning way beyond free access knowledge. Ensaio: Avaliação e Políticas Públicas em Educação , Rio de Janerio, v. 26, n. 98, p. 213-230, dez. 2017. http://dx.doi.org/10.1590/s0104-40362018002601320
  » http://dx.doi.org/10.1590/s0104-40362018002601320
• CLARK, D. MOOCs: taxonomy of 8 types of MOOC. Apr. 2013. Available from: https://goo.gl/yi74sS Access: 2019 Nov. 8.
  » https://goo.gl/yi74sS
• CONIJN, R.; VAN DEN BEEMT, A.; CUIJPERS, P. Predicting student performance in a blended MOOC. Journal of Computer Assisted Learning , [s. l.], v. 34, n. 5, p. 615-628, Oct. 2018. https://doi.org/10.1111/jcal.12270
  » https://doi.org/10.1111/jcal.12270
• CROSSLIN, M. et al. Customizable modalities for individualized learning: examining patterns of engagement in dual-layer MOOCs. Online Lerning , Newburyport, v. 22, n. 1, p. 19-38. Jan. 2018. http://dx.doi.org/10.24059/olj.v22i1.1080
  » http://dx.doi.org/10.24059/olj.v22i1.1080
• DAVIS, D. et al. Follow the successful crowd: raising MOOC completion rates through social comparison at scale. In: INTERNATIONAL LEARNING ANALYTICS & KNOWLEDGE CONFERENCE, 17., 2017, Vancouver. Proceedings […] Vancouver: ACM Press, 2017. Available from: http://dl.acm.org/citation.cfm?doid=3027385.3027411 Access: 2018 Oct. 14
  » http://dl.acm.org/citation.cfm?doid=3027385.3027411
• DAWSON, S. et al. Recognising learner autonomy: lessons and reflections from a joint x/c MOOC. In: HIGHER EDUCATION RESEARCH AND DEVELOPMENT SOCIETY OF AUSTRALIA 2015, Melbourne. Proceedings […] Melbourne: HERDSA, 2015. Available from: http://www.herdsa.org.au/system/files/HERDSA_2015_Dawson.pdf Access: 2019 Nov. 21.
  » http://www.herdsa.org.au/system/files/HERDSA_2015_Dawson.pdf
• DESPUJOL, I. M. et al. Effect of free certificate discontinuation in completion rates of MOOC. In: DELGADO KLOOS, C. et al. (eds.). Digital education: out to the world and back to the campus. Cham: Springer, 2017. p. 182-187. European MOOCs Stakeholders Summit, EMOOCs 2017, Madrid. Proceedings […].
• DHORNE, L. et al. Mentoring learners in MOOCs: a new way to improve completion rates? In: DELGADO KLOOS, C. et al. (eds.). Digital education: out to the world and back to the campus. Cham: Springer, 2017. p. 29-37. European MOOCs Stakeholders Summit, EMOOCs 2017, Madrid. Proceedings […].
• DIAS, E.; PINTO, F. C. F. Educação e sociedade. Ensaio: Avaliação e Políticas Públicas em Educação, Rio de Janeiro, v. 27, n. 104, p. 449-454, jul./set. 2019. https://doi.org/10.1590/s0104-40362019002701041
  » https://doi.org/10.1590/s0104-40362019002701041
• DOWNES, S. Learning networks and connective knowledge. Collective Intelligence and E-Learning , [s. l.], v. 2, p. 1-26, 2010. https://doi.org/10.4018/978-1-60566-729-4.ch001
  » https://doi.org/10.4018/978-1-60566-729-4.ch001
• ENGLE, D.; MANKOFF, C.; CARBREY, J. Coursera’s introductory human physiology course: Factors that characterize successful completion of a MOOC. International Review of Research in Open and Distance Learning , Edmonton, v. 16, n. 2, p. 46-68, 2015. https://doi.org/10.19173/irrodl.v16i2.2010
  » https://doi.org/10.19173/irrodl.v16i2.2010
• ESTÉVEZ-AYRES, I. et al. An algorithm for peer review matching in Massive courses for minimising students’ frustration. Journal of Universal Computer Science , Graz, v. 19, n. 15, p. 2173-2197, Sep. 2013.
• FIDALGO-BLANCO, Á.; SEIN-ECHALUCE, M. L.; GARCÍA-PEÑALVO, F. J. From massive access to cooperation: lessons learned and proven results of a hybrid xMOOC/cMOOC pedagogical approach to MOOCs. International Journal of Educational Technology in Higher Education , [s. l.], v. 13, n. 1, p. 1-13, 2016. https://doi.org/10.1186/s41239-016-0024-z
  » https://doi.org/10.1186/s41239-016-0024-z
• FREITAS, S. I.; MORGAN, J.; GIBSON, D. Will MOOCs transform learning and teaching in higher education? Engagement and course retention in online learning provision: engagement and course retention in online learning provision. British Journal of Educational Technology , [s. l.], v. 46, n. 3, p. 455-471, May 2015. https://doi.org/10.1111/bjet.12268
  » https://doi.org/10.1111/bjet.12268
• GARDNER, J.; BROOKS, C. Student success prediction in MOOCs. User Modeling and User-Adapted Interaction , [s. l.], v. 28, n. 2, p. 127-203, June 2018. https://doi.org/10.1007/s11257-018-9203-z
  » https://doi.org/10.1007/s11257-018-9203-z
• GASEVIC, D. et al. Where is research on massive open online courses headed? A data analysis of the MOOC research initiative. The International Review of Research in Open and Distributed Learning , Edmonton, v. 15, n. 5, p. 134-176, Oct. 2014. https://doi.org/10.19173/irrodl.v15i5.1954
  » https://doi.org/10.19173/irrodl.v15i5.1954
• GREENE, J. A.; OSWALD, C. A.; POMERANTZ, J. Predictors of retention and achievement in a massive open online course. American Educational Research Journal , [s. l.], v. 52, n. 5, p. 925-955, Oct. 2015. https://doi.org/10.3102/0002831215584621
  » https://doi.org/10.3102/0002831215584621
• HILL, P. Some validation of MOOC student patterns graphic . e-Literate, 2013. Available from: https://eliterate.us/validation-mooc-student-patterns-graphic/ Access: 2018 Oct. 14.
  » https://eliterate.us/validation-mooc-student-patterns-graphic/
• HO, A. et al. HarvardX and MITx: The first year of open online courses, fall 2012-summer 2013. HarvardX, an. 2014. (HarvardX and MITx Working Paper, vol. 1). Available from: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263 Access: 2019 Nov. 25.
  » https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2381263
• HONE, K. S.; EL SAID, G. R. Exploring the factors affecting MOOC retention: a survey study. Computers & Education , [s. l.], v. 98, p. 157-168, July 2016. https://doi.org/10.1016/j.compedu.2016.03.016
  » https://doi.org/10.1016/j.compedu.2016.03.016
• JANSEN, D.; ROSEWELL, J.; KEAR, K. Quality frameworks for MOOCs. In: KINSHUK, J. M.; KHRIBI, M. K. Open education: from OERs to MOOCs. [S.l.]: Springer, 2017. p. 261-281.
• JASNANI, P. Designing MOOCs: a white paper on instructional design for MOOCsTata Interactive Systems. Ranchi: Indian Institute of Management Ranchi, 2013. Available from: https://www.coursehero.com/file/24424663/Designing-MOOCs-A-White-Paper-on-ID-for-MOOCs-1pdf/ Access: 2019 Oct. 25
  » https://www.coursehero.com/file/24424663/Designing-MOOCs-A-White-Paper-on-ID-for-MOOCs-1pdf/
• JORDAN, K. MOOC completion rates: the Data. Juen 2015. Available from: http://www.katyjordan.com/MOOCproject.html Access: 2019 Oct. 3.
  » http://www.katyjordan.com/MOOCproject.html
• KLEMKE, R.; ERADZE, M.; ANTONACI, A. The flipped MOOC: using gamification and learning analytics in MOOC design: a conceptual approach. Education Sciences , Basel, v. 8, n. 1, p. 25, Feb. 2018. https://doi.org/10.3390/educsci8010025
  » https://doi.org/10.3390/educsci8010025
• LIAO, S. C.; HUNT, E. A.; CHEN, W. Comparison between inter-rater reliability and inter-rater agreement in performance assessment. Annals of the Academy of Medicine Singapore , Singapore, v. 39, n. 8, p. 613-618, 2010.
• LUIK, P. et al. Completion of programming mooc or dropping out: are there any differences in motivation? In: EUROPEAN CONFERENCE ON E-LEARNING, ECEL 2018, 17., 2018, Athens. [S. n. t.], Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85057999485&partnerID=40&md5=5d9cf09e1e33275c5c9ca542d1982c3a Access: 2019 Nov. 14.
  » https://www.scopus.com/inward/record.uri?eid=2-s2.0-85057999485&partnerID=40&md5=5d9cf09e1e33275c5c9ca542d1982c3a
• MILLIGAN, C.; LITTLEJOHN, A.; MARGARYAN, A. Patterns of engagement in connectivist MOOCs. Journal of Online Learning and Teaching , [s. l.], v. 9, n. 2, p. 149-159, June 2013.
• MOHAMED, M. H.; HAMMOND, M. MOOCs: a differentiation by pedagogy, content and assessment. International Journal of Information and Learning Technology , [s. l.], v. 35, n. 1, p. 2-11, Jan. 2018. https://doi.org/10.1108/IJILT-07-2017-0062
  » https://doi.org/10.1108/IJILT-07-2017-0062
• OSUNA-ACEDO, S.; MARTA-LAZO, C.; FRAU-MEIGS, D. From sMOOC to tMOOC, learning towards professional transference. Comunicar , [s. l.], v. 26, n. 55, p. 105-114, Apr. 2018.
• PADILLA RODRIGUEZ, B. C.; ARMELLINI, A.; CACERES VILLALBA, V. C. Massive open online courses (MOOCs) behind the scenes. In: KIRBY, P.; MARKS, G. (eds.) Proceedings of Global Learn 2016. Waynesville: Association for the Advancement of Computing in Education, 2016.
• PARK, J.-H.; CHOI, H. J. Factors influencing adult learners’ decision to drop out or persist in online learning. Journal of Educational Technology & Society , Taiwan, v. 12, n. 4, p. 207-217, 2009.
• RHOADS, R. A. et al. The massive open online course movement, xMOOCs, and faculty labor. The Review of Higher Education , Baltimore, v. 38, n. 3, p. 397-424, 2015. https://doi.org/10.1353/rhe.2015.0016
  » https://doi.org/10.1353/rhe.2015.0016
• ROSÉ, C. P.; FERSCHKE, O. Technology support for discussion based learning: From computer supported collaborative learning to the future of massive open online courses. International Journal of Artificial Intelligence in Education , [s. l.], v. 26, n. 2, p. 660-678, 2016. https://doi.org/10.1007/s40593-016-0107-y
  » https://doi.org/10.1007/s40593-016-0107-y
• SANCHEZ-GORDON, S.; LUJAN-MORA, S. An ecosystem for corporate training with accessible MOOCs and OERs. In: INTERNATIONAL CONFERENCE ON MOOCS, INNOVATION AND TECHNOLOGY IN EDUCATION (MITE) - 2015 IEEE, 3., 2015, Amritsr. Proceedings […] Amritsar: IEEE, 2015. p. 123-128. Available from: http://ieeexplore.ieee.org/document/7375301/ Access: 2020 July 20.
  » http://ieeexplore.ieee.org/document/7375301/
• SIEMENS, G. Connectivism: a learning theory for the digital age. Elearnspace.org. 2012a . Available from: http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1089.2000&rep=rep1&type=pdf Access: 2019 Oct. 10.
  » http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.1089.2000&rep=rep1&type=pdf
• SIEMENS, G. Knowing knowledge. [S.l.]: [s. N.], 2006.Available from: https://archive.org/details/KnowingKnowledge/page/n1/mode/2up Access: 2019 Nov. 20.
  » https://archive.org/details/KnowingKnowledge/page/n1/mode/2up
• SIEMENS, G. ¿Qué tiene de original el conectivismo? [S. l.]:IES Juan B. Justo, 2012b. Available from: https://jbjusto-caba.infd.edu.ar/sitio/conectivismo/ Access: 2019 Nov. 13.
  » https://jbjusto-caba.infd.edu.ar/sitio/conectivismo/
• SOKOLOVSKAYA, A. Connectivist knowledge building, collaborative learning, and social presence in a connectivist massive open online course: a study of PLENK2010. Thesis (Master of Arts) – Concordia University, Montreal, 2015.
• VIHAVAINEN, A.; LUUKKAINEN, M.; KURHILA, J. Multi-faceted support for MOOC in programming. In: ANNUAL CONFERENCE ON INFORMATIO TECHOLOGY EDUCATION, 13., 2012, Calgary. Proceedings […] Alberta: ACM Press, 2012. p. 171-176. Available from: http://dl.acm.org/citation.cfm?doid=2380552.2380603 Acess: 2018 Oct. 14.8
  » http://dl.acm.org/citation.cfm?doid=2380552.2380603
• VITIELLO, M. et al. MOOC dropouts: a multi-system classifier. In: LAVOUÉ, E. et al. (eds.). Data driven approaches in digital education. Cham: Springer, 2017. p. 300-314.
• WAARD, I. MOOC factors influencing teachers in formal education. Revista Mexicana de Bachillerato a Distancia, [s. l.], v. 7, n. 13, p. 1-12, 2015.
• WANG, Z.; ANDERSON, T.; CHEN, L. How learners participate in connectivist learning: An analysis of the interaction traces from a cMOOC . International Review of Research in Open and Distance Learning , Edmonton, v. 19, n. 1, p. 44-67, Feb. 2018. https://doi.org/10.19173/irrodl.v19i1.3269
  » https://doi.org/10.19173/irrodl.v19i1.3269
• WARRIEM, J. M.; MURTHY, S.; IYER, S. Shifting the focus from learner completion to learner perseverance: evidences from a teacher professional development MOOC. In: INTERNATIONAL CONFERENCE ON COMPUTERS IN EDUCATION - ICCE 2016 - THINK GLOBAL ACT LOCAL, 24., 2016, Mumbai. Proceedings […] Mumbai: Asia-Pacific Society for Computers in Education, 2016. Available from: https://www.scopus.com/inward/record.uri?eid=2-s2.0-85018948672&partnerID=40&md5=5e6fc73c77dcb71e0425d85d89f19068 Access: 2019 Nov. 18.
  » https://www.scopus.com/inward/record.uri?eid=2-s2.0-85018948672&partnerID=40&md5=5e6fc73c77dcb71e0425d85d89f19068
• WILEY, D. Openness as catalyst for an educational reformation. Educause Review , Louisville, v. 45, n. 4, p. 14-20, July/Aug. 2010.
• YIN, Y. Chinese learners’ perceptions of MOOCs: a case study. Thesis (Doctoral in Arts and Humanities) – Heinrich Heine University, Düsseldorf, 2016.
• YOUSEF, A. M. F. et al. MOOC: a review of the state-of-the-art. In: CSEDU 2014 - INTERNATIONAL CONFERENCE ON COMPUTER SUPPORTED EDUCATION – CSEDU 2014, 6., 2014, Barcelona. Proceedings […] Barcelona: SCITEPRESS - Science and Technology,2014. p. 9-20.
• ZHANG, Q. et al. Exploring the communication preferences of MOOC learners and the value of preference-based groups: iIs grouping enough? Educational Technology Research & Development , New York, v. 64, n. 4,p. 809-837, Aug. 2016. https://doi.org/10.1007/s11423-016-9439-4
  » https://doi.org/10.1007/s11423-016-9439-4
• ZHENG, S. et al. Understanding student motivation, behaviors and perceptions in MOOCs. In: ACM CONFERENCE, 18., 2015,Vancouver. Proceedings […] Vancouver: ACM Press, 2015.Available from: http://dl.acm.org/citation.cfm?doid=2675133.2675217 Access: 2018 Oct. 14.
  » http://dl.acm.org/citation.cfm?doid=2675133.2675217

Publication Dates

• Publication in this collection
  13 Nov 2020
• Date of issue
  Jan-Mar 2021

History

• Received
  08 Jan 2020
• Accepted
  03 Sept 2020