LEARNING MANAGEMENT SYSTEMS (LMS) AND E-LEARNING MANAGEMENT: AN INTEGRATIVE REVIEW AND RESEARCH AGENDA

AMBIENTES VIRTUAIS DE APRENDIZAGEM E GESTÃO DA EDUCAÇÃO A DISTÂNCIA: UMA REVISÃO INTEGRATIVA E AGENDA DE PESQUISA

ABSTRACT

Information Technology (IT) can be an important component for innovation as enables e-learning and it can provide conditions for an organization to be able to work with new businesses and improved processes. In this regard, Learning Management Systems (LMS) allow communication and interaction between teachers and students in virtual spaces. However, the literature indicates that there are gaps in research, especially concerning the use of IT for the management of e-learning. The purpose of this paper is to analyze the available literature about the application of LMS for the e-learning management, seeking to present possibilities for research in the field. An integrative literature review was performed considering the Web of Science, Scopus, Ebsco and Scielo databases, where 78 references were found, of which 25 were full papers. By eliminating duplication, 14 papers remained, which came to constitute the portfolio of the study. The analysis of the papers allowed to conclude that: 1) the most frequent research strategy was the quantitative; 2) survey was the most used research design; 3) the most frequent categories in the studied educational platforms belong to Instructional Resources and the less frequently ones belong to Interface and, 4) most of the studies are related to administrative function control; 5) LMS in e-learning management is still incipiently discussed in the literature. This analysis derives interesting characteristics from scientific studies, highlighting gaps and guidelines for future research, including learning analytics. The main contribution of this paper is related to the management of e-learning using LMS.

Learning Management Systems (LMS); E-learning; Management; Learning Analytics; Integrative Review

RESUMO

A Tecnologia da Informação (TI) pode ser um componente importante para a inovação, uma vez que permite a Educação a Distância (EaD) e pode fornecer condições a que a organização possa trabalhar com novos processos e negócios. Os Ambientes Virtuais de Aprendizagem (AVA) permitem a comunicação e interação entre professores e alunos em espaços virtuais. No entanto, a literatura indica que existem lacunas nas pesquisas, especialmente sobre o uso da TI para a gestão da EaD. O objetivo deste artigo é analisar a bibliografia sobre a utilização do AVA no gerenciamento da modalidade, bucando apresentar possibilidades para novas pesquisas sobre a temática. Foi realizada uma revisão integrativa, considerando as bases de dados: Web of Science, Scopus, Ebsco e Scielo. A revisão integrativa é um método qualitativo para a análise da literatura. Foram encontradas 78 referências, das quais 25 eram artigos completos. Ao eliminar as duplicações, 14 artigos passaram a constituir o portfólio de trabalho. As análises dos artigos permitiram concluir que: 1) a estratégia de pesquisa mais frequente foi a quantitativa; 2) survey foi o delineamento de pesquisa mais utilizado; 3) as categorias mais frequentes nas plataformas educacionais estudadas pertencem à Recursos Didáticos e as menos frequentes pertencem à Interface; 4) a maior parte dos estudos está relacionada com a função administrativa de controle e, 5) o papel do AVA na gestão da EaD ainda é discutido de maneira incipiente na literatura. A análise apresenta características dos estudos científicos, destacando as lacunas e uma agenda de pesquisas futuras, incluindo o aprendizado analítico. A principal contribuição do trabalho é a discussão da gestão da EaD por meio do AVA.

Ambientes Virtuais de Aprendizagem (AVA); Gestão da Educação a Distância (EaD); Aprendizado Analítico; Revisão Integrativa

1. INTRODUCTION

E-learning is both cause and result of significant changes in the definition of education concept, as well as changes in the understanding of how it should be organized and managed (Peters, 2003). With the e-learning advance, educational institutions managers started to deal with different activities, requiring the development of new procedures and finding alternatives to address emerging challenges that go beyond educational issues. An e-learning system consists of all components and processes that operate when distance learning and teaching occurs (Rosenberg, 2001). It includes learning, teaching, communication, creation and management (Belloni, 2001; Peters, 2003). According to Moore and Kearsley (2007) e-learning is a planned learning process that occurs in general, in a different place other than a regular school, and as a result, it requires special techniques of course design, special forms of instruction, special methods of communication through electronic and other technologies, as well as essential organizational and administrative arrangements.

Organizations that deploy e-learning should be studied and evaluated as systems. A system includes subsystems of knowledge sources, creation, transmission, interaction, learning and management. In practice, the more integrated they are, the greater the effectiveness of the e-learning organization (Moore & Kearsley, 2007). Meanwhile, the growing demand for Information Technology (IT), which can help the management and organization of e-learning, led to the development of Learning Management Systems (LMS). The LMS seek to automate the administration of the courses, to record users, to record courses, to record information about the learning process and to provide reports to the course administration (Coutinho, 2009). E-learning implies important changes in the culture and structure of the institutions that decide to adopt it (Moore & Kearsley, 2007). It also assists in the production of new knowledge backed by IT, and an integrated view of it can enable the creation and management of internal and external processes as parts of a great organizational system (Vieira et al., 2005).

Background studies such as the ones by Bach, Domingues and Walter (2013) and Zawacki-Richter, Bäcker and Sebastian Vogt (2009) points out that there are gaps in the e-learning literature. In this regard, for the authors, there is a need for studies that guide educational institutions and teachers so they can exploit the resources that only IT can provide and, consequently, improve teaching and management. The purpose of this paper is to analyze the available literature about the application of LMS technology for the e-learning management, seeking to present possibilities for researches in the field. The next section is devoted to the theoretical framework including the definition of Information Technology, LMS, LMS characteristics and e e-learning management. Section 3 presents the research objectives and methodology adopted, followed by the results, discussion, conclusions and further research.

2. THEORETICAL BACKGROUND

2.1. Information Technology

The function exercised by Information Technology (IT) in organizations has changed significantly over time (Laurindo, 2009; Rosini, 2013). Here it corroborates with Laurindo (2009) by understanding the concept of IT broadly, encompassing Information Systems (IS), telecommunications and automation, as well as a whole spectrum of hardware and software technologies used by organizations to provide data, information and knowledge. This comprehensive view of IT is present in the idea of "digital convergence", an expression that has been used in the technology industry.

IT can be an important component of innovation, not only by its direct application, but also because it is a vector for other innovations that facilitates, enhances, and, among others, highlights e-learning (Laurindo, 2009). In the scenario of globalization, in which virtual organizations and e-businesses develop, there are great expectations regarding the potential of Information Technology, which increases the importance of its role analysis. However, when using IT in the educational process, it is essential to identify the conceptions that underlie its development, having an adequate view of its possibilities and potentials, because depending of its use, it will be explicit the understanding that we have of the educational process in a space that includes the technology itself (Schlemmer, Saccol & Garrido, 2007).

The development of IT has generated interactive media allowing learning and collective construction of knowledge through networks, with interchangeability of the roles of source and receiver. But only in the 1990s the inclusion of IT in e-learning projects happened (Souza, 2005). In this context, the interest in thinking the interactivity offered by IT grows and its impact on education and organizational culture (Sartori & Garcia, 2009).

According to Oliveira (2012), the potential that IT offers may make e-learning closer to the classroom mode in relation to personal interaction and preserve the distance between teachers and students, in order to improve the process of mediated communication, systematic guidance and constant monitoring, focused on the formation of skills and attitudes that allow the student to have learning process autonomy in a continuous self-education. In this context, IT provides progressively greater flexibility and accessibility to education, culture and professional and personal development, contributing to the creation of educational systems. The pedagogical potential of IT in the pedagogical mediation of e-learning has as main pillar the building of the distance knowledge, thus modifying the paradigm that brings "knowledge as a state and not as a process" (Oliveira, 2012; Rosini, 2013).

Oliveira (2012) also emphasizes that the introduction of IT in education may not be a pedagogical innovation, once the use of old educational practices is no guarantee of a new education. Thus, the criterion to analyze a project of e-learning seems to be not only in the technological mediation, but in the didactic-pedagogical conception that is related to both technological support and its use in the pedagogical mediation. E-learning requires a pedagogical project different from the face-to-face education project and at the same time equal or even more rigorous than a face-to-face course.

2.2. Learning Management Systems

According to Araújo Júnior and Marquesi (2009) a Learning Management System, widely spread as LMS and, hence the use of this acronym in this study may be defined, in the user perspective, as a virtual environment that aims to simulate face-to-face learning environments with the use of Information Technology. In an LMS, the interaction happens through devices that enable communication either synchronously or asynchronously, allowing the creation of different strategies to encourage a dialogue and active participation of students. According to Lonn and Teasley (2009) Learning Management Systems are web-based systems that enable teachers and students to share materials, to submit and return assignments and to communicate online. Meanwhile Almrashdeh et al. (2011) point out that an LMS is software used to plan, implement and evaluate a specific learning process.

In LMS, mediation involves both the acquisition of competences and communication skills of all teachers and students, and a greater concern to create interaction moments and practical application possibilities of collaborative work, with that learning process happening in a participatory manner. For that, the teacher relies on communication devices, such as chat rooms, forums, blogs, video blogs (Souza, 2005; Sartori & Garcia, 2009; Rosini, 2013). To these authors, it is necessary to consider that an LMS must seek to get the best advances in technology available today, for reasons of efficiency and for enabling the maximum degree of interactivity and communication among users. Learning and collaborative work have become fundamental and technological advances should lead to the achievement of high interaction levels.

The first LMS appeared in the nineties, along with the first web browsers. According to Silva (2013), Learning Management Systems are often criticized, due to the belief that these technologies simply virtualize non-virtual classrooms. However, according to the author, they are not the main problem, but the way they are designed, structured and crafted. Furthermore, the use of an LMS requires careful studies particularly in relation to educational and financial aspects.

2.2.1. LMS Features

An LMS is characterized by integrating multiple media, different languages ​​and resources, enabling alternative technologies, and presenting information in an organized manner to fulfill its main purpose, which is the construction of learning through interaction. It is also important to note that a well-planned course, based on innovative teaching methodologies is also necessary when both e-learning quality and a greater adherence to this modality are longed (Santos, 2003; Matucheski & Lupion, 2010).

There are several LMS options in the market (Schlemmer, Saccol & Garrido, 2007), including commercial or proprietary and free software or free courses (Rosini, 2013; Silva, 2013). Among the LMS options that can be found in the international market we highlight the BlackBoard (proprietary environment), Breeze, Moodle (which has a public license), plus dotLRN and the Sakai Project (Santos, 2003; Itmazi et al., 2005; Romero, Ventura & García, 2008; Coutinho, 2009; Almrashdeh et al., 2011).

In Brazil, there are LMS options developed by private companies, universities and government departments. The WebAula is a proprietary LMS, while the Teleduc was designed by the State University of Campinas. The EduWeb and Aulanet were developed by PUC Rio de Janeiro. The E-Proinfo is an LMS developed and used by the Brazilian Federal Government (Coutinho, 2009).

In relation to quality standards in an LMS, considering the development of IT and the growing use of these environments, the importance of identifying those that comply with minimum requirements arises. These requirements can be expressed in terms of reliability, scalability, security, sustainability and adoption of international standards of quality. Reliability can be obtained through the experience of large universities to use virtual fields for the face-to-face or distance education. Scalability is needed to attend to the large numbers of students, a fundamental characteristic of e-learning. The adoption of international quality standards is a factor that depends on the team that developed the project and the options for meeting the needs and goals of users, and which can differentiate from virtual environments to virtual environments (Sartori & Garcia, 2009).

Concerning the criteria for adoption of an LMS, it is necessary that the institution take into account criteria such as the need to restrict access so that only the students enrolled in the subject/course can access the content and activities; the need to promote communication with students through the use of electronic mail, forums, chats; university courses that require tracking of the teaching and learning processes; the need to know where the students "walk", what they access, what they read, when they're doing in LMS, and also the need to evaluate them (Sartori & Garcia, 2009; Almrashdeh et al., 2011).

The evaluation of an LMS is essential to ensure its effective implementation and positive impact on the delivery of e-learning (Almrashdeh et al., 2011). According to Silva (2013), the best LMS choice for an institution depends on its characteristics and objectives. Coutinho (2009) points out that several researchers and users have been devoted to investigate what the necessary elements for choosing an LMS are. In 2004, for example, a team of the Information Technology, Education and Society Group at the Federal University of Rio de Janeiro (UFRJ), identified seven main categories of tools in an educational platform. These categories are: Interface, Navigation, Evaluation, Didactic Resources, Communication / Interaction, Coordination and Administrative Support.

According to Roque et al. (2004), Interface category is the element through which communication is established between the user and the system during the interaction process. It must be clean, objective, fast loading, consistent, respecting the user´s language, allowing resizing and accessibility options and portability. Navigation category is related to the free and easy movement between LMS pages. It should be straightforward, with the standardization of controls and easy to move from one screens to the other. The category includes Evaluation Forms to determine if a student has indeed assimilated the proposed content. They are tools that allow the teacher to evaluate and monitor the student; they must be flexible and allow monitoring or tracking of the learning activities. The Didactic Resources category consists of tools that the environment offers to the teacher. They should be easy to use and versatile, allowing the appropriate pedagogical application and use. The Communication / Interaction category consists of the flow of information between people in an LMS. Communication processes should be mapped and interaction should be encouraged (Roque et al., 2004).

The Coordination category focuses on the activities of planning, creation, execution and control of courses by the teacher. It should facilitate the organization of courses, enable monitoring the performance of students and tutors; and incorporate mechanisms to assess the student’s cognitive development and define the player’s roles. The Administrative Support category combines administration tools and environmental management. Its main objects are: teacher / student / course integration; production of statistical reports; definition of access privileges; processing applications and providing general information about the environment (Roque et al., 2004).

2.3. E-learning Management

Bof (2005) states that e-learning is complex and requires efficient management so that educational outcomes can be achieved. It is crucial to establish strategies and mechanisms by which one can ensure that this system will effectively work as intended, once the following components are defined: educational goals, instructional design, steps and activities, mechanisms to support the learning system, technologies to be used, evaluation system, formal academic procedures and functioning of the system as a whole, E-learning is made up of a number of components that must operate in an integrated manner. It is about the formalization of an operational structure since that involves the development of the course design, the production of didactic materials or information sources and the definition of an evaluation system, including the establishment of operational mechanisms for the distribution of subjects, the availability of learning support services and the establishment of academic procedures.

The origins of educational management, specifically of e-learning management, are related to the General Theory of Administration consolidated in the twentieth century. According to Sobral and Peci (2008), administration consists in the efficient and effective use of resources in an organization, so that its objectives can be achieved. In this regard, the process of contemporary administration involves four interrelated activities called administrative functions: planning, organization, managing and control, arising from the primordial definitions from French administrator Henri Fayol, early twentieth century (Fayol, 1990).

Planning means setting goals and developing strategies and actions to achieve them, organization means determining what should be done, how it should be done and who should do it, managing, on the other hand, implies to lead and motivate members of the organization and, ultimately, control involves monitoring performance to ensure that goals are achieved (Sobral & Peci, 2008). All administrative functions (planning, organization, managing and control) and resources (facilities, space, time, money, information and people) are present in educational management in general and particularly in the management of e-learning.

In e-learning management, as in regular educational management, refers to the action of planning, organizing, coordinating and controlling space, time, money, facilities, people and information, not losing focus on pedagogical principles, which is the purpose in both management systems.. But in this case, their specificities must be analyzed carefully, because the regular educational management (public, in particular) is based on scientific management (business management), but it keeps certain specificities that deserve special care from managers (Mill & Brito, 2009).

For being an institution of peculiar nature, the ways to plan, organize, manage and control a school or a university must be different from the traditional ways business managers make decisions. By the type of institution, the management of higher education differs from the management of basic education. Likewise, the management of e-learning must be treated distinctly. As well as in regular education, managers of e-learning should not disregard the pedagogical nature of their decisions which are turned into actions, but it should be clear that teaching and learning are distinct processes. The educational management of e-learning also provides planning decisions, organization, direction and control, similar to those of regular education in higher education and also concerned with facilities, space, time, money, information and people. However, it is necessary that e-learning managers are aware of the differences between both (Mill & Brito, 2009).

It is understood that, by combining an LMS and the management of e-learning, it is possible to improve the planning, organization, management and control of managers and enhance e-learning processes . In this regard Belloni (2001) highlights that a significant trend is the investment in IT, not only in equipment, but also in research of appropriate methodologies and in training for their application. This emphasis reflects the need for studies either in improving the efficiency and effectiveness of the existing methods of management, and in the creation of new mechanisms for improving e-learning processes.

Adding to this idea, Vaz (2007) says that an LMS is a well-defined and well-built pattern when it becomes a learning management application used in planning, execution and evaluation of a specific e-learning process. For her the focus of an LMS is on the learner and on the organization, and its main purposes are the management of learners, learning activities, the process of evaluation of e-learning and mapping skills of the organization providing education. The environment may also assist in the monitoring and management of relations between users and learning activities.

The application of IT in e-learning has enabled possibilities that include, from the administrative to the pedagogical elements, expansion and management contexts (Souza, 2005). The development of technology has made changes in the way of planning devices that allow the interaction, the content delivery, the offer of communication devices, which increase the complexity of an LMS developed to achieve educational goals. Educating in the Information Society is not only the discursive update of the educational paradigm, but also a deeper understanding of the contributions of technological devices of information and communication for the development of distinctive pedagogical practices, according to the social and cultural context (Sartori & Garcia, 2009).

Macfadyen and Dawson (2010)complement that significant students’ information can be extracted from an LMS and may help educators to extract and visualize real-time data on student engagement and probability of success in their courses. Nevertheless, there is a strong concern of researches in e-learning about the technological aspect, notably the use of Information Technology, and also about an LMS being able to exchange, dialogue, collaboration and joint elaboration ( Oliveira, 2012). Although, McGill and Klobas (2009) point out that LMS research is characterized by a diversity of studies conducted in a wide variety of contexts, considering different variables and explanatory models. For them, on that basis, it is difficult, if not impossible, for research results to be generalized in the field.

According to a literature review in e-learning, conducted by Berge & Mrozowski (2001), research has tended to emphasize student learning outcomes for individual courses rather than form a total academic program. In the study by Zawacki-Richter, Bäcker and Sebastian Vogt (2009), who conducted a review of 695 papers on distance education published in five of the major scientific journals between 2000 and 2008, the researchers showed that there is a strong imbalance between the three research levels in distance education. For them, research on distance education is dominated by issues that relate to the micro perspective, that is, teaching and learning in distance education, where more than 50% of all papers had focused on interaction and communication in communities of learning, instructional design and student characteristics. The authors indicate that the areas related to the management and organization of distance education, that is, methods of research and distance education knowledge transfer, globalization of education and cultural aspects, innovation and change, and the costs and benefits of distance education deserve more attention from researches.

Bach, Domingues and Walter (2013), in turn, performed a systematic review of the Brazilian scientific production on the use of IT in education between 1997 and 2011 and verified that there are large concentrations of studies on implementation and management of distance learning courses, use of IT in education, quality evaluation and satisfaction in using an LMS, pedagogy and didactics in the distance learning content, evaluation of professional skills and competencies related to distance education and contributions of IT to teaching and learning. For them, it reflects the transition of many universities to distance education as well as the existing arguments over their advantages and limitations. The authors also state that surveys could be carried out to guide higher education institutions and teachers to explore the resources that only IT can offer and, therefore, qualitatively improve education. It is worth highlighting that it was observed that the literature has made efforts with an emphasis on pedagogical (Santos, 2003; Gonzales, 2005; Souza, 2005; Araújo Júnior & Marquesi, 2009; Sartori & Garcia, 2009; Matucheski & Lupion, 2010; Macfadyen & Dawson, 2010) and technological aspects (Belanger & Jordan, 2000; Roque et al., 2004; VAZ, 2007; ROMERO, VENTURA & GARCÍA, 2008; MCGILL & KLOBAS, 2009), from the perspective of teachers/tutors and/or students (Derouin, Fritzsche & Salas, 2004; Coates, James & Baldwin, 2005; Mackay & Stockport, 2006; Mcgill & Hoobs, 2008; Lonn & Teasley, 2009; Almrashedh et al., 2011).

The evolution of IT presents new aspects to the quality advance and improvement of e-learning courses. Its technological structure has an important role in this context; it is understood that one can outline courses and provide the acquisition of knowledge from techniques and appropriate technologies (Pimentel, Freitas & Siqueira, 2011). Accordingly, it was observed that there are gaps in e-learning theories, especially regarding the use of an LMS for managing distance education, since none of the theoretical framework studies presented this issue in depth. The understanding of this aspect, suppressed by literature, deserves attention, corroborating with the aforemention vision of Bach, Domingues and Walter (2013).

3. RESEARCH OBJECTIVES AND METHODOLOGY

The objective of this paper is to search and analyze published studies about the application of LMS technology for e-learning management until 2012, to map the issues that have been investigated according to the categories proposed by Roque et al. (2001) and suggest guidelines for future research in the field.

To conduct this study we adopted the method of integrative review (Whittemore & Knafl, 2005). This type of review provides to professionals from diverse fields a quick access to relevant research findings that support decision making, providing critical knowledge (Jackson, 1980; Mendes, Silveira & Galvão, 2008). The integrative review method allows to systematize the scientific knowledge of a particular area of knowledge (Whittemore & Knafl, 2005; Botelho, Cunha & Macedo, 2011). To the extent that it presents an overview of the scientific literature related to a particular subject, the integrative review brings researchers and the problem to be studied together (Botelho, Cunha & Macedo, 2011). In this context, it helped to recognize the development of LMS studies on e-learning managing over time and it thus allowed to envision new possibilities for research. The same authors state that the integrative review should follow some well-defined steps:

Step 1: theme identification and research question selection,

Step 2: establishment of inclusion and exclusion criteria,

Step 3: identification of pre-selected and selected studies,

Step 4: categorization of the selected studies,

Step 5: analysis and interpretation of results and

Step 6: presentation of the review and synthesis of knowledge.

Steps 1 and 2 will be presented in this section, while the others will be presented in subsequent sections. Regarding the first step, the subject of this review involves "LMS and e-learning management" and the question that guides the development of this integrative review is: "What is the state of the art in the use of an LMS in e-learning management? ".

In the second step, the inclusion and exclusion criteria adopted were defined for the preparation of the review. This step involved the definition of databases (Web of Science, Scopus, Ebsco and Scielo). Then, for the survey the whole period available in the databases was considered until the date when the searches were made. A preliminary exploratory study was conducted in order to know the behavior and characteristics of the data. The final searches (which generated the data for this study) were made in December 2012. This study was a preliminary stage in a doctoral thesis, so, the authors opted to delay for some time the publishing of the results of this research.

The key words or descriptors used for the searches were: “Learning Management Syste*”, “Management” and “Strategy”. The asterisk was used to allow the inclusion of papers that mention either "system" and "systems" and the Boolean operator "and" to refine the search. Due to preliminary searches in the databases, we chose not to use the term "e-learning" in search expressions, not to overly restrict the results.

In the Web of Science database, we used the following search strategy: Topic=("Learning Management Syste*") AND Topic=(Management) AND Topic=(Strategy) Timespan=All Years. Databases=SSCI. Lemmatization=On. In the Scopus database, we used the following search strategy: TITLE-ABS-KEY("Learning Management Syste*" AND "Management" AND "Strategy") AND SUBAREA(mult OR arts OR busi OR deci OR econ OR psyc OR soci). In the Ebsco database, we used the following search strategy: TX "Learning Management Syste*" AND TX Management AND TX Strategy, Limiters - Full Text; Academic journals (analyzed by experts); Type of publication: Periodical; Search modes - Boolean / Phrase. In the Scielo database, we used the following search strategy: “Learning Management Syste*” AND “Management” AND “Strategy” in all indices with a regional extent.

The inclusion and exclusion criteria followed: the selection of databases, the exploratory searches in databases, definition of keywords and setting search strategies for each database. It is noteworthy that, from the results, only published papers recognized by the scientific community were selected, which disseminate relevant research in a field of knowledge. Therefore, we sought to conduct the inclusion and exclusion procedure in a rigorous and transparent manner, for the representativeness of the sample is an indicator of depth, quality and reliability of the final conclusions in an integrative review. Then the next steps of the review are presented.

4. RESULTS

In this section the third step of the integrative review is shown, i.e., the identification of selected and pre-selected studies for analysis. The searches returned 78 references. We considered only full papers available in the databases, written in Portuguese, English or Spanish, with the identification of the author, year, volume, title, objectives, methodology, results and conclusion. After reading the articles, 14 papers were selected for the analysis that fit the research objectives, as shown in Table 1:

Table 1
List of papers with its respective order numbers, journals and databases.

Nº	Article title	Journal	Database
1	Attitudes to the application of a Web-based learning system in a microbiology course	Computers & Education	Scopus Web of Science
2	Integration of metacognitive skills in the design of learning objects	Computers in Human Behavior	Scopus Web of Science
3	Integrating an educational 3D game in Moodle	Simulation & Gaming	Scopus
4	A learning style classification mechanism for e-learning	Computers & Education	Scopus Web of Science
5	Instructional technologies in social science instruction in South Africa	Computers & Education	Scopus Web of Science
6	Saving time or innovating practice: Investigating perceptions and uses of Learning Management Systems	Computers & Education	Web of Science
7	Strategies for the delivery of e-information services to support the e-learning environment at the University of Sharjah	The Electronic Library	Ebsco Web of Science
8	The Library’s role and challenges in implementing an elearning strategy: a case study from northern Australia	Health Information and Libraries Journal	Ebsco Scopus
9	A five-year study of on-campus Internet use by undergraduate biomedical students	Computers & Education	Scopus Web of Science
10	Analysis of learners’ navigational behavior and their learning styles in an online course	Journal of Computer Assisted Learning	Scopus Web of Science
11	Mining LMS data to develop an ‘‘early warning system” for educators: A proof of concept	Computers & Education	Scopus Web of Science
12	Using computer supported collaborative learning strategies for helping students acquire self-regulated problem-solving skills in mathematics	Computers & Education	Scopus Web of Science
13	Visualizing and monitoring effective interactions in online collaborative groups	British Journal of Educational Technology	Web of Science
14	Who needs to do what where?: Using learning management systems on residential vs. commuter campuses	Computers & Education	Scopus Web of Science

Table 1 allows observing that 57% of the papers were from Computers & Education journal. To show the aspects that may characterize LMS in e-learning management, we used the categories proposed by Roque et al. (2004), classifying them into Interface, Navigation, Evaluation, Didactic Resources, Communication/Interaction, Coordination and Administrative Support based on the following order: keywords, abstracts and conclusions of the papers analyzed, as presented below.

5. DISCUSSION

This section presents the fourth and fifth steps of the integrative review, i.e., the categorization of selected studies and analysis of the results and interpretation. In this integrative review fourteen papers which met the inclusion and exclusion criteria were analyzed. Table 2 shows the year, the authors, the objective and the Gil’s (2011) research design:

Table 2
List of papers by year, author(s), objective and research design.

Nº	Year	Author(s)	Objective	Research design
1	2005	Masiello, Ramberg & Lonka	Evaluate the validity of LMS Ping Pong as a tool for e-learning considering attitudes of teachers and students.	Survey
2	2007	Sánchez-Alonso & Vovides	Suggest the use of specific ontologies as the basis for incorporating information about metacognition in learning objects so that an LMS can select and recommend designed tasks for the development and / or improvement of metacognitive skills of students in the context of e-learning.	Experimental research
3	2008	González & Blanco	Suggest a prototype that integrates a 3D game with the Moodle LMS, enabling the exchange of information between the two systems.	Experimental research
4	2009	Chang et al.	Suggest a mechanism of learning style classification to classify and identify students learning styles in LMS.	Experimental research
5	2009	Louw et al.	Investigate the access that students of social sciences in South African universities and staff had in the use of ICTs in Western Cape.	Survey
6	2009	Lonn & Teasley	Investigate the uses and perceived benefits of using a LMS to support the teaching of traditional classroom by teachers and students of a big Midwestern American university.	Survey
7	2010	Boumarafi	Reflecting about the development of a new learning environment within the library of the University of Sharjah in the United Arab Emirates.	Study of case
8	2010	Ritchie	Explore the role of a health library in implementing an e-learning in an organization.	Study of case
9	2010	Judd & Kennedy	Reports on a five-year study (2005–2009) of biomedical students’ on-campus use of the Internet.	Survey
10	2010	Graf, Liu & Kinshuk	Investigate how students with different learning styles use the LMS regarding to their browsing behavior.	Study of case
11	2010	Macfadyen & Dawson	Investigate student’s online activities seeking to predict their academic performance.	Survey
12	2010	Lazakidou & Retalis	Investigate the efficacy of a proposed computer-based teaching using a method of self-regulation of problem solving.	Study of case
13	2010	Calvani et al.	Suggest a methodology to evaluate effective collaborative interactions within the module Forum for the Moodle learning management.	Study of case
14	2011	Lonn, Teasley & Krumm	To compare differences in the use of a LMS between instructors and students.	Survey

Regarding the year of publication, it is noted that no records were found previous to 2005. Most publications are from 2010, and no 2012 papers were found, possibly due to issues related to deadlines for the publication in journals indexed by the bases used in this analysis. Six of the analyzed papers, approximately 43%, are surveys. According to Gil (2011), this type of research is characterized by the direct interrogation of persons of whom you want to know the behavior. Information about the problem studied is requested by a significant group of individuals, in order to be able to obtain conclusions corresponding to the data collected through quantitative analysis.

The papers by Louw et al. (2009) and Judd and Kennedy (2010) evaluated the LMS from the student’s perspective, seeking to understand how their perception of issues such as benefits and limitations happens. The papers by Masiello, Ramberg and Lonka (2005), Lonn and Teasley (2009) and Jud and Kennedy (2010) included in the evaluation the vision of teachers or instructors. The paper by Macfadyen and Dawson (2010) sought to investigate the prediction of academic performance in relation to activities undertaken by students online. Five papers, approximately 36%, used the case study approach. According to Gil (2011), a case study is characterized by the exhaustive and deep study of one or a few objects so that it is possible to acquire a broad and detailed knowledge of the object, in which multiple sources of evidence are used. The papers by Boumarafi (2010) and Ritchie (2010) attempted to investigate the relationship between the integration of libraries and virtual and learning environments in two separate contexts. The papers by Graf, Liu and Kinshuk (2010) and Lazakidou and Retalis (2010) sought to investigate issues related to learning styles and LMS. The publication by Calvani et al. (2010) proposed a methodology to evaluate effective collaborative interactions within the forum module for the learning management in an LMS.

Of the fourteen selected papers, three, approximately 21%, used an experimental research design. To Gil (2011), the experiment comprises determining an object of study, selecting the variables that would be capable of influencing it, setting controlling and observation ways that a variable is produced in the object. Sánchez- The papers by Alonso and Vovides (2007), González and Blanco (2008) and Chang et al. (2009) seek to offer practical models and prototypes for solving problems observed in LMS. Sánchez-Alonso and Vovides (2007) propose the use of ontologies to incorporate information about metacognition in learning objects in LMS. The paper by González and Blanco (2008) seeks to integrate 3D games with LMS and Chang et al. (2009) propose a mechanism to adapt LMS to the student's learning style.

Table 3 lists the year, the authors and the LMS that each paper presented as a context for data collection, case study or as an environment to experiment:

Table 3
Papers list according to year, author(s) and LMS.

Nº	Year	Author(s)	LMS
1	2005	Masiello, Ramberg & Lonka	Ping Pong
2	2007	Sánchez-Alonso & Vovides	Not mentioned
3	2008	González & Blanco	Moodle
4	2009	Chang et al.	Not mentioned
5	2009	Louw et al.	Not mentioned
6	2009	Lonn & Teasley	Sakai
7	2010	Boumarafi	Blackboard
8	2010	Ritchie	Not mentioned
9	2010	Judd & Kennedy	Not mentioned
10	2010	Graf, Liu & Kinshuk	Moodle
11	2010	Macfadyen & Dawson	Blackboard
12	2010	Lazakidou & Retalis	Moodle
13	2010	Calvani et al.	Moodle
14	2011	Lonn, Teasley & Krumm	Sakai

Approximately 35% of papers, do not mention the LMS used as context. It was observed that the metacognitive integration proposed by Sanchez-Vovides and Alonso (2007) can be implemented in any LMS, so the authors do not mention a specific LMS. Chang et al. (2009) does not mention explicitly an LMS; however, he states that the mechanism for learning style classification is compatible with an LMS that follows the SCORM (Sharable Content Object Reference Model). The work of Louw et al. (2009) is not about an LMS in particular, but it presents technologies used in a South African university, among them LMS. Ritchie (2010) indicates that a new LMS focused on the context specificities of his study will be deployed. The study by Judah and Kennedy (2010) as well as the work of Louw et al. (2009) were not on a LMS in particular, but they noted the use of LMS by students, in a biomedicine program at an Australian university.

Meanwhile the papers by González and Blanco (2008), Graf, Liu and Kinshuk (2010), Lazakidou and Retalis (2010) and Calvani et al. (2010), approximately 28%, indicated the use of Moodle. The papers by Lonn and Teasley (2009) and Lonn, Teasley and Krumm (2011), approximately 14%, indicated the use of the Sakai environment (note that two of authors co-authored the paper). Boumarafi (2010) and Macfadyen and Dawson (2010) publications indicated the use of Blackboard and only the paper by Masiello, Ramberg and Lonka (2005) showed the use of an LMS called Ping Pong.

Table 4 lists the papers classified according to the categories proposed by Roque et al. (2001). These categories include Interface, Navigation, Evaluation, Didactic Resources, Communication / Interaction, Coordination and Administrative Support.

Table 4
List of papers according to year, author(s) and categories of LMS.

Nº	Year	Author(s)	LMS categories
			Interface	Navigation	Evaluation	Didactic Resources	Communication / Interaction	Coordination	Administrative Support
1	2005	Masiello, Ramberg &Lonka		X		X	X	X	X
2	2007	Sánchez-Alonso & Vovides	X	X	X	X	X	X	X
3	2008	González & Blanco	X	X	X	X		X
4	2009	Chang et al.		X	X	X		X
5	2009	Louw et al.			X	X	X	X	X
6	2009	Lonn & Teasley		X	X		X		X
7	2010	Boumarafi				X			X
8	2010	Ritchie				X			X
9	2010	Judd & Kennedy				X	X	X	X
10	2010	Graf, Liu & Kinshuk	X	X	X			X	X
11	2010	Macfadyen & Dawson			X			X	X
12	2010	Lazakidou & Retalis			X	X			X
13	2010	Calvani et al.			X	X	X	X
14	2011	Lonn, Teasley & Krumm		X	X	X	X	X	X

After analyzing the papers, it was observed that all of them deal with some issue related to e-learning management, insofar as they discuss topics in the Coordination category or in the Administrative Support category or both categories, demonstrating the papers adherence to the search criteria adopted for this review. Despite of this, administrative or system management issues showed operational focus rather than LMS strategic matters, which could have also been studied by the keywords used for the searches. This is evident to the extent that, for example, the same papers deal with issues of the Navigation or Didactic Resources categories, focusing on the system's functional aspects rather than its use for course planning or performance monitoring supported by an LMS.

Issues related to the e-learning planning or strategy was not considered a central theme in none of the studies. Also there was no evident relationship between the study category and the research design. The Interface category was the less frequent one in the analyzed papers, only occurring in the papers by Sánchez-Alonso and Vovides (2007), González and Blanco (2008) and Graf, Liu and Kinshuk (2010), while Didactic Resources category was observed in eleven papers, i.e., except in papers by Lonn and Teasley (2009), Graf, Liu and Kinshuk (2010) and Macfadyen and Dawson (2010). For its part, the paper by Sánchez-Alonso and Vovides (2007) dealt with all the analyzed categories, and the second paper that elaborated the most about the different categories was the one by Lonn, Teasley and Krumm (2011), in a total of six of the seven categories. The papers that dealt with the least number of categories (two) were the ones by Boumarafi (2010) and Ritchie (2010). The remaining papers had three to five categories. The relatively high number of categories discussed in the papers expresses a tendency to discuss LMS from a systemic perspective.

It is important to discuss how the studies show which and/or how they use the tools available in LMS for management purposes. The papers by Masiello, Ramberg and Lonka (2005), Lonn and Teasley (2009), Judd and Kennedy (2010), Graf, Liu and Kinshuk (2010), Lazakidou and Retalis (2010) Calvani et al. (2010) and Lonn, Teasley and Krumm (2011), corresponding to 50% of the analyzed papers, are related to the administrative function control (Sobral & Peci, 2008). These papers involve the definition of performance measures for e-learning, the systematic verification of actual performance of learning activities, the comparison between the patterns and planned objectives and effectively observed performance, even though they not established corrective measures in case of significant deviations. An example of this, in the study by Graf, Liu and Kinshuk (2010), the navigational behavior of students in an online course within a learning management system was investigated, looking at how students with different learning styles prefer to use and learn in such a course.

The papers by Sánchez-Alonso and Vovides (2007), Chang et al. (2009), Boumarafi (2010), Ritchie (2010) and Macfadyen and Dawson (2010), totalizing 36% of analyzed papers, are related to the administrative function planning (Sobral & Peci, 2008). These papers are geared towards a vision of the future where objectives are specified, strategies defined and actions are taken to achieve them. Planning allows e-learning managers to focus their actions on specific purposes, allowing them to concentrate their activities on what is most critical in the context of each course. As an example, the study by Chang et al. (2009) indicates that the proposed classification mechanism can effectively classify and identify students’ learning styles, contributing to the course planning.

Only the study by Louw et al. (2009), accounting for 7% of the analyzed papers, has an explicit relation with administrative function organization (Sobral & Peci, 2008). The article discusses practices of distribution of tasks and Information and Communication Technologies resources (ICT) among students of social sciences from South African universities. The research conclusions point out that the students in South Africa, in most cases, are not resistant to the adoption of ICTs, but feel constrained by practical issues such as the lack of infrastructure, support and time.

Also only the article by González and Blanco (2008), 7% of the analyzed papers, relates to the administrative function of direction (Sobral & Peci, 2008). The article clarifies people management processes in e-learning, specifically with the motivation of the students facing a prototype that integrates a 3D game with the LMS Moodle, making possible the exchange of information between the two systems. It is emphasized that this function requires more action than planning and organization because the manager has the responsibility to provide a favorable environment to the implementation of a quality work and in which workers feel satisfied. The authors conclude that identifying and detecting such emotional factors in interactions with video games and their consequences in the learning process seem to be a key to improving collaboration among members, motivating activities, and promoting learning.

The 14 analyzed studies highlight administrative functions seen as an isolated manner, focusing on control. The papers that discuss planning or are directed only at Blackboard (two papers) or are directed at any LMS. Still in relation to planning, the papers place great emphasis on educational issues, and low focus on the managers’ needs. It is also noteworthy that few papers put as a central element the administrative functions of organization and managing, revealing possibilities not yet explored for research, which will be outlined below.

6. CONCLUSIONS AND FURTHER RESEARCH

In this section, the sixth step of the integrative review is depicted, i.e. the presentation of the review and synthesis of knowledge. In this study, the integrative review method was used to map the studies in the Web of Science, Scopus, Ebsco and Scielo databases, regarding the use of LMS in the e-learning management. 78 papers were located, of which 14 full papers were selected to compose the analysis portfolio. The analysis focused on categories that can characterize the scientific production about the LMS use in the e-learning management. The development of this work allowed: 1) to observe the evolution of research; and 2) to identify possible trends of growth in the number of scientific papers dealing with the subject.

It was possible to outline the historical behavior of scientific production and realize that there is growing academic interest from different countries (England, Spain, USA, South Africa, Australia and United Arab Emirates) to develop research related to themes adjacent to this integrative review. This article also identified authors, objectives and designs of the researches that are being carried out on the subject of this integrative review. About 43% are survey, 36% used the design of study of case and approximately 21% used an experimental research design. Among the publication sources with the highest number of papers on the topic highlights the Computers & Education journal, with about 57% of the analyzed papers.

The list of journals presented in this work allows researchers to, for example, know where to start the deepening of research on the subject and know what are the main publications related. The LMS that each paper presented as context for data collection, case study or as a system for experiment constituted as objects of analysis. Of all publications investigated, 35% did not mention the LMS used as context, 28% indicated the use of Moodle, 14% used Blackboard or Sakai environment, and only one paper indicated the use of Ping Pong, the latter two poorly known in Brazil.

Regarding the categories considered important for analyzing and developing an LMS (Interface, Navigation, Evaluation, Didactic Resources, Communication / Interaction, Coordination and Administrative Support) might conclude that: 1) all papers analyzed showed some relation to the categories Coordination or Administrative Support or both; 2) the relationship between e-learning and LMS planning or strategy were not a central scope to none of the analyzed papers. Also in relation to the categories, the interface was the less frequent in the papers analyzed, while the Didactic Resources was the most frequent. Sánchez-Alonso and Vovides’s (2007) paper that aimed to assess the LMS Ping Pong validity as a tool for e-learning considering attitudes of teachers and students was the only study that analyzed included all categories, the other studies fulfilled much of the categories, with an emphasis on at least two of them, and may configure the complexity when dealing with research on LMS.

Regarding the proposal of LMS use in the analyzed studies, most of them are related to the administrative function control (50%), followed by planning (36%) and organization (7%) and management (7%). The studies approach the administrative functions on a non-integrated manner and focusing educational information. The e-learning management information remain on the margins of the discussion, which may represent interesting opportunities for research in this field.

It is also noticed that, even though having some tools to manage the e-learning courses, that is not an easy or ordinary task. Most managers use LMS only for operational needs or just to issues directly related to control the learning of the students. In contrast, LMS does not provide an adequate tool for management to analyze so many data. In this sense, Learning analytics, which is the measurement, collection, analysis and reporting of data about learners and their contexts, for purposes of understanding and optimizing learning and the environments in which it occurs, emerges as a fundamental need in the context of studies on LMS. The field of learning analytics has the potential to enable higher education institutions to increase their understanding of their students’ learning needs, and the managers to use that understanding to positively influence students learning and progression (Slade & Prinsloo, 2013).

There are some limitations that impact in the findings of this study, for example, the study’s implications are limited by a low number of selected papers, duo to the scarce periodicals addressing this subject and the lack of studies that approached explicitly the use of LMS in the e-learning management. Also, the resources. space, time, money and people using the available LMS were not discussed, due to the few clear evidence in this regard and not all studies have made clear the LMS used as context, which could distort the analysis.

The main contribution of this paper is related to the management of e-learning using LMS. This review also revealed that there is a lack of clear theoretical definitions on the relationship between the LMS and the e-learning management. It was noticed that different technological platforms are treated in a generic way and that there is few empirical research focused on the topic. The analyzed research approach superficially the theme and don't respond, actually, the IT management issues when utilized as a support to the managers of e-learning, supporting the view of Bach, Domingues and Walter (2013).

Questions like "how LMS influenced the e-learning planning, direction, execution and control from the manager perspective?" Or "what the e-learning manager needs in relation to the technological platform used?" Or "Is there an effective alignment between IT and the e-learning processes?" are examples of questions that indicate some research opportunities that can be developed, seeking to fill the gaps identified by this study. The implications for the e-learning management’s field by using LMS cannot be weighted unless there is a research agenda.

REFERENCES

• Almrashdeh, I.A., Sahari, N., Zin, N.A.M., & Alsmadi, M. (2011). Distance learning management system requirements from student’s perspective. Journal of Theoretical and Applied Information Technology, 24(1), 17-27.
• Araújo-Junior, C.F., & Marquesi, S.C. (2009). Atividades em ambientes virtuais de aprendizagem: parâmetros de qualidade. In: LITTO, F.M., & Formiga, M. (Eds.), Educação a distância: o estado da arte. Pearson, São Paulo. pp. 358-368.
• Bach, T.M., Domingues, M.J.C.S., & Walter, S.A. (2013). Tecnologias da informação e comunicação no ensino: um estudo bibliométrico e sociométrico de 1997-2011. Avaliação, 18(2), 393-416. doi: 10.1590/S1414-40772013000200009
  » https://doi.org/10.1590/S1414-40772013000200009
• Belanger, F.; & Jordan, D. (2000). Evaluation and implementation of distance learning: technologies, tools and techniques London: Idea Group Publishing.
• Belloni, M.L. (2001). Educação a distância. Autores Associados, Campinas.
• Berge, Z.L., & Mrozowski, S. (2001). Review of Research in Distance Education, 1990 to 1999. The American Journal of Distance Education 15(3), pp. 5-19 doi: 10.1080/08923640109527090
  » https://doi.org/10.1080/08923640109527090
• Beuren, I.M. (2007). Gerenciamento da informação: um recurso estratégico no processo de gestão empresarial. Atlas, São Paulo.
• Bof, M.B. (2005). Gestão de sistemas de educação a distância. In: Almeida, M.E.B., & Moran, J.M. (Eds.). Integração das tecnologias na educação: salto para o futuro. Ministério da Educação, Brasília. pp. 150-154.
• Botelho, L., Cunha, C., & Macedo, M. (2011). O método da revisão integrativa nos estudos organizacionais. Gestão e Sociedade, 5(11), 121-136.
• Boumarafi, B. (2010). Strategies for the delivery of e-information services to support the e-learning environment at the University of Sharjah. The Electronic Library, 28(2), 276-285. doi: 10.1108/02640471011033639
  » https://doi.org/ 10.1108/02640471011033639
• Calvani, A., Fini, A., Molino, M., & Ranieri, M. (2010). Visualizing and monitoring effective interactions in online collaborative groups. British Journal of Educational Technology, 41(2), 213-226. doi: 10.1111/j.1467-8535.2008.00911.x
  » https://doi.org/10.1111/j.1467-8535.2008.00911.x
• Coates, H., James, R., & Baldwin, G. (2005). A critical examination of the effects of learning management systems on university teaching and learning. Tertiary Education and Management, 11, pp. 19-36.
• Coutinho, L. (2009). Aprendizagem on-line por meio de estruturas de cursos. In: LITTO, F.M. & Formiga, M. (Eds.), Educação a distância: o estado da arte. Pearson, São Paulo. pp. 310-324.
• Derouin, R.E., Fritzsche, B.A., & Salas, E. (2004). Optimizing e-learning: research-based guidelines for learner-controlled training. Human Resource Management, 43(2-3), pp. 147-162, autumn/fall.
• Gil, A.C. (2011). Métodos e técnicas de pesquisa social Atlas, São Paulo.
• Chang, Y., Kao, W., Chu, C., & Chiu, C. (2009). A learning style classification mechanism for e-learning. Computers & Education, 53(2), 273-285. doi:10.1016/j.compedu.2009.02.008
  » https://doi.org/10.1016/j.compedu.2009.02.008
• Fayol, H. (1990). Administração industrial e geral: previsão, organização, comando, coordenação e controle. 10 ed. São Paulo: Atlas, 1990.
• Gonzales, M. (2005). Fundamentos da tutoria em educação a distância São Paulo: Avercamp.
• González, C., & Blanco, F. (2008). Integrating an educational 3D game in Moodle. Simulation & Gaming, 39(3), 399-413. doi:10.1177/1046878108319585
  » https://doi.org/10.1177/1046878108319585
• Graf, J., Liu, T., & Kinshuk, J. (2010). Analysis of learners’ navigational behavior and their learning styles in an online course. Journal of Computer Assisted Learning, 26, 116–131. doi: 10.1111/j.1365-2729.2009.00336.x
  » https://doi.org/10.1111/j.1365-2729.2009.00336.x
• Itmazi, J.A. Gea, M.M., Paderewski, P. & Gutiérrez, F.L. (2005). A comparison and evaluation of open source learning managment systems. In: Proccedings of IADIS, Internacional Conference Applied Computing 2005, Algarve, Portugal.
• Jackson, G. (1980). Methods for Integrative Reviews. Review of Educational Research, 50(3), 438-460. doi: 10.3102/00346543050003438
  » https://doi.org/10.3102/00346543050003438
• Judd, T., & Kennedy, G. (2010). A five-year study of on-campus internet use by undergraduate biomedical students. Computers & Education, 55(4), 1564-1571. doi: 10.1016/j.compedu.2010.06.022
  » https://doi.org/10.1016/j.compedu.2010.06.022
• Laurindo, F.J.B. (2008). Tecnologia da informação: planejamento e gestão de estratégias. Atlas, São Paulo.
• Lazakidou, G., & Retalis, S. (2010). Using computer supported collaborative learning strategies for helping students acquire self-regulated problem-solving skills in mathematics. Computers & Education, 54(1), 3-13. doi: 10.1016/j.compedu.2009.02.020
  » https://doi.org/10.1016/j.compedu.2009.02.020
• Lonn, S., & Teasley, S. (2009). Saving time or innovating practice: investigating perceptions and uses of learning management systems. Computers & Education, 53(3), 686-694. doi: 10.1016/j.compedu.2009.04.008
  » https://doi.org/10.1016/j.compedu.2009.04.008
• Lonn, S., Teasley, S., & Krummc, A. (2011). Who needs to do what where?: using learning management systems on residential vs. commuter campuses. Computers & Education, 56(3), 642-649. doi: 10.1016/j.compedu.2010.10.006
  » https://doi.org/10.1016/j.compedu.2010.10.006
• Louwa, J., Brown, C., Muller, J., & Soudien, C. (2009). Instructional technologies in social science instruction in South Africa. Computers & Education, 53(2), 234-242. doi: 10.1016/j.compedu.2009.02.001
  » https://doi.org/10.1016/j.compedu.2009.02.001
• Macfadyen, L., & Dawson, S. (2010). Mining LMS data to develop an ‘‘early warning system” for educators: a proof of concept. Computers & Education, 54(2), 588-599. doi: 10.1016/j.compedu.2009.09.008
  » https://doi.org/10.1016/j.compedu.2009.09.008
• Mackay, S., & Stockport, G. J. (2006). Blended learning, classroom and e-learning. The Business Review, Cambridge, v. 5, n. 1, p. 82-88, summer.
• Masiello, I., Ramberg, R., & Lonka, K. (2005). Attitudes to the application of a web-based learning system in a microbiology course. Computers & Education, 45(2), 171-185. doi: 10.1016/j.compedu.2004.07.001
  » https://doi.org/10.1016/j.compedu.2004.07.001
• Matucheski, F.L., & Lupion, P.T. (2010). Potencialidades e limitações do ambiente virtual de aprendizagem em um curso on-line. Revista Intersaberes, 5(10), 152-166.
• Mcgill, T., & Hobbs, V. (2008). How students and instructors using a virtual learning environment perceive the fit between technology and task. Journal of Computer Assisted Learning, 24(3), 191-202. doi: 10.1111/j.1365-2729.2007.00253.x
  » https://doi.org/10.1111/j.1365-2729.2007.00253.x
• Mcgill, T. J.; Klobas, J. (2009). A task-technology fit view of learning management system impact. Computers & Education, v. 52, p. 496-508.
• Mendes, K., Silveira, R., & Galvão, C. (2008). Revisão integrativa: método de pesquisa para a incorporação de evidências na saúde e na enfermagem. Texto Contexto Enferm, 17(4), 758-764. doi: 10.1590/S0104-07072008000400018
  » https://doi.org/10.1590/S0104-07072008000400018
• Mill, D., & Brito, N.D. (2009). Gestão da Educação a Distância: Origens e Desafios. In: Proccedings of 15º CIAED, 15th Congresso Internacional Abed de Educação a Distância, Fortaleza. Brazil.
• Moore, M., & Kearsley, G. (2007). Educação a distância: uma visão integrada. Thomson Learning, São Paulo.
• Oliveira, E.G. (2012). Educação a distância na transição paradigmática. Papirus, Campinas.
• Peters, O. (2003). A educação a distância em transição: tendências e desafios. Unisinos, São Leopoldo.
• Pimentel, N.A., Freitas, M., M., & Siqueira, J. (2011). A gestão da tecnologia da informação na educação a distância mediada pela internet In: Metodologia aplicada à educação a distância. Almeida, F.A.S., & Silva, A.M. (Org.). Porto, Portugal; Goiás, Brasil: Universidade Estadual de Goiás.
• Ritchie, A. (2010). The library’s role and challenges in implementing an e-learning strategy: a case study from northern Australia. Health Libraries Group Health Information and Libraries Journal, 28(1), 41-49. doi: 10.1111/j.1471-1842.2010.00923.x
  » https://doi.org/10.1111/j.1471-1842.2010.00923.x
• Romero, C., Ventura, S., & García, E. (2008). Data mining in course management systems: Moodle case study and tutorial. Computers & Education, 51(1), 368-384. doi: 10.1016/j.compedu.2007.05.016
  » https://doi.org/10.1016/j.compedu.2007.05.016
• Roque, G., Chamovitz, I., Araujo, J., Gouvea, M., Cardoso, R., Azambuja, S., & Moura, S. (2004). Aspectos relevantes para o desenvolvimento de ambientes educacionais para a web In: Proccedings of CISCI, 3rd Conferência Iberoamericana en Sistemas, Cibernética e Informática. Miami, United States.
• Rosini, A.M. (2013). As novas tecnologias da informação e a educação a distância Cengage Learning, São Paulo.
• Sánchez-Alonso, S., & Vovides, Y. (2007). Integration of metacognitive skills in the design of learning objects. Computers in Human Behavior, 23(6), 2585-2595. doi: 10.1016/j.chb.2006.08.010
  » https://doi.org/10.1016/j.chb.2006.08.010
• Santos, E.O. (2003). Ambientes virtuais de aprendizagem: por autorias livres, plurais e gratuitas. Revista da FAEEBA, 12(18), 425-435.
• Sartori, A.S., & Garcia, F.G. (2009). Ambientes virtuais de aprendizagem em experiências latino-americanas e espanholas: práticas pedagógicas no contexto da sociedade da informação. Revista Linhas, 10(2), 75-86.
• Schlemmer, E., Saccol, A., & Garrido, S. (2007). Um modelo sistêmico de avaliação de softwares para educação a distância como apoio à gestão de EaD. Revista de Gestão USP, 14(1), 77-91.
• Silva, R.S. (2013). Gestão de EaD: educação a distância na era digital. Novatec, São Paulo.
• Slade, S., & Prinsloo, P. (2013). Learning Analytics: Ethical Issues and Dilemmas. American Behavioral Scientist, 57(10), 1510–1529, doi: 10.1177/0002764213479366.
  » https://doi.org/10.1177/0002764213479366
• Souza, A.R.B. (2005). Movimento didático na educação à distância: análise e prospecções. 2005. 224 f. Tese (Doutorado em Engenharia de Produção), Programa de Pós-Graduação em Engenharia de Produção, Universidade Federal de Santa Catarina, Florianópolis.
• Peci, A., & Sobral, F. (2008). Administração: Teoria e Prática no Contexto Brasileiro. São Paulo: Pearson Prentice Hall.
• Souza, A.R.B. (2005). Movimento didático na educação à distância: análise e prospecções. Doctoral dissertation, Programa de Pós-Graduação em Engenharia de Produção, Universidade Federal de Santa Catarina, Brazil.
• Vaz, M.F.R. (2007). Modelagem e arquitetura de sistemas para monitoração e acompanhamento da aprendizagem eletrônica Doctoral dissertation, Programa de Pós-Graduação em Engenharia de Computação e Sistemas Digitais, Universidade de São Paulo, Brazil.
• Vieira, E., Schuelter, G., Kern, V., & Alves, J. (2005). A teoria geral de sistemas, gestão do conhecimento e educação a distância: revisão e integração dos temas dentro das organizações. Revista de Ciências da Administração (RCA), 7(14), 1-13.
• Wang, M., Ran, W., Liao, J., & Yang, S. (2010). A performance-oriented approach to e-learning in the workplace. Educational Technology & Society, 13(4), 167-179.
• Whittemore, R., & Knafl, K. (2005). The integrative review: updated methodology. Journal of Advanced Nursing, 52(5), 546-553.
• Zawacki-Richter, O., Bäcker, E.M., & Vogt, S. (2009). Review of distance education research (2000 to 2008): analysis of research areas, methods, and authorship patterns. International Review of Research in Open and Distance Learning, 10(6), 21-50.

Publication Dates

• Publication in this collection
  Aug 2016

History

• Received
  23 Dec 2015
• Accepted
  28 Mar 2016