Virtual and augmented reality application in production engineering teaching-learning processes

Paper aims: Structure virtual and augmented reality applications in engineering teaching-learning processes, with emphasis on production engineering, highlighting application gaps. Originality: First studies that, to which subjects applied virtual and augmented reality in engineering teaching-learning processes, the gaps of application in production engineering, and a discussion about the impacts. Research method: The research method applied was the systematic literature review. Main findings: Structuring of virtual or augmented reality applications in engineering, discussion about the application in production engineering, opportunities for future research, how low application affects graduated professional education, consequently, organizational competitiveness. Implications for theory and practice: The applications of virtual and augmented reality bring developing student skills more actively and cognitively, making training more complete, increasing their skills, and supporting the competitiveness of organizations through professionals who can contribute more broadly and effectively.

. Basic features related to the virtual world.

Immersion
No immersion; when the computationally developed environment is viewed through a desktop screen or projected view.
With immersion; when the user has the sensation of being totally inside a computational environment using visualization glasses or CAVEs (Cave Automatic Virtual Environment).

Interaction
No interaction; when the user does not interact with the virtual environment.
With interaction; when it is linked to the ability of the computer to detect user's inputs and instantly modify the virtual world and the actions on it.

Involvement
It is related to the condition of the level of involvement of a user with a specific action. It can be passive, such as receiving training, visualizing the virtual environment; or active, such as reading a book, participating in virtual surgery.
It is realized that the use of virtual and augmented reality technologies is increasingly used in teaching-learning processes. There are examples in the literature of application in the teaching of dentistry (Valdez et al., 2014), electrotechnical laboratories (Martins & Guimarães, 2013), chemistry (García-Hernández & Kranzlmüller, 2019), nursing (Fealy et al., 2019), early childhood education (Nainggolan et al., 2019), molecular biology (Safadel & White, 2019), architecture (Zalloom, 2019), among others. In such applications, the opportunity to bring the teaching environment something very close to that found in the real world provides students the opportunity to apply and develop concepts, interaction, immersion, and visualization in the teaching environments.
About Production Engineering currently in Brazil, the institution that organizes and evaluates Production Engineering guidelines is ABEPRO (Brazilian Association in Production Engineering). This association serves as a reference for all Brazilian educational institutions. In 2003, ABEPRO developed a Production Engineering curriculum reference document, considering ten major areas: One of the referenced areas in the curricular guidelines is Education in Production Engineering. The application of virtual and augmented reality can generate an environment very close to reality for the student, enabling the understanding of concepts, methods, tools, and approaches. Cardoso et al. (2013) believe computational resources can complement subjects developed in the classroom, making teaching-learning processes more attractive.
The following research questions arose from these esplanade ideas: What was the application of reality and augmented reality in the engineering teaching-learning process? What is the utilization of virtual and augmented reality in the teaching-learning process in production engineering? In which areas and concepts of production engineering could apply virtual reality or augmented reality? This paper's general objective is to identify, through a systematic literature review, the application of virtual and augmented reality in teaching-learning processes in production engineering courses. The intent is to present trends for this type of technology in the teaching-learning processes in production engineering and where it could be applied.
According to Gil (2010), teaching-learning processes must be acting on students' cognitive issues, providing opportunities for interaction between the student and the object which is being studied. It is necessary to seek teaching-learning processes with cognitive approaches.
From this perspective, the use of reality and augmented reality would allow the use of laboratories with concepts very close to reality, which offer an ideal environment for the development of learning. Experiences with virtual reality generate the opportunity for immersion and acting directly on students' cognitive issues, creating the opportunity for learning in an environment very close to those found in the real world (Braga, 2001;Schlemmer & Backes, 2014).
This research is justified once it approaches the possibilities and gaps in the use of virtual and augmented reality in the teaching-learning process of Production Engineering students. It is essential to provide opportunities for the development of teaching environments closer to reality, leading students to the active construction of knowledge.
Analyzing the document developed by Associação Brasileira de Engenharia de Produção (2003) about the production engineering curriculum guidelines and the research results, it is possible to notice gaps in the application of virtual and augmented reality in the Production Systems Management, Production Planning and Control areas, Material Handling, Production Simulation, Productive Process Management, Metrological Quality Organization, Work Organization, Accident Risk Analysis and Prevention, Work Safety, Product and Process Ergonomics.
These gaps cause an impact on production engineer training due to the lack of opportunity to learn in a teaching environment closer to reality. This impact generates reduced organizational competitiveness because graduates come to the labor market with many gaps in the applications of the theories studied. This article will have the methodology, results, result discussions, and conclusion sections.

Methodology
The method used for development was the systematic literature review. The systematic literature review has the role of analyzing and develop theories through extensive research in the existing literature (Dresch et al., 2015;Gough & Thomas, 2012) Dresch et al. (2015 explain that the systematic literature review is secondary studies that seek to understand, structure, and evaluate primary studies, answering actual research questions, identifying gaps in the literature, and synthesizing reports of the results. For the systematic literature review was selected the model proposed by Gough & Thomas (2012), whose steps are detailed in Figure 1.
Search strategy. It is necessary to define what to look for, where to search, how to minimize propensity, which studies to consider, and the extent of the search (Dresch et al., 2015).
In defining what to look for, the activity was organized into three steps. The first was to understand the historical behavior of publications dealing with teaching methodologies that apply virtual or augmented reality. This analysis was performed using the Web of Science databases to explore the relevance of the theme and its growth. The search terms used were: (teach * and method *) and ("virtual reali" or "augmented reali *" or "virtual world" or "real virtual *" or "VR" or "AR"). Data collection was the period from 2000 to 2018 because at that time, the applications of technologies such as virtual reality and augmented reality increased. The indicators extracted were ✓ the number of publications per year; ✓ the publications area; ✓ and the evolution of publications by research area.
In the second step, related publications, teaching methodologies in engineering that used virtual or augmented reality technology as a teaching-learning strategy. Searches were realized in keywords, title, and abstract in the Scopus and Web of Science databases. The search terms combined were: (teach* and method*) and ("virtual reali*" "or" augmented reali* "or" virtual world "or" virtual real*") and ("engineer*"). The research period covers from 2004 to the present day. The results of these searches and the inclusion and exclusion criteria of the collected data were applied.
The definition of the mentioned bases was because they have indexation with the most significant number of high impact factors and relevant journals.
It was used the model proposed in Figure 2 for the description and analysis of the papers. The criteria for exclusion of the studies were:  1) Exclusion of repeated papers, because it was analyzed two databases; 2) Exclusion of paper that didn't contain at least one of the following terms in their title: students, methodology, method, teaching, learning, simulation software, virtual reality, augmented reality, virtual world, AR (augmented reality), VR (virtual reality), engineering, virtual simulation, virtual lab, education, digital environment, virtual 3D, 3D, virtual platform, virtual simulator; 3) Exclusion of papers, after it was analyzed the abstract, did not broach the teaching methodology related to virtual reality or augmented reality; 4) Exclusion of publications that did not have full texts available in the databases; 5) Exclusion of the articles that were not about teaching in engineering areas, with an application to virtual and/ or augmented reality.
The publications that were not excluded were analyzed about the quality assessment of the study. To be not excluded in this evaluation was necessary for the publication to present: ✓ Teaching which engineering the study was about; ✓ Some results regarding the application of virtual and/or augmented reality in the teaching-learning process.
Publications that did not present these requirements were excluded from the study. One ecological triangulation approach was applied in the qualitative analysis of publications to generate the synthesis of the results. In this evaluation, it is tried to understand, from the literature, why such studies generated such results and what are their reflections on the human condition and its development (Barnett-Page & Thomas, 2009).
In this triangulation, the application area (which engineering and developed subject) of applied virtual reality or augmented reality method, characteristics related to the immersion, interaction, and involvement of the methods employed.
Finally, the third step of the method, the applications in production engineering were detailed, and the possibilities of application that have not been tested in production engineering, generating opportunities, and research gaps.

Results
The first step of data analysis was about the evolution of publications related to teaching methodologies that use virtual or augmented reality. The searches were made in the Web of Science database, with publications made between 2000 and 2018. The database returned 1345 publications. Graph 1 shows the number of publications per year. Graph 1. Publication of materials on teaching methodologies using virtual and/or augmented reality in the Web of Science database.
Graph 1 shows a growing trend of publications about the subject in the last 18 years in the Web of Science database. On average, in 10 years, there were 110.7 publications per year; in the last four years, the publications were around 164.75, showing increased averages. An analysis that shows the growth of the theme is that until 2009, there were 242 publications. However, from 2010 to 2018, there were 896 publications, showing an increase of 287% in these publications. The next analysis concerns the main areas of the theme publications, whose objective is to understand which areas of research are using virtual reality and augmented reality technology. The collected data are available in Graph 2.
According to Graph 2, three research areas stand out in the publication of materials that approach teaching methodologies with the application of virtual and/or augmented reality, namely: Education, Computer Science, and Engineering. They represent more than 60% of the total research areas of publications.
The next step refers to the growing trend of publications in the areas mentioned. The publications were segregated by periods of four years. The data are described in Table 2.  Analyzing the data available in Table 2, five areas that publish the most on the subject had a 90% increase in publications in Web of Science indexed journals, comparing the 2011-2014 and 2015-2018 quadrennium. Also, checking the same quadrennial, individually, it is possible to notice that the areas of Surgery and Health Care Science showed a 28.7% growth in its publications. In Education, this increase was 149% and in Engineering, 92%, showing strong growth in the application of virtual and/or augmented reality in these two research areas.
After the above analysis, a systematic literature review, as described in the second step of the method, was performed. It was found 671 publications in this step. Figure 3 presents the search results, the application of the criteria, and the inclusion or exclusion of publications. After applying all the exclusion criteria determined in the methodology, 40 articles remained for a complete analysis of their texts, and a bibliometric study. The analyses performed were: ✓ the evolution of publications by year; ✓ what kind of technology was used in the teaching-learning process; ✓ and the application in which engineering.
The first analysis was performed on the evolution of publications per year of the included texts, whose results are available in Graph 3.
Graph 3 shows that just one publication occurred in 2006, and after 2009, there is a publication every year. The type of technology pointed out in the publications included for data analysis was examined. The results are available in Graph 4.
The texts selected for the qualitative analysis show that the use of virtual reality in the teaching-learning process is practically 70%. This result is because many of the virtual reality applications are non-immersive, run from the desktop or virtual projection, and there's much software that performs this function.  It was also analyzed how much virtual and augmented reality is being used in different engineering courses, as shown in Graph 5.
Graph 5. Proportion of application of virtual or augmented reality technologies in engineering.
The analysis of this graph shows the total number of applications is larger than the number of texts analyzed, i.e. some publications have application scope in more than one engineering course.
The four engineering courses that apply virtual and/or augmented reality to their teaching-learning processes are mechanical engineering, civil engineering, computer, and production engineering, with respectively 30.0%, 25.0%, 9.0% (computer and production engineering) of the applications in the analyzed texts.
In the fourth position are electrical, industrial, and general engineering, both with 7.0%. The industrial engineering applications evaluated deal with fluid mechanics, metrology, and the application of metal structure calculations. In general engineering classification, applications are in basic engineering disciplines, which cover most courses (calculus, physics, etc.). The authors of the publication do not highlight which engineering course the student has attended.
The quantitative analysis to understand which engineering is using virtual and/or augmented reality. The data examined are available in Table 3. No specific trend was observed by analyzing the data provided in Table 3 for the application of particular technology in given engineering. Already, observing the two highest rates of engineering that use technology (mechanical and civil), it is noted that both use virtual and/or augmented reality in their teaching-learning process.
About the characteristics of immersion, interaction, and involvement provided to the virtual world and virtual objects applied in the research, the type of technology used and the characteristics presented in the study were compared. The information is shown in Table 4. Table 4 shows that a large part of the applications is without immersion, i.e., there are virtual reality applications with immersion being shyly applied in engineering teaching-learning processes. Analyzing the interaction process in the application of virtual or augmented reality technologies in teaching-learning processes is approximately 46% of relationships. Participant involvement is related to user engagement in their performance in the virtual world. Many articles do not cite this involvement.
Another point analyzed was the area of application of virtual reality and augmented reality and their characteristics of use (Martins & Guimarães, 2013). These data follow in Table 5.   Table 5 shows that, in several engineering environments, virtual or augmented reality are used to support teaching-learning processes, bringing students the opportunity to develop pedagogical activities supported by technologies.
About production engineering, it was evaluated all complete papers included in the qualitative analysis, which was detailed in step three of the methodology and after the full reading of the texts, leaving four texts for analyses. (Table 6).
After analyzed the publications highlighted in Table 6, it can be understood the applications of virtual reality in production engineering processes. The main contribution of the authors is presented as follows.
The paper written by Chang et al. (2018) studies a software simulation applied to supply chain management.  a fixed structure, not allowing the elaboration of different scenarios for simulation. The authors propose to develop software that aims to explore supply chain behavior from changing distribution network scenarios. In 2012, the author had used the software simulating supply chain management for an undergraduate course with 21 students. The students were separated into four groups; each group simulated scenarios and made different decisions. At the end of the simulations, their results were analyzed, compared, and questioned why their different choices had led to different results. In the publication by Laseinde et al. (2016), the authors propose to work in an integrated manner between process flow and equipment layout, through virtual reality. In this publication, the authors deal with the manufacture of plastic gears. First, all the gear manufacturing steps are modeled in virtual reality. The entire flow architecture is then analyzed, from the arrival raw material to the finalization of the product, including the arrival at the warehouse of finished products. The concepts were applied to a group of 15 students, comparing the evolution of knowledge retention. Without the application of virtual reality, 25% of students retained knowledge, and with the application of virtual reality, this rate increased to 80% retention.
The research developed by Gamlin et al. (2014) had its approach to the application of virtual reality into lean manufacturing training. The software was designed for training people on how a manufacturing cell works and how production waste and restrictions related to lean manufacturing appears. The project was developed with the support of PepsiCo, in which an example of a virtual manufacturing cell was created. The external software has acted in real-time with the production scenario. The system compared process performance before and after those layout changes. After training, it was expected that the users notice the production system-related losses and make decisions to reduce or eliminate them.
The paper by Cudney et al. (2011) discusses the application of virtual environments in undergraduate curricula for learning about lean methods and simulation environments on the subject. In this model, the first step is the creation of the development of a productive layout, in virtual reality, to apply the concepts. The objective is to evaluate the waste of the production process, and the model proposes tools to reduce the work in process. Then, it discusses the application of tools and their impacts on the process and inventory, such as SMED, unit flow, 5S, and Kaizen.
After analyzing these four papers, it appears that production engineering applications are in areas of the supply chain, production flow, production waste and lean manufacturing tools (Chang et al., 2018;Laseinde et al., 2016;Gamlin et al., 2014;Cudney et al., 2011). All cases demonstrate a more dynamic and interactive teaching environment. All the articles evaluated address the characteristics of cognitivism (Cunha, 2017), in which the student develops knowledge through simulation, challenge, research, and proposals for solving problems. Laseinde et al. (2016), mentions that the retention of knowledge of the subjects discussed increased from 25% to 80%, comparing the application of the traditional teaching method (lecture) to the method using virtual reality.

Discussion of results
This section presents the evolution of the results on the researched theme, its application in engineering, and future researches analysis of this application in production engineering.
It is possible to perceive the quantitative evolution of research on the execution of virtual environments in the teaching-learning processes. The application of these environments currently occurs in several research areas. This perception connects with what Kaliská (2012) says about how human beings can learn in many different ways, and technology is one of them. Libâneo (1994), Gil (2010), and Schlemmer & Backes (2014) relate this evolution. These authors emphasize that the educational processes must compromise with the cognitive question because the cognitive processes of teaching develop students' learning through the development of activities. The augmented and virtual reality bring conditions of immersion, interaction, and involvement, providing a much more interactive environment and student skills development. This consensus explains the growth of research on the subject, showing its relevance. Table 6. Selected articles after a full reading.

Title Authors (Publication year)
A novel network simulation software for supply chain management courses Chang et al. (2018) Educating Tomorrows Engineers: Reinforcing Engineering Concepts Through Virtual Reality (VR) Teaching Aid Laseinde et al. (2016) Immersive virtual reality deployment in a lean manufacturing environment Gamlin et al. (2014) Enhancing undergraduate engineering education of lean methods using simulation learning modules within a virtual environment Cudney et al. (2011) It is a new trend according to the understanding that the students can learn in various ways, that cognition is fundamental in the learning process, and that augmented and virtual reality act on the cognitive issue. The application of these technologies in teaching is growing. It covers many areas of research, and it is one of the ways used to develop cognition and bring the student very close to a real environment, making it easier to understand theoretical concepts.
When evaluating the practice of virtual and/or augmented reality in engineering, we notice a certain level of stability of publications, highlighting the areas of civil and mechanical engineering, both of which use virtual and augmented reality.
One finding is that the application occurs in several engineering-related subjects, generating significant learning process results in the evaluated publications. Valdez et al. (2014) and Braga (2001) support this understanding, emphasizing that virtual environments are multidisciplinary and applicable to teaching-learning processes, and produce the development of knowledge.
Practically all engineers use such technologies to support teaching-learning processes, but they scarcely explore their tools' immersion, interaction, and involvement characteristics. However, according to Schlemmer & Backes (2014), this application directly affects cognitive issues and is essential in the teaching-learning process. The implementation of these technologies, taking into account their characteristics, would generate a much more interactive environment, providing a realistic environment much closer to the ideal for student learning (Valdez et al., 2014;Braga, 2001).
It is perceived as a shy use of virtual and/or augmented reality in teaching-learning processes when evaluated the application in production engineering. It is found as themes of publications layout and production flow, analysis wastes of the production system, and supply chain management. Only one publication brings the practice of immersion in virtual reality when observing the characteristics of research applications in production engineering. However, all have the use of interaction and involvement with the developed environment (Chang et al., 2018;Laseinde et al., 2016;Gamlin et al., 2014;Cudney et al., 2011).
The document developed by Associação Brasileira de Engenharia de Produção (2003), about the curricular guidelines of production engineering, shows that many areas of study still do not use virtual and/or augmented reality as a tool to support the teaching-learning processes. However, when used, the impact of the teachinglearning process or satisfaction of the students is not measured.
Virtual or augmented reality application models could be developed for other issues related to production engineering, such as Production Systems Management, Production Planning, and Control, Material Handling, Production Simulation, Production Management. Productive Processes, Metrological Quality Organization, Work Organization, Accident Risk Analysis and Prevention, Work Safety, Product and Process Ergonomics, among others, including applications dealing with supply chain, productive flow, production wastes, and lean manufacturing, with different approaches and characteristics of the four articles evaluated in this research.
Based on the understanding that the use of virtual reality and augmented reality in teaching-learning processes, there is a tendency of application. When analyzing the notes of the areas of production engineering that could use the application, it appears that there are many application opportunities, and their non-use generates some impacts.
One of the impacts is the less attractive and very theoretical teaching-learning processes, failing to transmit to students all the necessary knowledge for the development of production engineering themes. This is because the traditional approach to teaching is focused on the teacher, with the student as an onlooker in the teachinglearning process. However, the cognitive approach genuinely addresses the interaction between subject and object. In this approach, the student should be allowed to conduct research based on trial and error, analysis, and problem-solving (Cunha, 2017).
The second impact is that students leave with learning gaps. These deficiencies often may not be resolved during professional development, causing loss of competitiveness of the production engineer in their professional performance. These gaps are noticed when Laseinde et al. (2016) mentions that without the application of virtual reality, 25% of the students retained knowledge. This index increased to 80% retention with the use of virtual reality.
The third impact is the inability of organizations to use all the concepts proposed by production engineering because professionals arrive with knowledge gaps. This impact makes organizations lose competitiveness, become more vulnerable to the market, and expose medium-and long-term sustainability risks. There are opportunities to apply these concepts within organizations in training, simulations, and other applications.
Based on the impacts seen in production engineering, it is stated that generating teaching-learning processes that form the student more completely, without gaps, potentially impact organizational competition at times when they enter the job market, due to the improvement of the learning and satisfaction rates of students who use it in the teaching environment (Ehmann & Wittenberg, 2018;Fiorentino et al., 2009;Inayat et al., 2016). Therefore, the application of teaching approaches that use virtual or augmented reality in production engineering would yield results, making the subject relevant for discussion.

Conclusion
This research shows that the application of virtual reality and augmented reality in teaching-learning processes increased significantly in the analyzed period. Virtual environments are now available for students from several areas of expertise.
When approached in the context of engineering, it can be seen that the application of virtual and augmented reality tools is diversified, acting in several engineering areas, primarily civil and mechanical engineering. It is noteworthy that the application occurs extremely broadly, in various subjects, such as physics, electronic circuits, fluid mechanics, among others.
Another important conclusion is that, in all the studied cases, the execution of virtual or augmented reality in support of engineering teaching-learning processes improved the learning index, showing that the application of technologies is relevant to these results and acts cognitively on students, accelerating the learning process.
In production engineering, there are still few publications on the subject, but the existing ones bring application examples, whose results are satisfactory.
It is worth noticing that there are many opportunities to implement this type of technology for teaching in production engineering. This use help students understand the concepts of production engineering and their functionality, generating more competitive professionals to work in the market, making organizations more competitive from the complete knowledge of production engineers.
In the analysis of qualitative texts, it was noticed that the application applied in the areas of the supply chain, productive flow, production wastes and lean manufacturing tools (Chang et al., 2018;Laseinde et al., 2016;Gamlin et al., 2014;Cudney et al., 2011).
From the applications, it was noticed other opportunities for future research. For example, virtual reality or augmented reality can be used to measure the performance of the teaching-learning process or the satisfaction of students in production engineering. Also, in the areas of Production Systems Management, Production Planning and Control, Material Handling, Production Simulation, Productive Process Management, Quality Metrological Organization, Work Organization, Accident Risk Analysis and Prevention, Work Safety, Product and Process Ergonomics.
Other future research opportunities are related to the characteristics of the application of virtual reality and augmented reality, such as, didactic-pedagogical characteristics, educational strategies, environments, tools, cases of success and failure and application barriers.