Virtual simulations for health education: how are user skills assessed?

Introduction: A virtual simulator, or one based on virtual reality, can computationally recreate real contexts. Objective: To analyze works on virtual simulations for training clinical procedures, focusing on the assessment of user skills. Method: Integrative literature review, carried out between 2010 and 2020. A total of 56 studies were selected 56 studies. Results: The selected studies showed that the variables and parameters of virtual simulators are usually obtained by consulting experts or through medical literature. These simulators mainly focus on developing psychomotor skills and assessing the learner’s performance through real-time alerts, progress indicators, and performance reports after the end of each training. Conclusion: Considering the expert’s knowledge exclusively to define the requirements of virtual simulators can limit their reliability and accuracy. The participation of experts in these projects does not follow standards regarding the selection and frequency with which they collaborate. Few simulators provide insightful and pertinent feedback on user performance.


INTRODUCTION
Medical education has changed, especially regarding the student's training method 1 . Due to the characteristics of the traditional training model, it is often impossible to experience more significant variability and complexity of clinical cases. In addition, another difficulty reported is the students' insecurity when treating patients for the first time 2 . Thus, health education professionals have been looking for new techniques to improve students' clinical skills and ensure the patient's integrity.
Traditionally, in health education, simulations are carried out to train clinical procedures. For these simulations, instructors use mannequins, animals, and corpses. However, these practices have the disadvantage of requiring on-demand preparation and raise ethical issues 3 . In this sense, virtual simulators emerged as an alternative to traditional training.
They allow the user to interact with a virtual environment that is similar to the interaction in the real world, reducing costs and enabling the experience of more significant variability of clinical cases.
There are minimum criteria for developing training with simulations, such as defining the expected objective, the target audience, the application usage scenario, the challenge difficulty, the main application subject, and the concepts related to it 4 . Thus, adequately designed simulation-based training can significantly reduce health professionals' errors and improve patient safety 2

RESEARCH METHODOLOGY
The integrative literature review was chosen as the methodological procedure for the selection and analysis of research related to the subject. Each step will be detailed below, as well as the obtained results.

Research questions
Virtual simulators for training address skills to be developed and the several ways to assess their acquisition. Therefore, the following research questions (RQ) were defined aiming to contemplate the different scenarios on the topic:

Papers screening
After defining the research questions, the next step was to select the studies for analysis. Terms commonly present in studies on virtual simulators for health training were used in the search to cover the largest number of articles and obtain a broad state-of-the-art view. As a result, the following search string was obtained: ("augmented reality" OR "virtual reality" OR "simulation" OR "simulator" OR "haptic" OR "haptics") AND ("medical education" OR "medical training").
The following digital libraries were searched: ACM Digital Library, IEEE Xplore, and PubMed, considering that the first two are important publication vehicles for Computer Science and Medical Informatics. The studies were selected based on the title/abstract search using the search string defined above.
In addition to the cited databases, we also included articles that we deemed relevant or appeared in the selected studies' references. Based on the search string and after reading the titles of the studies, we initially selected 210 articles. Finally, all articles were read in full and the exclusion criteria were applied.
The defined exclusion criteria ware: 1. not addressing health procedures, 2. not addressing skill development for health professionals, and 3. dealing with non-digital simulations. Most of the excluded studies included non-digital simulations. In this last phase, a total of 56 studies were selected, which were relevant to our objective.

RQ1: How are the variables and parameters of virtual simulators defined?
In this study, the variables of a clinical procedure mean the set of elements that constitute it, such as the instruments used and the professional's performance. The parameters mean the values assigned to the variables, which can be the presence of a step or clinical instrument, as well as a range of acceptable values, such as needle depth and angulation. Virtual simulators usually use the variables and parameters to assess user performance, and they influence the accuracy and reliability of the simulator.
Most of the analyzed studies defined the variables and parameters of the virtual simulators with the help of experts in the area, which is equivalent to more than 70% of the selected studies. The participation of these experts took place through multidisciplinary research groups or by inviting them to participate in specific stages of the virtual simulator production.
Their participation in the testing stage is also common. However, we observed that there is no defined standard regarding the participation of these professionals, and the publications do not provide clear indications about the methods and criteria for selecting experts. Furthermore, it is not clear how often their consulting activities takes place. Considering that the analysis and definition of a system's requirements are cyclical, this step will require those involved in more than one moment, for example. In turn, the multidisciplinary teams theoretically allow the more active and frequent participation of professionals from different fields of activity, being able to identify variables and parameters with precision, influencing the reliability and accuracy of the final product. Concerning quantity, two studies that only consulted one specialist were identified. However, this should not be a limitation if there are more professionals involved in the validation step.
Another way that was identified to obtain the requirements is from the medical literature; around 13% of the studies did this.
Only approximately 11% of the studies combined the literature with the participation of experts. The literature is one of the t resources traditionally present in professional training, so relying on its information is considered reliable, although the handson experience can add observations that will make the virtual simulation even more realistic.
The knowledge construction process consists of three stages: knowing, knowing how to do and knowing how to be.
The awareness of education enables the learner to acquire information. The knowing stage operates in the sense of giving meaning to the theory, practically transforming it into knowledge, which is related to the stage of knowledge.
Knowing how to do it is related to putting knowledge into use.
That is, the development of skills related to the studied field.
Finally, knowing how to be is about the learner's attitude. It lies in deciding to put knowledge (knowing) and skills (knowing how to do) into motion. In this sense, the literature can assist in identifying variables, parameters, and clinical procedures steps. However, the granularity of the problem and the criteria for evaluating the identified variables and parameters are often not found in the books, making the participation of experts necessary to assist in modeling the parameters that will be inserted in the simulator and defining the assessment criteria for the variables. Only experts can assist in obtaining this information with a focus on knowing how to do and knowing how to be. We consider that the exclusive use of the literature can limit the establishment of parameters for assessment while relying exclusively on the expert can limit the reliability and accuracy of the simulation. The ideal situation is to consult both sources in a complementary way. We also consider important that teams involved in developing virtual simulations use validated techniques to define criteria for selecting experts, such as the Fehring model 60 .
We also identified a considerable amount of proposals with little or no detail on how they defined the virtual simulator variables and parameters. A total of 10 studies did not specify this information. This amount is higher than that of studies based on the literature. Although in half of these studies, the text suggests the literature, mainly related scientific articles, they do not specify how they extracted the information in question. The other half is about tools that share tactile force feedback, whose later tests cover aspects of performance or usability validated by students, and in one case, by experts. One of the studies focuses exclusively on describing the proposed technique without detailing the source of information.

RQ3: How are the skills assessed?
In their study, Machado et al. 63  Finally, there are the progress indicators, which provides: 1.
scores for correct answers, 2. retention on a given task until completion, and 3. the possibility of repeating a particular task to improve practice. These indicators are objective ways of verifying whether the user achieved the expected performance.
In addition, they act as motivating agents, as they adopt gamelike characteristics.
There was a considerable number of studies (40% of them) whose objective is to present a new simulation technique, such as the texture and cutting of human tissues, or the advancement of an existing computational technique.
These studies focus almost exclusively on engineering aspects, leaving the assessment of skills as a future task or approaching it superficially. Such studies do not detail assessment information.
These virtual environments commonly bring only the tactileforce feedback. Although tactile feedback plays an essential role in aiding the development of clinical skills, in some cases, these studies do not take into account the online assessment process and condition the assessment of user skills to the presence of a supervisor (offline assessment). Approximately 43% of virtual simulators need instructors to carry out user skill assessments. We also observed that, regardless of this aspect, 56% of the analyzed simulators have their use conditioned to the instructor's presence. Another point that was observed is that most studies use simulated data when performing tests 37 .

RQ4: How is the virtual simulator effectiveness verified for the acquisition of skills?
The verification of the occurrence of skills acquired by students is a point of attention in the studies. In simulators that use alert mechanisms, is assumed that the reduction of these messages indicates fewer errors, hence a behavioral change. However, we are aware of the importance of timely feedback. The same goes for progress indicators, as it is necessary to provide the student with a meaningful reflection about their performance to avoid resorting to trial and error. In performance reports, some studies do not clarify how the information is made available. In addition to being quantitative, this feedback must also be qualitative in the sense of presenting a clear textual (or visual) feedback; otherwise, the presence of an instructor will still be necessary to complete the training cycle.
Among the selected studies, there are 36 (64% of them) in which the presence of an instructor is necessary to provide some feedback on the user's performance in virtual simulation. In 20 of these studies, it is assumed that an instructor is needed because, in addition to not providing feedback on student performance, these simulators do not have mechanisms capable of situating learners about the completion of their tasks. There were two of these studies 10,17 in which the instructor's presence was only necessary for some simulation modules. In these cases, the presence of instructors was due to network interaction that allowed discussions between students and professionals.
Bloom's taxonomy 66 classifies learning as a plural and interactive phenomenon that co-occurs in the cognitive, affective, and psychomotor domains. Virtual simulations that operate on motor skills can directly contribute to achieving educational goals related to the psychomotor domain. The psychomotor domain deals with behaviors that imply the development of neuromuscular coordination. It related, therefore, to the acquisition of skills that combine muscle actions, cognition, skills to manipulate objects or perform a procedure 67 . Although Bloom and his team never defined a taxonomy for the psychomotor domain, others did it. Dave's Dave's classification 68 for the psychomotor domain is the most often cited interpretation. It consists of five categories: imitation (observing the skill and trying to repeat), manipulation (following instructions, memorizing a procedure and being able to reproduce it), precision (performing the skill accurately and without help), articulation (combining skills to achieve a non-standard goal) and naturalization (unconscious domain of activity, when one becomes an expert).
The naturalization category is not typically covered in virtual simulations since to become an expert, the apprentice will need time of practical experience with actual patients. Regarding the precision category, as already mentioned, although computer simulations have the potential for selfguided use, the absence of this functionality is still common. It implies challenges in advancing the goals of the psychomotor domain, since the presence of an instructor is still necessary. Regarding the articulation category, although there efforts have been made in this direction, it constitutes a challenge for simulations to present greater variability in clinical cases, aiming to predict the wide range of possible clinical scenarios for the same procedure. This demand reinforces the need for constant and active participation of domain experts.
In addition to reducing costs and support the tactile and visual aspects, another differential of training mediated by virtual simulators is the possibility of using it at any time, without requiring supervision and receiving accurate and agile feedback about the user's performance. In this sense, studies should further explore this potential. We understand that virtual simulators can be efficient tools for skill acquisition. However, feedback provided to the user about their actions during the simulation plays a fundamental role in the knowledge construction process. Therefore, these tools must provide personalized, relevant, and timely feedback on user performance. As noted, few studies have addressed the assessment of learning, and as it is an automatic assessment, few studies have implemented this functionality. Thus, we understand this potential area of the research landscape as a possible open problem.
Finally, when developing a virtual simulator for training, considering the user's skills assessment, the results of this study show that it is essential to observe the following steps: • Design: a step that will deal with the survey and definition of the tool's requirements. The criteria for selecting specialists must follow standards already reported in the literature, and the times and frequency of their participation must be specified. In addition, for the definition of variables and parameters, the knowledge established in the literature should be considered, complemented by the experts' knowledge. We recommend developing documents such as concept and navigation maps and using case diagrams to help the interdisciplinary team communicate.
• Skill feedback: a step that will verify the correct operation of the simulator, its educational effectiveness, and the user's performance in the simulation. The presence of experts is necessary to validate the requirements defined in the design stage and collaborate modeling skill assessment methods by the simulator (online assessment). As for evaluating educational effectiveness, we observe that the singleuse effects may not reflect the learning reality. Thus, it is essential to analyze how this construction of knowledge occurs over time and its effects on the work process. The acquisition of knowledge can be carried out in the short, medium, or long term, and what we observed in the studies are the single-use effects. Virtual simulators must provide personalized, relevant, and timely feedback to the student about their performance.

CONCLUSION
This review aimed to analyze studies about simulations in virtual graphic environments for training and education in the health area, focusing on the user skill assessments procedures. We conclude that using virtual simulators for training and assessment of clinical skills can be effective by using reliable and well-established variables and parameters, supporting teaching and learning in the field of health education. Considering the primary question of this research, we found that most simulator evaluations are not related to user skills but usability aspects. Thus, there is a significant research area to be explored in this direction, specifically related to user skill assessment, since this kind of evaluation has been little addressed in virtual simulators. It demands efforts to integrate research from computer science, engineering, and health areas.