Augmented reality in interventional radiology education: a systematic review of randomized controlled trials

ABSTRACT BACKGROUND: Augmented reality (AR) involves digitally overlapping virtual objects onto physical objects in real space so that individuals can interact with both at the same time. AR in medical education seeks to reduce surgical complications through high-quality education. There is uncertainty in the use of AR as a learning tool for interventional radiology procedures. OBJECTIVE: To compare AR with other learning methods in interventional radiology. DESIGN AND SETTING: Systematic review of comparative studies on teaching techniques. METHODS: We searched the Cochrane Library, MEDLINE, Embase, Tripdatabase, ERIC, CINAHL, SciELO and LILACS electronic databases for studies comparing AR simulation with other teaching methods in interventional radiology. This systematic review was performed in accordance with PRISMA and the BEME Collaboration. Eligible studies were evaluated using the quality indicators provided in the BEME Collaboration Guide no. 11, and the Kirkpatrick model. RESULTS: Four randomized clinical trials were included in this review. The level of educational evidence found among all the papers was 2B, according to the Kirkpatrick model. The Cochrane Collaboration tool was applied to assess the risk of bias for individual studies and across studies. Three studies showed an improvement in teaching of the proposed procedure through AR; one study showed that the participants took longer to perform the procedure through AR. CONCLUSION: AR, as a complementary teaching tool, can provide learners with additional skills, but there is still a lack of studies with a higher evidence level according to the Kirkpatrick model. SYSTEMATIC REVIEW REGISTRATION NUMBER: DOI 10.17605/OSF.IO/ACZBM in the Open Science Framework database.


INTRODUCTION
Learning is the process of acquiring new knowledge and skills, and this process has its difficulties and pitfalls. 1,2 In medicine, acquiring new abilities can lead to improvement in outcomes, as in the field of surgery, in which open surgical procedures have been replaced by minimally invasive procedures, and fresh devices are created to refine surgical abilities, and teaching processes as well. 3,4 The "learning before doing" concept is rapidly replacing the conventional "see one, do one, teach one" technique, in order to avoid potential mistakes. 5,6 According to British National Health Service data, preventable injuries and deficient medical training are responsible for 10% of hospitalizations. 7 In consonance, "warm-up" can be applied to students and experienced professionals, thus boosting performance and self-confidence. 8 This could form another application for augmented reality (AR).
AR involves digitally overlapping virtual objects onto physical objects in real space so that individuals can interact with both at the same time. Virtual reality produces immersion of the user in a given environment, which may or may not be controlled, by depriving the perception of the local environment through use of a computerized scenario or one previously captured on video, and experiencing an environment as if it existed. [9][10][11][12][13][14][15] With AR, users visualize the real situation in which they are immersed along with a virtual projection of a 3D image.
This immersion can be enhanced with sound, touch and smell through integrated external components. 10,11,13,[16][17][18] Increasingly, use of mobile AR (mAR) makes time and location flexible and expands training time. 10,19,20 Interventional radiology consists of imaging-guided minimally invasive procedures that enable lower morbidity and shorter hospitalization time. 7 Spatial and cognitive proprioception are the main difficulties identified during training. 21,22 Acquisition of skills to use new devices is also a common issue, which can cause tragic outcomes, especially at the start of a career. 21,22 Therefore, AR may improve medical teaching and enhance skills relating to given procedures. 23,24 Preliminary studies comparing use of AR with traditional teaching methods have produced promising results. 3,4,25 There is no systematic review about augmented reality in interventional radiology.

OBJECTIVE
The aim of this study was to identify, systematically analyze and summarize the best available evidence comparing AR teaching techniques with various other methods in interventional radiology.

Inclusion criteria
We included studies that compared the AR method with several other teaching methods -phantom, cadavers, porcine method and didactic teaching (books, articles, lectures without the use of AR) -in interventional radiology. No restrictions concerning the language, publication status of the study or population were imposed.

Selection of studies and data extraction
Eligible studies were identified using a two-stage method by two independent reviewers (AYPG, MLD). Disagreements were settled by reaching a consensus. First, after eliminating duplicates, titles and abstracts retrieved through the search strategy were evaluated, thus yielding potentially eligible studies. Second, fulltext evaluation of the pre-selected studies was performed to confirm eligibility; this process was carried out through the Rayyan platform (https://rayyan.qcri.org). 26

Evaluation of methodological quality
The Cochrane Collaboration tool was applied to assess the risk of bias in individual studies and across studies. 27 Eligible randomized controlled trials (RCTs) were analyzed using the quality indicators from Best Evidence Medical Education (BEME) Collaboration Guide no. 11 28 (Annex 1) and the Kirkpatrick model (BEME Guide no. 8) ( Table 1). 29,30 According to BEME Guide no. 11, higher quality studies meet a minimum of seven out of eleven indicators. The tools are well established and cover a wide spectrum of methodological issues.
Articles that did not compare teaching methods, along with those with a population dropout rate ≥ 50% (as prescribed in BEME Collaboration Guide no. 11) and those analyzing factors other than medical teaching, were excluded.

Search methods for choosing studies
Electronic searches were performed in the PubMed, Cochrane Library, Embase, ERIC, CINAHL, Tripdatabase and SciELO References from the studies included and from the main reviews on the subject were also analyzed. The search strategies were carried out on July 29, 2020, for each database, and are shown in Table 2.

RESULTS
The search yielded 5189 articles; 50 of these were duplicates and were excluded. Through analysis on the titles and abstracts, 56 articles were selected for full-text evaluation, out of which four were included (Figure 1). Among these 56 articles, Grasso et al. 31 did not evaluate the learning that resulted from the teaching methods and was excluded from the analysis.
Two of the four studies were from Canada 15,17 and used a pre-experience questionnaire; the other two were from the United States 14,18 and used both a pre-experience and a post-experience questionnaire. We found that heterogeneity was present among both the participants and the procedures analyzed. All of these studies were RCTs in which, differently from the intervention group, the control group did not have access to an AR device; while the remaining instructions and other materials (books and didactic lessons) were equal for the two groups.
All of these four studies reported that changes in perspective or judgment occurred in the groups of participants, concerning teaching and learning (Kirkpatrick evidence level 2B).
Regarding procedures, two studies analyzed central venous catheter placement, 14,18 one study evaluated the lumbar puncture procedure 15 and one investigated injection into the interfacetal joint. 17 Although a diversity of issues were analyzed among these trials, the performance achieved through the technique was the main outcome in all of them. Regarding the populations investigated, the participants comprised respiratory therapists, sleep technicians, pre-medical and medical undergraduate students, emergency medicine and surgery residents and anesthesiologists.
In three of the studies analyzed, 14,15,17 it was concluded that AR could increase students' skills in interventional radiology.
AR is used in a variety of areas of medicine and no systematic review or clinical trial has been carried out using a homogenous population. Because of the heterogeneous nature of the populations studied, different AR devices analyzed and different medical procedures used in these four RCTs, we did not perform any meta-analysis. Table 3 shows the quality assessment and risk of bias analysis conducted using the Cochrane Collaboration tool.
Huang et al. 14  in the median total duration of the procedure was found between the groups. Most participants (71%; n = 23) were successful in cannulating the internal jugular upon the first attempt (12 in the AR group versus 11 in the non-AR group). A significant difference in adherence level between the two groups (22.9 ± 4.1 in the AR group versus 18.1 ± 6.3 in the non-AR group; η 2 = 0.90; P = 0.003) was detected. In the post-exercise questionnaire for the AR group, more than 80% of the participants stated that the instrument did not cause any fatigue and was not too heavy to be uncomfortable.
Nonetheless, 30% admitted that the equipment affected their action skills and that it was not easy to regulate. On the other hand, 94% reported that the hand, head and foot interactions were undemanding and 80% stated that the information presented on-screen was suitable and reacted fast enough.
In the study by Wu et al., 18  showed that the majority of the participants were not previously familiar with AR or with wearable computing technology (75% and 60%, respectively); however, 73% reported having some degree of knowledge about Google Glass. Nonetheless, 87% of the participants randomized to Google Glass reported that the instrument was comfortable to use for the procedure.
Keri et al. 15 evaluated the effectiveness of Perk Tutor (GPS extension, Ultrasonix, Canada) in relation to a phantom, as a teaching method among anesthesiology and surgery residents for lumbar puncture procedures. Perk Tutor is a training All of these results are summarized in Table 4. 14,15,17,18   Records excluded due to title and abstract (n = 5,083) Full-text articles assessed for eligibility (n = 56) 52 full-text articles excluded, for following reasons: -20 did not apply to the specialty of interventional radiology -18 did not compare the teaching methods -10 evaluated other technologies -3 without control group -1 did not evaluate the learning that resulted from the teaching methods Studies included in qualitative synthesis (n = 4) Records excluded due to duplicates (n = 50)

DISCUSSION
The objective of our study was to examine the current evidence on training using AR in interventional radiology and its performance, along with the impact of AR on educational outcomes and skills, and its main advantages, disadvantages and challenges during the teaching-learning process.
New teaching techniques such as virtual reality (VR), AR or mixed reality (MR) are being introduced in medical education. 5,32 AR combines virtual and real-world through use of wearable technology that provides a live feed from computer workstations (i.e. from an ultrasound device). 18 Images and information are shown in the user's line of sight through the device. 18 Everyday use of mobile devices facilitates implementation of this instructional tool in teaching processes, which permits access to learning at any moment. 33,34 However there is still a lack of research regarding the competence of this technology. 33 AR methods have stood out in the surgical environment over recent years, through providing educational simulation practice free from potential ethical/hygiene concerns. 35 Furthermore, the pressure imposed on healthcare systems during the COVID-19 pandemic has hastened implementation of new technologies, thereby accelerating the learning of healthcare professionals. 36 Students are now used to dealing with technologies such as the internet, 3D video games, cell phones and others. 19,20,[37][38][39][40] Teachers can avail themselves of this familiarity to upgrade teaching methods and aids, so as to encourage students. 19

2B
The main purpose of AR involves the concept of "practice makes perfection", given that efficient performance in procedures requires experience. 45 AR simulation provides the possibility of repetition to boost self-confidence, within a safe method. 46 Over five million central venous catheters are fitted each year in the United States. The complication rates are 5%-8% higher per procedure when these are handled by novice professionals. 14 Teaching with AR aids could result in lower morbidity, hospitalization time and costs.
Comparison of learning between novice physicians and experienced interventional radiologists could enable evaluation of whether AR has the capacity to accelerate learning. Studies comparing different kinds of AR in one specified procedure need to be performed in order to determine which technology is better for that particular procedure. From the current information available, AR is a useful additional tool for teaching interventional radiology, but not a substitute for the traditional methodology.
From the students' perspective, AR can contribute to mastery and confidence in a new procedure, through enabling students to memorize details, thus decreasing the tension in real-life situations.
Regarding classroom ambience, AR may enable a shift from the monotonous routine of expository classes, thus providing evolution of the learning experience. Assembling education with technology would engage young people, thereby transforming learning into a pleasant experience and improving learning, as well as clinical practice.
One limitation of this systematic review was that only two studies analyzed the same procedure. 14,18 Numerous procedures are involved in interventional radiology, but in the four studies evaluated, only three different procedures were investigated: central venous catheter placement, lumbar punctures and interfacetal joint injection. Two different types of AR devices were tested: Perk Tutor and AR glasses. Different AR devices could be compared in the future. Moreover, the small samples used in the studies represented another limitation, thus hampering generalization. Another limitation was the lack of evaluation among experienced professionals. The participants included in these studies were novice physicians or non-physicians; none of these studies investigated radiology residents.
The level of evidence of the studies was also a limitation: all of them were classified as 2B in the Kirkpatrick model. 29 Our searches did not retrieve any studies with educational evidence at level 3 (behavioral change), 4A (change in the organizational system/ practice) or 4B (change among participants, students, residents or colleagues). Despite the current interest 22 in using simulators, it remains to be delineated which types of simulation and simulator should be used, and what population this teaching method will be applied to. Hence, a higher level of evidence is needed.
Hardware needs are also a concern, considering that running the application produced intense energy usage as well as device heating. 34 These technical difficulties could be resolved by the smartphone industry. Use of faster networks enables a shared environment through cloud services and shared real-time information. Introduction of artificial intelligence to AR-based learning programs can also provide more positive learning.
The costs of AR devices are expected to decrease along with the evolution of production and increased market competition, thus bringing these technologies to low-income countries. Moreover, AR-based medical training could facilitate teaching for people with reading limitations and could also facilitate remote teaching.

CONCLUSION
It was demonstrated through this study that AR, as a complementary tool, can add skills to learners and thus can improve the teaching-learning process. It needs to be noted that only level 2B studies were found in this systematic review and, thus, that a higher level of evidence is required. Moreover, comparison of beginner physicians and expert interventional radiologists could enable appraisal of the hastening of the learning curve through AR, as well as investigation of which set of AR tools is most adequate for each procedure.