Augmented reality in interventional radiology education: a systematic review of randomized controlled trials

ABSTRACT

BACKGROUND:

Augmented reality (AR) involves digitally overlapping virtual objects onto physical objects in real space so that individuals can interact with both at the same time. AR in medical education seeks to reduce surgical complications through high-quality education. There is uncertainty in the use of AR as a learning tool for interventional radiology procedures.

OBJECTIVE:

To compare AR with other learning methods in interventional radiology.

DESIGN AND SETTING:

Systematic review of comparative studies on teaching techniques.

METHODS:

We searched the Cochrane Library, MEDLINE, Embase, Tripdatabase, ERIC, CINAHL, SciELO and LILACS electronic databases for studies comparing AR simulation with other teaching methods in interventional radiology. This systematic review was performed in accordance with PRISMA and the BEME Collaboration. Eligible studies were evaluated using the quality indicators provided in the BEME Collaboration Guide no. 11, and the Kirkpatrick model.

RESULTS:

Four randomized clinical trials were included in this review. The level of educational evidence found among all the papers was 2B, according to the Kirkpatrick model. The Cochrane Collaboration tool was applied to assess the risk of bias for individual studies and across studies. Three studies showed an improvement in teaching of the proposed procedure through AR; one study showed that the participants took longer to perform the procedure through AR.

CONCLUSION:

AR, as a complementary teaching tool, can provide learners with additional skills, but there is still a lack of studies with a higher evidence level according to the Kirkpatrick model.

SYSTEMATIC REVIEW REGISTRATION NUMBER:

DOI 10.17605/OSF.IO/ACZBM in the Open Science Framework database.

KEYWORDS (MeSH terms):
Radiology, interventional; Augmented reality; Education, medical; Simulation training

AUTHORS’ KEYWORDS:
Video game; Perk Tutor; Google Glass; Wearable technology

INTRODUCTION

Learning is the process of acquiring new knowledge and skills, and this process has its difficulties and pitfalls.¹,² In medicine, acquiring new abilities can lead to improvement in outcomes, as in the field of surgery, in which open surgical procedures have been replaced by minimally invasive procedures, and fresh devices are created to refine surgical abilities, and teaching processes as well.³,⁴

The “learning before doing” concept is rapidly replacing the conventional “see one, do one, teach one” technique, in order to avoid potential mistakes.⁵,⁶ According to British National Health Service data, preventable injuries and deficient medical training are responsible for 10% of hospitalizations.⁷ In consonance, “warm-up” can be applied to students and experienced professionals, thus boosting performance and self-confidence.⁸ This could form another application for augmented reality (AR).

AR involves digitally overlapping virtual objects onto physical objects in real space so that individuals can interact with both at the same time. Virtual reality produces immersion of the user in a given environment, which may or may not be controlled, by depriving the perception of the local environment through use of a computerized scenario or one previously captured on video, and experiencing an environment as if it existed.⁹–¹⁵ With AR, users visualize the real situation in which they are immersed along with a virtual projection of a 3D image. This immersion can be enhanced with sound, touch and smell through integrated external components.¹⁰,¹¹,¹³,¹⁶–¹⁸ Increasingly, use of mobile AR (mAR) makes time and location flexible and expands training time.¹⁰,¹⁹,²⁰

Interventional radiology consists of imaging-guided minimally invasive procedures that enable lower morbidity and shorter hospitalization time.⁷ Spatial and cognitive proprioception are the main difficulties identified during training.²¹,²² Acquisition of skills to use new devices is also a common issue, which can cause tragic outcomes, especially at the start of a career.²¹,²² Therefore, AR may improve medical teaching and enhance skills relating to given procedures.²³,²⁴ Preliminary studies comparing use of AR with traditional teaching methods have produced promising results.³,⁴,²⁵

There is no systematic review about augmented reality in interventional radiology.

OBJECTIVE

The aim of this study was to identify, systematically analyze and summarize the best available evidence comparing AR teaching techniques with various other methods in interventional radiology.

METHODS

The PICO technique (Population, Intervention, Comparison, Outcome) was used to define the study, as follows:

P = Undergraduate healthcare students; postgraduate trainees; continuous professional development training – independent of the specialties.

I = Augmented reality to teach interventional radiology.

C = Traditional methodology versus AR.

O = Improve ultrasound skills to achieve an accurate diagnosis

Study model

This systematic review was executed in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and Best Evidence Medical Education (https://www.bemecollaboration.org/), and was registered in the Open Science Framework (https://osf.io/wn762). This study was exempted from institutional review as no live subjects were studied.

Inclusion criteria

We included studies that compared the AR method with several other teaching methods – phantom, cadavers, porcine method and didactic teaching (books, articles, lectures without the use of AR) – in interventional radiology. No restrictions concerning the language, publication status of the study or population were imposed.

Selection of studies and data extraction

Eligible studies were identified using a two-stage method by two independent reviewers (AYPG, MLD). Disagreements were settled by reaching a consensus. First, after eliminating duplicates, titles and abstracts retrieved through the search strategy were evaluated, thus yielding potentially eligible studies. Second, full-text evaluation of the pre-selected studies was performed to confirm eligibility; this process was carried out through the Rayyan platform (https://rayyan.qcri.org).²⁶

Evaluation of methodological quality

The Cochrane Collaboration tool was applied to assess the risk of bias in individual studies and across studies.²⁷ Eligible randomized controlled trials (RCTs) were analyzed using the quality indicators from Best Evidence Medical Education (BEME) Collaboration Guide no. 11²⁸ (Annex 1) and the Kirkpatrick model (BEME Guide no. 8) (Table 1).²⁹,³⁰ According to BEME Guide no. 11, higher quality studies meet a minimum of seven out of eleven indicators. The tools are well established and cover a wide spectrum of methodological issues.

Table 1
Kirkpatrick’s hierarchy³⁰

Level	Feature	Evaluation
1	Reaction	Participants’ opinions about the learning experience, its organization, presentation, content, teaching methods and quality of instruction
2A	Learning - Change in attitude	Changes in attitudes or perceptions among participating groups concerning teaching and learning
2B	Learning - Modification of knowledge or skills	For knowledge, this refers to acquisition of concepts, procedures and principles For skills, this refers to acquisition of thinking/problem-solving, psychomotor and social skills
3	Behavior - Behavioral change	Documents transfer learning to the workplace or students’ willingness to apply new knowledge and skills
4A	Results - Change in the organizational system/practice	Refers to broader changes in the organization, attributable to the educational program
4B	Results - Change between participants, students, residents or colleagues	Refers to the improvement in learning/performance of students or residents as a direct result of educational intervention

Articles that did not compare teaching methods, along with those with a population dropout rate ≥ 50% (as prescribed in BEME Collaboration Guide no. 11) and those analyzing factors other than medical teaching, were excluded.

Search methods for choosing studies

Electronic searches were performed in the PubMed, Cochrane Library, Embase, ERIC, CINAHL, Tripdatabase and SciELO databases, using the following MeSH terms: Interventional Radiology; Virtual Reality; Augmented Reality; Video Games; Computer Simulation; Education, Medical; Teaching; and Simulation Training.

References from the studies included and from the main reviews on the subject were also analyzed. The search strategies were carried out on July 29, 2020, for each database, and are shown in Table 2.

Table 2
Search strategy according to the corresponding database

Database	Search strategy
Cochrane Library	#1: MeSH descriptor: [Radiology, Interventional] explode all trees #2: MeSH descriptor: [Virtual Reality] explode all trees #3: MeSH descriptor: [Augmented Reality] explode all #4: MeSH descriptor: [Video Games] explode all trees #5: MeSH descriptor: [Computer Simulation] explode all trees #6: MeSH descriptor: [Education, Medical] explode all trees #7: MeSH descriptor: [Teaching] explode all trees #8: MeSH descriptor: [Simulation Training] explode all trees #9: #1 AND #2 OR #3 OR #4 OR #5 AND #6 OR #7 OR #8
MEDLINE	#1: “Radiology, Interventional”[MeSH] OR (Interventional Radiology) #2: “Virtual Reality” [MeSH] OR (Reality, Virtual) OR (Virtual Reality, Educational) OR (Educational Virtual Realities) OR (Educational Virtual Reality) OR (Reality, Educational Virtual) OR (Virtual Realities, Educational) OR (Virtual Reality, Instructional) OR (Instructional Virtual Realities) OR (Instructional Virtual Reality) OR (Realities, Instructional Virtual) OR (Reality, Instructional Virtual) OR (Virtual Realities, Instructional) OR “Augmented Reality“[MeSH] OR (Augmented Realities) OR (Realities, Augmented) OR (Reality, Augmented) OR (Mixed Reality) OR (Mixed Realities) OR (Realities, Mixed) OR (Reality, Mixed) OR “Video Games”[MeSH] OR (Game, Video) OR (Games, Video) OR (Video Game) OR (Computer Games) OR (Computer Game) OR (Game, Computer) OR (Games, Computer) OR “Computer Simulation”[MeSH] OR (Computer Simulations) OR (Simulation, Computer) OR (Simulations, Computer) OR (Computerized Models) OR (Computerized Model) OR (Model, Computerized) OR (Models, Computerized) OR (Models, Computer) OR (Computer Models) OR (Computer Model) OR (Model, Computer) OR (In Silico) OR (In Silicos) OR (Silico, In) OR (Silicos, In) #3: “Education, Medical”[MeSH] OR (Medical Education) OR “Teaching”[MeSH] OR (Training Techniques) OR (Technique, Training) OR (Techniques, Training) OR (Training Technique) OR (Training Technics) OR (Technic, Training) OR (Technics, Training) OR (Training Technic) OR (Pedagogy) OR (Pedagogies) OR (Teaching Methods) OR (Method, Teaching) OR (Methods, Teaching) OR (Teaching Method) OR (Academic Training) OR (Training, Academic) OR (Training Activities) OR (Activities, Training) OR (Training Activity) OR (Techniques, Educational) OR (Technics, Educational) OR (Educational Technics) OR (Educational Technic) OR (Technic, Educational) OR (Educational Techniques) OR (Educational Technique) OR (Technique, Educational) OR “Simulation Training”[MeSH] OR (Training, Simulation) OR (Interactive Learning) OR (Learning, Interactive) #4: #1 AND #2 AND #3
EMBASE	#1: interventional radiology/exp #2: virtual reality/exp #3: augmented reality/exp #4: video game/exp #5: computer simulation/exp #6: medical education/exp #7: simulation training/exp #8: teaching/exp #9: #1 OR #2 AND #3 OR #4 OR #5 OR #6 AND #7 OR #8
LILACS	#1: mh: “Radiologia Intervencionista” OR (Radiología Intervencional) OR (Radiology, Interventional) OR (H02.403.740.675) #2: mh: “Realidade Virtual” OR (Realidad Virtual) OR (Virtual Reality) OR (Educational Virtual Realities) OR (Educational Virtual Reality) OR (Instructional Virtual Realities) OR (Instructional Virtual Reality) OR (Realities, Instructional Virtual) OR (Reality, Educational Virtual) OR (Reality, Instructional Virtual) OR (Reality, Virtual) OR (Virtual Realities, Educational) OR (Virtual Realities, Instructional) OR (Virtual Reality, Educational) OR (Virtual Reality, Instructional) OR (L01.224.160.875) OR (L01.296.555) OR (SP4.011.127.428.806.030) #3: mh: “Realidade Aumentada” OR (Realidad Aumentada) OR (Augmented Reality) OR (Augmented Reality for Health) OR (Augmented Reality in Clinical Simulations) OR (Augmented Reality in Health Care Education) OR (Augmented Reality in Health) OR (Augmented Reality in Healthcare Education) OR (SP4.011.127.428.806.020) #4: mh: “Jogos de Vídeo” OR (Juegos de Video) OR (Video Games) OR (Computer Game) OR (Computer Games) OR (Game, Computer) OR (Game, Video) OR (Games, Computer) OR (Games, Video) OR (Video Game) OR (I03.450.642.693.930) OR (L01.224.900.930) #5: mh: “Educação de Graduação em Medicina” OR (Educación de Pregrado en Medicina) OR (Education, Medical, Undergraduate) OR (Education, Undergraduate Medical) OR (Medical Education, Undergraduate) OR (Undergraduate Medical Education) OR (I02.358.399.450) #6: mh: “Treinamento por Simulação” OR (Entrenamiento Simulado) OR (Simulation Training) OR (Interactive Learning) OR (Learning, Interactive) OR (Training, Simulation) OR (I02.903.847) #7: #1 AND #2 OR #3 OR #4 AND #5 OR #6
Tripdatabase	(Interventional radiology)(Virtual reality OR Augmented reality OR Video game OR Computer simulation)(Medical education OR Simulation training OR Teaching)
ERIC	#1: Interventional radiology #2: Virtual reality #3: Augmented reality #4: Video game #5: Computer simulation #6: Medical education #7: Simulation training #8: Teaching #9: #1 OR #2 AND #3 OR #4 OR #5 OR #6 AND #7 OR #8
CINAHL	#1: Interventional radiology #2: Virtual reality OR vr OR augmented reality OR video games OR computer simulation #3: Medical education OR simulation training or simulation education or simulation learning OR teaching #4: #1 AND #@ AND #3
SciELO	#1: Interventional radiology #2: Virtual reality #3: Augmented reality #4: Video game #5: Computer simulation #6: Medical education #7: Simulation training #8: Teaching #9: #1 OR #2 AND #3 OR #4 OR #5 OR #6 AND #7 OR #8

RESULTS

The search yielded 5189 articles; 50 of these were duplicates and were excluded. Through analysis on the titles and abstracts, 56 articles were selected for full-text evaluation, out of which four were included (Figure 1). Among these 56 articles, Grasso et al.³¹ did not evaluate the learning that resulted from the teaching methods and was excluded from the analysis.

Figure 1
PRISMA flow diagram of study selection.

Two of the four studies were from Canada¹⁵,¹⁷ and used a pre-experience questionnaire; the other two were from the United States¹⁴,¹⁸ and used both a pre-experience and a post-experience questionnaire. We found that heterogeneity was present among both the participants and the procedures analyzed. All of these studies were RCTs in which, differently from the intervention group, the control group did not have access to an AR device; while the remaining instructions and other materials (books and didactic lessons) were equal for the two groups.

All of these four studies reported that changes in perspective or judgment occurred in the groups of participants, concerning teaching and learning (Kirkpatrick evidence level 2B).

Regarding procedures, two studies analyzed central venous catheter placement,¹⁴,¹⁸ one study evaluated the lumbar puncture procedure¹⁵ and one investigated injection into the interfacetal joint.¹⁷ Although a diversity of issues were analyzed among these trials, the performance achieved through the technique was the main outcome in all of them. Regarding the populations investigated, the participants comprised respiratory therapists, sleep technicians, pre-medical and medical undergraduate students, emergency medicine and surgery residents and anesthesiologists. In three of the studies analyzed,¹⁴,¹⁵,¹⁷ it was concluded that AR could increase students’ skills in interventional radiology.

AR is used in a variety of areas of medicine and no systematic review or clinical trial has been carried out using a homogenous population. Because of the heterogeneous nature of the populations studied, different AR devices analyzed and different medical procedures used in these four RCTs, we did not perform any meta-analysis. Table 3 shows the quality assessment and risk of bias analysis conducted using the Cochrane Collaboration tool.

Table 3
Quality assessment/risk of bias analysis using the Cochrane Collaboration tool

	Underlying bias	Resource bias	Setting bias	Educational bias	Content bias
Huang et al.14
Keri et al.15
Moult et al.17
Wu et al.18

• Low risk of bias.
• Unclear risk of bias.

Huang et al.¹⁴ enrolled 32 adult novice central line operators (physicians, respiratory therapists and sleep technicians) with no visual or auditory impairments. Comparisons were made between simulations using AR reality glasses and conventional instruction; the AR glasses used were Brother AiRScouter WD-200B AR glasses (Brother International Corp., Bridgewater, New Jersey, United States). The authors did not comment on the cost of the teaching techniques. The AR simulation group undertook a five to ten-minute hands-on instructional course on the AR device; the mean time taken for AR head placement was 71 seconds. No significant difference in the median time taken for internal jugular cannulation or in the median total duration of the procedure was found between the groups. Most participants (71%; n = 23) were successful in cannulating the internal jugular upon the first attempt (12 in the AR group versus 11 in the non-AR group). A significant difference in adherence level between the two groups (22.9 ± 4.1 in the AR group versus 18.1 ± 6.3 in the non-AR group; η² = 0.90; P = 0.003) was detected. In the post-exercise questionnaire for the AR group, more than 80% of the participants stated that the instrument did not cause any fatigue and was not too heavy to be uncomfortable. Nonetheless, 30% admitted that the equipment affected their action skills and that it was not easy to regulate. On the other hand, 94% reported that the hand, head and foot interactions were undemanding and 80% stated that the information presented on-screen was suitable and reacted fast enough.

In the study by Wu et al.,¹⁸ 20 medical students and 20 emergency medicine residents were compared with regard to learning central venous catheter positioning. All the participants watched a video explaining how to use Google Glass and how to place an internal jugular central venous access catheter under ultrasound guidance in a simulation task trainer. The participants were randomized into two groups: with and without Google Glass. The ultrasound machine setup was the same between the groups; the intervention group participants were asked to execute the procedure by viewing ultrasound images displayed on their Google Glass screen, while the control group executed the procedure by viewing ultrasound images shown on the ultrasound screen. The Google Glass group took longer to perform the procedure, with longer times spent looking at the patient and monitor and greater numbers of needle redirections, at both training levels (medical students and emergency medicine residents). This may have been due to unfamiliarity with Google Glass, thus requiring more attention throughout the procedure. The responses to the post-exercise questionnaire showed that the majority of the participants were not previously familiar with AR or with wearable computing technology (75% and 60%, respectively); however, 73% reported having some degree of knowledge about Google Glass. Nonetheless, 87% of the participants randomized to Google Glass reported that the instrument was comfortable to use for the procedure.

Keri et al.¹⁵ evaluated the effectiveness of Perk Tutor (GPS extension, Ultrasonix, Canada) in relation to a phantom, as a teaching method among anesthesiology and surgery residents for lumbar puncture procedures. Perk Tutor is a training platform that was designed to display ultrasound images along with real-time three-dimensional images, using wearable technology. The authors did not comment on the participants’ experience levels regarding AR. There were 24 participants, who were divided into two groups (ten anesthesiologists and two surgeons): Perk Tutor with phantom and phantom alone. All the participants received a short presentation on spinal anatomy, ultrasound basics and how to use the device. They were also trained to perform ultrasound-guided procedures on three different lumbar spine models. The participants were then tested using conventional ultrasound guidance on an abnormal spinal model that they had not previously seen, for ten minutes at most or until positive fluid backflow ws observed at the needle hub. The potential tissue damage, needle path in tissue, total duration of the procedure and time taken to insert the needle were measured. Eleven participants in the phantom group and all participants in the Perk Tutor with phantom group performed the task successfully. The potential for tissue lesion was significantly lower in the Perk Tutor with phantom group (39.7 [range 21.3-42.7] square centimeters (cm²) versus 128.3 [50.3-208.2] cm²). Moreover, the needle tissue path was shorter (426.0 [range 164.9-571.6] millimeters (mm) versus 629.7 [306.4-2,879.1] mm), as also was the time taken to insert the needle (30.3 [14.0-51.0] seconds (sec) versus 59.1 [26.0-136.2] sec). The total duration of the procedure was similar (203.8 [range 135.1-274.9] sec versus 266.9 [221.6-416.2] sec).

Moult et al.¹⁷ compared the performances of 26 pre-medical undergraduate students (with no prior needle insertion experience) in a task of injection into the interfacetal joint. Participants were divided equally into two groups: Perk Tutor with phantom and phantom only. The authors did not comment on the cost of the teaching techniques. Both groups received a ten-minute introductory class on anatomy, procedure, ultrasound image interpretation and needle handling techniques. Afterwards, both groups had a ten-minute practice session on ultrasound-guided facet joint injections on the phantom; the Perk Tutor group had access to ultrasound and Perk Tutor, while the phantom group only had access to the ultrasound machine. The Perk Tutor and phantom group had a mean success rate of 61.5%, while this rate was 38.5% in the phantom only group; the total duration of the procedure was longer in the phantom only group (73 ± 8 versus 66 ± 6 seconds). The total needle distance travel (inside and outside of the phantom body) was greater in the phantom only group (1803 ± 290 versus 1366 ± 185 mm), but the inside distance traveled was shorter (25 ± 3 versus 42 ± 16 mm) in this group. Moreover, within the phantom body, the needle tip time was greater in the Perk Tutor and phantom group (296 ± 45 seconds versus 243 ± 28 seconds).

All of these results are summarized in Table 4.¹⁴,¹⁵,¹⁷,¹⁸

Table 4
Summary of studies’ findings.

Study	Country	Design	Participants	Procedure	Intervention	Comparison	Results	Kirkpatrick
Huang et al.14	United States	Randomized clinical trial with pre and post-experience questionnaire	32 novice central line operators including physicians, respiratory therapists and sleep technicians	Positioning of the central venous catheter	Augmented reality glasses Brother AiRScouter WD-200B AR glasses 17 participants	No augmented reality glasses 15 participants	Participants who trained in AR needed fewer attempts to perform the procedure.	2B
Keri et al.15	Canada	Randomized clinical trial with pre-experience questionnaire	24 anesthesiology and surgery residents	Lumbar puncture	Perk Tutor + phantom 12 participants	Phantom only 12 participants	Participants who trained in AR injured less tissue and were quicker to insert the needle	2B
Moult et al.17	Canada	Randomized clinical trial with pre-experience questionnaire	26 pre-medical undergraduate students	Injection into interfacetal joint	Perk Tutor + phantom 13 participants	Phantom only 13 participants	Participants who trained in AR obtained much better results, especially regarding the total duration of the procedure.	2B
Wu et al.18	United States	Randomized clinical trial with pre and post-experience questionnaire	10 1st and 10 4th year medical students; 10 1st year emergency residents and 10 3rd year residents	Positioning of a central venous catheter	Google Glass 5 participants	Without Google Glass 5 participants	Glass users took longer to complete the procedure.	2B

DISCUSSION

The objective of our study was to examine the current evidence on training using AR in interventional radiology and its performance, along with the impact of AR on educational outcomes and skills, and its main advantages, disadvantages and challenges during the teaching-learning process.

New teaching techniques such as virtual reality (VR), AR or mixed reality (MR) are being introduced in medical education.⁵,³² AR combines virtual and real-world through use of wearable technology that provides a live feed from computer workstations (i.e. from an ultrasound device).¹⁸ Images and information are shown in the user’s line of sight through the device.¹⁸

Everyday use of mobile devices facilitates implementation of this instructional tool in teaching processes, which permits access to learning at any moment.³³,³⁴ However there is still a lack of research regarding the competence of this technology.³³

AR methods have stood out in the surgical environment over recent years, through providing educational simulation practice free from potential ethical/hygiene concerns.³⁵ Furthermore, the pressure imposed on healthcare systems during the COVID-19 pandemic has hastened implementation of new technologies, thereby accelerating the learning of healthcare professionals.³⁶

Students are now used to dealing with technologies such as the internet, 3D video games, cell phones and others.¹⁹,²⁰,³⁷–⁴⁰ Teachers can avail themselves of this familiarity to upgrade teaching methods and aids, so as to encourage students.¹⁹,³⁷,⁴¹ Kotcherlakota et al. evaluated the utility of clinical simulation through applying AR technology to education outcomes for nurse practitioners in pediatric asthma management.⁴² The students showed high motivation, satisfaction and confidence scores.⁴² A systematic review by Barteit et al., on AR, VR and MR in several medical specialties, showed similar outcomes that revealed that these techniques were at least not inferior to traditional teaching methods.⁴³ Moreover, these technologies offer opportunities for scalability and repetition without risk to patients.⁴³

A systematic review by Rad et al. demonstrated that, in thoracic surgery, AR-enhanced intraoperative knowledge of anatomy decreased preoperative preparation time and workload.³⁵ However, with regard to anatomy education, Bölek et al. concluded from a meta-analysis on five studies with a total of 508 participants that AR did not have any meaningful advantages or disadvantages for students’ education, compared with several traditional educational tools.⁴⁴

AR could form a viable tool within traditional anatomy teaching in more technological environments.⁴⁴ Küçük et al. found that neuroanatomy learning using AR with a smartphone provided support for students, through reducing cognitive effort and increasing educational pleasure.¹⁹ According to our systematic review, the results regarding AR are similar in several medical specialties.

The main purpose of AR involves the concept of “practice makes perfection”, given that efficient performance in procedures requires experience.⁴⁵ AR simulation provides the possibility of repetition to boost self-confidence, within a safe method.⁴⁶

Over five million central venous catheters are fitted each year in the United States. The complication rates are 5%-8% higher per procedure when these are handled by novice professionals.¹⁴ Teaching with AR aids could result in lower morbidity, hospitalization time and costs.

Comparison of learning between novice physicians and experienced interventional radiologists could enable evaluation of whether AR has the capacity to accelerate learning. Studies comparing different kinds of AR in one specified procedure need to be performed in order to determine which technology is better for that particular procedure. From the current information available, AR is a useful additional tool for teaching interventional radiology, but not a substitute for the traditional methodology.

From the students’ perspective, AR can contribute to mastery and confidence in a new procedure, through enabling students to memorize details, thus decreasing the tension in real-life situations. Regarding classroom ambience, AR may enable a shift from the monotonous routine of expository classes, thus providing evolution of the learning experience. Assembling education with technology would engage young people, thereby transforming learning into a pleasant experience and improving learning, as well as clinical practice.

One limitation of this systematic review was that only two studies analyzed the same procedure.¹⁴,¹⁸ Numerous procedures are involved in interventional radiology, but in the four studies evaluated, only three different procedures were investigated: central venous catheter placement, lumbar punctures and interfacetal joint injection. Two different types of AR devices were tested: Perk Tutor and AR glasses. Different AR devices could be compared in the future. Moreover, the small samples used in the studies represented another limitation, thus hampering generalization.

Another limitation was the lack of evaluation among experienced professionals. The participants included in these studies were novice physicians or non-physicians; none of these studies investigated radiology residents.

The level of evidence of the studies was also a limitation: all of them were classified as 2B in the Kirkpatrick model.²⁹ Our searches did not retrieve any studies with educational evidence at level 3 (behavioral change), 4A (change in the organizational system/practice) or 4B (change among participants, students, residents or colleagues). Despite the current interest²² in using simulators, it remains to be delineated which types of simulation and simulator should be used, and what population this teaching method will be applied to. Hence, a higher level of evidence is needed.

Hardware needs are also a concern, considering that running the application produced intense energy usage as well as device heating.³⁴ These technical difficulties could be resolved by the smartphone industry. Use of faster networks enables a shared environment through cloud services and shared real-time information. Introduction of artificial intelligence to AR-based learning programs can also provide more positive learning.

The costs of AR devices are expected to decrease along with the evolution of production and increased market competition, thus bringing these technologies to low-income countries. Moreover, AR-based medical training could facilitate teaching for people with reading limitations and could also facilitate remote teaching.

CONCLUSION

It was demonstrated through this study that AR, as a complementary tool, can add skills to learners and thus can improve the teaching-learning process. It needs to be noted that only level 2B studies were found in this systematic review and, thus, that a higher level of evidence is required. Moreover, comparison of beginner physicians and expert interventional radiologists could enable appraisal of the hastening of the learning curve through AR, as well as investigation of which set of AR tools is most adequate for each procedure.

• Evidence-Based Health Department, Universidade Federal de São Paulo (UNIFESP), São Paulo (SP), Brazil
• Sources of funding: No funding was received for this study

REFERENCES

• ¹
  Silva AB, de Amorim AC. A Brazilian educational experiment: teleradiology on web TV. J Telemed Telecare. 2009;15(7):373-6. PMID: 19815908; https://doi.org/10.1258/jtt.2009.090204
  » https://doi.org/10.1258/jtt.2009.090204
• ²
  Duarte ML, Santos LR, Guimarães Júnior JB, Peccin MS. Learning anatomy by virtual reality and augmented reality. A scope review. Morphologie. 2020;104(347):254-66. PMID: 32972816; https://doi.org/10.1016/j.morpho.2020.08.004
  » https://doi.org/10.1016/j.morpho.2020.08.004
• ³
  Cates CU, Lönn L, Gallagher AG. Prospective, randomised and blinded comparison of proficiency-based progression full-physics virtual reality simulator training versus invasive vascular experience for learning carotid artery angiography by very experienced operators. BMJ Stel. 2016;2:1-5. https://doi.org/10.1136/bmjstel-2015-000090
  » https://doi.org/10.1136/bmjstel-2015-000090
• ⁴
  Kreiser K, Gehling K, Zimmer C. Simulation in Angiography - Experiences from 5 Years Teaching, Training, and Research. Rofo. 2019;191(6):547-52. PMID: 30754054; https://doi.org/10.1055/a-0759-2248
  » https://doi.org/10.1055/a-0759-2248
• ⁵
  Chaer RA, Derubertis BG, Lin SC, et al. Simulation improves resident performance in catheter-based intervention: results of a randomized, controlled study. Ann Surg. 2006;244(3):343–52. PMID: 16926560; https://doi.org/10.1097/01.sla.0000234932.88487.75
  » https://doi.org/10.1097/01.sla.0000234932.88487.75
• ⁶
  Mirza S, Athreya S. Review of Simulation Training in Interventional Radiology. Acad Radiol. 2018;25(4):529-39. PMID: 29221857; doi: https://doi.org/10.1016/j.acra.2017.10.009
  » https://doi.org/10.1016/j.acra.2017.10.009
• ⁷
  Johnson SJ, Guediri SM, Kilkenny C, Clough PJ. Development and validation of a virtual reality simulator: human factors input to interventional radiology training. Hum Factors. 2011;53(6):612-25. PMID: 22235524; https://doi.org/10.1177/0018720811425042
  » https://doi.org/10.1177/0018720811425042
• ⁸
  Willaert WI, Aggarwal R, Daruwalla F, et al. European Virtual Reality Endovascular Research Team EVEResT. Simulated procedure rehearsal is more effective than a preoperative generic warm-up for endovascular procedures. Ann Surg. 2012;255(6):1184-9. PMID: 22566016; https://doi.org/10.1097/SLA.0b013e31824f9dbf
  » https://doi.org/10.1097/SLA.0b013e31824f9dbf
• ⁹
  Chytas D, Johnson EO, Piagkou M, et al. The role of augmented reality in anatomical education: An overview. Ann Anat. 2020;229:151463. PMID: 31978568; https://doi.org/10.1016/j.aanat.2020.151463
  » https://doi.org/10.1016/j.aanat.2020.151463
• ¹⁰
  Aebersold M, Voepel-Lewis T, Cherara L, et al. Interactive Anatomy-Augmented Virtual Simulation Training. Clin Simul Nurs. 2018;15:34-41. PMID: 29861797; https://doi.org/10.1016/j.ecns.2017.09.008
  » https://doi.org/10.1016/j.ecns.2017.09.008
• ¹¹
  Mahmood F, Mahmood E, Dorfman RG, et al. Augmented Reality and Ultrasound Education: Initial Experience. J Cardiothorac Vasc Anesth. 2018;32(3):1363-7. PMID: 29452879; https://doi.org/10.1053/j.jvca.2017.12.006
  » https://doi.org/10.1053/j.jvca.2017.12.006
• ¹²
  Dhar P, Rocks T, Samarasinghe RM, Stephenson G, Smith C. Augmented reality in medical education: students’ experiences and learning outcomes. Med Educ Online. 2021 Dec;26(1):1953953. PMID: 34259122; https://doi.org/10.1080/10872981.2021.1953953
  » https://doi.org/10.1080/10872981.2021.1953953
• ¹³
  Sandrone S, Carlson CE. Future of Neurology & Technology: Virtual and Augmented Reality in Neurology and Neuroscience Education: Applications and Curricular Strategies. Neurology. 2021:10.1212/WNL.0000000000012413. PMID: 34187858; https://doi.org/10.1212/WNL.0000000000012413
  » https://doi.org/10.1212/WNL.0000000000012413» https://doi.org/10.1212/WNL.0000000000012413
• ¹⁴
  Huang CY, Thomas JB, Alismail A, et al. The use of augmented reality glasses in central line simulation: “see one, simulate many, do one competently, and teach everyone”. Adv Med Educ Pract. 2018;9:357-63. PMID: 29785148; https://doi.org/10.2147/AMEP.S160704
  » https://doi.org/10.2147/AMEP.S160704
• ¹⁵
  Keri Z, Sydor D, Ungi T, et al. Computerized training system for ultrasound-guided lumbar puncture on abnormal spine models: a randomized controlled trial. Can J Anaesth. 2015;62(7):777-84. PMID: 25804431; https://doi.org/10.1007/s12630-015-0367-2
  » https://doi.org/10.1007/s12630-015-0367-2
• ¹⁶
  Ebner F, De Gregorio A, Schochter F, et al. Effect of an Augmented Reality Ultrasound Trainer App on the Motor Skills Needed for a Kidney Ultrasound: Prospective Trial. JMIR Serious Games. 2019;7(2):e12713. PMID: 31042155; https://doi.org/10.2196/12713
  » https://doi.org/10.2196/12713
• ¹⁷
  Moult E, Ungi T, Welch M, et al. Ultrasound-guided facet joint injection training using Perk Tutor. Int J Comput Assist Radiol Surg. 2013;8(5):831-6. PMID: 23329279; https://doi.org/10.1007/s11548-012-0811-5
  » https://doi.org/10.1007/s11548-012-0811-5
• ¹⁸
  Wu TS, Dameff CJ, Tully JL. Ultrasound-guided central venous access using Google Glass. J Emerg Med. 2014;47(6):668-75. PMID: 25281180; https://doi.org/10.1016/j.jemermed.2014.07.045
  » https://doi.org/10.1016/j.jemermed.2014.07.045
• ¹⁹
  Küçük S, Kapakin S, Göktaş Y. Learning anatomy via mobile augmented reality: Effects on achievement and cognitive load. Anat Sci Educ. 2016;9(5):411-21. PMID: 26950521; https://doi.org/10.1002/ase.1603
  » https://doi.org/10.1002/ase.1603
• ²⁰
  Jamali SS, Shiratuddin MF, Wong KW, Oskam CL. Utilising Mobile-Augmented Reality for Learning Human Anatomy. Procedia - Social and Behavioral Sciences 2015;197:659-68. https://doi.org/10.1016/j.sbspro.2015.07.054
  » https://doi.org/10.1016/j.sbspro.2015.07.054
• ²¹
  Gallagher AG, Smith CD. From the operating room of the present to the operating room of the future. Human-factors lessons learned from the minimally invasive surgery revolution. Semin Laparosc Surg. 2003;10(3):127-39. PMID: 14551655; https://doi.org/10.1177/107155170301000306
  » https://doi.org/10.1177/107155170301000306
• ²²
  Aeckersberg G, Gkremoutis A, Schmitz-Rixen T, Kaiser E. The relevance of low-fidelity virtual reality simulators compared with other learning methods in basic endovascular skills training. J Vasc Surg. 2019;69(1):227-35. PMID: 30579447; https://doi.org/10.1016/j.jvs.2018.10.047
  » https://doi.org/10.1016/j.jvs.2018.10.047
• ²³
  Markovic J, Peyser C, Cavoores T, et al. Impact of endovascular simulator training on vascular surgery as a career choice in medical students. J Vasc Surg. 2012;55(5):1515-21. PMID: 22464708; https://doi.org/10.1016/j.jvs.2011.11.060
  » https://doi.org/10.1016/j.jvs.2011.11.060
• ²⁴
  Stromberga Z, Phelps C, Smith J, Moro C. Teaching with Disruptive Technology: The Use of Augmented, Virtual, and Mixed Reality (HoloLens) for Disease Education. Adv Exp Med Biol. 2021;1317:147-62. PMID: 33945136.25; https://doi.org/10.1007/978-3-030-61125-5_8
  » https://doi.org/10.1007/978-3-030-61125-5_8
• ²⁵
  Berry M, Reznick R, Lystig T, Lönn L. The use of virtual reality for training in carotid artery stenting: a construct validation study. Acta Radiol. 2008;49(7):801-5. PMID: 18608009; https://doi.org/10.1080/02841850802108438
  » https://doi.org/10.1080/02841850802108438
• ²⁶
  Ouzzani M, Hammady H, Fedorowicz Z, Elmagarmid A. Rayyan - a web and mobile app for systematic reviews. Syst Rev. 2016;5(1):210. PMID: 27919275; https://doi.org/10.1186/s13643-016-0384-4
  » https://doi.org/10.1186/s13643-016-0384-4
• ²⁷
  Sterne JAC, Hernan MA, McAleenan A, Reeves BC, Higgins JPT. Chapter 25: assessing risk of bias in a non-randomized study. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane handbook for systematic reviews of interventions version 6.0. Hoboken: John Wiley & Sons; 2019.
• ²⁸
  Buckley S, Coleman J, Davison I, et al. The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Med Teach. 2009;31(4):282-98. PMID: 19404891; https://doi.org/10.1080/01421590902889897
  » https://doi.org/10.1080/01421590902889897
• ²⁹
  Steinert Y, Mann K, Centeno A, et al. A systematic review of faculty development initiatives designed to improve teaching effectiveness in medical education: BEME Guide No. 8. Med Teach. 2006;28(6):497-526. PMID: 17074699; https://doi.org/10.1080/01421590600902976
  » https://doi.org/10.1080/01421590600902976
• ³⁰
  Kirkpatrick D, Kirkpatrick J. Evaluating training programs: The four levels. 3^rd ed. San Francisco, CA: Berrett-Koehler; 2006.
• ³¹
  Grasso RF, Faiella E, Luppi G, et al. Percutaneous lung biopsy: comparison between an augmented reality CT navigation system and standard CT-guided technique. Int J Comput Assist Radiol Surg. 2013;8(5):837-48. PMID: 23377707; https://doi.org/10.1007/s11548-013-0816-8
  » https://doi.org/10.1007/s11548-013-0816-8
• ³²
  Gelmini AYP, Duarte ML, de Assis AM, Guimarães Junior JB, Carnevale FC. Virtual reality in interventional radiology education: a systematic review. Radiol Bras. 2021;54(4):254-60. PMID: 34393293; https://doi.org/10.1590/0100-3984.2020.0162
  » https://doi.org/10.1590/0100-3984.2020.0162
• ³³
  Rodríguez-Abad C, Fernández-de-la-Iglesia JD, Martínez-Santos AE, Rodríguez-González R. A Systematic Review of Augmented Reality in Health Sciences: A Guide to Decision-Making in Higher Education. Int J Environ Res Public Health. 2021;18(8):4262. PMID: 33920528; https://doi.org/10.3390/ijerph18084262
  » https://doi.org/10.3390/ijerph18084262
• ³⁴
  Mladenovic R, Matvijenko V, Subaric L, Mladenovic K. Augmented reality as e-learning tool for intraoral examination and dental charting during COVID-19 era. J Dent Educ. 2021. PMID: 34420217; https://doi.org/10.1002/jdd.12780 Epub ahead of print.
  » https://doi.org/10.1002/jdd.12780
• ³⁵
  Arjomandi Rad A, Vardanyan R, Thavarajasingam SG, et al. Extended, virtual and augmented reality in thoracic surgery: a systematic review. Interact Cardiovasc Thorac Surg. 2021:ivab241. PMID: 34542639; https://doi.org/10.1093/icvts/ivab241
  » https://doi.org/10.1093/icvts/ivab241
• ³⁶
  Ha J, Parekh P, Gamble D, et al. Opportunities and challenges of using augmented reality and heads-up display in orthopaedic surgery: A narrative review. J Clin Orthop Trauma. 2021;18:209-15. PMID: 34026489; https://doi.org/10.1016/j.jcot.2021.04.031
  » https://doi.org/10.1016/j.jcot.2021.04.031
• ³⁷
  Ferrer-Torregrosa J, Jiménez-Rodríguez MÁ, Torralba-Estelles J, et al. Distance learning ects and flipped classroom in the anatomy learning: comparative study of the use of augmented reality, video and notes. BMC Med Educ. 2016;16(1):230. PMID: 27581521; https://doi.org/10.1186/s12909-016-0757-3
  » https://doi.org/10.1186/s12909-016-0757-3
• ³⁸
  Moro C, Štromberga Z, Raikos A, Stirling A. The effectiveness of virtual and augmented reality in health sciences and medical anatomy. Anat Sci Educ. 2017;10(6):549-59. PMID: 28419750; https://doi.org/10.1002/ase.1696
  » https://doi.org/10.1002/ase.1696
• ³⁹
  McNulty JA, Sonntag B, Sinacore JM. Evaluation of computer-aided instruction in a gross anatomy course: a six-year study. Anat Sci Educ. 2009;2(1):2-8. PMID: 19217066; https://doi.org/10.1002/ase.66
  » https://doi.org/10.1002/ase.66
• ⁴⁰
  McNulty JA, Halama J, Espiritu B. Evaluation of computer-aided instruction in the medical gross anatomy curriculum. Clin Anat. 2004;17(1):73-8. PMID: 14695594; https://doi.org/10.1002/ca.10188
  » https://doi.org/10.1002/ca.10188
• ⁴¹
  Lynch TG, Steele DJ, Johnson Palensky JE, Lacy NL, Duffy SW. Learning preferences, computer attitudes, and test performance with computer-aided instruction. Am J Surg. 2001;181(4):368-71. PMID: 11438276; https://doi.org/10.1016/s0002-9610(01)00575-x
  » https://doi.org/10.1016/s0002-9610(01)00575-x
• ⁴²
  Kotcherlakota S, Pelish P, Hoffman K, Kupzyk K, Rejda P. Augmented Reality Technology as a Teaching Strategy for Learning Pediatric Asthma Management: Mixed Methods Study. JMIR Nurs. 2020;3(1):e23963. PMID: 34406970; https://doi.org/10.2196/23963
  » https://doi.org/10.2196/23963
• ⁴³
  Barteit S, Lanfermann L, Bärnighausen T, Neuhann F, Beiersmann C. Augmented, Mixed, and Virtual Reality-Based Head-Mounted Devices for Medical Education: Systematic Review. JMIR Serious Games. 2021;9(3):e29080. PMID: 34255668; https://doi.org/10.2196/29080
  » https://doi.org/10.2196/29080
• ⁴⁴
  Bölek KA, De Jong G, Henssen D. The effectiveness of the use of augmented reality in anatomy education: a systematic review and meta-analysis. Sci Rep. 2021 Jul 27;11(1):15292. PMID: 34315955; https://doi.org/10.1038/s41598-021-94721-4
  » https://doi.org/10.1038/s41598-021-94721-4
• ⁴⁵
  Patel R, Dennick R. Simulation based teaching in interventional radiology training: is it effective? Clin Radiol. 2017;72(3):266.e7-.e14. PMID: 27986263; https://doi.org/10.1016/j.crad.2016.10.014
  » https://doi.org/10.1016/j.crad.2016.10.014
• ⁴⁶
  Aggarwal R, Black SA, Hance JR, Darzi A, Cheshire NJ. Virtual reality simulation training can improve inexperienced surgeons’ endovascular skills. Eur J Vasc Endovasc Surg. 2006;31(6):588-93. PMID: 16387517; https://doi.org/10.1016/j.ejvs.2005.11.009
  » https://doi.org/10.1016/j.ejvs.2005.11.009

Publication Dates

• Publication in this collection
  08 Aug 2022
• Date of issue
  Jul-Aug 2022

History

• Received
  11 July 2021
• Reviewed
  24 Nov 2021
• Accepted
  27 Dec 2021