Desenvolvimento e Validação de um Aplicativo para Smartphone para Ensino de Residência em Ortopedia

rbem Revista Brasileira de Educação Médica Rev. bras. educ. med. 0100-5502 1981-5271 Associação Brasileira de Educação Médica Resumo: Introdução: Além de proporcionar vários benefícios, o aprendizado móvel oferece novos ambientes de ensino. Desenvolvemos e validamos um aplicativo de smartphone para residentes de ortopedia e traumatologia, a fim de ajudá-los no estudo e, consequentemente, na aprovação do exame do Conselho de Ortopedia e Traumatologia. Métodos: Trata-se de um estudo quantitativo para validar a aplicação móvel. O instrumento foi desenvolvido para as plataformas iOS® e Android®, em português e gratuito. Os 132 participantes do estudo, divididos em três grupos, usaram a ferramenta. Adotaram-se dois questionários: a já validada System Usability Scale (SUS) e um outro criado pelos autores (devidamente validado) para avaliar a praticidade e viabilidade do aplicativo como ferramenta de aprendizagem. O modo, a frequência absoluta e o percentual das variáveis estudadas foram cruzados pelo teste de Fisher ou qui-quadrado de Pearson. Resultados: Dos 132 participantes, 55,3% possuem iOS®. Todos os participantes afirmaram que já haviam usado um aplicativo em seus smartphones; seis (4,5%) mencionaram que nunca utilizaram aplicativos para fins acadêmicos (valor de p < 0,001); 100% destacaram a utilidade dessa tecnologia no desenvolvimento teórico do residente; e 124 (93,9%) concordaram que se trata de um método auxiliar de aprendizagem para médicos ortopedistas em geral. Obteve-se um escore médio da SUS de 84,2 (DP 10,8) com margem de erro de 1,9. O escore da SUS variou de 82,4 a 86,1 (IC95%). O coeficiente alfa de Cronbach foi de 0,797. Conclusão: O aplicativo desenvolvido foi bem-sucedido nos exames realizados e pode ser uma alternativa no ensino médico na área ortopédica. INTRODUCTION Communication and Information Technologies (ICTs) started being used to help the acquisition of knowledge and facilitate the teaching-learning process1. The use of technology through mobile devices or mobile learning (m-learning) is known to offer many benefits, such as access to content anywhere and at any time, the ability to perform tasks and activities and note down ideas or queries on the Internet. Teaching through m-learning enables the simulation of complex systems, along with the benefit of promoting a large number of perspectives with a high quality of visualization and interaction2. In this context, ICTs, according to each modality, can enable the creation of new learning environments, allowing remote and/or online access to training platforms3, increasing communication and distance learning. Thus, these support tools incorporated into an electronic health management system exponentially increase the growth of medical information acquisition4. There are several practical training methods for orthopedic surgeons, including hands-on training with synthetic bones or cadavers, and the use of software and computer simulators for planning and simulating situations in 3D environments5. This reduces financial costs and brings about improvements in skills development, the handling of surgical instruments, competence to reach the learning curve earlier, better assessment of the acquired techniques, and perhaps most importantly, the maximization of the patient’s safety6. Within the ICT-based learning modalities, cognitive simulation is one of the most recent examples of medical training innovation. It is the process by which medical residents or students evaluate and rehearse actions within their minds without physical movement and, among them, is the training related to solving questions for theoretical-practical training. It is hypothesized that residents may improve their intraoperative performance and surgical skills through appropriate preoperative cognitive simulation, with or without appropriate preceptors to help them7. The use of smartphone applications for the orthopedic surgeon training is a powerful tool for improving the quality of these professionals’ training8. Currently, 67.1% of medical students choose an application to help them with their daily academic doubts, and 42.2% use it to assist them in their professional practice9. Among the several approaches to the teaching-learning process using mobile applications, the knowledge acquired through the resolution of various issues, also called retrieval learning, can be highlighted, which has proved to be more effective than just reading the content to be studied10. A search was conducted in the virtual stores Applestore® and GooglePlay® and it was found that most of the applications available for the orthopedic student based on retrieval learning were not properly scientifically validated. Therefore, the aim of this research was to develop and validate an application for the improvement and training of orthopedics and traumatology residents, which allows the resolution of preparatory questions for the Brazilian Board of Orthopedics and Traumatology (TEOT) exam. MATERIALS AND METHODS App development The development of this application involved the following steps: review of the available education and training applications in orthopedics and comparison with evidence-based practices; creation of an initial database of questions based on the bibliography proposed by the Teaching and Training Committee (TTC) of the Brazilian Society of Orthopedics and Traumatology (SBOT); selection of strategies to change behavior and guidelines to be used in the application; development of the interfaces and designs of the application; and finally, selection of participants for the application of the validation test. The application was developed in both Android® and iOS® platforms. Regarding the development, the following tools were used: IDE® (Integrated Development Environment) Android Studio; Android® Software Development Kit (SDK); the Android® Emulator System with Google's Application Programming Interface (APIs); and the OpenCV (Open Source Computer Vision Library) library for the development of the image processing functions present in the application. The application allows the customization of the content of the simulated questions, the visualization of the elapsed time and right or wrong answers with the respective bibliographic references. Also, the user’s results progression can be followed. The participants were randomly selected using Microsoft Excel® software. They were then divided into three groups: group I, 4-year medical students of the Christus University Center (Unichristus) (33 participants) and the Federal University of Ceará (UFC) (33 participants); group II, residents of orthopedics and traumatology from the state of Ceará (33 participants); group III, consisting of orthopedists and traumatologists who are members of SBOT and medical residency staff (33 participants). This sample size was calculated based on the study conducted in Leipzig, Germany9, which showed the preference of applications for health training, when evaluating two groups of students, divided among those who want to use an application (69.9%) versus those who do not (28.0%); it was considered necessary to evaluate a sample of 33 individuals per study group in order to obtain a sample that represented 90% of power and 95% confidence of the null hypothesis in this study (Fleiss method with continuity correction). Calculations were performed using the Open Epi Stat Calc® (http://www.openepi.com/Menu/OE_Menu.htm). The inclusion criteria were: age above 18 years old and participants who signed the Free and Informed Consent Form. Those participants who wished to discontinue their participation in the study were withdrawn from the study. Regarding the ethical aspects, the basic principles of ethics in human research were considered, such as autonomy, justice, beneficence, and nonmaleficence. The study was submitted to the Human Research Ethics Committee (CEP) of Unichristus, protocol number 65487617.5.0000.5049. There are no conflicts of interest to be disclosed by the authors. Data Collection The questionnaire (Appendix A) was organized into three sessions: 1) related to the user’s authorization and experience with applications; 2) related to the SUS® (System Usability Scale) questionnaire11, validated in Portuguese to quantify the applicability of the developed application12; and 3) related to the importance of the application in learning, being developed based on Davis Technology Acceptance Model13 and the Likert Scale, which measures the system practicability level perceived by users during the resolution of questions and evaluates the usefulness of specific questions about orthopedics and traumatology, as well as learning. The data were tabulated in Microsoft Excel® and exported to the Statistical Package for Social Sciences (SPSS) software, version 17.0 for Windows®, where the analyses were performed with a confidence interval of 95%. The mode, absolute frequency and percentage of the variables studied were cross-checked using Fisher's exact test or Pearson's chi-squared test. RESULTS The developed application was named Quiz Ortopedia ®, and its patent was registered in the National Institute of the Industrial Property (INPI) for iOS® (registration number BR 512018000006-1) and Android® (registration number BR 512018000005-3). The application allows the user’s registration with name, e-mail, year of residence, a 6-digit password and password confirmation. On the initial screen, it is possible to choose between a customized test, a simulation according to the suggestion by SBOT, a summary of the user's profile and information about the development and version of the application. Regarding personalized questions, the user has access to main topics (Orthopedics, Trauma, and Basic) and subtopics divided mainly by body joints (hip, knee, etc.). It is also possible to choose several simultaneous topics and then one can choose the number of questions and the timing (per question or total time). Throughout the resolution of the questions, the user will be able to see the elapsed time, the total number of questions, as well as the number of correct or incorrect questions up to that time. After that, a result in percentage should be displayed and the user is given the option to access the comments about both right and wrong questions. In the user’s profile, there is a screen with the results, so that the user can check study progression up to that point. The images of some representative screens of the application are depicted in Figure 1. Figure 1 Quiz screens: Topic selection, questions, and test results from Quiz Ortopedia®. Regarding the profile of the study participants and their previous experiences with the use of applications, 55.3% of the participants (73/132) have iOS® as their operating system, while 44.7% (59/132) use Android®. All 132 participants reported that they had already used some type of application on their smartphones, and only 6 (4.5%) mentioned never having used applications for academic purposes (Table 1). Table 1.Statistical analysis among students, residents, and orthopedists regarding the operating system and the use of applications for academic purposes. Occupation Student Resident Orthopedist p-Value Operating System Android 32 (48.5%) 18 (54.5%) 9 (27.3%) 0.057 IOs 34 (51.5%) 15 (45.5%) 24 (72.7%) Used application for academic purposes No 3 (4.5%) 0 (.0%) 3 (9.1%) 0.208 Yes 63 (95.5%) 33 (100.0%) 30 (90.9%) * p <0.05, chi-square. Data expressed as absolute and percentage frequencies. In group I, 63 participants (95.5%) had already used mobile applications for academic purposes. In group II, all of them had already used the mobile phone with the same purpose, and in group I, 30 (90.9%) used academic applications (Table 1). The type of operating system (iOS® or Android®) showed a significant statistical difference between the groups (p-value <0.001). The use of applications for academic purposes did not show a statistical difference between the groups. When we analyzed the usability of the application, evaluated according to the SUS® questionnaire, we observed that it showed an average SUS score of 84.2 (standard deviation of 10.8) with a margin of error of 1.9. To verify the reliability of the obtained results, Cronbach's alpha coefficient was used14. The value obtained for this coefficient was 0.797, showing the sample had a good level of reliability. The analysis of the answer to each question in the SUS questionnaire (part II) is found in Appendix B. We observed a statistical difference between groups regarding the following statements: 1) I think I would like to use this system frequently, in which the resident group entirely agreed with this statement, while the other groups only agreed with the statement (p-value = 0.018); 2) I found that the various functions in this system were well integrated, in which the resident and the orthopedic groups entirely agreed with this statement, whereas the academic group only agreed (p-value = 0.024); and 3) I was very comfortable using this system, in which the resident and the orthopedic groups fully agreed with this statement, whereas the academic group only agreed (p-value = 0.027). The analysis of part III, regarding the importance of the application in learning, evidenced differences between the groups with statistical significance for all questions (0.001-0.032), except for question 6 (p-value = 0.177) (Table 2). Table 2 Absolute frequencies and percentages of the answers to the questions of part III of the evaluation tool among medical students, orthopedic residents, and orthopedists with specialist qualifications. Occupation Student Resident Orthopedist p-Value 1) Is problem-solving training part of learning? Neutral 1 (1.5%) 0 (.0%) 0 (.0%) 0.003 Agree 26* (39.4%) 3 (9.1%) 4 (12.1%) Strongly agree 39 (59.1%) 30* (90.9%) 29* (87.9%) 2) Can the resolution of questions help in the preparation for the Board of Orthopedics and Traumatology (BOT) exam? Neutral 7* (10.6%) 0 (.0%) 0 (.0%) 0.001 Agree 23* (34.8%) 3 (9.1%) 5 (15.2%) Strongly agree 36 (54.5%) 30* (90.9%) 28* (84.8%) 3) Do I believe that the practicality of a mobile application on problem-solving in orthopedics and traumatology could help in preparing residents for exams? Neutral 1 (1.5%) 0 (.0%) 0 (.0%) 0.005 Agree 25* (37.9%) 3 (9.1%) 4 (12.1%) Strongly agree 40 (60.6%) 30* (90.9%) 29* (87.9%) 4) Does the application seem to be a useful technology in the theoretical development of the orthopedic resident during their training? Agree 26* (39.4%) 6 (18.2%) 6 (18.2%) 0.027 Strongly agree 40 (60.6%) 27* (81.8%) 27* (81.8%) 5) Can the application be used as an auxiliary method of learning for medical students studying Orthopedics and Traumatology? Disagree 2 (3.0%) 4 (12.1%) 7 (21.2%) 0.032 Neutral 1 (1.5%) 3* (9.1%) 2 (6.1%) Agree 32 (48.5%) 17* (51.5%) 14 (42.4%) Strongly agree 31* (47.0%) 9 (27.3%) 10* (30.3%) 6) Can the application be useful as an auxiliary learning method for general orthopedic physicians? Disagree 1 (1.5%) 0 (.0%) 3 (9.1%) 0.177 Neutral 2 (3.0%) 2 (6.1%) 0 (.0%) Agree 34 (51.5%) 15 (45.5%) 19 (57.6%) Strongly agree 29 (43.9%) 16 (48.5%) 11 (33.3%) * p <0.05, chi-square. Data expressed as absolute and percentage frequencies. DISCUSSION A substantial increase in the availability and use of mobile applications for smartphones and tablets for medical education has been reported in recent years, including the handling of work tools and care in the learning mode15. They are of great importance for enabling remote and online access to the information, allowing greater flexibility in learning. Students are more comfortable regarding when, how and where to study, making it possible for them to escape the traditional method of teaching conferences in the classrooms16. This teaching alternative can bring safety to the patient treated by the future doctors, since it challenges students regarding decision-making4. The advantages for the medical sciences include the encouragement of and engagement in independent and group learning, as well as flexibility in the method of study17. The use of ICTs in the medical area also includes improved patient care and multi-professional communication18. Among the disadvantages that can be mentioned, in addition to the cost of developing the technology, are the failures of the mobile device method, such as access to multiple contents, offering distraction through other tools, or even superficial learning, inadequate screen size and insufficient memory, the lack of validation and, sometimes, lack of scientific evidence related to some application categories7. However, with the improvements in science, it is believed that many of these difficulties can be solved19. In recent years, the growth of ICTs and their involvement in medical education has become significant, revolutionizing the access to information and learning methods. In previous literature review studies, it was observed that smartphone application technology can be integrated into successful orthopedic practice18. However, currently, the market does not have a large availability of applications targeting this specialty. Among users of these technology devices, approximately 50% use the Android® 22.5% use Blackberry® and 18.5% use iOS / Apple®(20 operating system. Among orthopedic surgeons, 84% use smartphones, mostly iPhone® (55%) and 53% use it in clinical practice. At the time of the survey, there were 61 orthopedic applications for the iOS / iPhone® platform and 13 applications for the Android® platform21. Moreover, there are concerns about the content available in this tool, since published information validation tools are pointed out as a challenge to be put into practice21. In this perspective, a study evaluated the main applications used by orthopedic specialists in hand surgery. Among the educational purposes, the reading of scientific journals and recent publications were the main modes of teaching and medical training. As a result, there was no application for direct resident training regarding the resolution of issues or clinical situations22. As for other studies, it could be inferred that, concerning orthopedic applications for teaching and medical training, the great majority consisted of guidelines, training of surgical techniques and videos3),(20, as well as for anatomy, medical calculators, laboratory results and drugs and medicines23. A pilot study of a simulation application aimed at the management of septic arthritis for orthopedic residents showed positive results when comparing the percentage of correct answers in the tests versus the correctness of the simulations, concluding that the subject training application has great potential to educate and evaluate the user in an efficient and cost-effective way24. Faced with this brief bibliographic review, the Quiz Ortopedia ® is a revolutionary application for teaching and medical training, related to an innovative and promising improvement in medical education. The developed application followed certain characteristics to meet the objectives of proposing an auxiliary tool in the preparation of residents of orthopedics and traumatology for the TEOT exam, based on the principle of retrieval learning. In addition, the method used to create the training application for orthopedic residents was based on some pillars of education by simulation: positive feedback, repetitive practice, difficulty scale, variable clinical catch, individual or group learning and validation of the simulator25. All these pillars were taken into account during the creation of the application assessed in this study. It has been observed that witnessing the development and improvement of their skills throughout the training increases the residents' confidence, as well as receiving constructive, direct and specific feedback26. The developed application allows the user to follow their evolution throughout the test, in addition to receiving the comments related to each question answered by them. The other applications on the market do not have comments related to the questions, do not have the bibliographic reference from which the question was taken, do not allow the comparison of the user's own results and do not allow a comparison of results with the other users. In the context of the questionnaire applied to the participants, we can infer the high acceptance of the practicality of the application by the sample group, since 96.2% (127) of the users agreed that it is easy to use, whereas 90.7% of the participants affirmed they would like to use the app frequently. The lowest agreement regarding the other evaluations (85.6%, corresponding to 113 users) was related to whether the use of the application was considered as an auxiliary study method for students; this can be justified by the answers of the group of students, since the purpose of the developed application is not to assist, at this moment, the study of the discipline of Orthopedics and Traumatology, but help the residents who will take the Board Exam. When asked about the usability of the application (part II and III of the questionnaire), the response of the majority of users was positive among the different sample groups. Regarding the usability, 100% (132) stated that it was a useful technology for the resident’s theoretical learning, in addition to the 93.4% (124) that agreed it is an auxiliary method of learning for general orthopedic physicians. These data indicate the users’ recognition regarding the importance of alternative ways to support the teaching-learning process, as well as references from previous studies, which identified the increase in the number of residents and the limited opportunities to develop surgical and technical skills. Many different solutions have been proposed in education centers around the world, including e-training programs, simulation, and mandatory training to maximize learning opportunities within existing resource constraints27. In this context, the use of simulation applications allows residents to practice skills in a safe and controlled environment, demonstrating their improved confidence and minimizing patient risk. For comparative purposes, a study related to the development of a teaching and training application in anesthesiology, analyzed 20 participants (medical students and residents) based on the SUS evaluation score of 90.628, which is referred to as a high-value usability index28. Although the SUS® score of our study was slightly lower (84.2), the size of our sample denotes a more significant impact and scientific representativeness, as well as greater variability, being translated by a similar sample reliability between the two studies (0.79). These data were measured taking into account Cronbach's alpha coefficient, which represents the best method to evaluate variation between samples14. These data demonstrate the importance of the SUS® score obtained by the application developed by this study. Another study assessing a medical education application for residents in the area of surgical simulation was able to conclude that this type of tool is a good complementary alternative for the training of these students’ profile by increasing cognitive performance and thus ensuring greater patient safety29. The Quiz Ortopedia®, albeit through a different proposal, can achieve the same goal by helping residents take the Board Exam after having developed more knowledge of the covered content. Among the limitations of the study, we can mention the lack of a comparative efficiency test between residents who used the application and the residents who studied using only the traditional methods. Another limitation is that it included residents from only one federation unit, not covering all the other states. More comparative studies are required, in the near future, to consolidate the efficiency of the Quiz Ortopedia® application as a support teaching and training method aimed at the education of medical residents of orthopedics and traumatology. CONCLUSION This study demonstrated the development, use, and evaluation, through a questionnaire validated in the study, of a mobile application created for the iOS® and Android® platforms aimed at the training of residents in orthopedics and traumatology based on the resolution of questions. The application showed high usability with good sample variability when validated by specialists in Orthopedics and Traumatology, as well as by medical students and orthopedic residents. The developed application was successful in the tests performed and may be an alternative in medical education in the orthopedic area. REFERENCES 1 1. Pereira FGF, Silva DVD, Sousa LMOD, Frota NM. Construção de um aplicativo digital para o ensino de sinais vitais. Rev. Gaúcha Enferm. 2016;37(2):e59015. Pereira FGF Silva DVD Sousa LMOD Frota NM Construção de um aplicativo digital para o ensino de sinais vitais Rev. Gaúcha Enferm 2016 37 2 e59015 2 2. Marçal E, Viana W, Andrade RMC, Rodrigues D. A mobile learning system to enhance field trips in geology. 2014 IEEE Frontiers in Education Conference (FIE) Proceedings. 2014;1-8. Marçal E Viana W Andrade RMC Rodrigues D A mobile learning system to enhance field trips in geology 2014 IEEE Frontiers in Education Conference (FIE) Proceedings 2014 1 8 3 3. Kulendran M, Lim M, Laws G, Chow A, Nehme J, Darzi A, et al. Surgical smartphone applications across different platforms: their evolution, uses, and users. Surgical Innovation. 2014;21(4):427-440. Kulendran M Lim M Laws G Chow A Nehme J Darzi A Surgical smartphone applications across different platforms: their evolution, uses, and users Surgical Innovation 2014 21 4 427 440 4 4. Cook DA, Blachman MJ, Price DW, West CP, Thomas BLB, Berger RA, et al. Educational Technologies for Physician Continuous Professional Development. Acad Med. 2018;93(1)1:104-112. Cook DA Blachman MJ Price DW West CP Thomas BLB Berger RA Educational Technologies for Physician Continuous Professional Development Acad Med 2018 93 1 104 112 5 5. Akhtar K, Sugand K, Sperrin M, Cobb J, Standfield N, Gupte C. Training safer orthopedic surgeons. Acta Orthop. 2015;86(5):616-621. Akhtar K Sugand K Sperrin M Cobb J Standfield N Gupte C Training safer orthopedic surgeons Acta Orthop 2015 86 5 616 621 6 6. Atesok K, Mabrey JD, Jazrawi LM, Egol KA. Surgical Simulation in Orthopaedic Skills Training. J Am Acad Orthop Surg. 2012;20(7):410-422. Atesok K Mabrey JD Jazrawi LM Egol KA Surgical Simulation in Orthopaedic Skills Training J Am Acad Orthop Surg 2012 20 7 410 422 7 7. Masalimova AR, Levina EY, Platonova RI, Yakubenko KY, Mamitova NV, Arzumanova LL, et al. Cognitive Simulation as Integrated Innovative Technology in Teaching of Social and Humanitarian Disciplines. EURASIA Journal of Mathematics, Science and Technology Education. 2017;13(8):4915-4928. Masalimova AR Levina EY Platonova RI Yakubenko KY Mamitova NV Arzumanova LL Cognitive Simulation as Integrated Innovative Technology in Teaching of Social and Humanitarian Disciplines EURASIA Journal of Mathematics, Science and Technology Education 2017 13 8 4915 4928 8 8. Shaw CM, Tan SA. Integration of Mobile Technology in Educational Materials Improves Participation: Creation of a Novel Smartphone Application for Resident Education. J Surg Educ. 2015;72(4):670-673. Shaw CM Tan SA Integration of Mobile Technology in Educational Materials Improves Participation: Creation of a Novel Smartphone Application for Resident Education J Surg Educ 2015 72 4 670 673 9 9. Sandholzer M, Deutsch T, Frese T, Winter A. Medical students’ attitudes and wishes towards extending an educational general practice app to be suitable for practice: A cross-sectional survey from Leipzig, Germany. Eur J Gen Pract. 2016;22(2):141-146. Sandholzer M Deutsch T Frese T Winter A Medical students’ attitudes and wishes towards extending an educational general practice app to be suitable for practice: A cross-sectional survey from Leipzig, Germany Eur J Gen Pract 2016 22 2 141 146 10 10. Karpicke JD, Roediger HL. The Critical Importance of Retrieval for Learning. Science. 2008;319(5865):966-968. Karpicke JD Roediger HL The Critical Importance of Retrieval for Learning Science 2008 319 5865 966 968 11 11. Brooke J. SUS-A quick and dirty usability scale. Usabil Eval Ind. 1996;189(194):4-7. Brooke J SUS-A quick and dirty usability scale Usabil Eval Ind 1996 189 194 4 7 12 12. Tenório JM, Cohrs FM, Sdepanian VL, Pisa IT, Marin HF. Desenvolvimento e Avaliação de um Protocolo Eletrônico para Atendimento e Monitoramento do Paciente com Doença Celíaca. Revista de Informática teórica e aplicada. 2010;17(2):210-220. Tenório JM Cohrs FM Sdepanian VL Pisa IT Marin HF Desenvolvimento e Avaliação de um Protocolo Eletrônico para Atendimento e Monitoramento do Paciente com Doença Celíaca Revista de Informática teórica e aplicada 2010 17 2 210 220 13 13. Davis FD. Perceived usefulness, perceived ease of use and user acepptance of information technology. MIS Quarterly. 1989;13(3):319-340. Davis FD Perceived usefulness, perceived ease of use and user acepptance of information technology MIS Quarterly 1989 13 3 319 340 14 14. Bonett DG, Wright TA. Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning. Journal of Internet Banking and Commerce. 2015;36(1):3-15. Bonett DG Wright TA Cronbach’s alpha reliability: Interval estimation, hypothesis testing, and sample size planning Journal of Internet Banking and Commerce 2015 36 1 3 15 15 15. Singler K, Roth T, Beck S, Cunningham M, Gosch M. Development and initial evaluation of a point-of-care educational app on medical topics in orthogeriatrics. Archives Of Orthopaedic And Trauma Surgery. 2016;136(1):65-73. Singler K Roth T Beck S Cunningham M Gosch M Development and initial evaluation of a point-of-care educational app on medical topics in orthogeriatrics Archives Of Orthopaedic And Trauma Surgery 2016 136 1 65 73 16 16. Shaw CM, Tan SA. Integration of Mobile Technology in Educational Materials Improves Participation: Creation of a Novel Smartphone Application for Resident Education. J Surg Educ . 2015;72(4):670-673. Shaw CM Tan SA Integration of Mobile Technology in Educational Materials Improves Participation: Creation of a Novel Smartphone Application for Resident Education J Surg Educ 2015 72 4 670 673 17 17. Juanes JA, Ruisoto P. Computer Applications in Health Science Education. J Med Syst. 2015;39(9):97. Juanes JA Ruisoto P Computer Applications in Health Science Education J Med Syst 2015 39 9 97 97 18 18. Ozdalga E, Ozdalga A, Ahuja N. The smartphone in medicine: a review of current and potential use among physicians and students. J Med Internet Res. 2012;14(5):e128. Ozdalga E Ozdalga A Ahuja N The smartphone in medicine: a review of current and potential use among physicians and students J Med Internet Res 2012 14 5 e128 19 19. Walsh K. Mobile Learning in Medical Education: Review. Ethiop J Health Sci. 2015;25(4):363-366. Walsh K Mobile Learning in Medical Education: Review Ethiop J Health Sci 2015 25 4 363 366 20 20. Al-Hadithy N, Gikas PD, Al-Nammari SS. Smartphones in orthopaedics. Int Orthop. 2012;36(8):1543-1547. Al-Hadithy N Gikas PD Al-Nammari SS Smartphones in orthopaedics Int Orthop 2012 36 8 1543 1547 21 21. Franko OI. Smartphone Apps for Orthopaedic Surgeons. Clin Orthop Relat Res. 2011;469(7):2042-2048. Franko OI Smartphone Apps for Orthopaedic Surgeons Clin Orthop Relat Res 2011 469 7 2042 2048 22 22. Franko OI. Mobile software applications for hand surgeons. J Hand Surg, 2012;37(6):1273-1275. Franko OI Mobile software applications for hand surgeons J Hand Surg 2012 37 6 1273 1275 23 23. Duncan SF, Hendawi TK, Sperling J, Kakinoki R, Hartsock L. iPhone and iPad use in orthopedic surgery. Ochsner J. 2015;15(1):52-57. Duncan SF Hendawi TK Sperling J Kakinoki R Hartsock L iPhone and iPad use in orthopedic surgery Ochsner J 2015 15 1 52 57 24 24. Shore BJ, Miller PE, Noonan KJ, Bae DS. Predictability of Clinical Knowledge Through Mobile App-based Simulation for the Treatment of Pediatric Septic Arthritis: A Pilot Study. J Pediatr Orthop. 2018;38(9):e541-e545. Shore BJ Miller PE Noonan KJ Bae DS Predictability of Clinical Knowledge Through Mobile App-based Simulation for the Treatment of Pediatric Septic Arthritis: A Pilot Study J Pediatr Orthop 2018 38 9 e541 e545 25 25. Guze PA. Using technology to meet the challenges of medical education. Trans Am Clin Climatol Assoc,2015;126:260. Guze PA Using technology to meet the challenges of medical education Trans Am Clin Climatol Assoc 2015 126 260 260 26 26. Lees MC, Zheng B, Daniels LM, White JS. Factors Affecting the Development of Confidence Among Surgical Trainees. J Surg Educ . 2019;76(3):674-683. Lees MC Zheng B Daniels LM White JS Factors Affecting the Development of Confidence Among Surgical Trainees J Surg Educ 2019 76 3 674 683 27 27. Stirling ER, Lewis TL, Ferran NA. Surgical skills simulation in trauma and orthopaedic training. Journal of orthopaedic surgery and research. 2014;9(1):126. Stirling ER Lewis TL Ferran NA Surgical skills simulation in trauma and orthopaedic training Journal of orthopaedic surgery and research 2014 9 1 126 126 28 28. Pereira RVS, Kubrusly M, Marçal E. Desenvolvimento, utilização e avaliação de uma aplicação móvel para educação médica: Um estudo de caso em Anestesiologia. RENOTE. 2017;15(1):1-10. Pereira RVS Kubrusly M Marçal E Desenvolvimento, utilização e avaliação de uma aplicação móvel para educação médica: Um estudo de caso em Anestesiologia RENOTE 2017 15 1 1 10 29 29. Sugand K, Mawkin M, Gupte C. Training effect of using Touch Surgery™ for intramedullary femoral nailing. Injury. 2016;47(2):448-452. Sugand K Mawkin M Gupte C Training effect of using Touch Surgery™ for intramedullary femoral nailing Injury 2016 47 2 448 452