Use of a mobile platform in rheumatology by undergraduate medical students during the COVID-19 pandemic

Uso de uma plataforma móvel em reumatologia por acadêmicos de Medicina durante a pandemia por Covid-19

Abstract:

Introduction:

Mobile applications are considered relevant in the health area. As rheumatology is a complex and prevalent specialty in clinical practice, the development of tools that favor learning becomes necessary.

Objective:

To evaluate the impact of a mobile platform on learning, obtaining the degree of satisfaction with the teaching tool and the effects on different scenarios of educational practice, including performance evaluation regarding questions about the theoretical contents of the platform.

Methods:

Quantitative, descriptive study, carried out in Christus University Center, located in Fortaleza, Brazil. Questionnaires prepared by the authors were used to assess the impact on the different active methodologies used in the institution and the degree of student satisfaction regarding their use. The estimate of the students’ cognitive gain was measured through pre-test and post-test, using the multiple-choice format.

Results:

The questionnaires were applied to 71 students to assess the effects and satisfaction. A total of 90 students participated in the pre-test and 32 in the post-test phase. The platform showed a positive perception of learning for 83.1% of the students, with greater impact on lectures, although there was a greater impact on the simulation scenarios in the eighth semester. A good degree of satisfaction was observed in 94.4% of the answers, with improved performance in the tests, increasing from 43.7% to 63.3% (P < 0.001).

Conclusions:

The development and application of the mobile platform in rheumatology showed excellent results, with favorable effects on the teaching of the specialty, disclosing a good degree of satisfaction regarding its use and better performance on the questioning about the theoretical contents of the platform.

Keywords:
Medical education; Rheumatology; Mobile applications

Resumo:

Introdução:

Aplicativos móveis são considerados relevantes na área da saúde. Sendo a reumatologia uma especialidade complexa e prevalente na prática clínica, torna-se necessário o desenvolvimento de ferramentas que favoreçam a aprendizagem.

Objetivo:

Este estudo teve como objetivo avaliar o impacto no aprendizado que uma plataforma móvel proporcionou, obtendo o grau de satisfação com a ferramenta de ensino e a repercussão nos diferentes cenários de prática educacional, incluindo avaliação de performance quanto a questionamentos do conteúdo teórico da plataforma.

Método:

Trata-se de estudo transversal com delineamento descritivo e abordagem quantitativa do tipo analítico, realizado no Centro Universitário Christus, localizado em Fortaleza, Brasil. Pesquisaram-se, por meio de questionários elaborados pelos autores, a repercussão nas diferentes metodologias ativas presentes na instituição e o grau de satisfação dos alunos quanto ao uso. A estimativa do ganho cognitivo dos alunos foi mensurada por meio de pré-teste e pós-teste, no formato de múltiplas escolhas.

Resultado:

Participaram 71 alunos durante do questionário sobre a repercussão e a satisfação. Nas questões de múltiplas escolhas, 90 alunos participaram do pré-teste e 32 do pós-teste. A plataforma apresentou percepção positiva na aprendizagem para 83,1% dos alunos, havendo maior impacto nas aulas expositivas, embora no oitavo semestre tenha havido maior repercussão nos cenários de simulação. Houve boa satisfação em 94,4% das respostas, com melhora no aproveitamento quanto às questões de múltiplas escolhas, evoluindo de 43,7% para 63,3% (p < 0,001).

Conclusão:

A elaboração e a aplicação da plataforma móvel em reumatologia obtiveram excelentes resultados, com repercussão favorável no ensino da especialidade e boa satisfação quanto ao uso e melhor performance no que concerne ao questionamento sobre os conteúdos teóricos da plataforma.

Palavras-chave:
Educação Médica; Reumatologia; Aplicativos Móveis

INTRODUCTION

Since March 2020, from the moment the World Health Organization (WHO) declared that the infection by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus was a pandemic, technology has been essential for the continuity of the teaching plan, because there was no access, justified by the rules of social distancing, to the main learning scenarios: classroom, laboratories, simulation centers, community, hospitals, and primary care centers. Thus, the virtual environment became the most appropriate one¹. The need for social distancing made pedagogical programming unfeasible, making it necessary to provide a temporary system for teaching and educational support with a rapid configuration, characterizing the concept of emergency remote teaching, a term used to differentiate the previously developed online teaching modality , which was designed aimed at continuity². This teaching modality impacted the practical ability of teachers to offer support for the development of skills by some students, especially those with internet access difficulty, as well as affected student motivation and performance, promoting reflection on the sustainability of emergency teaching, which does not make the modality unviable, but makes it possible to use its complementary form, or blended learning³. One way to minimize these adversities was the use of mobile applications or platforms⁴.

Mobile applications are programs developed for a given operating system of a mobile device, such as tablets, smartphones, with interaction occurring through touch, due to the existence, in this type of technology, of a touchscreen, with different topics: games, personal organizers, text editors, chat apps, among others⁵. In the health area, it is currently considered extremely relevant, and has become useful for remote monitoring, diagnostic support, and decision-making⁶. This technological advance has also been used in the medical area, , either through data analysis, distance learning, artificial intelligence, or through digital/mobile health, advanced technology, and social networking⁷.

Rheumatology, as other medical specialties, has taken advantage of these technological tools. Review articles have reported the importance of some applications in rheumatology, facilitating therapeutic planning in common diseases of the area, such as systemic lupus erythematosus and osteoarthritis, not only regarding pharmacological management but also the rehabilitation regimen and physical activity. Other applications can be useful for checking the criteria for disease diagnosis, as well as the development of electronic portals that have been encouraged by relevant entities, such as the American College of Rheumatology⁸.

Few studies have addressed this issue in rheumatology. There is information on the digital impact on the specialty, concerning social media, and how it can influence medical education, career advancement, dissemination of research, and the increase of professional links (networking). The European League Against Rheumatism (EULAR) found that 68% of its members access social media for professional reasons, such as networking, searching for new research, acquiring new skills, and participation in online events⁹. In Brazil, an experimental, randomized, case-control study involving two pediatric centers evaluated an online virtual learning environment in pediatric rheumatology, aimed at pediatric residents, analyzing its effectiveness and satisfaction rates, observing that 75% of the participants agreed that a good level of learning could be achieved through the methodology¹⁰.

After the beginning of the pandemic caused by the coronavirus disease 2019 (COVID-19), a change of posture by the rheumatologists was necessary regarding assistance and teaching, finding in technology a flexible, innovative, and safe way to perform their activities¹¹. Adaptations of existing activities were made, such as bedside visits, conferences, case discussions, and articles for virtual platforms, as well as the increase of new teaching tools, such as the use of interactive games and the development of multidisciplinary sessions¹².

Rheumatology is considered a complex specialty. In a study on medical students’ perception of clinical practice, rheumatology was considered the third most difficult area by the students, together with intensive care medicine and endocrinology¹³. There is also a high prevalence of musculoskeletal diseases, especially rheumatological ones, in clinical practice. A Canadian study highlighted the difference between the time spent in teaching about the musculoskeletal system during medical school and the time spent in the practice of the profession by family physicians, showing that only 2.26% of the undergraduate time is dedicated to this specialty, and in clinical practice, 13.7% to 27.8% of patients have complaints directly related to the locomotor system¹⁴. This disproportion reflects in the degree of confidence and retention of knowledge about musculoskeletal diseases exhibited by medical students, as observed at the Faculty of Medicine, Alexandria University, Egypt, where 80% of the interviewed students reported a low level of confidence, and 75% of the participants scored 65% or less on a Rheumatology test¹⁵. In Brazil, a study conducted in a private higher education institution showed that 69% of the interviewed students considered their capacity to perform the osteomuscular physical examination to be bad or regular¹⁶. Thus, it is likely that a mobile platform, new and focused on teaching the specialty, will be well received by the student body.

The objective of this study is to measure, among the students at the Medical School of Centro Universitário Christus (Unichristus), the degree of satisfaction with the use of a new rheumatology platform, as well as to evaluate its impact on the different scenarios of educational practices, including performance evaluation regarding questions about the theoretical content of the teaching tool.

METHODS

This is a quantitative, experimental and descriptive study. After approval by the Research Ethics Committee on September 04, 2020 (N. 4.260.432), data collection was performed using an electronic form (Google Forms) during the period from October 2020 to July 2021. The study was conducted at Centro Universitário Christus, a private teaching institution located in the city of Fortaleza, in the state of Ceará, northeast Brazil. The Medicine course has a hybrid teaching plan, involving active teaching methodologies such as dialogical presentations, flipped classroom, simulation laboratory, problem-based learning (PBL), team-based learning (TBL), and clinical experiences, which take place in some partner hospitals and the School of Health Clinic (SHC) of the university center.

The study population consisted of medical students from the university center, considering as inclusion criteria the use of the mobile platform and being enrolled in the fourth or eighth semester of 2020.2 and 2021.1, periods of the curriculum with a greater workload of topics in rheumatology. A total of 360 ​​students were enrolled, considering the two semesters. The exclusion criterion was the student’s disengagement from the institution during the study period. The study was conducted in two stages: the first, of a methodological nature, characterized by the development of a teaching instrument; and the second of a quantitative nature.

The development of a technological product, a mobile platform in rheumatology called ‘ReumaChristus’, characterized the first stage of the study, aiming to support the teaching of rheumatology in the undergraduate medical course of the institution. This was registered by the National Institute of Industrial Property (INPI), under registration number BR512021000157-5. The platform is web-based, with free access through the link reumatologia.web.app, without the need for any registration or login for use. It has several topics, such as abstracts on different rheumatological pathologies, lectures given at the institution medical course, images from the authors’ database, free-access articles/guidelines, and videos of interviews with patients, professionals, and teachers linked to rheumatology. The platform usability analysis was performed using the System Usability Scale (SUS), an instrument previously validated for the Portuguese language¹⁷. This scale is based on the degree of agreement according to a Likert scale, assigning a score ranging from 1 to 5, with the lowest score being consistent with “strongly disagree” and the highest score being consistent with “strongly agree”. To calculate the SUS score, for items 1,3,5,7 and 9, the score corresponds to the position on the scale minus one. For items 2,4,6,8 and 10, the score corresponds to 5 minus the position on the scale. A summation of the scores for each item is made and the sum of the scores is multiplied by 2.5 to obtain the overall value of the usability scale. The odd-numbered questions correspond to questions of a positive nature, whereas the even-numbered ones correspond to the opposite situation¹⁸. A total of 71 students participated in this research, who were attending from the fourth to the eighth semesters, during which they were invited to voluntarily participate. This was considered a representative sample, thus resulting in a score of 87.39, and a value above 70 (acceptable), was considered good when >80^19),(20. It is important to emphasize that the participants’ informed consent form was created by the authors and applied before participation in the study was started, after being approved by the ethics committee.

In the second stage, a cognitive pre-test was applied before using the platform, containing 10 multiple-choice questions on rheumatology topics. After using the ReumaChristus platform, a post-test was applied, with the same questions as the initial test, aiming to estimate the students’ learning gain regarding the theoretical content of the platform. Finally, a questionnaire was applied on the student degree of satisfaction and perception of learning.

The pre-test and post-cognitive test questionnaires were applied, respectively, before and after the use of the mobile platform. They were prepared by the authors themselves and contained 10-multiple choice questions of varying difficulty and involved various topics, such as soft tissue rheumatism, inflammatory arthropathies, collagenoses, and infectious arthropathies. The entire content was present in the teaching tool.

All tests were applied remotely through Google Forms due to the biosecurity measures necessary for protection against SARS-CoV2 infection, reinforcing social distancing established by state government decree N. 33.519 of March 19, 2020²¹.

The data were tabulated in Microsoft Excel for Windows, and exported to the Statistical Package for the Social Sciences (SPSS) software, version 20.0 (IBM Corp., Armonk, New York, United State) with a 95% confidence level, and a P-value < 0.05 was considered statistically significant.

Subsequently, the evaluation semesters (4^th and 8^th) were associated with the other categories using Pearson’s chi-square test. Regarding the pre and post-test, the average percentage of correct answers for each class was calculated and compared using the Mann-Whitney test.

RESULTS

A total of 71 students answered the questionnaire on satisfaction and perception of cognitive gain after using the platform, with a predominance of females, with an average age of 24 years, and a slightly higher number of participants attending the eighth semester (Table 1).

Table 1
Demographic data of the students who answered the questionnaire on satisfaction and perception of cognitive gain after using the ReumaChristus platform, Fortaleza (CE), Brazil, 2020-2021 (n = 71).

Variable	Value
Age in Years
Average	24.3 ± 4.81
Minimum	19
Maximum	38
Sex (%)
Female	55 (77.5%)
Male	16 (22.5%)
Previous professional degree (%)
No	56 (78.9%)
Yes	15 (21.1%)
Semester (%)
Fourth	34 (47.9%)
Eighth	37 (52.1%)

• Source: Prepared by the authors.

It was observed that 59 (83.1%) of the students considered that the platform had enough impact on learning in rheumatology during the semester (Figure 1), with different percentages when evaluating the different scenarios of the active teaching methodologies (Figure 2).

Figure 1
Students’ perception of the positive impact of the mobile platform on learning in rheumatology during the semester, Fortaleza (CE), Brazil, 2020-2021 (n = 71).

Figure 2
Students’ perception of the positive impact of the mobile platform on each scenario of the institution’s active methodologies, Fortaleza (CE), Brazil, 2020-2021 (n = 71).

When students were asked which active teaching methodology had the greatest positive impact on learning after using the platform, they answered that the lectures were the most important, corresponding to 31 answers (43.7%), maintaining the option when analyzing the data related exclusively to the fourth semester; however, there was a predominance of simulation as the answer in the eighth semester. All these results showed statistical significance (Table 2).

Table 2
Students’ perception of which active teaching methodology had the greatest positive impact on learning after using the platform, Fortaleza (CE), Brazil, 2020-2021 (n = 71).

	Total	Value	Semester		P-Value
			4th	8th
Expository lessons	31 (43.7%)*	< 0.001	20 (58.8%)*	11 (29.7%)	0.025
Simulations	27 (38.0%)		10 (29.4%)	17 (45.9%)*
Tutorials	8 (11.3%)		4 (11.8%)	4 (10.8%)
Experiences (School Clinic)	5 (7.0%)		0 (0.0%)	5 (13.5%)*

• P < 0.05, Pearson’s chi-square test (n, %).
• Source: Prepared by the authors.

It was observed that 67 (94.4%) of the total number of students were satisfied with the mobile platform, while 4 (5.6%) were indifferent to it, and there was no response with a negative evaluation (Figure 3). No statistical significance was observed in the comparison between the semesters.

Figure 3
Students’ level of satisfaction with the ReumaChristus platform, Fortaleza (CE), Brazil, 2020-2021 (n = 71).

Regarding the estimation of knowledge acquisition, 90 students answered the cognitive pre-test, while 32 answered the post-test, with a mean score of 43.7 ± 15,6% and 63.3± 15,7%, respectively, with statistical significance (P < 0.001) according to the Mann-Whitney test.

DISCUSSION

The ReumaChristus platform showed, according to the students’ viewpoint, good performance and contributed positively to the knowledge and learning of students in the various scenarios of active methodologies. The learning gain is estimated by the ascending variation of performance in the multiple choice tests before and after using the platform.

The ReumaChristus platform, according to the students’ perception, had a positive effect on the learning in rheumatology, with a very satisfactory agreement of 83.1%. As a comparison, a Brazilian study involving the development and application of a virtual learning environment in pediatric rheumatology for pediatric residents found a percentage of 75% of the responses when evaluating the tool as useful for teaching¹⁰.

When considering the insertion of the platform in the active teaching methodologies, a positive impact was observed in all scenarios during the analysis of all semesters, with a predominance of greater impact on the lectures, the most frequent methodology in the institution during the period of the COVID-19pandemic, since face-to-face activities were reduced, directly impacting the number and format of simulations and experiential activities. This analysis is very pertinent when analyzing the fourth semester alone, in which almost all simulations were remote, via Google Meet^®, and there were no experiential activities, justifying the absence of impact of this activity on the aforementioned semester. The eighth semester had simulations in rheumatology before the beginning of the pandemic, as well as face-to-face experiences, respecting the current biosafety protocols at the time. The impact of the platform on the tutorials was similar in both groups.

Despite the favorable effect of the platform on remote activities, it is not possible to conclude such information regarding the students’ adherence to these activities, whether synchronous or asynchronous. Some factors can be seen as impediments, such as the lack of students’ motivation and expectations, the exhaustive use of digital resources by the teacher, the inadequacy of some contents, and some students’ inability regarding the use of technology²². The students’ stress level, the verbal and non-verbal communication impairment, the prolonged activity time, and the technical problems with hardware and software can also be considered obstacles to the development of the activity²³.

Although a satisfactory level was observed related to the students’ perception of learning gain and degree of satisfaction with the platform, there was no change in most of the student body regarding their interest in rheumatology as a medical specialty, since 64.8% were indifferent or disagreed when questioned. The literature is scarce regarding the discussion of the students’ interest in following different medical areas; however, the few existing results in Brazil do not show rheumatology as one of the most sought after specialties^24),(25. The workload, the time available for family, the place of professional practice, the early financial return, and the possibility of clinical-surgical context are aspects that influence the preference for a particular medical area²⁶.

When evaluating the degree of student satisfaction with the ReumaChristus platform, a very favorable result was observed, since there was a 94% agreement in the answers. Among the specific sections of the tool, a lower percentage of agreement (84%) was evidenced in the one related to scientific articles, probably due to the limited number of available articles, since, due to copyrights, only open-access papers were available on the platform.

The students’ performance regarding the theoretical contents of the platform was estimated through a pre-test and a post-test, both containing the same questions, showing an improvement in performance from 43.7% to 63.3% (P < 0.001). A similar result was observed when comparing the results of the pre-test with those of the post-test after using a virtual pediatric rheumatology learning environment, since the aforementioned study showed an increase from 50% to 63.1% with statistical significance (P <0.001), although the interviewed sample did not comprise undergraduate students, but pediatric residents¹⁰.

There are several digital tools in rheumatology for patients with rheumatological diseases; however, there are few applications focused on medical education in this specialty, and most of the existing ones do not have a validation analysis, which gives this study an unprecedented positive impact on medical education. It is noteworthy that in Brazil, there is only one study in rheumatology that analyzes the introduction of a digital learning environment for pediatric residents¹⁰. It is observed that a platform in rheumatology, in general, is something innovative, since there is no similar research in adults, as seen in Pubmed and Google Scholar²⁰.

The platform emerges as an innovative tool for medical education, resulting in positive points, which gives the tool to disseminate this in training centers aiming at the use and positive impact of the same training of students.

There are some limitations in the present the study and, consequently, in the interpretation of the data. The research was carried out during a pandemic and consequent social distancing, which greatly limited the number of face-to-face activities, thus making it necessary to obtain the answers to the questionnaires online, resulting in a reduction in adherence throughout the study period. This reduction is demonstrated by the number of responses to the applied post-test questionnaires, diverging from what was expected during the study design. Another important fact to be raised is that the researchers are teachers of the students evaluated by the questionnaires; however, during the research, the students answered the questionnaires voluntarily, and at no time was the grade for the school semester linked to the answering of the questionnaires. The scarcity of studies on the use of digital tools in the teaching of rheumatology makes it impossible to use other results for comparative purposes. It is worth noting that the impact of the ReumaChristus platform could be better assessed using other observational methodologies, such as cohort or case-control studies, which provide greater strength of evidence; however, they may involve ethical issues, logistical difficulties and higher financial costs²⁷. It is also noted that, although the study is a single-center one, this digital format can be used as an example and basis for future studies, mainly multicentric studies. Another relevant limitation is the number of multiple-choice questions in the pre-test and post-test, which can interfere with the reliability of the assessment; a total number of 100 questions is considered reproducible, which is unfeasible from a logistical point of view to be applied in the study^28).

Apps and mobile platforms are appropriate means to be used in medical education amid the continuing development of technology and the need for social distancing due to the SARS-CoV-2 pandemic, being no different for the ReumaChristus platform. The availability of abstracts on the main topics, lectures, scientific articles, guidelines, images, knowledge on support groups for rheumatic patients, and videos on the viewpoint of specialists, future rheumatologists, and patients makes the platform extremely useful in undergraduate courses, with potential use also by residents, rheumatologists, and doctors from other specialties.

The developed technological tool can be used as a complementary method in the teaching of rheumatology in higher education institutions, taking into account that it had a favorable impact on the learning gain and aided in different scenarios of active methodologies (lectures, tutorials, simulations, and experiences), as well as having been evaluated as a satisfactory tool.

CONCLUSIONS

The development and use of a mobile platform focused on rheumatology achieved excellent results, not only regarding its usability but also regarding the student degree of satisfaction and performance related to questions about the theoretical content of the platform, helping in various scenarios of the active teaching methodologies. We also believe that the ReumaChristus platform is an unprecedent and relevant action in the dissemination and knowledge of rheumatology in the academic environment, favoring better assistance from the medical graduates to patients with musculoskeletal complaints. Thus, these are favorable characteristics of a tool to be used for teaching in undergraduate medical school, and more specifically, in rheumatology.

REFERENCES

• ¹
  Valdez-García JE, Eraña-Rojas IE, Elizondo JAD, Cordero-Diaz MA, Torres-Quintanilla A, Esperón-Hernández RI, et al. The role of the medicine student in COVID-19 pandemic. A shared responsibility. Cir Cir. 2020;88(4):399-401. PMID: 32567589.
• ²
  Rahim AFA. Guidelines for online assessment in emergency remote teaching during the COVID-19 pandemic. Education in Medicine Journal. 2020;12(2):58-69.
• ³
  Code J, Ralph R, Forde K. Pandemic designs for the future: perspectives of technology education teachers during COVID-19. Infor Learn Sci. 2020;121(5/6):419-31.
• ⁴
  Shah NL, Miller JB, Bilal M, Shah B. Smartphone Apps in Graduate Medical Education Virtual Recruitment During the COVID-19 Pandemic. J Med Syst. 2021;45(3):36. PMID: 33559756.
• ⁵
  Souza AL, Murta CAR, Leite LGS. Tecnologia ou metodologia: aplicativos móveis na sala de aula. In: Anais do Encontro Virtual de Documentação em software livre e Congresso Internacional de linguagem e tecnologia online; 2016. Minas Gerais: Universidade Federal de Minas Gerais.
• ⁶
  Tibes CMS, Dias JD, Zem-Mascarenhas SH. Aplicativos móveis desenvolvidos para a área da saúde no Brasil: revisão integrativa da literatura. Revista Mineira de Enfermagem. 2014;18(2):471-86.
• ⁷
  Young AJ. Novas tecnologias e Medicina de Família. Rev Bras Med Fam Comunidade. 2017;12(39):1-6.
• ⁸
  Miedany YE. e-Rheumatology: are we ready?. Clin Rheumatol. 2015;34(5):831-7. PMID: 25708153.
• ⁹
  Venuturupalli RS, Sufka P, Bhana S. Digital Medicine in Rheumatology: Challenges and Opportunities. Rheum Dis Clin North Am. 2019;45(1):113-26. PMID: 30447740.
• ¹⁰
  Cunha ALG, Terreri MT, Len CA. Virtual learning environment in pediatric rheumatology for pediatric residents. Rev Paul Pediatr. 2020 Jan 13;38:e2018189.
• ¹¹
  Koumpouras F, Helfgott S. Stand Together and Deliver: Challenges and Opportunities for Rheumatology Education During the COVID-19 Pandemic. Arthritis Rheumatol. 2020 Jul;72(7):1064-1066. PMID: 32270923.
• ¹²
  Bilal S, Shanmugam VK. Enhancing rheumatology education during the COVID-19 Pandemic. Rheumatol Int. 2021 Mar;41(3):503-508.
• ¹³
  Romero MLDB, Dourado MHB, Júnior JM, Caetano DAP, Marinho RB, Luz AL. A clínica médica sob a percepção do estudante de medicina/The medical clinic under the perception of the medical student. Brazilian Journal of Health Review. 2018;1(2):328-31.
• ¹⁴
  Caldas CAM, Paz OAG, Negrão JNC, Caldato MCF. A reumatologia em um curso de medicina com aprendizagem baseada em problemas. Revista Brasileira de Educação Médica. 2013;37(4):584-90.
• ¹⁵
  Abou-Raya A, Abou-Raya S. The inadequacies of musculoskeletal education. Clin Rheumatol . 2010;29(10):1121-6. PMID: 20607334.
• ¹⁶
  Pícoli RP, Domingo ALA, Santos SC, Andrade AHG, Araujo CAF, Kosloski RMM, et al. Competências propostas no currículo de medicina: percepção do egresso. Revista Brasileira de Educação Médica . 2017;41(4):525-32.
• ¹⁷
  Boucinha RM, Tarouco LMR. Avaliação de ambiente virtual de aprendizagem com o uso do sus-system usability scale. RENOTE. 2013;11(3).
• ¹⁸
  Brooke J. SUS: A quick and dirty usability scale. Usability Eval. 1995; 189:11-30.
• ¹⁹
  Bangor A, Kortum PT, Miller J. Determining what individual SUS scores mean: Adding an adjective rating scale. Journal of usability studies. 2009;4(3):114-23.
• ²⁰
  Silva FTM, Pinheiro GA, Sousa CGS, Leitão ACSB, Kubrusly M, Barros Filho EMd, et al. Desenvolvimento e usabilidade de uma plataforma móvel para apoio ao ensino de reumatologia. Research, Society and Development. 2021;10(11):e527101119960.
• ²¹
  Ceará. Decree No. 33.519, of March 19, 2020. Intensifies measures to fight human infection by the new coronavirus. Official State Gazette; March 19, 2020; 3 (56):1-2.
• ²²
  Regmi K, Jones L. A systematic review of the factors - enablers and barriers - affecting e-learning in health sciences education. BMC Med Educ. 2020 Mar 30;20(1):91.
• ²³
  Reinhart A, Malzkorn B, Döing C, Beyer I, Jünger J, Bosse HM. Undergraduate medical education amid COVID-19: a qualitative analysis of enablers and barriers to acquiring competencies in distant learning using focus groups. Med Educ Online. 2021 Dec;26(1):1940765. PMID: 34128776; PMCID: PMC8208109.
• ²⁴
  Filho ABM, Cavalcante DF, Santos LGM, de Lima VAM, Maia AMLR, Fernandes TAAM. Especialização precoce em medicina: um estudo com acadêmicos dos três primeiros anos de um curso no Semi-Árido Nordestino. Brazilian Journal of Development. 2021;7(5):49895-908.
• ²⁵
  Purim KS, Borges LMC, Possebom AC. Profile of the newly graduated physicians in southern Brazil and their professional insertion. Rev Col Bras Cir. 2016 Jul-Aug;43(4):295-300. English, Portuguese. PMID: 27679952.
• ²⁶
  Martins JB, Rodriguez FP, Coelho ICMM, Silva EM. Fatores que Influenciam a Escolha da Especialização Médica pelos Estudantes de Medicina em uma Instituição de Ensino de Curitiba (PR). Rev. bras. educ. med. 43 (2) • Apr-Jun 2019.
• ²⁷
  Fronteira I. Estudos Observacionais na Era da Medicina Baseada na Evidência: Breve Revisão Sobre a Sua Relevância, Taxonomia e Desenhos [Observational studies in the era of evidence based medicine: short review on their relevance, taxonomy and designs]. Acta Med Port. 2013 Mar-Apr;26(2):161-70. Portuguese. Epub 2013 May 31. PMID: 23809749.
• ²⁸
  Case SM, Swanson DB. Constructing written test questions for the basic and clinical sciences: National Board of Medical Examiners Philadelphia, PA; 1998.

• 2
  Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Edited by

Chief Editor: Rosiane Viana Zuza Diniz. Associate Editor: Roberto Esteves.

Publication Dates

• Publication in this collection
  26 June 2023
• Date of issue
  2023

History

• Received
  15 Oct 2022
• Accepted
  07 Mar 2023