Web-Based Undergraduate Medical Education in a Virtual Learning Environment Using an Original Pedagogical Approach: an Observational Longitudinal Study

Educação médica de graduação via web em um ambiente virtual de aprendizagem utilizando uma abordagem pedagógica original: um estudo observacional longitudinal

ABSTRACT

Background

Asynchronous Web-based Medical Education in Virtual Learning Environments (VLEs) has grown steadily because of its many advantages. Various configurations and instructional methods are presently available. The existing proposals are poorly structured and/or not very effectively used for teaching diagnostic skills to undergraduate medical students in Brazil. A robust instructional method with positive pedagogical characteristics is needed. Thus, we have proposed a pedagogically-structured method for VLEs that includes a motivating initial reading (Medical Chronicle – MC), and a knowledge building program, using real cases coupled with audiovisual resources (Diagnostic Workshop – DW). We aimed to verify its acceptance, as well as the efficacy of the MC/DW method in improving the diagnostic ability of medical students, in the long term.

Methods

An opinion survey, two MC/DW materials and two Knowledge and Diagnostic Skills (KDS) questionnaires on stroke and epilepsy were developed, and two medical student groups were followed up in this 2013 longitudinal observational study. The students answered a KDS1, and attended a traditional lecture on one of the topics. They also accessed a VLE to apply the MC/DW method on stroke or epilepsy. We applied the same questionnaire (KDS2 and KDS3, respectively), one month and 5-6 months after the KDS1. We analyzed the mean KDS1 score of all the students, and the mean pairwise of those who accessed and those who did not access the VLEs during these three stages. An opinion survey was applied, and the results were analyzed by descriptive statistics.

Results

87 students participated in the study, but six were excluded as they did not answer the questionnaires. The KDS1 general mean score was 1.59 (SD0.71). We found that 66 students (81.5%) accessed the VLE, showing a significant improvement in diagnostic skills in the KDS2 (mean5.65, p<0.05) and KDS3 (mean 4.57, p<0.05), with non-significant variations for those who did not access it. The MC was considered at least good for 62 students (94%), with 52 students (78.8%) finding that a checklist was sufficient to clear up all their DW doubts.

Conclusions

The MC/DW method in VLE proved to be effective for improving the diagnostic capability of the undergraduate medical students in the long term, and it was well accepted by the students. It presents several positive pedagogical characteristics and can be replicated.

Education in Health; Education; Medicine

RESUMO

Contexto

A educação médica assíncrona via web, em Ambientes de Aprendizagem Virtual (AVA), cresceu muito, em virtude de suas diversas vantagens. Há várias configurações e métodos de instrução disponíveis. As propostas existentes são pouco estruturadas e/ou não são efetivamente utilizadas para o ensino de habilidades de diagnóstico para estudantes de graduação em Medicina, no Brasil. É necessário um método instrutivo robusto, com características pedagógicas positivas. Assim, propomos um método pedagogicamente estruturado para AVA, que inclui uma leitura inicial motivadora (Crônica Médica – CM) e um programa de construção de conhecimento, utilizando casos reais juntamente com recursos audiovisuais (Oficina Diagnóstica – OD). Objetivamos verificar a aceitação e a eficácia do método proposto na melhoria da capacidade diagnóstica dos estudantes de Medicina, em longo prazo.

Métodos

Foi desenvolvido um questionário de opiniões, dois materiais CM/OD e dois questionários de Conhecimento e Habilidades Diagnósticas (CHD) sobre acidente vascular encefálico e epilepsia. Dois grupos de estudantes de Medicina foram acompanhados neste estudo observacional longitudinal, em 2013. Os alunos responderam ao CHD1 e participaram de uma aula teórica tradicional, sobre um dos tópicos. Eles também acessaram um AVA para realizar o método CM/OD sobre um dos dois temas. O mesmo questionário (CHD2 e CHD3) foi aplicado, respectivamente, um mês e cinco-seis meses após o CHD1. Analisamos a média CHD1 de todos os alunos e as médias pareadas nos três momentos, dos que acessaram e daqueles que não acessaram o AVA. O questionário de opiniões foi aplicado e analisado por estatística descritiva.

Resultados

Oitenta e sete alunos participaram, mas seis foram excluídos por não responderem a todos os questionários. A média geral do CHD1 foi de 1,59 (desvio-padrão 0,71). Dos 81 participantes, 66 (81,5%) acessaram o AVA, mostrando melhora significativa nas habilidades de diagnóstico no CHD2 (média 5,65, p <0,05) e CHD3 (média 4,57, p < 0,05), com variações não significativas para aqueles que não acessaram. A CM foi considerada pelo menos boa para 62 alunos (94%), e 52 alunos (78,8%) julgaram que o gabarito disponível foi suficiente para sanar todas as dúvidas da OD.

Conclusões

O método CM/OD no AVA provou ser eficaz para melhorar a capacidade de diagnóstico dos estudantes de graduação em Medicina, em longo prazo, e foi bem aceito pelos alunos. Ele apresenta várias características pedagógicas positivas e pode ser replicado.

Educação em Saúde; Ensino; Medicina

BACKGROUND

Computers have been used in medical education since 1960^1,2. Recently, there has been evidence of increased investment in Web-Based Medical Education (WBME) worldwide, following the globalization promoted by the Internet and the numerous teaching-learning opportunities available^3,4,5. WBME is gaining increasing acceptance through activities and courses offered by Virtual Learning Environments (VLEs). These spaces help minimize problems in the classroom, such as material availability, since it is a form of self-directed and potentially interactive study carried out at different times and places, but focused on the same content and same skills^6,7.

However, WBME using VLEs is not a single entity, and various configurations are used, such as e-mail and multimedia⁸, and instructional methods, such as recorded lectures⁹, patient cases¹⁰, self-assessment, feedback, and interactive discussions¹¹. How and when to use specific designs is something that must be learned¹². Unfortunately, the existing proposals, including massive open online courses (MOOCs), are poorly structured and/or not very effectively used for teaching diagnostic skills to undergraduate medical students, especially in Brazil, being mostly applied in the basic sciences^3,13,14. VLEs and MOOCs are often used only for classroom “virtualization”, privileging technology instead of the pedagogical method¹⁵. But studying clinical cases is not sufficient; students must also be exposed to a robust instructional method, with contextualized, practical, and interactive experiences, feedback, and repetition¹⁶. In practice, even modern Web technologies, such as Web2.0 and Second Life, though potentially fit for collective knowledge construction, have played little role in medical education so far^17,18,19.

We recently proposed an original pedagogical method for teaching of neurology in classroom medical education²⁰. We adapted our proposal for VLEs, taking into account the growth of WBME in VLEs and its advantages, pedagogical features of VLE learning and the lack of well-structured and effective-proven proposals for WBME under VLEs to improve undergraduates’ diagnostic skills. The method was named Medical Chronic/Diagnostic Workshop (MC/DW), and is based on Charles Maguerez’s Arch and on constructivist theories²¹. Its aim is to develop the clinical method, with emphasis on the medical interview and relevant physical examination, which account for up to 88% of diagnoses²². MC/DW is patient-centered, focused on solving virtual, contextualized and interactive real cases, with feedback, repetition and analysis²³. It involves a motivating initial reading (MC), and a knowledge-building structured program (DW).

We present the Medical Chronicle (MC) as a simple language narrative. We developed a reflection-inducing, reliable chronological record of events, with a critique by the author, inspired by real-life events in the daily life of the medical services²⁴. Several critical incidents are included in the MCs, to open an agenda of topics to be learned. These topics should be those that general practitioners need to know. The MC therefore acts as a learning motivating tool, applicable to VLE²⁵. The second tool used in the method is a Diagnostic Workshop (DW), focusing on topics defined by the agenda. It consists of digitized real medical cases, presented in ascending order of complexity, in a trans-disciplinary mode, allowing a gradual and cumulative, mainly diagnostic, construction of knowledge. Each question contains one or more virtual “stations” (e.g. images, scientific article excerpts, videos, and complementary exams) for its resolution. We do give frequent summaries and we approach diagnostic heuristics. We allow classroom or virtual interaction among the participants. Feedback (with post-references) is provided at the end of the DW, always seeking the best available scientific evidence. We encourage the students to return to the questions/“stations” several times, thus promoting repetitive study and respect for individual learning time, an aspect that is severely neglected in class. We respect copyright law and restrict VLE access. All these characteristics make DW an original tool with many differential characteristics for use in VLEs²⁰.

The questions that motivated this research were: Would students access this method in the VLE? How would they subjectively analyze our proposal? and Would the MC/DW method be effective in improving the diagnostic skills of the medical students in a long term, if applied after a traditional lecture class? This work aimed to verify the acceptance of our proposal by the students, as well as the efficacy of the VLE MC/DW method in improving the diagnostic ability of medical students in the long term.

METHODS

Elaboration of the research material

We developed two MC/DW materials, one on stroke and the other on epilepsy, both prevalent topics that need to be mastered by general practitioners. These topics belong to the authors’ area of specialization (neurology). Both materials were inserted into a VLE – Moodle (modular object-oriented dynamic learning environment) called PVANet at the Universidade Federal de Viçosa (UFV/MG, Brazil). We developed an opinion survey and two Knowledge and Diagnostic Skills (KDS) questionnaires, one on stroke and the other on epilepsy. The opinion survey contained alternative responses using numerical or Likert scales, aiming to gather the student’s subjective reactions. The KDS questions assessed not only the students’ theoretical knowledge but also, in particular, their “know-how” (diagnosis), as well as their attitudes and practices. The KDS and opinion survey scales were constructed based on Bunchaft & Cavas²⁶. The final stroke questionnaire consisted of ten open-ended questions, while the epilepsy questionnaire contained seven open-ended and three multiple-choice questions. Half of the questions were devised by an experienced neurologist, rather than a researcher. The open-ended questions (e.g., “What is the diagnosis?”), required objective answers.

Ethics approval and consent to participate

This research work was approved by the Research Ethics Committee at Unicamp and by the UFV (Ruling no. 468/2010, CAAE 0358.0.146.000-10).The participants signed a consent form.

Application and pre-experimental results

Three medical doctors connected to the university (a general practitioner, an internist, and a neurologist) initially evaluated the two MC/DW materials and the KDS questionnaires. The medical researcher asked experts to analyze the content, which was found to be comprehensive and cumulative, but too extensive, and with navigation difficulties (the experts therefore suggested it be reduced and divided into topics to make it easier to handle). The teaching method was considered educational, thought-provoking, and trans-disciplinary, stimulating preliminary study and highlighting the clinical method. The professors considered KDSs (evaluation methods of the proposed MC/DW) to be adequate, presenting practical and routine situations to the general practitioner. The DWs on stroke and epilepsy contained 15 and 26 questions, respectively, some with sub-items. Two 4^th year medical student groups (total 32 students) at Unicamp (Campinas/SP, Brazil) accessed the VLE and rated the MC/DW sets and the opinion survey. Twenty-nine students (90.6%) considered the MCs at least good. The DWs’ feedbacks, with post-references, were found to be sufficient to clear up the doubts of 26 students (81.2%). All students (100%) reported that the feedback complemented learning and prompted a search for new information, providing a few suggestions to improve the material. After adjustments, the materials and KDS/opinion surveys were found to be suitable for application in a longitudinal observational study.

Design and participants

A longitudinal observational study involving two groups of undergraduate medical students at the UFV during 2013 (2^nd and 3^rd year students, familiar with basic science and clinical examination, were selected). The research was conducted using the curriculum course.

Interventions

We applied the KDS (KDS1) stroke questionnaire to the 2^nd year medical students and the epilepsy questionnaire to the 3^rd year medical students. We further gave a traditional lecture on stroke or epilepsy. We then invited all the students to access the VLE and to apply the MC/DW method. The students had 30 days to access the VLE, being allowed to access it as many times and for as long as they found necessary, outside the classroom. The researcher in charge controlled access to the VLEs and answered questions by mail, sending a prompt to those who had not yet entered the system. All the students were invited to respond, without previous notice, to the same KDS (KDS2) and the opinion survey, 30 days after KDS1, and the same KDS (KDS3) five to six months after KDS1. Throughout the course, there was no formal exposure to these topics between KDS1 and KDS3. The researcher attributed the scores to KDS1, 2, and 3 questionnaires using a checklist. Each KDS was worth ten points (1 point for each correct answer, with partially correct answers being accepted). We must emphasize that the KDS questions were not included in the VLE or discussed in the classroom.

Statistical analysis

We tabulated the scores of the KDS1, 2, and 3 questionnaires in Microsoft Excel 2007 as a single set (stroke KDS + epilepsy KDS) with paired data. We used the software Statistical Package for the Social Sciences – SPSS (version 20) (SPSS Inc., USA), performing an initial descriptive analysis of the sample, and obtaining the mean and standard deviation in the KDS1 of the students who completed the study. Next, we calculated the means of those who accessed and did not access the VLE, during those three instances. We conducted a descriptive analysis to characterize the samples, after the classification made by the students, and applied the Anova statistics for repeated measures, to verify whether there was a significant difference between the scores of the KDS1, KDS2 and KDS3 survey questionnaires. The assumptions for performing Anova were met (N≥30; allowing to apply the Central Limit Theorem; equal variances in the different groups; observations made by each group independent from one another; and to measure the variable under study at an interval scale). Once the differences in KDS1, 2, and 3 had been confirmed, we applied Bonferroni post hoc analysis to verify whether there was a significant difference between the scores of those who accessed and those who did not access the VLE. We analyzed the opinion survey data tabulated in Microsoft Excel 2007 using descriptive statistics tools.

RESULTS

Analysis of the KDS questionnaires

Ninety-two 2nd and 3rd year medical students were invited to participate in this 2013 study; 87 students participated, but six were excluded because they did not answer all the questionnaires. Thus, we evaluated data obtained from questionnaires KDS1, 2, 3 and opinion survey of 81 students, 41 males (50.1%), mean age of 21.9 years (19-34, Standard Deviation or SD 2.41). We obtained the mean of the 81 participants in KDS1, 1.59 points (SD 0.71). We confirmed that 66 students (81.5%) accessed the VLEs and 15 students (18.5%) did not; we sorted the responses, separated them, and obtained the means and standard deviations, as summarized in Table 1.The reasons why 15 students did not access the VLE were not verified.

TABLE 1
Mean and standard deviations of the scores of the students who accessed and who did not access the VLE

	Access to the VLEs (PVANet)	N	Mean	Std. deviation
KDS1	Yes	66	1.64	0.72
	No	15	1.35	0.62
KDS2	Yes	66	5.65	1.59
	No	15	1.76	1.20
KDS3	Yes	66	4.57	1.32
	No	15	1.57	0.90

We used Anova for repeated measures and verified a significant difference between the scores of the students obtained in the three KDS questionnaires [F (1.76; 139.33) = 72.76, p<0.01]. We found that, statistically, there was an interaction between the scores obtained from the questionnaires applied and access to VLE [F (1.76; 139.33) = 48.43, p<0.01]. We then used the Bonferroni post hoc analysis, showing no significant difference in the KDS1 grades between those who accessed and did not access the VLE (t = 1.48, p>0.05). However, the students who accessed the VLE had higher scores in KDS2 (t = 8.88, p<0.05) and in KDS3 (t = 8.39, p<0.05). These results indicate that access to the VLE had a direct influence on the KDS2 and KDS3 grades (Figure 1).

FIGURE 1
Mean of the KDS1 scores of all students and of KDS2 and 3 of those who accessed and did not access the VLE, confidence interval of 95%

Opinion questionnaire analysis

Of the 66 participants who accessed the VLE, 34 (51.5%) were female. We found that 37 students (56.1%) completed the entire activity (reading the MC and at least one primary reference, answering all the DW questions and checklist), with 62 students (94%) considering the MC at least good. We verified that 62 students (94%) checked their answers; 52 (78.8%) considered that it was sufficient to clear up all their doubts; and 57 students (86.4%) considered it necessary to hold a final collective discussion on the subject.

DISCUSSION

We aimed to verify the acceptance of our proposal by the students, as well as the efficacy of the VLE MC/DW method in improving the diagnostic ability of medical students in the long term. The vast majority (81.5%) of the students accessed the VLE MC/DW method. The MC was considered at least good by 62 students (94%); 52 (78.8%) considered the checklist sufficient to clear up all their doubts; and 57 students (86.4%) considered it necessary to hold a final collective discussion on the subject. The results of the KDS questionnaire showed that those who accessed the VLE had a mean score of 4.57 out of 10, while those who did not access it had a mean score of 1.57 out of 10, after 5-6 months of access.

These data corroborate the assumption that the MC/DW method is feasible, even if performed in isolation in subjects or modules²⁷, stimulating students to study before the activity. It was well accepted, with the high participation rate indicating that students showed great interest, although the Hawthorne effect may have played a role²⁸. The answer key sheet cleared up any doubts, providing considerable individual learning, but most of the students expressed the need for a final collective discussion, to clear up any remaining doubts and reinforce the learning. We understand this would be a very good discussion, but only after individual doubts have been cleared up, through e-mails or chat. The MC/DW method led to a statistically significant improvement in the ability to diagnose routine neurological problems in the long term. This improvement in performance may be explained by the fact that MC/DW is patient-centered method, focused on solving virtual, contextualized and interactive real cases. Feedback and repetition promote greater motivational and critical thinking, and emphasize differential diagnosis of the so-called minor symptoms (nonspecific symptoms, a real epidemic of the 21^st Century)^29,30,31. According to Kirkpatrick and Kirkpatrick³², training systems evaluation takes place at four levels: reaction (satisfaction), learning, behavioral change, and results. The evaluation of the last two stages is only possible with a consolidated course. Thus, we evaluated the reactions of students exposed to the MC/DW method through the opinion survey, and long-term learning through analysis of questionnaires KDS1, 2, and 3. The real change in behavior and outcomes in patients is rarely evaluated in undergraduate research. It is unlikely that the students sought answers for the KDS, since they had no access to the questions and answers, and were not told they would be taking tests. Our study did not aim to compare our method with other WBME methods, since our proposal is original and existing methods (WBME and face-to-face) are not mutually exclusive^33,34. A longer exposure time to any methodology can produce a positive effect³⁵, but we showed that long-term improvement in diagnostic skills was better for those who accessed the VLE, with exposure time to classes having little influence on the results.

The MC/DW method seems to help eliminate one of the major difficulties faced by undergraduate students: the use of factual knowledge for the elaboration of diagnoses and clinical decision-making^36,37, which is probably the most serious bottleneck in the health system. The use of MC/DW can foster the development of minimum competencies generated from an exponentially growing knowledge and scarce medical course time. It is still uncertain whether individual learning styles influence WBME learning^38,39. However, it seems that a strong instructional method, like MC/DW, minimizes these effects⁴⁰. Furthermore, MC helps identify medical heuristics (processes through which medical reasoning is made), especially diagnostic heuristics, thus attempting to minimize reasoning biases by the students, since not every medical decision is based on scientific data, not always available⁴¹. An updated answer-key sheet is the only task the professor must provide in the long-term to maintain the MC/DW method. We found this to be a positive point, as it contributes to continuing medical education (with one of the greatest challenges for many doctors being to find the best evidence at the appropriate time)⁴².

We recognize that some topics may seem to be better taught through regular teaching models. The MC/DW method does not replace face time learning, such as patient assessments, internships, and skills labs⁴³. We carried out an observational (not randomized) study in two separate curriculum subjects at the same college. Further multi-centric randomized studies are necessary to compare the MC/DW method with other proposals structured for WBME in VLEs to measure the actual effect size for improved diagnostic capability in the long term.

CONCLUSIONS

The MC/DW method for WBME in VLE was found to be effective in improving diagnostic skills of undergraduate medical students in the long term. It was well accepted and positively evaluated by the students, and applied to isolated subjects. Since it presents several pedagogical characteristics needed for greater VLE learning, the MC/DW method seems to be a good choice of instructional structured method by any professor or researcher who applies VLE. The description of the proposal allows its replication.

REFERENCES

• ¹
  Kahn AB. Computer education within medical education. J Med Educ 1972;47:137.
• ²
  Juanes JA, Ruisoto P. Computer Applications in Health Science Education.J Med Syst. 2015;39(9):97.
• ³
  Vasconcelos DFP, Vasconcelos ACCG. Desenvolvimento de um Ambiente Virtual de Ensino em Histologia para Estudantes da Saúde. RevBrasEducMéd 2013; 37(1): 132-137.
• ⁴
  Johnson PT, Eng J, Rowell MR, Fishman EK. Evolving physician perception of worldwide web education: 2007 update and review of the literature. AcadRadiol. 2007;14:1092–1101.
• ⁵
  Battat R, Jhonson M, Wiseblatt L, Renard C, Habib L, Normil M, Remillard B, Brewer TF, Sacajiu G. The Haiti Medical Education Project: development and analysis of a competency based continuing medical education course in Haiti through distance learning. BMC Med Educ. 2016;16(1):275.
• ⁶
  Berman AH, Biguet G, Stathakarou N, Westin-Hägglöf B, Jeding K, McGrath C, Zary N, Kononowicz AA. Virtual Patients in a Behavioral Medicine Massive Open Online Course (MOOC): A Qualitative and Quantitative Analysis of Participants’ Perceptions. Acad Psychiatry. 2017; 41(5): 631-641.
• ⁷
  Pourmand A, Raymond L, Nouraie M. Asynchronous web-based learning, a practical method to enhance teaching in emergency medicine. Telemedicine and e-Health. 2013;19(3):169–172.
• ⁸
  Hsiao CC, Tiao MM, Chen CC. Using interactive multimedia e-Books for learning blood cell morphology in pediatric hematology. BMC Med Educ. 2016;16(1): 290.
• ⁹
  Topale L. The strategic use of lecture recordings to facilitate an active and self-directed learningapproach. BMC Med Educ. 2016;16(1):201.
• ¹⁰
  Gaupp R, Körner M, FabryG. Effects of a case-based interactive e-learning course on knowledge and attitudes about patient safety: a quasi-experimental study with third-year medical students.BMC Med Educ. 2016;16:172.
• ¹¹
  Detroyer E, Dobbels F, Debonnaire D, Irving K, Teodorczuk A, Fick DM, Joosten E, MilisenK. The effect of an interactive delirium e-learning tool on healthcare workers’ delirium recognition, knowledge and strain in caring for delirious patients: a pilot pre-test/post-test study. BMC Med Educ.2016;16:17.
• ¹²
  Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? A systematic review of the variability of interventions. Med Educ 2010;44:765-74.
• ¹³
  Swinnerton BJ, Morris NP, Hotchkiss S, Pickering JD. The integration of an anatomy massive open online course (MOOC) into a medical anatomy curriculum. AnatSci Educ. 2017;10(1):53-67.
• ¹⁴
  Oliveira I, Reis ZS, Araújo MM, Freire CM. Cardiac auscultation simulator embedded in virtual learning environment to support medical teaching. Stud Health Technol Inform. 2015;216:976
• ¹⁵
  Cook DA, Dupras DM. A practical guide to developing effective web-based learning. J Gen Intern Med. 2004;19(6):698–70
• ¹⁶
  Cook DA, Levinson AJ, Garside S, Dupras DM, Erwin PJ, Montori VM. Instructional design variations in Internet-based learning for health professions education: A systematic review and meta-analysis. AcadMed2010;85:909-22.
• ¹⁷
  Sandars J, Schroter S. Web 2.0 technologies for undergraduate and postgraduate medical education: an online survey. Postgrad Med J. 2007;83:759–762.
• ¹⁸
  Beard L, Wilson K, Morra D, Keelan J. A survey of health-related activities on second life. J Med Internet Res. 2009;11(2):e17.
• ¹⁹
  Rasmussen A, Lewis M, White J. The application of wiki technology in medical education. Med Teach. 2013;35(2):109–114.
• ²⁰
  Magalhães LVB, Fernandes PT, Magalhães DSF, Bastos RR, Li LM. A Brazilian original pedagogical approach to the teaching of neurology. ArqNeuropsiquiatr 2014;72(10):747-52.
• ²¹
  Whitman N. A review of constructivism. Understanding and using a relatively new theory. Fam Med. 1993;25:517–521.
• ²²
  Peterson MC, Holbrook JH, Hales D, Smith NL, Staker LV. Contributions of the history, physical examination, and laboratory investigation in making medical diagnoses. West J Med 1992;156(2):163-5.
• ²³
  Mamede S, van Gog T, Moura AS, Faria RMD, Peixoto JM, Schmidt HG. How can students’ diagnostic competence benefit most from practice with clinical cases? The effects of structured reflection on future diagnosis of the same and novel diseases. Acad Med. 2014;89:121-127
• ²⁴
  Charon R. Narrative medicine: A model for empathy, reflection, profession, and trust. JAMA 2001;286(15):1897-902.
• ²⁵
  Flanagam JC. The critical incident technique. Psychol Bull 1954;51(4):327-58.
• ²⁶
  Bunchaft, G. & Cavas, C. S. T. (2002). Sob medida: um guia sobre a elaboração de medidas do comportamento e suas aplicações. São Paulo, Vetor.
• ²⁷
  Lee TY, Fang YL. The effectiveness of an e-learning program on pediatric medication safety for undergraduate students: A pretest-post-test intervention study. Nurse Education Today. 2013;33: 378-383.
• ²⁸
  Adair JG. The hawthorne effect: A reconsideration of the methodological artifact. Journal of Applied Psychology. 1984;69(2):334–345.
• ²⁹
  Ochoa JG, Wludyka P. Randomized comparison between traditional and traditional plus interactive web-based methods for teaching seizure disorders. Teaching and Learning in Medicine. 2008;20(2):114–117.
• ³⁰
  Subramanian A, Timberlake M, Mittakanti H et al. Novel educational approach for medical students: Improved retention rates using interactive medical software compared with traditional lecture-based format. J Surg Educ. 2012;69(4):449–452.
• ³¹
  Benseñor IJM. Minor symptoms: the illness of the 21^st century. São Paulo Med J. 2001;119(2):46–7.
• ³²
  Kirkpatrick DL, Kirkpatrick JD. Evaluating training programs: the four levels. 3rd ed. San Francisco: Berret-Koehler Publisher, Inc; 2006.
• ³³
  Rowe M, Frantz J, Bozalek V. The role of blended learning in the clinical education of healthcare students: A systematic review. Med Teach. 2012;34:e216–e221.
• ³⁴
  Atan H, Sulaiman F, Idrus RM. The effectiveness of problem-based learning in the web-based environment for the delivery of an undergraduate physics course. International Education Journal.2005;6(4):430–437.
• ³⁵
  Cook DA, Hamstra SJ, Brydges R, Zendejas B, Szostek JH, Wang AT, Erwin PJ, Hatala R. Comparative effectiveness of instructional design features in simulation-based education: Systematic review and meta-analysis. Med Teach. 2013;35:e867–e898.
• ³⁶
  Buytaert W, Baez S, Bustamente M, Dewulf A. Web-based environmental simulation: bridging the gap between scientific modeling and decision-making. Environ Sci Technol. 2012;46:1971–1976.
• ³⁷
  McGregor CA, Paton C, Thomson C, Chandratilake M, Scott H. Preparing medical students for clinical decision-making: A pilot study exploring how students make decisions and the perceived impact of a clinical decision-making teaching intervention. Med Teach. 2012;34:e508–e517.
• ³⁸
  Cook DA. Learning and cognitive styles in web-based learning: Theory, evidence, and application. Acad Med. 2005;80:266–278
• ³⁹
  Taylor DCM, Hamdy H. Adult learning theories: Implications for learning and teaching in medical education: AMEE Guide No. 83. Med Teach. 2013;35:1561–1572.
• ⁴⁰
  Cook DA. Revisiting Cognitive and Learning Styles in Computer-Assisted Instruction: Not so useful after all. Acad Med. 2012;87:778–784.
• ⁴¹
  Kappesser J, de C Williams AC. Clinical Judgement heuristics: Methods and models. Eur J Pain. 2013;17(10):1423–4.
• ⁴²
  Barnes BE. Creating the practice-learning environment: using information technology to support a new model of continuing medical education. Acad Med. 1998;73(3):278-81.
• ⁴³
  Smith SR, Cookson J, McKendree J, Harden RM. Patient-centred learning – back to the future. Med Teach. 2007;29 (1): 33–37

Publication Dates

• Publication in this collection
  Jan-Mar 2019

History

• Received
  9 Feb 2018
• Accepted
  9 Nov 2018