Acessibilidade / Reportar erro

The development and evaluation of a distance learning system in ophthalmology

A elaboração e avaliação de um sistema de ensino a distância em oftalmologia

Abstracts

INTRODUCTION: Web-based e-learning is a teaching tool increasingly used in many medical schools and specialist fields, including ophthalmology. AIMS: this pilot study aimed to develop internet-based course-based clinical cases and to evaluate the effectiveness of this method within a graduate medical education group. METHODS: this was an interventional randomized study. First, a website was built using a distance learning platform. Sixteen first-year ophthalmology residents were then divided into two randomized groups: one experimental group, which was submitted to the intervention (use of the e-learning site) and another control group, which was not submitted to the intervention. The students answered a printed clinical case and their scores were compared. RESULTS: there was no statistically significant difference between the groups. CONCLUSION: We were able to successfully develop the e-learning site and the respective clinical cases. Despite the fact that there was no statistically significant difference between the access and the non access group, the study was a pioneer in our department, since a clinical case online program had never previously been developed.

Internet; Distance learning; Computer simulation; Strabismus


INTRODUÇÃO: O ensino a distância pela Internet é uma ferramenta de educação cada vez mais utilizada em muitas faculdades de medicina e áreas de medicina especializadas, inclusive a oftalmologia. OBJETIVOS: Este estudo piloto teve como objetivo elaborar casos clínicos baseados no curso online e avaliar a eficácia deste método num grupo de estudantes de pós-graduação de medicina. MÉTODOS: foi um estudo de intervenção randomizado. Em primeiro lugar, um website foi construído usando uma plataforma de ensino à distância. Dezesseis residentes de oftalmologia do primeiro ano então foram divididos em dois grupos aleatórios: um grupo experimental, o qual foi submetido à intervenção (uso do site de ensino a distância) e outro grupo de controle, que não foi submetido à intervenção. Os alunos responderam a um caso clínico impresso e seus resultados foram comparados. RESULTADOS: não houve diferença estatisticamente significativa entre os grupos. CONCLUSÃO: Conseguimos desenvolver com sucesso o website de ensino a distância e os respectivos casos clínicos. Apesar do fato de que não houve diferença estatisticamente significativa entre o grupo com acesso e o grupo sem acesso, o estudo foi pioneiro no nosso departamento, uma vez que nunca havia sido desenvolvido anteriormente um programa online de casos clínicos.

Internet; Educação a Distância; Simulação por computador; Estrabismo


RESEARCH

The development and evaluation of a distance learning system in ophthalmology

A elaboração e avaliação de um sistema de ensino a distância em oftalmologia

Josie Naomi IyeyasuI; Stella Maris da Costa e CastroI; Renato Marcos Endrizzi SabbatiniII; Keila Monteiro de CarvalhoI

IUniversidade Federal do Paraná, Curitiba, PR, Brasil

IIUniversidade Estadual de Campinas, Campinas, SP, Brasil

Correspondence CORRESPONDENCE ADDRESS Rua Azarias de Melo, 377 apto 52 Taquaral – Campinas CEP: 13076-008 – SP Email: jobatatinha@yahoo.com.br

RESUMO

INTRODUÇÃO: O ensino a distância pela Internet é uma ferramenta de educação cada vez mais utilizada em muitas faculdades de medicina e áreas de medicina especializadas, inclusive a oftalmologia.

OBJETIVOS: Este estudo piloto teve como objetivo elaborar casos clínicos baseados no curso online e avaliar a eficácia deste método num grupo de estudantes de pós-graduação de medicina.

MÉTODOS: foi um estudo de intervenção randomizado. Em primeiro lugar, um website foi construído usando uma plataforma de ensino à distância. Dezesseis residentes de oftalmologia do primeiro ano então foram divididos em dois grupos aleatórios: um grupo experimental, o qual foi submetido à intervenção (uso do site de ensino a distância) e outro grupo de controle, que não foi submetido à intervenção. Os alunos responderam a um caso clínico impresso e seus resultados foram comparados.

RESULTADOS: não houve diferença estatisticamente significativa entre os grupos.

CONCLUSÃO: Conseguimos desenvolver com sucesso o website de ensino a distância e os respectivos casos clínicos. Apesar do fato de que não houve diferença estatisticamente significativa entre o grupo com acesso e o grupo sem acesso, o estudo foi pioneiro no nosso departamento, uma vez que nunca havia sido desenvolvido anteriormente um programa online de casos clínicos.

Palavras-Chave: Internet; Educação a Distância; Simulação por computador; Estrabismo.

ABSTRACT

INTRODUCTION: Web-based e-learning is a teaching tool increasingly used in many medical schools and specialist fields, including ophthalmology.

AIMS: this pilot study aimed to develop internet-based course-based clinical cases and to evaluate the effectiveness of this method within a graduate medical education group.

METHODS: this was an interventional randomized study. First, a website was built using a distance learning platform. Sixteen first-year ophthalmology residents were then divided into two randomized groups: one experimental group, which was submitted to the intervention (use of the e-learning site) and another control group, which was not submitted to the intervention. The students answered a printed clinical case and their scores were compared.

RESULTS: there was no statistically significant difference between the groups.

CONCLUSION: We were able to successfully develop the e-learning site and the respective clinical cases. Despite the fact that there was no statistically significant difference between the access and the non access group, the study was a pioneer in our department, since a clinical case online program had never previously been developed.

Key words: Internet; Distance learning; Computer simulation; Strabismus.

INTRODUCTION

Computer and web-based learning have been widely used in medical education. Examples include modern imaging software(1), surgical simulators(2-4), the use of telemedicine(5-8), and clinical simulations, etc. The advantages of this kind of method are - among many others - agility(9), updated information(10), the possibility of multiple access(11), the possibility of making mistakes(12), uniformity in learning(13, 14), and freedom at work(14).

Several studies have already been undertaken on the subject, even including comparisons between pre- and post-scores for students taking web-courses and comparisons between traditional learning methods and web-based learning methods. On the role of clinical simulation specifically, however, there are still few studies regarding the effect of web-based simulations as an adjunct tool in residency courses(13).

We therefore created a pilot study which aimed to develop an internet-based course using clinical cases' presentation scenarios, geared toward residents at the Ophthalmo-Otolaryngology Department of the State University of Campinas and to describe our experience in evaluating the effectiveness of this method in this graduate medical student group.

METHODS

The study was interventional and randomized. The project was approved by the Institutional Research Ethics Committee.

The methodology employed in the study comprised three steps: clinical case collection, the construction of the website, and the test application.

Case Collection

15 cases were selected according to their relevance for basic residency learning among all of the cases discussed by our staff and several guests, including ophthalmologists and orthoptists. Patients were invited to participate after receiving an explanation of the study, and all of them or their legally-responsible guardians signed the informed consent term.

The records (history, ophthalmological examination, complementary examination and photos) for the selected cases were then transcribed to a text processor and saved in .pdf format. Only the case discussions were collected in a separate file in order to keep them hidden from the users.

Site Construction

The site was built using the Moodle (Modular Object-Oriented Dynamic Learning Environment)(15), which is free software developed for producing modular internet-based courses. The course was developed using the Moodle platform from the Edumed Institute and is available at the following website: http://www.ead.edumed.org.br.

The opening page contains information on the study, user instructions, an index of all cases and the final questionnaire. All of the 15 cases follow a standard template: first, the .pdf file containing the case's records (initials, sex, age, anamnesis, ophthalmological examination and photos) is presented, and then the user is asked three questions (on semiology, diagnostic hypothesis and conduct). Their answers were stored in the site's own database.

There was also a satisfaction questionnaire available from the site containing 13 multiple choice questions regarding its usability and the residents' satisfaction with the course. It was based on the ACSI (American Customer Satisfaction Index)(16) questionnaire, which was created as a tool for customer satisfaction surveys and has been used by the NIH (National Institutes of Health) for evaluation of their websites(17). The scores were also stored in the site database.

Test Application

The participants were all of the 16 first-year residents at the Ophthalmo-Otolaryngology Department of the Faculty of Medical Sciences of the State University of Campinas. Since this was an educational study, and students' participation in the residency program is mandatory, no informed consent was required.

At the end of the first year of residency, students received a pressed clinical case for resolution (called cc00), which was distinct from all of those on the site. The case contained three questions: one on semiology (q1), one on diagnostic hypothesis (q2) and one on patient management (q3).

They were then divided into two randomized groups. One group, formed by eight residents, was invited to access the site, while the other group, also composed of eight residents, had no access to it. In order to access the site, the residents first had to register a username and a password. Once this was complete, they were able to freely access the site, except for the discussion of the cases, which remained unavailable.

When all members of the access group had finished their work, the discussions, which included criticisms of semiology, diagnostic hypothesis and conclusions on each case, were published on the site so that the users could see and compare their answers as a form of self-assessment.

All 16 residents then received another pressed clinical case for resolution (called cc01), which was also distinct from all of those on the site and from the first one, with the same three questions on semiology, diagnostic hypothesis and patient management.

Both the first and the last pressed clinical cases were corrected by two assessors who did not know which group each resident belonged to.

Statistical analysis was divided into two parts: one regarding the results of the site questionnaire and one regarding the notes of the pressed cases (evaluation of the method itself).

The results of the site questionnaire were taken from the operational system itself and were performed using the program Epi Info Version 3.5.1 for Windows. A descriptive analysis was made for all questions.

The evaluation of the method itself was divided into three parts: first, means of the scores for the clinical cases before (cc00) and after (cc01) site access were compared for the access group and the non-access group; second, means of the scores for each question included in the clinical cases (q1, q2 and q3) before (cc00) and after (cc01) site access were compared on an individual basis in the access group and the non-access group; and finally, the average scores for the clinical cases before and after site access were compared with the scores for the compulsory regular written test the residents are submitted to at the end of the first year of residency.

A statistical comparative analysis between the groups was made using the non-parametric Mann-Whitney test. A comparison between groups and time was made using the non-parametric ANOVA for repeated measures with matched groups. A non-parametric analysis was used due to the small sample size of each group. The software used was the SAS System (Statistical Analysis System) Version 9.2 for Windows.

A P-value less than 0.05 was considered statistically significant for all analyses.

RESULTS

All 16 invited residents (100%) answered the printed clinical cases.

All eight residents (100%) from the access group answered all of the questions available on the site.

Only 6 residents (75%) answered the site questionnaire, despite multiple emails and presence reminders.

The results are divided into two parts: the site questionnaire result and the evaluation of the effectiveness of the method.

Site questionnaire result

Each question in the questionnaire asked the residents to attribute scores from 0 (terrible) to 10 (excellent). The results are summarized in Table 1.

The scores obtained in the site evaluation questionnaire were generally high. The course loading speed was the item with the lowest mean, median and minimum score; whereas the quality and relevance of the course content was the item with the highest mean, median and minimum score.

Evaluation of the method's effectiveness

First, the scores for the participants of each group were analyzed within their own group in order to check whether they were homogeneous. Once this was confirmed for all analyses (p=0.8567 for the global scores, p=0.4968 for Question 1 scores, p=0.7386 for Question 2 scores, p=0.8407 for Question 3 scores and p= 0.7152 for the written test scores), the scores for the access group (Group 1) and the non-access group (Group 2) were compared. The results are summarized in Tables 2 to 6.

Table 2

For all analyses, Group 1 corresponds to the site access group, Group 2 to the non-access group, Note 1 corresponds to the average scores before the site access, Note 2 corresponds to the average scores after site access and Note 3 corresponds to the average scores for the written test.

As it may be observed, there was no statistically significant difference between the groups for both global scores (p=0.9677) and isolated question scores (p=0.6814 for Question 1, p=0.4970 for Question 2 and p=0.5309 for Question 3.

The written regular test scores were higher than the written clinical case scores for both groups (p= 0.0092 in comparison with the scores before access, and p= 0.0021 in comparison with scores after access for the experimental group).

Thus, in short, there was no statistically significant difference between the scores before and after site access (p= 0.8277). However, there was statistically significant difference for both comparisons between the scores before and after the site access with the scores of the written test (p= 0.0092 and p= 0.0021, respectively). The analysis showed that the residents achieved better scores in the written test than in the clinical print cases.

DISCUSSION

Our site questionnaire revealed good scores, which means that the residents generally approved of the site as well as the use of this sort of tool in medical education. This result is in accordance with many previous studies undertaken on e-learning, in which questionnaires also revealed that participants were satisfied with this kind of resource(13, 18), it was considered effective(19) and fun(20) and that they would like to participate in more online courses(21).

In terms of the validity of the test itself, several studies have been performed in other countries. In all of them, an internet-based course was developed and pre- and post- test scores for the participants (who were either medical students or residents) between intervention (access site) and non-intervention (non-access site) groups were compared. Some of them found statistical differences between the groups(13), with better post-scores in the intervention group; while others, including ours, did not encounter such a difference(19). The authors attribute this result to the small number of participants in the study, and this may also have affected our results.

In some studies in which an online clinical simulation course was performed, a comparison of the written tests between a site access group and a non-access group showed that the first group achieved better grades than the second one(20, 22), which means that the access group improved their knowledge. Perhaps this did not occur in our study because of the small number of participants. However, a possible bias cited by the authors is that the tests were performed at different moments: the access group took the test soon after the course, while the other group did it at the end of the month for the ambulatory stage(22). This did not occur in our study, since all of the participants did the post-test together. The fact that the residents' participation was compulsory could also be considered as another bias(22), and this may also have been applied to our study. The problem is that if we had let the residents voluntarily join the course, our sample size would have been even smaller and another bias would have been created, since only those in favor of online courses would have joined the study.

A University Telemedicine Net was created in Brazil in 2006, as a means of allowing access to medical schools and university hospitals that develop projects in the telemedicine area for the National Teaching and Research Net communication system. This resource provides remote communication among national research centers, facilitating the exchange of knowledge, qualifying courses, diagnosis and treatment discussions, teleconferences, etc. resulting in a better service for poorer populations.

The Federal Government and the Ministry of Health also created in 2007 a national program known as the Brazil Telehealth Program(23), whose objectives are to develop a means of supporting healthcare and providing continual education on family health. The methods used by the program are the utilization of modern information technologies, such as telecommunications (virtual libraries, videoconferences, video streaming and chats) for the continuous distance education of the professionals involved, and development of a consultancy system among specialists and information and communication technologies. A pilot study has been implemented in nine Brazilian states, involving approximately 2,700 health teams and 11 million inhabitants(24).

In a country covering an area as large as Brazil, these kinds of tools are very helpful, since they offer homogeneous knowledge to everyone regardless of distance from technology poles. The present study may therefore be very useful in the Telehealth context.

Based on these premises, we are able to say that the main limitations of the study were the small sample size and the presence of a confounding factor, since despite having the same standardized curriculum, we know that knowledge may vary from one person to another due to individual factors such as the university where the graduate studies were performed, previous knowledge on the subject, time spent studying at home and personal interest. It is practically impossible to control this confounding factor, which may even be considered inherent to any study performed on residents and involving individual knowledge. Perhaps it was due to this confounding factor that the hypothesis that the site would affect residents' knowledge could not be confirmed. Finally, mandatory resident participation may have been a limitation. If residents had volunteered to participate, they may not have expressed resistance at having to complete all the tasks. However, if we had allowed them to volunteer, our sample size would have been even smaller, in addition to the fact that there would have been another bias, since only those who favor e-learning would have joined the course.

CONCLUSION

We were able to successfully develop an e-learning site based on clinical cases. In terms of student evaluation results, the method was approved by most participants who had access to the course. Despite the fact that there was no statistically significant difference between the access and the non-access groups, this pilot study was a pioneer in our department, since an online clinical case program had never been previously developed. Therefore, this study was able to demonstrate itself to be at least equivalent to the e-learning method in terms of the traditional method in the participating group, with the added advantage that the students may access the course wherever and whenever they wish.

This research may encourage future studies in our department, including a larger number of participants (the calculation of the theoretical sample space for a definitive conclusion was found to be 80 participants), both at our hospital and other medical ophthalmological centers in Brazil.

ACKNOWLEDGEMENTS

I wish to acknowledge the help and encouragement received from Dr. Nilza Minguini, Dr. José Mário de Andrade, Dr. Fábio Bosso, Maria Stella Simões, Vera Lúcia Pereira and I would especially like to thank all of the residents and patients who participated in the study.

Recebido em: 27/06/2011

Reencaminhado em: 23/11/2011

Reencaminhado em: 09/09/2012

Aprovado em: 26/02/2013

CONFLICTS OF INTEREST

The authors confirm that no financial support or financial conflicts of interest affected the study.

AUTHORS' CONTRIBUTION

Stella Maris da Costa e Castro: coleta de dados e revisão do manuscrito Renato Marcos Endrizzi Sabbatini: revisão do manuscrito Keila Monteiro de Carvalho: revisão do manuscrito e coordenação da equipe

  • Kuzmak PM, Dayhoff RE. The use of digital imaging and communications in medicine (DICOM) in the integration of imaging into the electronic patient record at the Department of Veterans Affairs. J Digit Imaging 2000 May;13(2 Suppl 1):133-7.
  • Henderson BA, Neaman A, Kim BH, Loewenstein J. Virtual training tool. Ophthalmology 2006 Jun;113(6):1058-9.
  • Koch F, Koss MJ, Singh P, Naser H. [Virtual reality in ophthalmology]. Klin Monbl Augenheilkd 2009 Aug;226(8):672-6.
  • Feudner EM, Engel C, Neuhann IM, Petermeier K, Bartz-Schmidt KU, Szurman P. Virtual reality training improves wet-lab performance of capsulorhexis: results of a randomized, controlled study. Graefes Arch Clin Exp Ophthalmol 2009 Jul;247(7):955-63.
  • Helveston EM, Orge FH, Naranjo R, Hernandez L. Telemedicine: Strabismus e-consultation. J AAPOS 2001 Oct;5(5):291-6.
  • Peter J, Piantadosi J, Piantadosi C, Cooper P, Gehling N, Kaufmann C, et al. Use of real-time telemedicine in the detection of diabetic macular oedema: a pilot study. Clin Experiment Ophthalmol 2006 May-Jun;34(4):312-6.
  • Wei JC, Valentino DJ, Bell DS, Baker RS. A Web-based telemedicine system for diabetic retinopathy screening using digital fundus photography. Telemed J E Health 2006 Feb;12(1):50-7.
  • Yogesan K, Constable IJ, Morgan W, Soebadi DY. International transmission of tele-ophthalmology images. J Telemed Telecare 2000;6(1):41-4.
  • Lee AG, Boldt HC, Golnik KC, Arnold AC, Oetting TA, Beaver HA, et al. Using the Journal Club to teach and assess competence in practice-based learning and improvement: a literature review and recommendation for implementation. Surv Ophthalmol 2005 Nov-Dec;50(6):542-8.
  • Hugenholtz NI, de Croon EM, Smits PB, van Dijk FJ, Nieuwenhuijsen K. Effectiveness of e-learning in continuing medical education for occupational physicians. Occup Med (Lond) 2008 Aug;58(5):370-2.
  • Kulier R, Hadley J, Weinbrenner S, Meyerrose B, Decsi T, Horvath AR, et al. Harmonising evidence-based medicine teaching: a study of the outcomes of e-learning in five European countries. BMC Med Educ 2008;8:27.
  • Alverson DC, Saiki SM, Jr., Jacobs J, Saland L, Keep MF, Norenberg J, et al. Distributed interactive virtual environments for collaborative experiential learning and training independent of distance over Internet2. Stud Health Technol Inform 2004;98:7-12.
  • Wong RW, Lochnan HA. A web-based simulation of a longitudinal clinic used in a 4-week ambulatory rotation: a cohort study. BMC Med Educ   2009;9:8.
  • Rendas A, Pinto PR, Gamboa T. A computer simulation designed for problem-based learning. Med Educ 1999 Jan;33(1):47-54.
  • Moodle. Available from: http://moodle.org/
    » link
  • The American Customer Satisfaction Index. Available from: http://www.theacsi.org/
    » link
  • Wood FB, Siegel ER, Feldman S, Love CB, Rodrigues D, Malamud M, et al. Web evaluation at the US National Institutes of Health: use of the American Customer Satisfaction Index online customer survey. J Med Internet Res 2008;10(1):e4.
  • Meier AH, Henry J, Marine R, Murray WB. Implementation of a Web- and simulation-based curriculum to ease the transition from medical school to surgical internship. Am J Surg 2005 Jul;190(1):137-40.
  • Sakowski HA, Rich EC, Turner PD. Web-based case simulations for a primary care clerkship. Acad Med 2001 May;76(5):547.
  • Stahl A, Boeker M, Ehlken C, Agostini H, Reinhard T. [Evaluation of an internet-based e-learning ophthalmology module for medical students]. Ophthalmologe 2009 Nov;106(11):999-1005.
  • Godinho RN, Kamil S, Keogh I, Sih T, Eavey R. The Internet-current trends and future expectations among Brazilian pediatricians and otolaryngologists with a special interest in pediatric otolaryngology. Int J Pediatr Otorhinolaryngol 2003 Jul;67(7):729-32.
  • Westmoreland GR, Counsell SR, Tu W, Wu J, Litzelman DK. Web-based training in geriatrics for medical residents: a randomized controlled trial using standardized patients to assess outcomes. J Am Geriatr Soc 2010 Jun;58(6):1163-9.
  • Saúde Md. Programa Telessaúde Brasil. 2007; Available from: http://www.telessaudebrasil.org.br/php/level.php?lang=pt&component=42&item=1
  • Rezende EJ, Melo Mdo C, Tavares EC, Santos Ade F, Souza C. [Ethics and eHealth: reflections for a safe practice]. Rev Panam Salud Publica Jul;28(1):58-65.
  • CORRESPONDENCE ADDRESS
    Rua Azarias de Melo, 377 apto 52
    Taquaral – Campinas
    CEP: 13076-008 – SP
    Email:
  • Publication Dates

    • Publication in this collection
      18 June 2013
    • Date of issue
      Mar 2013

    History

    • Received
      27 June 2011
    • Accepted
      26 Feb 2013
    • Reviewed
      09 Sept 2012
    Associação Brasileira de Educação Médica SCN - QD 02 - BL D - Torre A - Salas 1021 e 1023 | Asa Norte, Brasília | DF | CEP: 70712-903, Tel: (61) 3024-9978 / 3024-8013, Fax: +55 21 2260-6662 - Brasília - DF - Brazil
    E-mail: rbem.abem@gmail.com