The Development and Evaluation of a Distance Learning System in Ophthalmology

– Estrabismo.. RESUMO Introdução: O ensino a distância pela Internet é uma ferramenta de educação cada vez mais utilizada em muitas faculdades de medicina e áreas de medicina especializadas, inclusive a oftalmologia. Objetivos: Este estudo piloto teve como objetivo elaborar casos clínicos baseados no curso online e avaliar a eficácia deste método num grupo de estudantes de pós-graduação de medicina. Métodos: foi um estudo de intervenção randomizado. Em primeiro lugar, um website foi construído usando uma plataforma de ensino à distância. Dezesseis residentes de oftalmologia do primeiro ano então foram divididos em dois grupos aleatórios: um grupo experimental, o qual foi submetido à intervenção (uso do site de ensino a distância) e outro grupo de controle, que não foi submetido à intervenção. Os alunos responderam a um caso clínico impresso e seus resultados foram comparados. Resultados: não houve diferença estatisticamente significativa entre os grupos. Conclusão: Conseguimos desenvolver com sucesso o website de ensino a distância e os respectivos casos clínicos. Apesar do fato de que não houve diferença estatisticamente significativa entre o grupo com acesso e o grupo sem acesso, o estudo foi pioneiro no nosso departamento, uma vez que nunca havia sido desenvolvido anteriormente um programa online de casos clínicos.

Several studies have already been undertaken on the subject, even including comparisons between pre-and post-scores for students taking web-courses and comparisons between traditional learning methods and web-based learning methods. On the role of clinical simulation specifically, however, there are still few studies regarding the effect of web-based simulations as an adjunct tool in residency courses (13) .
We therefore created a pilot study which aimed to develop an internet-based course using clinical cases' presentation scenarios, geared toward residents at the Ophthalmo-Otolaryngology Department of the State University of Campinas and to describe our experience in evaluating the effectiveness of this method in this graduate medical student group.

METHODS
The study was interventional and randomized. The project was approved by the Institutional Research Ethics Committee.
The methodology employed in the study comprised three steps: clinical case collection, the construction of the website, and the test application.

Case Collection
15 cases were selected according to their relevance for basic residency learning among all of the cases discussed by our staff and several guests, including ophthalmologists and orthoptists. Patients were invited to participate after receiving an explanation of the study, and all of them or their legally--responsible guardians signed the informed consent term.
The records (history, ophthalmological examination, complementary examination and photos) for the selected cases were then transcribed to a text processor and saved in .pdf format. Only the case discussions were collected in a separate file in order to keep them hidden from the users.

Site Construction
The site was built using the Moodle (Modular Object--Oriented Dynamic Learning Environment) (15) , which is free software developed for producing modular internet-based courses. The course was developed using the Moodle platform from the Edumed Institute and is available at the following website: http://www.ead.edumed.org.br.
The opening page contains information on the study, user instructions, an index of all cases and the final questionnaire. All of the 15 cases follow a standard template: first, the .pdf file containing the case's records (initials, sex, age, anamnesis, ophthalmological examination and photos) is presented, and then the user is asked three questions (on semiology, diagnostic hypothesis and conduct). Their answers were stored in the site's own database.
There was also a satisfaction questionnaire available from the site containing 13 multiple choice questions regarding its usability and the residents' satisfaction with the course. It was based on the ACSI (American Customer Satisfaction Index) (16) questionnaire, which was created as a tool for customer satisfaction surveys and has been used by the NIH (National Institutes of Health) for evaluation of their websites (17) . The scores were also stored in the site database.

Test Application
The participants were all of the 16 first-year residents at the Ophthalmo-Otolaryngology Department of the Faculty of Medical Sciences of the State University of Campinas. Since this was an educational study, and students' participation in the residency program is mandatory, no informed consent was required.
At the end of the first year of residency, students received a pressed clinical case for resolution (called cc00), which was distinct from all of those on the site. The case contained three questions: one on semiology (q1), one on diagnostic hypothesis (q2) and one on patient management (q3).
They were then divided into two randomized groups. One group, formed by eight residents, was invited to access the site, while the other group, also composed of eight residents, had no access to it. In order to access the site, the residents first had to register a username and a password. Once this was complete, they were able to freely access the site, except for the discussion of the cases, which remained unavailable.
When all members of the access group had finished their work, the discussions, which included criticisms of semiology, diagnostic hypothesis and conclusions on each case, were published on the site so that the users could see and compare their answers as a form of self-assessment.
All 16 residents then received another pressed clinical case for resolution (called cc01), which was also distinct from all of those on the site and from the first one, with the same three questions on semiology, diagnostic hypothesis and patient management.
Both the first and the last pressed clinical cases were corrected by two assessors who did not know which group each resident belonged to.
Statistical analysis was divided into two parts: one regarding the results of the site questionnaire and one regarding the notes of the pressed cases (evaluation of the method itself).
The results of the site questionnaire were taken from the operational system itself and were performed using the program Epi Info Version 3.5.1 for Windows. A descriptive analysis was made for all questions.
The evaluation of the method itself was divided into three parts: first, means of the scores for the clinical cases before (cc00) and after (cc01) site access were compared for the access group and the non-access group; second, means of the scores for each question included in the clinical cases (q1, q2 and q3) before (cc00) and after (cc01) site access were compared on an individual basis in the access group and the non-access group; and finally, the average scores for the clinical cases before and after site access were compared with the scores for the compulsory regular written test the residents are submitted to at the end of the first year of residency.
A statistical comparative analysis between the groups was made using the non-parametric Mann-Whitney test. A comparison between groups and time was made using the non-parametric ANOVA for repeated measures with matched groups. A non-parametric analysis was used due to the small sample size of each group. The software used was the SAS System (Statistical Analysis System) Version 9.2 for Windows.
A P-value less than 0.05 was considered statistically significant for all analyses.

RESULTS
All 16 invited residents (100%) answered the printed clinical cases.
All eight residents (100%) from the access group answered all of the questions available on the site.
Only 6 residents (75%) answered the site questionnaire, despite multiple emails and presence reminders.
The results are divided into two parts: the site questionnaire result and the evaluation of the effectiveness of the method.

Site questionnaire result
Each question in the questionnaire asked the residents to attribute scores from 0 (terrible) to 10 (excellent). The results are summarized in Table 1.  The scores obtained in the site evaluation questionnaire were generally high. The course loading speed was the item with the lowest mean, median and minimum score; whereas the quality and relevance of the course content was the item with the highest mean, median and minimum score.

Evaluation of the method's effectiveness
First, the scores for the participants of each group were analyzed within their own group in order to check whether they were homogeneous. Once this was confirmed for all analyses (p=0.8567 for the global scores, p=0.4968 for Question 1 scores, p=0.7386 for Question 2 scores, p=0.8407 for Question 3 scores and p= 0.7152 for the written test scores), the scores for the access group (Group 1) and the non-access group (Group 2) were compared. The results are summarized in Tables 2 to 6.  Table 2 shows the descriptive analysis and the comparison of the global scores between the groups. Table 3 shows the descriptive analysis and the comparison of Question 1 (regarding semiology), 2 (regarding diagnostic hypothesis) and 3 (regarding patient management) scores between the groups. Table 4 shows the descriptive analysis and comparison of global scores between the groups and among situations (before the site, after site access and written test).
For all analyses, Group 1 corresponds to the site access group, Group 2 to the non-access group, Note 1 corresponds to the average scores before the site access, Note 2 corresponds to the average scores after site access and Note 3 corresponds to the average scores for the written test.   As it may be observed, there was no statistically significant difference between the groups for both global scores (p=0.9677) and isolated question scores (p=0.6814 for Question 1, p=0.4970 for Question 2 and p=0.5309 for Question 3. The written regular test scores were higher than the written clinical case scores for both groups (p= 0.0092 in comparison with the scores before access, and p= 0.0021 in comparison with scores after access for the experimental group).
Thus, in short, there was no statistically significant difference between the scores before and after site access (p= 0.8277). However, there was statistically significant difference for both comparisons between the scores before and after the site access with the scores of the written test (p= 0.0092 and p= 0.0021, respectively). The analysis showed that the residents achieved better scores in the written test than in the clinical print cases.

DISCUSSION
Our site questionnaire revealed good scores, which means that the residents generally approved of the site as well as the use of this sort of tool in medical education. This result is in 100 Josie Naomi Iyeyasu et al.
Distance Learning System accordance with many previous studies undertaken on e-learning, in which questionnaires also revealed that participants were satisfied with this kind of resource (13,18) , it was considered effective (19) and fun (20) and that they would like to participate in more online courses (21) . In terms of the validity of the test itself, several studies have been performed in other countries. In all of them, an internet-based course was developed and pre-and post-test scores for the participants (who were either medical students or residents) between intervention (access site) and non-intervention (non-access site) groups were compared. Some of them found statistical differences between the groups (13) , with better post-scores in the intervention group; while others, including ours, did not encounter such a difference (19) . The authors attribute this result to the small number of participants in the study, and this may also have affected our results.
In some studies in which an online clinical simulation course was performed, a comparison of the written tests between a site access group and a non-access group showed that the first group achieved better grades than the second one (20,22) , which means that the access group improved their knowledge. Perhaps this did not occur in our study because of the small number of participants. However, a possible bias cited by the authors is that the tests were performed at different moments: the access group took the test soon after the course, while the other group did it at the end of the month for the ambulatory stage (22) . This did not occur in our study, since all of the participants did the post-test together. The fact that the residents' participation was compulsory could also be considered as another bias (22) , and this may also have been applied to our study. The problem is that if we had let the residents voluntarily join the course, our sample size would have been even smaller and another bias would have been created, since only those in favor of online courses would have joined the study.
A University Telemedicine Net was created in Brazil in 2006, as a means of allowing access to medical schools and university hospitals that develop projects in the telemedicine area for the National Teaching and Research Net communication system. This resource provides remote communication among national research centers, facilitating the exchange of knowledge, qualifying courses, diagnosis and treatment discussions, teleconferences, etc. resulting in a better service for poorer populations.
The Federal Government and the Ministry of Health also created in 2007 a national program known as the Brazil Telehealth Program (23) , whose objectives are to develop a means of supporting healthcare and providing continual education on family health. The methods used by the program are the utilization of modern information technologies, such as telecommunications (virtual libraries, videoconferences, video streaming and chats) for the continuous distance education of the professionals involved, and development of a consultancy system among specialists and information and communication technologies. A pilot study has been implemented in nine Brazilian states, involving approximately 2,700 health teams and 11 million inhabitants (24) .
In a country covering an area as large as Brazil, these kinds of tools are very helpful, since they offer homogeneous knowledge to everyone regardless of distance from technology poles. The present study may therefore be very useful in the Telehealth context.
Based on these premises, we are able to say that the main limitations of the study were the small sample size and the presence of a confounding factor, since despite having the same standardized curriculum, we know that knowledge may vary from one person to another due to individual factors such as the university where the graduate studies were performed, previous knowledge on the subject, time spent studying at home and personal interest. It is practically impossible to control this confounding factor, which may even be considered inherent to any study performed on residents and involving individual knowledge. Perhaps it was due to this confounding factor that the hypothesis that the site would affect residents' knowledge could not be confirmed. Finally, mandatory resident participation may have been a limitation. If residents had volunteered to participate, they may not have expressed resistance at having to complete all the tasks. However, if we had allowed them to volunteer, our sample size would have been even smaller, in addition to the fact that there would have been another bias, since only those who favor e-learning would have joined the course.

CONCLUSION
We were able to successfully develop an e-learning site based on clinical cases. In terms of student evaluation results, the method was approved by most participants who had access to the course. Despite the fact that there was no statistically significant difference between the access and the non-access groups, this pilot study was a pioneer in our department, since an online clinical case program had never been previously developed. Therefore, this study was able to demonstrate itself to be at least equivalent to the e-learning method in terms of the traditional method in the participating group, with the added advantage that the students may access the course wherever and whenever they wish. This research may encourage future studies in our department, including a larger number of participants (the calculation of the theoretical sample space for a definitive conclusion was found to be 80 participants), both at our hospital and other medical ophthalmological centers in Brazil.