Satisfaction of the ophthalmology residents with online progress test

Satisfação dos residentes de oftalmologia com testes de progresso online

ABSTRACT

Objective

To investigate if there is knowledge improvement over the Residency years, as well as students’ satisfaction about progress test. Finally, to describe the implementation of the online progress test in some Ophthalmological Residency programs in Brazil.

Methods

It was an observational cross-section study. The participants were all Ophthalmology residents who accepted to join the study. They did an online test and answered demographic and satisfaction questionnaires.

Results

Two hundred and fifty-nine residents joined the study. 42,86% of the residents answered the demographic questionnaire and 6,95% answered the satisfaction questionnaire. In general, the residents approved the website as a tool for self-learning and to improve residency programs.

Conclusion

Over the years, the residents acquired knowledge during the residency program. In general, residents were satisfied with the test. The implementation of the online progress test system in the Ophthalmological Residency schools in Brazil was successful.

Internship and residency; Educational measurement; Clinical competence; Education, medical; Education, distance; Ophthalmology; Personal satisfaction; Surveys and questionnaires

RESUMO

Objetivo

Avaliar se ocorre melhora do desempenho na prova dos residentes ao longo dos anos de curso e o grau de satisfação dos residentes com o teste de progresso. Além disso, descrever a implementação do teste de progresso em alguns serviços de residência em oftalmologia no Brasil.

Métodos

Estudo observacional prospectivo. Os participantes foram todos os residentes de Oftalmologia que aceitaram participar do estudo. Eles fizeram um teste online e responderam a um questionário pessoal e um de satisfação.

Resultados

Duzentos e cinquenta e nove residentes participaram do estudo. 42,86% dos residentes responderam o questionário pessoal e 6,95% responderam o questionário de satisfação. Em geral, os residentes aprovaram o site como instrumento de autoaprendizado e como instrumento para melhoria dos programas de residência.

Conclusão

Houve aquisição de conhecimento dos residentes ao longo do curso. De um modo geral, os residentes ficaram satisfeitos com o teste. A implementação do teste de progresso nos serviços de residência de Oftalmologia foi bem-sucedida.

Internato e residência; Avaliação educacional; Competência clínica; Educação médica; Educação a distância; Oftalmologia; Satisfação pessoal; Inquéritos e questionários

INTRODUCTION

Online assessment has been widely used to assess learners’ knowledge across different fields including medical education. The main advantages of online assessment are the automated marking and instant feedback to students,¹ when compared to paper and pencil assessment. Most online assessment has the design of modular test in which each cohort of residents take a different test. However, this type of assessment has been criticized because they promote short-term memorization and are one-point measurement.²

Therefore, to benefit students’ long-term retention, longitudinal testing has been suggested. The progress testing (PT) is the most known and widely used to assess knowledge growth. The PT allows to assess students’ knowledge growth during medical training. In the PT, all learners simultaneously took the same test, following the same blueprint, irrespective of their year of training. The test is repeatedly taken during the course and it is based on a specific blueprint.³

The advantages of the PT over the traditional methods of knowledge evaluation, such as the modular, intermediate or final tests, are that it forces students to study continuously³and allow to measure growth of knowledge over time.⁴ Progress tests may be organized at a national and international level, which allows us to compare the results of students from different schools² and identify students at risk of failing (in qualifying examinations, for example). With this information, possible interventions during the course, like remediation, can be made. Previous studies have already showed students’ satisfaction with the PT, considering it a tool that supports knowledge improvement and offers them academic support as future physicians.⁵

The PT is already well-stablished in the undergraduate context. However, there is a lack of studies in the graduate context,⁶ with few Residency programs using PT showing promising results. An online test in some Ophthalmology Services in Brazil was introduced following the principles of progress test.³

This study aims to investigate if there is a difference between residents’ knowledge in the different years of residency training and students’ satisfaction on this new method of assessment.

METHODS

This research was an observational cross-sectional study conducted on an online platform.

This study was approved by the ethics committee of the Universidade Estadual de Campinas (Unicamp) on December 17, 2018 (CAAE number: 02613718.9.0000.5404).

Data were collected in 2020 and participants were ophthalmology residents. The participation was voluntary, and all participants signed a consent form.

The blueprint and test application were similar to the previous study,⁸ but we used a new set of 125 questions using the Moodle® platform regarding clinical and surgical issues in ophthalmology.

Before the test, residents were asked to fill in a general questionnaire (year of residency, sex, age, previous ophthalmology residency elsewhere and if they were taking preparatory courses for the Ophthalmology Title Test). After the test, they answered a satisfaction questionnaire. Fields for open comments and suggestions were also provided.

Data analysis

Each correct answer received a value of 0.08 value (which corresponds to 10/125), attributing a total score varying from zero to ten.

Descriptive analysis using frequency tables for categorical variables and positions and dispersion measures for numeric variables were used. To compare the difference between years, we conducted the Kruskal-Wallis test, followed by the Dunn test. To compare the participants’ knowledge growth, the Friedman test or the Wilcoxon test was conducted.

For the satisfaction questionnaire, we used the mean scores for each question.

Statistical significance level of 0.05 was used.

Data were analyzed using the Statistical Analysis System (SAS) for Windows®, version 9.4 (SAS Institute Inc, 2002-2012, Cary, NC, USA).

RESULTS

In total, 259 residents from 13 services around Brazil joined the study: 47 (18.1%) were in the first year of residency, 104 (40.2%) were in the second year, and 108 (41.7%) were in the third year. Of those, only six took the test in 2018, 2019, and 2020.

In total, 111 (42.86%) residents answered the demographic questionnaire. From those, 58 were women and 53 were men, with a mean age of 31. Nine residents had already taken residency in ophthalmology elsewhere, and 43 were taking preparatory courses for the Ophthalmology Title test.

We had also applied the test in 2018, 2019 and 2021, but the number of residents who took the tests for the 3 years was too small (only six). Therefore, unfortunately, it was not possible to make an individual progress analysis.

Knowledge growth

The mean score ranged from 5.0 to 6.4 (Table 1). We found a significant difference between the first and second years and between the first and third years of residency (p-value<0.0001), but no difference between the second and the third years (p-value =0,0619).

Table 1
Descriptive analysis of the progress test scores for each year

Residency year	n	Mean	Standard deviation	Minimum	Median	Maximum
1	47	5.0	1.3	2.7	4.8	7.3
2	52	6.4	1.3	3.6	6.5	9.0
3	54	6.5	1.3	4.2	6.2	9.2

Satisfaction questionnaire

Only 18 residents (6,95%) answered the satisfaction questionnaire. All mean scores obtained were high. The highest mean score was 10.0 for the validity of the site as a self-learning tool and for the validity of the site as a tool for the improvement of the residency programs, and the lowest mean score was 7.0 for the quality of the discussion of the questions (Table 2).

Table 2
Scores of the satisfaction questionnaire

Question	Mean score (0-10.0)
Website appearance	9.0
Quality of the questions	7.8
Quality of the discussion of the questions	7.0
General score for the website	9.5
Validity of the website as a self-learning tool	10.0
Validity of the website as a tool for the improvement of the residency programs	10.0

There were only three comments: “The test was too long”, “It was a good exercise for the brain” and “The test was too long and exhausting. It could be available longer, in order to allow people to do it at any other time”.

DISCUSSION

The study could demonstrate that online tests can be used in Ophthalmology Residency training. This study demonstrated that, by following the principles of progress test, it was possible to differentiate residents’ knowledge in different years of residency training, in line with previous studies that demonstrated the improvement of knowledge during residency training.³,⁸In general, residents showed a positive attitude towards this kind of test, although only 6.95% of the residents answered the satisfaction questionnaire.

Previous studies investigating students’ motivation on computer test found that they were stimulated and satisfied with online tests. They showed a positive attitude towards it, as in our study. The most positive aspects of online tests found were the autonomy, the faster feedback and the fact that online tests are more practically feasible.¹

Even though some of the residents considered the test too long and exhausting, we followed the guidelines for progress test and the same pattern of the specialist test. The aim was to introduce a test similar to the real specialist test and which could monitor residents’ knowledge growth. Although computerized adaptative testing could decrease the number of items, the implication process is not easy to achieve since it needs support from a psychometrician and a delivery platform.⁹

Limitations of the study

Due to the pandemic, the residents took the test at home; therefore, cheating was not controlled. Since the participation in the progress test was voluntary and not part of the official Residency program, we believe that residents had no reason to cheat. Unfortunately, the number of residents who answered the satisfaction questionnaire was very low, either because it was not compulsory, or because they were tired.

Lastly, it was not possible to make an individual analysis of the progress since it was a cross-sectional study. The data collected were just from the year 2020. We had indeed applied the test in 2018, 2019, and 2021, but unfortunately the number of residents who took the tests for the 3 years was too small (only six). Further research should investigate whether this online assessment would differentiate resident’s knowledge growth over time, as it is possible in undergraduate training.¹⁰

CONCLUSION

Progress test was able to demonstrate residents’ knowledge growth, and the residents approved the new method of assessment. The implementation of the online tests in the Ophthalmological Residency schools in Brazil was successful.

ACKNOWLEDGEMENTS

To the Statistical Sector of Faculty of Medical Sciences and the team of students and professors of the Statistics Department of the Institute of Mathematic, Statistics and Scientific Computing of the Universidade Estadual de Campinas for the data analysis

REFERENCES

• ¹
  Snekalatha S, Marzuk SM, Meshram SA, Maheswari KU, Sugapriya G, Sivasharan K. Medical students’ perception of the reliability, usefulness and feasibility of unproctored online formative assessment tests. Adv Physiol Educ. 2021;45(1):84-8.
• ²
  Verhoeven BH, Snellen-Balendong HA, Hay IT, Boon JM, van der Linde MJ, Blitz-Lindeque JJ, et al. The versatility of progress testing assessed in an international context: a start for benchmarking global standardization? Med Teach. 2005;27(6):514-20.
• ³
  Rutgers DR, van Raamt F, van Lankeren W, Ravesloot CJ, van der Gijp A, Ten Cate TJ, et al. Fourteen years of progress testing in radiology residency training: experiences from The Netherlands. Eur Radiol. 2018;28(5):2208-15.
• ⁴
  Cecilio-Fernandes D, Bicudo AM, Hamamoto Filho PT. Progress testing as a pattern of excellence for the assessment of medical students’ knowledge: concepts, history, and perspective. Med Ribeirão Preto. 2021;54(1):e173770.
• ⁵
  Sattar K, Ahmad T, Khalil MS, Soliman MM, Punnamperuma GG, Abdulghani HM. Medical students’ perception of the progress test as a quality-controlled assessment tool for improving learning and teaching, at a public sector medical college in Saudi Arabia. JPMA J Pak Med Assoc. 2017;67(2):300-4.
• ⁶
  Ravesloot C, van der Schaaf M, Haaring C, Kruitwagen C, Beek E, Ten Cate O, et al. Construct validation of progress testing to measure knowledge and visual skills in radiology. Med Teach. 2012;34(12):1047-55.
• ⁷
  Iyeyasu JN, Cecilio-Fernandes D, Carvalho KM. Longitudinal evaluation of the Ophthalmology residents in Brazil: an observational prospective study. Sao Paulo Med J. 2022;141(3):e202292.
• ⁸
  Dournes G, Bricault I, Chateil JF. Analysis of the French national evaluation of radiology residents. Diagn Interv Imaging. 2019;100(3):185-93.
• ⁹
  Collares CF, Cecilio-Fernandes D. When I say … computerised adaptive testing. Med Educ. 2019 Feb;53(2):115-116.
• ¹⁰
  Cecilio-Fernandes D, Aalders WS, de Vries J, Tio RA. The impact of massed and spaced-out curriculum in oncology knowledge acquisition. J Cancer Educ. 2018;33(4):922-5.

• Institution: Universidade Estadual de Campinas

• Financial support: no financial support for this work.

Publication Dates

• Publication in this collection
  14 Aug 2023
• Date of issue
  2023

History

• Received
  09 Jan 2023
• Accepted
  21 Apr 2023