Time of clerkship rotations’ interruption during COVID-19 and differences on Progress Test’s scores

SUMMARY

OBJECTIVES:

The transition from face-to-face to remote teaching is yet to be fully understood. In clinical training, traditional teaching must prevail because it is essential for the acquisition of skills and professionalism. However, the responses of each school to the pandemic and the decision on when to resume clerkship rotations were mixed. In this study, we aimed to analyze whether the time to resume clerkship rotations was associated with the performance of the students by using a multi-institutional Progress Test.

METHODS:

This is a cross-sectional study conducted at nine different Brazilian medical schools that administer the same annual Progress Test for all students. We included information from 1,470 clerkship medical students and analyzed the time of clinical training interruption as the independent variable and the student’s scores as the dependent variable.

RESULTS:

The comparisons of the students’ scores between the schools showed that there are differences; however, they cannot be attributed to the time the clerkship rotations were paused. The correlation between the schools’ average scores and the time to resume clerkship rotations was not significant for the fifth year (r= -0.298, p=0.436) and for the sixth year (r= -0.440, p=0.240). By using a cubic regression model, the time to resume clerkship rotations could explain 3.4% of the 5-year students’ scores (p<0.001) and 0.9% of the 6^–year students, without statistical difference (p=0.085).

CONCLUSIONS:

The differences between the students’ scores cannot be attributed to the time when the schools paused the clerkship rotations.

KEYWORDS:
COVID-19; Knowledge assessment; Medical students

INTRODUCTION

The COVID-19 pandemic has led many medical schools to interrupt their activities to keep their students safe and minimize the spread of infection¹. For medical schools, this decision was more challenging because the students, especially those in the final years of their undergraduate training, need to be trained in real scenarios. However, many just-graduated physicians would eventually look after patients with COVID-19 without adequate training²,³.

In the beginning of the pandemic, less was known about the clinical and epidemiologic characteristics of the disease and, therefore, most of the schools and universities stopped giving classroom lectures and initiated remote teaching. Conventional lectures were replaced by synchronous or asynchronous online classes, and clinical teaching was replaced by case-based online discussions. Upon gaining some understanding about the causes and methods of prevention of the disease, some schools and universities decided to reopen, provided that social distancing and personal protective equipment were adopted⁴. The decision for reopening was not unanimous across countries and even within the same geographical region⁵. In Brazil, the lack of national guidelines for health and sanitary measures led the states and cities’ leaders to decide the most appropriate time to reopen scholar activities. Accordingly, the universities resumed their presential classes at different times⁶,⁷.

The effects of online teaching on the students’ knowledge are yet to be properly evaluated. Although several studies show that faculty members and students have adapted to the new situation, the literature is scarce on objective demonstration of equivalence between traditional and online teaching for clinical training⁸–¹⁰. In a previous study, we showed that knowledge acquisition was impaired among final-years medical students during the pandemic. A possible explanation is that even after resuming clerkship, the students had less exposure to real clinical scenarios due to the closure of outpatient clinics and reduced variety of diseases in the wards (that were allocated to COVID-19 patients) [unpublished data].

Considering that different Brazilian medical schools resumed their clerkship rotations at different times and that clinical teaching is pivotal for knowledge acquisition, we aimed to investigate whether the time to resume clerkship rotations was associated with the performance of the students on a Progress Test.

METHODS

An annual Progress Test was conducted at nine medical schools of Brazil for all enrolled students. In this cross-sectional study, we included information of the 5- and 6-year medical students (clerkship years). We analyzed the time of clinical training interruption as the independent variable and the students’ scores as the dependent variable.

Ethical considerations

Since we dealt with secondary data and no student was identified, ethics committee approval was not necessary, according to Brazilian legislation.

Settings and progress test information

The participating schools are Universidade Estadual Paulista (UNESP), Universidade Estadual de Campinas (UNICAMP), Universidade de São Paulo (USP), Universidade Federal de São Paulo (UNIFESP), Universidade Federal de São Carlos (UFSCAR), Faculdade de Medicina de Marília (FAMEMA), Faculdade de Medicina de São José do Rio Preto (FAMERP), Universidade Estadual de Londrina (UEL), and Universidade Regional de Blumenau (FURB). The undergraduate program lasts 6 years and the clerkship rotations take place in the fifth and sixth years.

These 9 schools are joint in a 16-year-old consortium for the annual administration of the Progress Test. Except for USP, all the schools stopped clerkship rotations in March 2020.

The Progress Test consists of 120 multiple-choice items equally divided into 6 content areas: basic sciences, internal medicine, pediatrics, surgery, obstetrics and gynecology, and public health. The items are clinical vignette-based aiming for applied knowledge rather than knowledge recall. In 2020, the exam was administered in November and was computerized with a safe exam browser instead of the traditional paper-based question book. The test has only formative purposes (i.e., the student’s score does not have high-stake implications for progression in the educational program)¹¹.

Data analysis

The scores of all clerkship rotations were eligible to enter in the analysis. However, tests with less than 30 correct answers (casual mark) were excluded. The data of students who did not apply for the test were also excluded.

To compare the performances between the schools, the Kruskal-Wallis test was used. It was followed by Dunn’s test for post-hoc analysis.

After verifying whether there was a difference in scores between the schools, we analyzed the correlation between the time of clinical interruption and the schools’ average scores and the students’ individual scores. For parametric and nonparametric data, the Pearson’s and Spearman’s correlation were used, respectively.

In the second step, we analyzed if a linear model could explain the relationship between the variables. The correlation between the residuals was tested with the Durbin-Watson test and acceptable values should range between 1.5 and 2.5. In the final step, we analyzed which equation model could better explain the relationship and provide the coefficient of determination, i.e., how much of the time of interruption would explain the students’ scores. We set the statistical significance level at a p-value of 0.05.

The statistical analyses were performed using the IBM Statistical Package for the Social Sciences (SPSS), version 24.0 (IBM SPSS Statistics for MacBook, IBM Corp., Armonk, NY, USA).

RESULTS

Scores of 1,470 students (728 and 742 from the fifth and sixth years, respectively) were included in the study. The mean time of clerkship interruption was 141.6±52.0 and 111.4±49.1 days for the fifth and sixth years, respectively. Table 1 summarizes the number of students and the time of interruption for each school.

Table 1
Number of students(n) and days of clerkship interruption by school.

School	Fifth year		Sixth year
	n	Days of interruption	n	Days of interruption
UNESP	76	164	89	115
UNICAMP	113	168	104	125
USP	92	0	81	0
UNIFESP	117	154	126	63
UFSCAR	35	180	32	168
FAMEMA	73	199	89	171
FAMERP	74	80	76	70
UEL	77	140	78	119
FURB	71	47	67	47

The comparisons of the students’ scores between the schools showed that there are differences for both clerkship years (i.e., fifth and sixth years), with p<0.001 (Figure 1). Post-hoc analysis showed significant differences for some pairwise comparisons.

Figure 1
Boxplots of students’ scores according to school. The identification of schools is not presented due to secrecy agreements of the Progress Test consortium policy.

The correlation between the schools’ average scores and the time to resume clerkship rotations was not significant both fifth year (r= -0.298, p=0.436) and sixth year (r= -0.440, p=0.240).

The correlation between the students’ score and the time to resume clerkship rotations was significant for the fifth year (r= -0.082, p=0.026), though not for the sixth year (r= -0.064, p=0.084). A linear model showed R² values of 0.8 and 0.5% for the fifth and sixth years, respectively. However, it was unable to explain the relationships between the variables (score and time) because of the lack of homoscedasticity (constant variance of errors). Durbin-Watson values were 0.268 and 0.223 for the fifth and sixth years, respectively.

A curve estimation analysis showed a better fit of model using a cubic regression (higher R² values and lower p values) for both the clerkship years. The model summary and parameters estimates are presented in Table 2. The R² values for the fifth and sixth years are 3.4 and 0.9%, respectively, with nonsignificant value for the sixth year. The parameters estimates are too low, demonstrating that the dependent variable (students’ scores) is poorly explained by the independent variable (time).

Table 2
Regression model for students’ scores in function of the time to resume clerkship rotations.

	Model summary						Parameter estimates
Year	Equation	R2	F	Df1	Df2	Sig.	Constant	B1	B2	B3
5	Cubic	.034	8.526	3	724	.000	75.551	-271	.003	-1.003E−5
6	Cubic	.009	2.211	3	738	.085	82.728	-136	.002	-6.216E−6

• For both the years, a cubic regression has the better fit. The equation is y=k+b¹*x+b²*x²+b³*x³, where y: score (dependent variable); k: constant (intercept); x: time (independent variable); and (b¹*x+b²*x²+b³*x³): slope.

DISCUSSION

The COVID-19 pandemic has changed several aspects of human life, especially the social interactions. Health and education were affected and, due to the unprecedented situation, some decisions were difficult to make, such as the interruption and resumption of clinical and educational activities of medical undergraduates¹¹,¹².

The effects of the pandemic on the medical students’ learning are yet to be understood properly. Initial observations concluded that the transition from face-to-face to remote activities did not decrease the students’ knowledge. An American survey with 19 students enrolled in a third-year surgical clerkship during COVID-19 concluded that scores of students did not change in comparison with those who studied in the year before the outbreak of the pandemic¹³. However, this sample size is too small to provide unequivocal and generalizable conclusions. Another American study, with 335 first-year students, also showed in differences on the students’ scores before and during the COVID-19 pandemic¹⁴.

An Indonesian cross-sectional study with 270 fourth- and fifth-year medical students in surgery clerkship found no differences between the multiple-choice questions examinations conducted before and during the pandemic. However, in the objective-structured clinical examination, the students evaluated during the pandemic achieved higher scores than those evaluated before the pandemic¹⁵.

Importantly, none of the aforementioned studies described equating methods to compare different tests administered to different students reliably. Equating is an important tool to guarantee a reliable comparison¹⁶,¹⁷. By linking and equating different tests, we have previously demonstrated that the clerkship students’ knowledge acquisition was impaired during the pandemic [unpublished data].

It is possible that pre-clerkship and clerkship students have been affected differently by the pandemic and the transition to remote teaching¹⁸. Much of the educational material developed during the pandemic was helpful and replaced some traditional face-to-face activities, such as lectures and tutorial sessions, quite well¹⁹,²⁰. In some instance, the pandemic induced a positive effect on developing such material.

However, the scenario may be a little different for clerkship students who need direct contact with patients and scenarios to fully develop their professional competence with knowledge, skills, and positive and empathic attitudes toward the patients²¹. Therefore, the clerkship students were the first to resume their educational programs (even in the new normal context). Even so, the decisions on the most appropriate time to resume the clerkship rotations were different across many schools.

We hypothesized that the students who were away from clerkship rotations for a longer period would have lower scores on the test. Our findings do not support this hypothesis. Since there are differences in the median scores of the schools, we believe that other factors such as the nature and quality of how remote teaching was employed play more important roles in determining the students’ knowledge scores. Besides, we cannot discard the possibility of previous differences of the students’ scores across the schools before the pandemic.

A limitation of this study is that we are unable to provide information regarding the direct impact of the social distancing and remote teaching for each school individually, as no comparison with the pre-pandemic scores was done.

Another concern to be discussed is the possible interference of the computer-based exam on students’ performance, which could have changed the difficulty of the exam. However, there are several previous studies demonstrating that there is no difference on students score by using either computer- or paper-based exams, including for Progress Test²²,²³. Besides, as our Progress Test has no high-stake implications, cheating behavior is minimal and, if present, might be homogeneous across the schools.

CONCLUSIONS

We found that the students’ scores in the Progress Test were different across the schools. However, these differences cannot be attributed to the time during which the students were kept away from the clerkship rotations.

• Funding: PTHF and AMB received a grant from the National Board of Medical Examiners (NBME) – Latin America Grants 2020.

REFERENCES

• ¹
  Franklin G, Martin C, Ruszaj M, Matin M, Kataria A, Hu J, et al. How the COVID-19 pandemic impacted medical education during the last year of medical school: a class survey. Life (Basel). 2021;11(4):294. https://doi.org/10.3390/life11040294
  » https://doi.org/10.3390/life11040294
• ²
  Sani I, Hamza Y, Chedid Y, Amalendran J, Hamza N. Understanding the consequence of COVID-19 on undergraduate medical education: medical students’ perspective. Ann Med Surg (Lond). 2020;58:117-9. https://doi.org/10.1016/j.amsu.2020.08.045
  » https://doi.org/10.1016/j.amsu.2020.08.045
• ³
  Moreira MRC, Aquino RF, Barros LL, Parente NC, Machado MFAS, Oliveira AMF, et al. Do medical interns feel prepared to work in the COVID-19 pandemic? Rev Assoc Med Bras. 2020;66(7):973-8. https://doi.org/10.1590/1806-9282.66.7.973
  » https://doi.org/10.1590/1806-9282.66.7.973
• ⁴
  Giordano L, Cipollaro L, Migliorini F, Maffulli N. Impact of Covid-19 on undergraduate and residency training. Surgeon. 2021;19(5):e199-206. https://doi.org/10.1016/j.surge.2020.09.014
  » https://doi.org/10.1016/j.surge.2020.09.014
• ⁵
  Bozkurt A, Jung I, Xiao J, Vladimirschi V, Schuwer R, Egorov G, et al. A global outlook to the interruption of education due to COVID-19 pandemic: navigating in a time of uncertainty and crisis. Asian J Distance Educ. 2020;15(1):1-126. https://doi.org/10.5281/zenodo.3878572
  » https://doi.org/10.5281/zenodo.3878572
• ⁶
  Marinho PRD, Cordeiro GM, Coelho HFC, Brandão SCS. Covid-19 in Brazil: a sad scenario. Cytokine Growth Factor Rev. 2021;58:51-4. https://doi.org/10.1016/j.cytogfr.2020.10.010
  » https://doi.org/10.1016/j.cytogfr.2020.10.010
• ⁷
  Castioni R, Melo AAS, Nascimento PM, Ramos DL. Brazilian federal universities in the Covid-19 pandemic: student internet access and emergency remote teaching. Ensaio Aval Pol Públ Educ. 2021;29(111):399-419. https://doi.org/10.1590/S0104-40362021002903108
  » https://doi.org/10.1590/S0104-40362021002903108
• ⁸
  Desai D, Sen S, Desai S, Desai R, Dash S. Assessment of online teaching as an adjunct to medical education in the backdrop of COVID-19 lockdown in a developing country – an online survey. Indian J Ophthalmol. 2020;68(11):2399-403. https://doi.org/10.4103/ijo.IJO_2049_20
  » https://doi.org/10.4103/ijo.IJO_2049_20
• ⁹
  Bastos RA, Carvalho DRDS, Brandão CFS, Bergamasco EC, Sandars J, Cecilio-Fernandes D. Solutions, enablers and barriers to online learning in clinical medical education during the first year of the COVID-19 pandemic: a rapid review. Med Teach. 2022;44(2):187-95. https://doi.org/10.1080/0142159X.2021.1973979
  » https://doi.org/10.1080/0142159X.2021.1973979
• ¹⁰
  Torda A. How COVID-19 has pushed us into a medical education revolution. Intern Med J. 2020;50(9):1150-3. https://doi.org/10.1111/imj.14882
  » https://doi.org/10.1111/imj.14882
• ¹¹
  Hamamoto Filho PT, Silva E, Ribeiro ZMT, Hafner MLMB, Cecilio-Fernandes D, Bicudo AM. Relationships between Bloom’s taxonomy, judges’ estimation of item difficulty and psychometric properties of items from a progress test: a prospective observational study. São Paulo Med J. 2020;138(1):33-9. https://doi.org/10.1590/1516-3180.2019.0459.R1.19112019
  » https://doi.org/10.1590/1516-3180.2019.0459.R1.19112019
• ¹²
  Rose S. Medical student education in the time of COVID-19. JAMA. 2020;323(21):2131-2. https://doi.org/10.1001/jama.2020.5227
  » https://doi.org/10.1001/jama.2020.5227
• ¹³
  Newman NA, Lattouf OM. Coalition for medical education-A call to action: a proposition to adapt clinical medical education to meet the needs of students and other healthcare learners during COVID-19. J Card Surg. 2020;35(6):1174-5. https://doi.org/10.1111/jocs.14590
  » https://doi.org/10.1111/jocs.14590
• ¹⁴
  Kronenfeld JP, Ryon EL, Kronenfeld DS, Hui VW, Rodgers SE, Thorson CM, et al. Medical student education during COVID-19: electronic education does not decrease examination scores. Am Surg. 2021;87(12):1946-52. https://doi.org/10.1177/0003134820983194
  » https://doi.org/10.1177/0003134820983194
• ¹⁵
  Grand D, Schuster VL, Pullman JM, Golestaneh L, Raff AC. Medical student experience and outcomes, as well as preceptor experience, with rapid conversion of a preclinical Medical School course to a remote-based learning format in the setting of the COVID-19 pandemic. 2021;31(6):1895-901. https://doi.org/10.1007/s40670-021-01379-8
  » https://doi.org/10.1007/s40670-021-01379-8
• ¹⁶
  Setiawan E, Sugeng B, Luailiyah A, Makarim FR, Trisnadi S. Evaluating knowledge and skill in surgery clerkship during COVID-19 pandemics: a single-center experience in Indonesia. Ann Med Surg (Lond). 2021;68:102685. https://doi.org/10.1016/j.amsu.2021.102685
  » https://doi.org/10.1016/j.amsu.2021.102685
• ¹⁷
  Jodoin MG, Keller LA, Swaminathan H. A comparison of linear, fixed common item, and concurrent parameter estimation equating procedures in capturing academic growth. J Exp Educ. 2003;71(3):229-50.
• ¹⁸
  Chulu BW, Sireci SG. Importance of equating high-stakes educational measurements. Int J Test. 2011;11(1):38-52. https://doi.org/10.1080/15305058.2010.528096
  » https://doi.org/10.1080/15305058.2010.528096
• ¹⁹
  Dhillon J, Salimi A, ElHawary H. Impact of COVID-19 on Canadian medical education: pre-clerkship and clerkship students affected differently. J Med Educ Curric Dev. 2020;7:2382120520965247. https://doi.org/10.1177/2382120520965247
  » https://doi.org/10.1177/2382120520965247
• ²⁰
  Alsoufi A, Alsuyihili A, Msherghi A, Elhadi A, Atiyah H, Ashini A, et al. Impact of the COVID-19 pandemic on medical education: medical students’ knowledge, attitudes, and practices regarding electronic learning. PLoS One. 2020;15(11):e0242905. https://doi.org/10.1371/journal.pone.0242905
  » https://doi.org/10.1371/journal.pone.0242905
• ²¹
  Gaur U, Majumder MAA, Sa B, Sarkar S, Williams A, Singh K. Challenges and opportunities of preclinical medical education: COVID-19 crisis and beyond. SN Compr Clin Med. 2020;2(11):1992-7. https://doi.org/10.1007/s42399-020-00528-1
  » https://doi.org/10.1007/s42399-020-00528-1
• ²²
  Hochlehnert A, Brass K, Moeltner A, Juenger J. Does medical students’ preference of test format (computer-based vs. paper-based) have an influence on performance? BMC Med Educ. 2011;11:89. https://doi.org/10.1186/1472-6920-11-89
  » https://doi.org/10.1186/1472-6920-11-89
• ²³
  Karay Y, Schauber SK, Stosch C, Schüttpelz-Brauns K. Computer versus paper--does it make any difference in test performance? Teach Learn Med. 2015;27(1):57-62. https://doi.org/10.1080/10401334.2014.979175
  » https://doi.org/10.1080/10401334.2014.979175

Publication Dates

• Publication in this collection
  21 Nov 2022
• Date of issue
  2022

History

• Received
  29 June 2022
• Accepted
  16 July 2022