Open-access Remote team-based learning during COVID-19: lower academic performance is associated with the exclusion of test grades from final grades

ABSTRACT

Introduction:  The COVID-19 pandemic necessitated a shift from in-person to remote learning, including team-based learning activities. Many studies have explored several aspects of team-based learning; however, there is still a lack of knowledge regarding the factors affecting students’ attitudes and performance.

Objective:  To evaluate the impact of the COVID-19 pandemic on medical students’ performance in individual readiness assurance tests during team-based learning activities, given that their grades were excluded from the final course grades.

Methods:  We compared the number of team-based learning applications, student absences, and individual readiness assurance tests grades of medical undergraduate students in three different scenarios: Group 1, in-person individual readiness assurance tests before the pandemic; Group 2, remote individual readiness assurance tests during the lockdown period when their grades were excluded from the final grade calculation; and Group 3, in-person individual readiness assurance tests during the pandemic after lockdown restrictions eased.

Results:  Remote team-based learning led to significantly lower grades than in-person evaluations. Moreover, the number of students with failing or insufficient grades (below seven) increased during the remote learning period, when their grades were excluded from the final grade calculation, compared to in-person individual readiness assurance tests before the pandemic. Furthermore, absenteeism was higher when the online strategy was applied.

Conclusion:  We observed lower academic performance by medical students during remote team-based learning sessions, likely owing to the exclusion of individual readiness assurance tests grades from the final course grade calculations during the COVID-19 pandemic.

Keywords:
Team-based learning; Academic performance; School, medical; Students, medical; Learning; Problem-based learning; Educational measurement; Group process

Highlights

  • Remote team-based learning sessions were conducted during the COVID-19 pandemic, and their grades were not included in the final course grade calculation.

  • Remote team-based learning resulted in lower iRAT grades than in-person team-based learning, independent of the semester.

  • Remote team-based learning resulted in more students failing the test than in-person team-based learning.

  • Remote team-based learning resulted in lower student attendance than in-person team-based learning.


In Brief

Magdalon et al. demonstrated that excluding iRAT grades from the final grade calculation of the course negatively impacted student performance and attendance in team-based learning sessions.

INTRODUCTION

The COVID-19 pandemic forced all in-person classes and assessments to occur remotely. This transition has been termed emergency remote teaching to distinguish it from existing online learning courses.(1) This shift severely impacted assessment, which had to be conducted online using available technology.(2) The lack of online proctoring posed a particular challenge to all educational institutions in maintaining academic integrity during assessments. To mitigate cheating risks, many institutions adopted open-book examinations, although studies have reported significant grade differences compared with traditional closed-book examinations.(3,4)

Team-based learning (TBL) is an active learning method that focuses on achieving specific learning objectives in an environment in which students are held responsible for both individual and group learning. In TBL, teachers assign students to teams and present them with challenging problems that require collaborative problem-solving. Students are expected to attend the online class prepared and equipped with the necessary knowledge to answer group application questions.(5) The use of TBL in health education has increased over recent decades. Michaelsen's conceptual model for TBL addresses several elements that may impact learner engagement and learning outcomes, such as teacher decisions, individual characteristics, contextual factors, and team characteristics.(5) A systematic review has shown that most studies in this area focus on professor and student attitudes, as well as learning outcomes. Most of these studies show that TBL results in higher satisfaction than traditional classes, concluding that this is an effective instructional technique. However, several aspects of TBL require further investigation, including learner engagement and factors affecting academic outcomes.(6)

During the COVID-19 pandemic, TBL faced several challenges, including technological limitations (including Internet access), reduced interaction, and difficulty monitoring student engagement. Therefore, educators shared strategies for successful online TBL adaptations.(7-10) However, concerns about academic integrity in graded activities, such as individual and team readiness assurance tests (iRAT and tRAT), remain significant, especially with the use of open-book tests undermining the objectives of TBL.

The medical school investigated in this study continued to use TBL as a learning strategy during emergency remote teaching but decided not to factor iRAT scores into the final grade owing to concerns about academic integrity. This provided an opportunity to explore gaps in the literature regarding some dimensions of TBL, particularly the lack of information regarding TBL performance during the COVID-19 pandemic. The primary aim of this study was to evaluate the impact of emergency remote teaching during the COVID-19 pandemic on iRAT grades during TBL activities. We compared the iRAT grades across three scenarios: 1) in-person iRATs before the pandemic, 2) remote iRATs during the lockdown period when their grades were excluded from the final course grade, and 3) in-person iRATs during the pandemic after lockdown restrictions were eased.

OBJECTIVE

To evaluate the impact of the COVID-19 pandemic on medical students’ individual readiness assurance tests grades during team-based learning activities, specifically in the context where these grades were excluded from their final course grades.

METHODS

Setting

This study was performed at the medical school Faculdade Israelita de Ciências da Saúde Albert Einstein, Brazil, where TBL is employed as the main learning strategy. Team-based learning occurred in all courses over the first seven semesters, two to three times per week. The remaining five semesters were dedicated to internships. Each class consisted of 60 students, who were randomly divided into nine groups of six to seven students. At the beginning of each semester, the groups were re-divided.

Team-based learning at the institution follows all core elements.(11) In the preparatory phase, the teacher provides learning goals and assigns materials that students must study individually, such as book chapters, videos, and articles. In class, learning is assessed through an iRAT, followed by a tRAT after a consensus-building discussion. Both tests are identical and consist of approximately ten multiple-choice questions about the most important concepts from prior studies. During the tRAT, students receive immediate feedback if the answer is correct. If an incorrect answer is assigned, they have another chance to identify a new consensus answer, which is repeated until the correct answer is detected, thus decreasing the final grade of each question. Subsequently, the teacher clarifies the students’ doubts about the previous tests and rapidly begins the application phase, in which students are presented with different scenarios and problems to be solved. They must discuss the answer in the same group as in the tRAT, make a specific choice, and display it simultaneously with the other groups. The teacher promotes debates between groups, and students must explain their choices. Student participation is peer evaluated and counted as part of the TBL grade together with the other phases of the activity (iRAT counts for 20%, tRAT counts for 30%, and peer assessment counts for 50% of the TBL grade).

Both iRAT and tRAT are performed through the web-based learning platform Canvas® (Instructure® Inc., Salt Lake City, Utah, USA), regardless of whether the class is in person or remote. Each discipline includes two to 13 TBL sessions during each semester, along with other active learning methodologies, depending on the number of hours dedicated to that discipline every week.

From March 16th, 2020, until the end of the semester (June 2020), every activity in the first seven semesters (before the internship) was performed online and synchronously, including TBL sessions. After an academic break in July 2020, some in-person activities resumed in August 2020, with social distancing measures in place. During this period, some TBL sessions were held in person (split between two rooms with two teachers), although most continued online. When TBL was conducted remotely, students still received iRAT and tRAT grades, but they did not count toward the final course grade and were used solely for performance evaluation. In contrast, when TBL was conducted in person, even during the COVID-19 pandemic, iRAT and tRAT grades were included in the final course grades, influencing students’ pass or fail outcomes.

Data collection

To evaluate the effects of the COVID-19 pandemic on TBL activities, the anonymized iRAT grades and absence rates from all classes were collected from the Canvas® platform across three different scenarios:

Group 1 - Baseline group (control): This group included data from the first and second semesters of 2019, as well as the first semester of 2020 before the official start of the COVID-19 lockdown in Brazil (March 16th, 2020), when the iRATs were performed in person.

Group 2 - Remote pandemic group: This group included data from the first semester of 2020 during the lockdown in Brazil (starting March 16th, 2020) and the second semester of 2020, when iRATs were performed exclusively remotely and not included in the final course grades (online by Zoom Meeting® Tool; Zoom Video Communications, San Jose, CA, USA).

Group 3 - In-person pandemic group: This group included data from the second semester of 2020 and the first semester of 2021, during the COVID-19 pandemic, when iRATs were performed in person but with social distancing measures in place (activities were split between two rooms with two teachers).

This study was approved by the Institutional Research Ethics Committee of Hospital Israelita Albert Einstein (CAAE: 49031821.6.0000.0071; # 5.363.350).

Statistical analyses

All collected data were analyzed and expressed as means with corresponding standard deviations (SDs), standard errors (SEs), medians, interquartile ranges, minimum and maximum values, and 95% confidence intervals (95%CI). Group comparisons of means were performed using one-way analysis of variance (ANOVA) followed by Bonferroni's post-hoc test. Categorical comparisons between groups were analyzed using the χ2 test. All statistical analyses were performed using SPSS® version 27.0 (IBM® Inc., New York, USA), and statistical significance was set at p<0.05.

RESULTS

During the 30-months study period, 842 TBL sessions were conducted, yielding 41,849 iRAT grades. In 2019, 381 in-person TBL sessions took place across two semesters. During the first semester of 2020, at the beginning of the COVID-19 pandemic in Brazil, 153 TBL sessions were conducted, 65.4% of which were conducted online. In the second semester of 2020, 151 TBL sessions occurred, of which 72.8% were held online. By the first semester of 2021, students had engaged in 157 TBL sessions, with 63.7% conducted online. Following the easing of lockdown restrictions (second semester of 2020), 98 iRATs were performed using a new in-person strategy (Group 3), with students divided into two rooms with two teachers to maintain social distance. Descriptive data for all TBL sessions and comparisons between the different strategies are presented in table 1.

Table 1
Descriptive data and comparisons between team-based learning and absences across periods of the year, course semesters and rotations for different groups (basic–between 1st and 4th semesters; clinical–between 5th and 7th semester), according to each different scenario (Group)

The remote TBL strategy (Group 2) resulted in significantly lower grades (7.826; 95%CI= 7.796-7.856; p<0.001; one-way ANOVA) when compared to in-person evaluations, where TBL grades contributed to the final course grades, regardless of whether they were assessed before or during the COVID-19 pandemic (8.225; 95%CI = 8.203-8.247 for the baseline period [Group 1] and 8.464; 95%CI = 8.420-8.508 for the period after the lockdown [Group 3]; Table 2 and Figure 1). Moreover, the iRAT grades were significantly higher after the lockdown period than in the control group (p<0.001; Bonferroni's post-hoc test, one-way ANOVA). The remote iRAT grades were lower regardless of the semester or phase (basic or clinical), as shown in table 3.

Table 2
Means differences between iRAT grades, according to each different scenario (Group)
Figure 1
Boxplot charts demonstrating lower grades for iRAT (p<0.001, one-way ANOVA) for the online remote strategy (Group 2), when grades were excluded from the final course grade, comparing to in-person team-based learning before the COVID-19 pandemic (Group 1) and in-person team-based learning during the pandemic after the lockdown period (Group 3)
Table 3
Comparisons between iRATs grades and period of the year, course period (semester) and rotation (basic: between 1st and 4th semesters; clinical: between 5th and 7th semester), according to each scenario (Group)

To demonstrate the impact of lower iRAT grades from remote TBL sessions, which did not count toward the final grades, we categorized these grades into three different groups, as shown in table 4 and figure 2: (1) <4.0, indicating the students who failed the test; (2) ≥7.0, representing students who passed the test; (3) 4.00-6.99, the group of students with insufficient grades. The number of students who failed the test or had insufficient grades increased from 15.4% in the baseline period (Group 1) to 23.9% when the online strategy was adopted (Group 2), and returned to 13.7% after the lockdown period (Group 3).

Table 4
Differences between categorized values of iRAT grades, according to each different scenario (Group)
Figure 2
Bar chart demonstrating increased rates of students failing (orange bars) or those with insufficient grades (blue bars) during the iRAT conducted under the remote team-based learning strategy (Group 2), when grades did not contribute to final course approval

The frequency of remote TBL sessions (Group 2), which did not count toward final grades, was not significantly affected by the semester of the medical course (p=0.117, χ2 test) or the phase (basic or clinical) (Table 1). This indicates that the occurrence of failing grades was widespread and affected the entire student population. After he lockdown period, fewer in-person TBL sessions occurred in the basic phase (Group 3) (Table 1; p=0.002, χ2 test). This decline was attributed solely to the increased number of teachers at higher risk for COVID-19, preventing the adoption of in-person strategies for those classes. Furthermore, absenteeism in TBL sessions was higher during the online strategy, where grades did not count for final approval (Table 1; 9.5% for Group 2 versus 3.0% and 5.2% for Groups 1 and 3, respectively; p=0.001, χ2 test). These absence rates were not affected by the time of year, period of the medical course, or phase.

DISCUSSION

The COVID-19 pandemic has presented several challenges and necessitated adaptations to the learning environment, leading to a shift toward online classes and assessments.(12) The aim of this study was to evaluate the impact of the COVID-19 pandemic on medical students’ iRAT grades in TBL activities conducted remotely during emergency remote teaching. Our results show that the remote TBL strategy-and particularly the fact that these grades did not count toward final course approval-resulted in significantly lower grades and a higher incidence of failing or insufficient grades compared to in-person evaluations or when grades contributed to the final grade. The frequency of remote TBL sessions was unaffected by the semester or phase of the medical course, suggesting that the occurrence of failing grades was widespread and affected the entire student cohort. In addition to the lower grades, more students were absent during iRATs in the remote TBL sessions compared to the in-person pre- and post-pandemic periods. This suggests that students were less engaged in the remote learning environment and did not adequately prepare for classes as requested by the teachers. Moreover, the exclusion of grades from final course grades likely contributed to lower academic performance, making this a key confounding factor in the results.

These findings align with existing literature indicating that grading individual performance enhances student engagement with the content and encourages pre-class preparation.(11) A recent systematic review comparing face-to-face and remote TBL found no clear differences between the two approaches,(13) underscoring the importance of studies like the present one in assessing various strategies for TBL application in terms of academic achievement, learning, and satisfaction. However, we cannot exclude the possibility that the low grade identified in our study stemmed from the remote learning environment, as other authors have reported similar challenges during emergency remote teaching, including loss of interest, difficulty concentrating and understanding the content, reduced interactivity and technological issues.(14-21) Other possible explanation for the increased number of absences in iRATs during remote learning could be poor internet connectivity and technological barriers, although it was uncommon for students in the study to report such issues. Nonetheless, some students reported that they appreciated online learning during the pandemic, especially because of enhanced flexibility and more time for sleep or self-care(22,23) although students felt that they did not prepare themselves well in advance for online learning.(22)

The major concern regarding students’ poor performance in online formats is their learning outcomes. The TBL iRAT phase assesses student knowledge acquired in the preparatory phase and includes questions about the most important concepts of the theme. In cases where students are not prepared for class, all subsequent phases (the tRAT and application phases) may be compromised, affecting overall student learning. Furthermore, considering that students have studied less for remote TBL sessions or that their grades were not counted for final approval, it is likely that they also performed similarly in all other remote non-TBL classes, which may have impacted the entire learning of those disciplines during the COVID-19 pandemic. During remote TBL sessions, fostering a collaborative learning environment proved more challenging than in in-person classes. Teachers relied on Zoom Meeting® breakout rooms to perform tRATs and group discussions during the application phase. However, since teachers could not monitor all the group conversations simultaneously, it was common to close the breakout rooms before discussions were completed, which likely hindered deep learning opportunities for students who had prepared themselves and had higher grades on the iRAT. Deep learning often depends on opportunities for students to explain their reasoning to others.(24)

Several studies conducted during the COVID-19 pandemic have evaluated the effectiveness of synchronous online TBL as a valuable tool for continuing education for health professional students.(25-27) These studies indicated that remote and in-person TBL was approved by students and even led to improved test performance compared to students who did not voluntarily sign up to use the TBL method. Nonetheless, we could not find any studies that specifically investigated the impact of remote TBL when grades did not affect the final course grades. While we acknowledge that TBL is one of the most effective deep-learning tools, face-to-face teaching may yield better outcomes. We believe that the iRAT and tRAT results should count toward final grades to encourage students to be well prepared during pre-class study.

Another important aspect to consider is academic integrity during remote TBL. Given that TBL grades were not factored into final course grade, most teachers did not alter questions from the previous semesters. Conversely, when TBL sessions were conducted in-person, teachers were instructed to change the questions of the iRAT/tRAT, as students had access to past tests. Although students could have consulted previous tests during remote iRATs, which might have resulted in higher grades, our findings suggest otherwise. This indicates that most students upheld academic integrity during remote TBL sessions, despite an increased perception of cheating in online assessments.(28-30)

Strengths and limitations

To our knowledge, this is the first study to compare TBL grades before and during the COVID-19 pandemic, particularly in a medical school that uses TBL as its main learning strategy. One significant advantage is that all teachers and students (except for first semester students) were already proficient in the methodology, minimizing the possibility that any observed difficulties regarding remote TBL were due to a lack of knowledge of the method. Additionally, the high number of student grades analyzed provided robust data, enhancing the reliability of the results.

However, this study has some limitations. First, although we carefully collected and analyzed all data, we may have missed some data from TBL activities, particularly during remote learning, since some teachers may have used platforms other than Canvas® for assessments, and only tests recorded in Canvas® were considered. Second, during the study period, two courses changed faculty, which may have affected the difficulty level of the iRAT/tRAT questions. However, given the extensive dataset, we believe that such variations were isolated and did not significantly influence the overall findings. Third, the study lacked a fourth group: an in-person setting where TBL grades did not count toward the final grade. This group could have helped determine whether the lower academic achievement in remote TBL sessions was due to the remote learning environment or a lack of grade-based consequences. Unfortunately, this scenario was not included in the present study because it was not part of the medical school's educational program. Future qualitative research using student focus groups could enhance our understanding of how remote learning affects academic achievement.

CONCLUSION

This study indicated that medical students showed lower academic performance in remote individual readiness assurance tests during the COVID-19 pandemic, possibly because the team-based learning grades did not count toward their final course grades.

REFERENCES

  • 1 Hodges C, Moore S, Lockee B, Trust T, Bond A. The difference between emergency remote teaching and online learning. Educause. 2020 March 27 [cited 2024 Sep 2]. Available from: https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning
    » https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning
  • 2 Reid MD, Sam AH. Reflections on assessment in the wake of change from the COVID-19 pandemic. Medical Education. Blackwell Publishing Ltd; 2021. pp. 128-30.
  • 3 Eurboonyanun C, Wittayapairoch J, Aphinives P, Petrusa E, Gee DW, Phitayakorn R. Adaptation to Open-Book Online Examination During the COVID-19 Pandemic. J Surg Educ. 2021;78(3):737-9.
  • 4 Prigoff J, Hunter M, Nowygrod R. Medical Student Assessment in the Time of COVID-19. J Surg Educ. 2021;78(2):370-4.
  • 5 Michaelsen LK, Parmelee DX, McMahon KK, Levine RE. Team-based Learning for Health Professions Education: A Guide to Using Small Groups for Improving Learning. Stylus; 2008. 229 p.
  • 6 Reimschisel T, Herring AL, Huang J, Minor TJ. A systematic review of the published literature on team-based learning in health professions education. Med Teach. 2017;39(12):1227-37.
  • 7 Gaber DA, Shehata MH, Amin HA. Online team-based learning sessions as interactive methodologies during the pandemic. Med Educ. 2020;54(7):666-7.
  • 8 Wong P, Jumat MR, Lee IC, Foo KX, Goh SP, Ganapathy S, et al. Redesigning team-based learning facilitation for an online platform to deliver preclinical curriculum: a response to the COVID-19 pandemic. MedEdPublish. 2020;9:135.
  • 9 Malik AS, Malik RH. Twelve tips for conducting team-based learning session online in synchronous setting. Med Teach. 2022;44(5):486-93.
  • 10 Vieira JE, Matos LL, de Paula EG, Carrera RM. Teaching surgery during COVID-19: The experience of Albert Einstein Medical School, Brazil. Med Educ. 2021;55(5):628-9.
  • 11 Haidet P, Levine RE, Parmelee DX, Crow S, Kennedy F, Kelly PA, et al. Perspective: Guidelines for reporting team-based learning activities in the medical and health sciences education literature. Vol. 87, Academic Medicine. Lippincott Williams and Wilkins; 2012. p. 292-9.
  • 12 Pokhrel S, Chhetri R. A Literature Review on Impact of COVID-19 Pandemic on Teaching and Learning. Higher Education for the Future. 2021;8(1):133-41.
  • 13 Burton R, Kellett U, Mansah M, Sriram D. A systematic review of online team based learning approaches in health professional education. Nurse Educ Today. 2024;138:106156.
  • 14 Tan C. The impact of COVID-19 on student motivation, community of inquiry and learning performance. Asian Education and Development Studies. 2021;10(2):308-21.
  • 15 Mahdy MA. The Impact of COVID-19 Pandemic on the Academic Performance of Veterinary Medical Students. Front Vet Sci. 2020;7:594261.
  • 16 Onyema EM, Eucheria NC, Obafemi FA, sen S, Atonye FG, Sharma A, et al. Impact of Coronavirus Pandemic on Education. J Educ Pract. 2020;11(13):108-21.
  • 17 Abbasi S, Ayoob T, Malik A, Memon SI. Perceptions of students regarding e-learning during covid-19 at a private medical college. Pak J Med Sci. 2020;36(COVID19-S4):S57-61.
  • 18 Amir LR, Tanti I, Maharani DA, Wimardhani YS, Julia V, Sulijaya B, et al. Student perspective of classroom and distance learning during COVID-19 pandemic in the undergraduate dental study program Universitas Indonesia. BMC Med Educ. 2020;20(1):392.
  • 19 Dost S, Hossain A, Shehab M, Abdelwahed A, Al-Nusair L. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020;10(11):e042378.
  • 20 Daniels LM, Goegan LD, Parker PC. The impact of COVID-19 triggered changes to instruction and assessment on university students’ self-reported motivation, engagement and perceptions. Soc Psychol Educ. 2021;24(1):299-318.
  • 21 Shin M, Hickey K. Needs a little TLC: examining college students’ emergency remote teaching and learning experiences during COVID-19. J Furth High Educ. 2021;45(7):973-86.
  • 22 Schlenz MA, Schmidt A, Wöstmann B, Krämer N, Schulz-Weidner N. Students’ and lecturers’ perspective on the implementation of online learning in dental education due to SARS-CoV-2 (COVID-19): a cross-sectional study. BMC Med Educ. 2020;20(1):354.
  • 23 Zheng M, Bender D, Lyon C. Online learning during COVID-19 produced equivalent or better student course performance as compared with pre-pandemic: empirical evidence from a school-wide comparative study. BMC Med Educ. 2021;21(1):495.
  • 24 Pires EM, Daniel-Filho DA, de Nooijer J, Dolmans DH. Collaborative learning: elements encouraging and hindering deep approach to learning and use of elaboration strategies. Med Teach. 2020;42(11):1261-9.
  • 25 Anas S, Kyrou I, Rand-Weaver M, Karteris E. The effect of online and in-person team-based learning (TBL) on undergraduate endocrinology teaching during COVID-19 pandemic. BMC Med Educ. 2022;22(1):120.
  • 26 Silva EC, Lino-Neto T, Ribeiro E, Rocha M, Costa MJ. Going virtual and going wide: comparing Team-Based Learning in-class versus online and across disciplines. Educ Inf Technol. 2022;27(2):2311-29.
  • 27 Al-Neklawy AF, Ismail AS. Online anatomy team-based learning using blackboard collaborate platform during COVID-19 pandemic. Clin Anat. 2022;35(1):87-93.
  • 28 Watson G, Sottile J. Cheating in the Digital Age: Do Students Cheat More in Online Courses? [cited 2024 Sep 2]. Available from: http://www.ebscohost.com
    » http://www.ebscohost.com
  • 29 Miller A, Young-Jones AD. Academic integrity: Online classes compared to face-to-face classes. J Instructional Psychol. 2012;39(3-4):138-45.
  • 30 King DL, Case CJ. E-cheating: incidence and trends among college students. Issues Inf Syst. 2014;15(1):20-7.

Edited by

Publication Dates

  • Publication in this collection
    03 Feb 2025
  • Date of issue
    2025

History

  • Received
    31 Jan 2024
  • Accepted
    23 Aug 2024
location_on
Instituto Israelita de Ensino e Pesquisa Albert Einstein Avenida Albert Einstein, 627/701 , 05651-901 São Paulo - SP, Tel.: (55 11) 2151 0904 - São Paulo - SP - Brazil
E-mail: revista@einstein.br
rss_feed Acompanhe os números deste periódico no seu leitor de RSS
Reportar erro