Insights from a 3-year experience: depicting the flourishing of rheumatologists through entrustable professional activitiesABSTRACT
Introduction: The 14 entrustable professional activities (EPAs) approved by the American College of Rheumatology (ACR) were developed to be “the comprehensive set of tasks or responsibilities that any practicing rheumatologist should be capable of performing”. Our goal was to originally present and discuss the 3-year experience of regularly assessing rheumatology residents through ACR-approved EPAs.
Experience report: In this case series, all first-year and second-year rheumatology residents were assessed every May and November (the residency program starts in March) via online anonymous forms. For the EPA assessment, preceptors chose 1 of 5 levels of entrustability. Unsupervised practice was defined as levels 4 (“I had to provide minor directions”) and 5 (“I did not need to provide directions for safe and independent care”) combined. Individual reports were discussed by the program supervisor in separate feedback meetings. Between 2021 and 2023, 276 EPA reports from 11 residents assessed by 10 preceptors were analyzed. The number of EPAs with over 90% of unsupervised practice rating in semesters 1, 2, 3 and 4 were, respectively, 1, 0, 2 and 11. Overall progression was statistically significant for 13 EPAs and greater between semesters 2 and 3. Feedback meetings with an agenda for residents and for the program supervisor contributed to individual improvement and training qualification.
Discussion: Implementing EPAs in rheumatology residency was an opportunity to increase feedbacks and to qualify the program. The residents’ autonomy was practically nonexistent before the last assessment. Although a valuable formative instrument in residency, the entrustable professional activity-based assessment needs careful discussion before the adoption of cut-off values for summative purposes.
Conclusions: Assessing rheumatology residents through the ACR-approved EPAs establishes a regular feedback culture, providing opportunity for graduate students’ improvement and program qualification. Unsupervised practice was mostly achieved by the end of the training and a greater increment coincided with the change in the year of residency.
Keywords:
Medical education; medical residency
RESUMO
Introdução: As 14 atividades profissionais confiabilizadoras (entrustable professional activities -EPA) aprovadas pelo American College of Rheumatology (ACR) foram desenvolvidas para serem “o conjunto abrangente de tarefas ou responsabilidades que qualquer reumatologista deve ser capaz de executar”. Nosso objetivo foi apresentar e discutir a experiência de três anos de avaliação regular de residentes de reumatologia pelas EPA aprovadas pelo ACR.
Relato de experiência: Nessa série de casos, todos os residentes de reumatologia do primeiro e do segundo ano foram avaliados a cada mês de maio e novembro (a residência começa em março) por meio de formulários anônimos on-line. Para a avaliação por EPA, os preceptores escolheram um de cinco níveis de confiabilidade. A prática não supervisionada foi definida como níveis 4 (“Tive que fornecer orientação menor”) e 5 (“Não precisei fornecer orientação para um cuidado seguro e independente”) combinados. Os relatórios individuais foram discutidos pelo supervisor do programa em reuniões de feedback separadas. Entre 2021 e 2023, analisaram-se 276 relatórios de 11 residentes avaliados por dez preceptores. O número de EPA com mais de 90% de classificação de prática não supervisionada nos semestres 1, 2, 3 e 4 foi, respectivamente, 1, 0, 2 e 11. A progressão geral foi estatisticamente significativa para 13 EPA e maior entre os semestres 2 e 3. Reuniões de feedback com uma agenda para os residentes e para o supervisor do programa contribuíram para o aprimoramento individual e a qualificação da formação.
Discussão: A implementação de EPA na residência de reumatologia foi uma oportunidade para aumentar os feedbacks e qualificar o programa. A autonomia dos residentes era praticamente inexistente antes da última avaliação. Embora seja um valioso instrumento formativo na residência, a avaliação baseada em EPA necessita de uma discussão cuidadosa antes da adoção de pontos de corte para fins somativos.
Conclusão: A avaliação de residentes de reumatologia pelas EPA aprovadas pelo ACR estabelece uma cultura de feedback regular, proporcionando oportunidade para a melhoria dos pós-graduandos e para a qualificação do programa. A prática não supervisionada foi conseguida majoritariamente no final da formação, e um maior incremento coincidiu com a mudança de ano de treinamento.
Palavras-chave:
Educação Médica; Residência Médica
INTRODUCTION
In competency-based medical education, trainees are expected to acquire competencies that include knowledge, skills and attitude, and one single EPA may require multiples integrated competencies from the apprentice1. The 14 entrustable professional activities (EPAs) endorsed by the American College of Rheumatology (ACR) according to the Accreditation Council for Graduate Medical Education (ACGME) core competencies were developed to be “the comprehensive set of tasks or responsibilities that any practicing rheumatologist should be capable of performing”2.
A resident to be considered “trusted” needs enough experience to make the right choices when submitted to unpredicted challenges, and educators are expected to infer the trainees’ autonomy based on a limited range of observations3. Unsupervised practice is endorsed by in-training appropriate autonomy and levels of supervision4. Interestingly, there seems to be a gap between what residents are expected to accomplish and what preceptors perceive as readiness for practice5.
In Brazil, rheumatology residency consists of a 2-year training after at least 6 years of the undergraduate course in medicine and 2 years of internal medicine residency. Recently, a national resolution recommended EPAs as a possible basis for verifying the preparedness of resident doctors for progression at autonomous practice levels.6 However, what should be considered satisfactory per postgraduate year has not been defined and the progression of rheumatology residents’ autonomy through an EPA-based assessment during the training has not been reported so far. Therefore, this study aims to present and discuss a 3-year experience of regularly assessing residents through the ACR-approved EPAs.
EXPERIENCE REPORT
This case series followed the STROBE reporting guideline and was approved by the institutional review board and written informed consent was obtained from Rheumatology residents between 2021 and 2023. During this period, all first-year and second-year residents were assessed every May and November (the residency program starts in March) via online forms anonymously completed by all preceptors with objective questions for EPAs and one open-answer question for comments on skills and attitudes, mainly agency, reliability, integrity, capability and humility (“A RICH” acronym).7 In every EPA question, preceptors were instructed to choose, for each resident, 1 out of 5 levels of entrustability (1. “I had to do it”; 2. “I had to talk them through”; 3. “I needed to prompt”; 4. “I had to provide minor directions”; 5. “I did not need to provide directions for safe and independent care”)8.
Quantitative and qualitative data were compiled in individual reports presented by the program supervisor in face-to-face individual feedback meetings. Percentages of levels of entrustability per semester were presented in a table and comments were anonymously added. After a careful discussion of the report, a consensual agenda containing tasks for the resident and suggestions for the program was created.
The variables herein presented were described by absolute and relative frequencies and compared between semesters using the chi-square test of linear trend. Changes between semesters were compared using the generalized linear model with binomial distribution and Bonferroni’s complementary test. The level of significance adopted was 5% (p<0.05) and the analyses were performed using SPSS version 27.0.
Between 2021 and 2023, there were 276 EPA reports from 11 residents (1 resident withdrew to join another specialty residency and was excluded from the analyses) assessed by 10 preceptors. Throughout the 2 years of residency, the percentage of unsupervised practice rating increased significantly for 13 of the 14 EPAs (Table 1). When put in a radar chart, the overall increment of readiness from all residents throughout the course resembles a flower blossom and was used to illustrate individual progress in each report as well (Figure 1). Also, there was a major increment of perceived autonomy between semesters 2 and 3, as observed in 6 of the 14 EPAs (Table 2).
Radar chart with the overall percentages of levels 4 and 5 combined for each EPA from 2021 through 2023.
Regarding the agenda for program qualification created during the feedback meetings, examples of changes implemented were the greater number of sessions on physical examination training, more flipped-classroom seminars and the greater interface with other core specialties in case discussions and seminars.
DISCUSSION
Our EPA-based assessment of rheumatology residents included both a quantitative and a qualitative analysis. The combination of both broadened the perspective of the residents’ performance and established regular feedback meetings, creating a protected space to deal with expectations from residents and educators. Interestingly, the expected readiness was practically nonexistent before the final assessment, performed 3 months prior to the end of the training. In addition to this non-linear growth pattern, the more pronounced increment between two consecutive semesters was noticed when the residents became second-year trainees.
Entrustability scales have been reported in different residency programs and are considered an intuitive assessment tool that provides raters with elements of day-to-day clinical decisions, helping them recognize readiness for autonomous practice based on real-world judgements8)-(12. For trainees’ progression measurement, we considered levels 4 and 5 of combined entrustability as an achievable outcome, since residents were often under supervision in our program and we acknowledged the need of a learning curve for preceptors to consider level 5 alone as unsupervised practice. As previously reported, unsupervised practice in residency varied largely by EPAs and was achieved by 90% of pediatric residents in only 8 of 17 EPAs5. Although we did not look at the percentage of residents achieving level 5 of entrustability, our results show a similar heterogeneous behavior among ACR-approved EPAs (11 of 14 with over 90% of unsupervised practice rating).
Originally, our intention was not to use EPAs for summative purposes. We decided to implement an EPA-based assessment in the rheumatology training, because we wanted to nurture the feedback culture among preceptors and trainees, and because we felt that we needed to be closer to our trainees, considering the severe educational impacts of the COVID-19 pandemics. Along the course of the experience, we acknowledged that EPA could also help monitor the progress of the trainees’ knowledge, skills and attitudes throughout the program. Hence, we do not see the EPA-based assessment as purely summative with cut-offs for every training step, but rather an instrument to improve the development of competencies by timely identifying weaknesses and expanding on the feedback practice.
One of our experience’s limitations is the fact that we only performed the evaluation twice a year. We believe that, as an opportunity for feedback and identification of individual learning problems, EPA-based assessment could be performed more often. Most likely, a more frequent assessment could track trainees progress and provide preceptors with more consistent and timely data for pedagogical interventions. For an easier, ad libitum, point-of-learning assessment, the on-line questionnaires could be made available permanently and preceptors could be regularly reminded to complete them at their own conveniences. Also, by increasing the frequency of assessments, the impact of other confounders, such as memory bias, could be softened.
Different from the Medical School scenario, where students usually are assessed by multiple teachers after shorter periods of observation, in our residency, trainees are closely followed up by the entire group of preceptors throughout the 2 years of training. Since rating a trainee lower in entrustability could constitute a failure of the preceptors themselves, a multifaceted approach, such as 360-degree feedback, could optimize the accuracy of measuring readiness for practice13),(14. By including other-program trainees, supervisors, administrative staff, patients and families, the breadth of perspectives is expected to improve the accuracy and to grasp a more comprehensive guidance. Also, considering that the greater increment between semesters 2 and 3 could represent the influence of the postgraduate year on the perception of raters (first versus second year of residency), the creation of time-variable EPAs could offer an interesting alternative to measure the ongoing development of trainees at every stage of the course15.
Our small sample size should be acknowledged as a limitation for the generalizability of the results herein presented. Implementing an EPA-based assessment in a larger program would probably offer greater challenges, including lower preceptors’ adherence, less proximity between preceptors and residents, and larger variability of personal dilemmas experienced by residents along the training course. On the other hand, a larger experience could add to our findings a richer perspective in terms of diversity of practice scenarios and cultural backgrounds, for example.
Due to its limited sample size, our study does not allow drawing definite conclusions regarding the establishment of cut-off values for unsupervised practice in the different stages of training. On the contrary, EPA-based readiness in postgraduate education seems to be a heterogenous outcome, still falling behind the expectations of educators5),(12),(16. Therefore, we understand that the EPA-based assessment in postgraduate training constitutes a valuable formative approach, deserving future research for clarifying its role as a summative instrument. Notably, our results need to be validated in different rheumatology programs. Also, other medical specialties could demonstrate the applicability of EPAs across different contexts, mainly by exploring the impact of more frequent assessments or multidimensional feedback approaches to enhance the conversation around continuous improvement in medical education.
CONCLUSIONS
Assessing rheumatology residents through the ACR-approved EPAs establishes a regular feedback culture, providing opportunity for the residents’ improvement and program qualification. Unsupervised practice was mostly achieved by the end of the training and a greater increment coincided with the change in the year of residency. Although a valuable formative instrument in residency, the entrustable professional activity-based assessment needs careful discussion before the adoption of cut-off values for summative purposes.
ACKNOWLEDGEMENTS
The authors acknowledge and would like to thank Prof. Danilo Blank, Prof. Cristina Neumann and Prof. Francisco Arsego from Hospital de Clínicas de Porto Alegre, Universidade Federal do Rio Grande do Sul, for their guidance and support.
References
-
1 ten Cate O. An Updated Primer on Entrustable Professional Activities (EPAs). Rev bras educ med 2019;43(1 suppl 1):721-730. doi:10.1590/1981-5271v43suplemento1-20190238.ING
» https://doi.org/10.1590/1981-5271v43suplemento1-20190238.ING -
2 Brown CR, Criscione-Schreiber L, O’Rourke KS, et al. What Is a Rheumatologist and How Do We Make One? Arthritis Care Res 2016;68(8):1166-1172. doi:10.1002/acr.22817
» https://doi.org/10.1002/acr.22817 -
3 Ten Cate O, Carraccio C, Damodaran A, et al. Entrustment Decision Making: Extending Miller’s Pyramid. Acad Med 2021;96(2):199-204. doi:10.1097/ACM.0000000000003800
» https://doi.org/10.1097/ACM.0000000000003800 -
4 Schumacher DJ, Bria C, Frohna JG. The Quest Toward Unsupervised Practice: Promoting Autonomy, Not Independence. JAMA 2013;310(24):2613-2614. doi:10.1001/jama.2013.282324
» https://doi.org/10.1001/jama.2013.282324 -
5 Schumacher DJ, West DC, Schwartz A, et al. Longitudinal Assessment of Resident Performance Using Entrustable Professional Activities. JAMA Netw Open 2020;3(1):1-14. doi:10.1001/jamanetworkopen.2019.19316
» https://doi.org/10.1001/jamanetworkopen.2019.19316 - 6 CARVALHO DP DE. RESOLUÇÃO No 4, DE 1o DE NOVEMBRO DE 2023. DIÁRIO Of DA UNIÃO 2023;203(1):23-26.
-
7 ten Cate O, Chen HC. The ingredients of a rich entrustment decision. Med Teach 2020;42(12):1413-1420. doi:10.1080/0142159X.2020.1817348
» https://doi.org/10.1080/0142159X.2020.1817348 -
8 Gofton WT, Dudek NL, Wood TJ, Balaa F, Hamstra SJ. The Ottawa Surgical Competency Operating Room Evaluation (O-SCORE): A Tool to Assess Surgical Competence. Acad Med . 2012;87(10). https://journals.lww.com/academicmedicine/fulltext/2012/10000/the_ottawa_surgical_competency_operating_room.24.aspx
» https://journals.lww.com/academicmedicine/fulltext/2012/10000/the_ottawa_surgical_competency_operating_room.24.aspx -
9 Rekman J, Gofton W, Dudek N, Gofton T, Hamstra SJ. Entrustability Scales: Outlining Their Usefulness for Competency-Based Clinical Assessment. Acad Med 2016;91(2). https://journals.lww.com/academicmedicine/fulltext/2016/02000/entrustability_scales__outlining_their_usefulness.18.aspx
» https://journals.lww.com/academicmedicine/fulltext/2016/02000/entrustability_scales__outlining_their_usefulness.18.aspx -
10 Rose S, Fix OK, Shah BJ, Jones TN, Szyjkowski RD. Entrustable professional activities for gastroenterology fellowship training. Gastroenterology 2014;147(1):233-242. doi:10.1053/j.gastro.2014.04.038
» https://doi.org/10.1053/j.gastro.2014.04.038 -
11 ten Cate O. Transforming primary care training - Patient-centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med 2013;28(6):801-809. doi:10.1007/s11606-012-2193-3
» https://doi.org/10.1007/s11606-012-2193-3 -
12 Weiss PG, Schwartz A, Carraccio CL, et al. Achieving entrustable professional activities during fellowship. Pediatrics 2021;148(5). doi:10.1542/peds.2021-050196
» https://doi.org/10.1542/peds.2021-050196 -
13 Cousar M, Huang J, Sebro R, Levin D, Prabhakar H. Too Scared to Teach? The Unintended Impact of 360-Degree Feedback on Resident Education. Curr Probl Diagn Radiol 2020;49(4):239-242. doi:https://doi.org/10.1067/j.cpradiol.2019.04.006
» https://doi.org/https://doi.org/10.1067/j.cpradiol.2019.04.006 -
14 Lim SY, Bolster MB. Challenges in Optimizing Medical Education for Rheumatologists. Rheum Dis Clin North Am 2019;45(1):127-144. doi:10.1016/j.rdc.2018.09.008
» https://doi.org/10.1016/j.rdc.2018.09.008 -
15 Karpinski J, Frank JR. The Role of EPAs in Creating a National System of Time-Variable Competency-Based Medical Education. Acad Med . 2021;96(7):S36-S41. doi:10.1097/ACM.0000000000004087
» https://doi.org/10.1097/ACM.0000000000004087 -
16 Hanson MN, Pryor AD, Jeyarajah DR, et al. Implementation of entrustable professional activities into fellowship council accredited programs: a pilot project. Surg Endosc 2023;37(4):3191-3200. doi:10.1007/s00464-022-09502-5
» https://doi.org/10.1007/s00464-022-09502-5
Source: preparaed by the authors.