Assessment of the Performance of Orthopedic Residents in Clinical Practice^*

Abstract

Objective

To evaluate the performance of orthopedic residents while conducting clinical examinations.

Methods

The Mini Clinical Evaluation Exercise (Mini-CEX) was applied by three teaching doctors at four different moments. The instrument was adapted by the authors for use in orthopedics, with the development of descriptors for each evaluated skill. Supervisors were trained to use the Mini-CEX by the principal investigator through teaching materials and discussions, with standardization of the instrument descriptors.

Results

The mean scores obtained in the 4 evaluations for each of the 21 residents reveal improvement in the performances of residents in all skills assessed from the 1^st to the 4^th meeting.

Conclusions

We have found that the performance of orthopedic residents presented a satisfactory evolution, with progressive improvement in all skills.

Keywords
learning; orthopedics; physical examination; teaching

Resumo

Objetivo

Avaliar o desempenho de residentes de ortopedia na realização de exames clínicos.

Métodos

O Mini-CEX foi aplicado por três docentes em quatro momentos distintos. O instrumento foi adaptado pelos autores para uso em ortopedia, com desenvolvimento de descritores para cada habilidade avaliada. Os supervisores foram treinados pelo pesquisador principal para utilização do Mini-CEX por meio de materiais didáticos e discussões e os descritores do instrumento foram padronizados.

Resultados

As pontuações médias obtidas nas quatro avaliações de cada um dos 21 residentes revelaram melhora no desempenho de todas as habilidades entre o primeiro e o quarto encontro.

Conclusões

O desempenho dos residentes em ortopedia apresentou evolução satisfatória, com melhora progressiva de todas as habilidades.

Palavras-chave
aprendizagem; ensino; exame físico; ortopedia

Introduction

Reliable methods for assessing clinical resident's performance are a problem faced by teaching specialists worldwide.¹ Direct observation is a good tool for analyzing them in the workplace, allowing for the opportunity to observe what they know and can do. However, although teaching physicians and residents work in the same place, observation is traditionally informal and performed inconsistently in almost all medical specialties.²

To encourage teaching doctors to perform observations, the American Board of Internal Medicine proposed the Mini Clinical Evaluation Exercise (Mini-CEX), a 10- to 20-minute direct observation assessment of trainees' clinical skills. The evaluation takes place during trainee-patient meetings; interaction between the resident and a patient is observed, after which the evaluator provides the resident with feedback in the form of one-page written assessment rating such skills as “overall clinical competence” and “humanistic qualities/professionalism,” which must be signed by both the observer and the trainee.³

This method has been used in such specialties as cardiology,⁴ anesthesia,⁵ pediatrics,⁶ internal medicine,⁷ and chiropractic.⁸ However, no studies on its use in orthopedics have been found. Thus, our objective was to evaluate the performance of orthopedic residents while conducting clinical examinations.

Methods

Study Design and Location

The study was a longitudinal prospective study, performed from January to July 2016, in two residency programs in orthopedics, accredited by the Ministry of Education and recognized by the Brazilian Society of Orthopedics and Traumatology, in a capital of the Central-West region of Brazil.

Participants

Nonprobability sampling was employed. We included the postgraduate first-year (R1), second-year (R2), and third-year (R3) orthopedic residents in the two residency programs that agreed to participate in the study, and the participants signed informed consent forms. Excluded from the study were those belonging to (i) nonaccredited services of the Brazilian Society of Orthopedics or (ii) trainees of subspecialties of orthopedics (R4), such as spine, knee, shoulder, and foot and ankle surgery.

Data Collection

The Mini-CEX was applied by three teaching physicians of the respective hospitals in orthopedics. They were trained to use the Mini-CEX by the principal investigator through teaching materials and discussions, with standardization of the instrument descriptors. The Mini-CEX was used during individual resident-patient interactions in the emergency room, infirmary, and ambulatory departments. Its application occurred on dates formally scheduled by the appraisers to ensure that all residents experienced the evaluation on the same day, in their respective places of operation. Each resident was evaluated at four different times, with an interval of at least 30 days between evaluations.

Instrument Used

The Mini-CEX, created by the American Board of Internal Medicine, presents six clinical skills, as follows: (i) medical interviewing (clinical history); (ii) physical examination, (iii) humanistic quality/professionalism; (iv) clinical judgment; (v) counselling skills (communication and advice); and (vi) organization/efficiency. The instrument was adapted by the authors for use in orthopedics, with the development of descriptors for each evaluated skill. The process of adaptation of the instrument took place in specific meetings, in which 4 teaching physicians with more than 10 years of participation in the residency program in orthopedics, and 2 heads of residency training of this specialty were present. After 5 meetings with an average duration of 120 minutes, the instrument was adapted and defined such that each skill had 9 descriptors. If the objective of the respective descriptor was achieved, the resident received a point, respecting the original's maximum of nine points. The scoring scale was divided into 1 to 3 (unsatisfactory); 4 to 6 (satisfactory), and 7 to 9 (outstanding). The methodology used by Abadie et al. was used to develop the descriptors.

Outcome

The dependent variable was the performance of the orthopedic residents in the physical examination, evaluated by the Mini-CEX instrument.

Independent Variables

The following variables were used: age (in years lived); gender (male/female), and year of residency (R1/R2/R3).

Availability of Data and Materials

The datasets used and analyzed during the current study are available from the corresponding author on reasonable request.

Data Analysis

The data were recorded in a Microsoft Excel (Microsoft Corp., Remdond, WA, USA) spreadsheet, and the statistical analyses were performed using the IBM SPSS Statistics for Windows, Version 23.0 (IBM Corp., Armonk, NY, USA). The quantitative variables were presented as mean and standard deviation. The total Cronbach α coefficient was calculated at each scale to evaluate the internal validity of the Mini-CEX instrument. Normality was verified using the Shapiro-Wilk test. Nonparametric statistics were applied if data were not normal. The evolution of the residents (R1, R2, and R3) throughout the duration of the study was performed using the Friedman test, followed by analysis of multiple posthoc comparisons. The probability of rejecting the null hypothesis was 5%.

Results

The total number of residents was 33, 12 of whom were excluded because 3 were R4, and 9 were nonaccredited. Thus, the final sample consisted of 21 residents, 7 in each year of residence, and all male. The mean age was 25.86 years ( ± 2.27) for the R1s, 27.50 years ( ± 1.74) for the R2s, and 27.14 ( ± 1.97) for the R3s.

The total Cronbach α coefficient was 0.88 when all the skills of the Mini-CEX instrument were considered, with a 95% confidence interval ranging from 0.79 to 0.94. The “clinical reasoning” and “clinical history” skills were the ones with the highest values, 0.83 (0.67–0.92 CI) and 0.76 (0.52–0.89 CI), respectively.

The general evaluations using the Mini-CEX of the performance of residents during the clinical examinations on the 4 test days are shown in ►Table 1. A comparison of the scores from the first evaluations with the last ones showed significant improvement of all the clinical skills.

Table 1
Assessment of clinical skills using the mini-CEX in four meetings

Competence	Meetings (mean ± standard deviation)				p *
	First	Second	Third	Fourth
Physical examination	2.67 ± 1.02d	3.71 ± 0.96c	5.86 ± 0.96b	6.57 ± 1.25a	< 0.001
Medical interview skill	2.86 ± 1.39d	3.95 ± 1.02c	6.00 ± 0.95b	7.00 ± 0.89a	< 0.001
Clinical judgment	4.19 ± 1.33c	4.90 ± 0.83c	6.57 ± 1.16b	7.38 ± 0.86a	< 0.001
Humanistic quality	4.14 ± 1.24c	5.19 ± 1.03b	6.14 ± 1.11a	6.76 ± 1.04a	< 0.001

• Friedman test followed by Post-Hoc analysis.
• ^*
  Different letters indicate significant differences.

The mean scores obtained in the 4 evaluations for each of the 21 residents are presented in ►Table 2, broken out individually by residency year (R1, R2, R3) and skills. The results reveal improvement in the performances of the R1, R2, and R3 residents in all skills assessed from the first to the fourth evaluation.

Table 2
Mean scores obtained in four meetings according to resident's postgraduate year

Competence	Y*	First	Second	Third	Fourth	p **
Physical examination	R1	1.86 ± 1.07d	3.00 ± 0.58c	4.86 ± 0.90b	6.29 ± 0.49a	<0.001
	R2	3.17 ± 0.75b	3.67 ± 1.21b	6.33 ± 0.52a	6.83 ± 0.75a	0.001
	R3	3.00 ± 0.76b	4.38 ± 0.52b	6.38 ± 0.52a	6.63 ± 1.92a	0.001
Medical interview skill	R1	1.43 ± 1.13c	3.00 ± 0.58c	5.14 ± 1.07b	6.29 ± 0.49a	0.001
	R2	3.50 ± 0.84c.b	4.50 ± 1.22b	6.17 ± 0.41a	7.00 ± 0.89a	< 0.001
	R3	3.63 ± 0.92c	4.38 ± 0.52c	6.63 ± 0.52b	7.63 ± 0.74a	0.001
Clinical judgment	R1	2.86 ± 1.07c	4.29 ± 0.49c	5.57 ± 1.13a	6.57 ± 0.53a	0.001
	R2	4.67 ± 1.03c	5.33 ± 0.82c	6.67 ± 0.82b	7.83 ± 0.75a	0.001
	R3	5.00 ± 0.76b	5.13 ± 0.83b	7.38 ± 0.74a	7.75 ± 0.71a	< 0.001
Humanistic quality	R1	3.71 ± 1.11b	4.86 ± 1.21b	5.71 ± 1.11a	6.71 ± 1.11a	0.001
	R2	4.17 ± 1.47b	5.00 ± 1.26b	6.00 ± 1.10a	6.33 ± 0.52a	0.002
	R3	4.50 ± 1.20d	5.63 ± 0.52c	6.63 ± 1.06b	7.13 ± 1.25a	< 0.001

• ^**
  Friedman test followed by Posthoc analysis.
• ^*
  Y – postgraduate year. Different letters indicate significant differences. R1–first year resident, R2–second year resident, R3–third year resident.

Discussion

A comparison of the mean scores of the Mini-CEX scores of the four evaluations showed that there was a significant improvement in the performance of the residents. It is interesting to note that this evidence contrasts with what existing data in the literature. A systematic review of 119 papers on the tools for evaluation and direct observation of clinical skills, using the same instrument, showed that there was no improvement in clinical skills and patient care.⁹

Another study, published in Pediatrics, used the Mini-CEX in 4 to 6 evaluations of 23 residents over a year. In that study, the clinical skill that showed the most improvement was humanistic qualities/professionalism, and the skill with the least improvement was physical examination.¹⁰ This partially corroborates our findings as it supports a general progression over the course of the evaluation; however, it differs from our findings in that we found significant improvement in the residents' physical examination skill: the scores increased from 2.67 for the first test day to 6.57 (p< 0.001) for the last test day. Physical examination and medical interviewing were the skills that our study found to improve most over the course of the evaluations. This can be explained by differences in feedback styles and focus (in our study, more feedback was given on physical examination) between the different Mini-CEX studies.

Other medical specialties also presented satisfactory results in relation to physical examination skills. Another study's evaluations of pediatric residents found an average of 6.1 for this skill.⁶ Similar results were found in 108 cardiology residents, with a mean of 7.1, 7.5, 7.5, and 8.0 found in R1, R2, R3, and R4 residents, respectively.⁴

The present study shows that the average score for physical examination skills for the R1s on the first test day was the lowest one found. This can be attributed to the R1s' relative newness to the specialty, less time in theoretical/practical classes, and less observation by supervisors in their consultations. In addition, the first evaluation happened shortly after the R1s' entry in the of “orthopedics program, when they were only equipped with the knowledge acquired in the undergraduate level; this suggests a lack of specialty teaching during medical school and the need to correct this gap during medical residency.

It can be observed that the residents showed improvement in their clinical skills during the serial evaluations; however, the R2s and R3s did not evolve from the third to the fourth assessment in the area of physical examination skills, probably due to having been in the specialty for a longer time and having already achieved greater skills and satisfactory scores at the time of the study.

It is important that orthopedic residents perform optimally, especially in the physical examination. To achieve this goal, it is necessary for teaching physicians to use the best teaching and learning resources possible. Teaching is most effective when residents are involved both physically and mentally—namely, through direct involvement in patient care—by encouraging residents to take notes while studying, especially about relevant questions to ask supervisors and/or patients.¹¹ Another way to improve residents' performances would be to focus on their “development zone”—that is, to start from the exact point of their gaps or misconceptions rather than teaching what is already known. Thus, it is crucial to identify the limits of residents' knowledge.¹¹

Teaching orthopedic physical examination is challenging. There are several variables involved in the process, such as the clinical skill of the supervisor, the willingness of the patient to be evaluated in a group as an illustrative example as physical examination signs are taught, and the interests of the residents. The best strategy for teaching this skill is to start with a supervisor with top-level skills and a strong aptitude for teaching. Attributing this function to someone without either ability would only perpetuate residents' bad habits.

Another effective strategy to achieve a quantitative increase in skill is teaching orthopedic physical examination at the bedside. This is perhaps one of the best ways to promote its improvement. This learning scenario occurred routinely in the 1960s, but it became infrequent in the 1990s.¹² Physical examination skills cannot be taught effectively in theory lectures without the presence of a real or simulated patient.⁹

The present study becomes relevant in the context of a lack of research in this area, showing the positive evolution of the performance of residence with the Mini-CEX, as no similar studies were found related to orthopedics. Our findings contribute to the knowledge of this subject, with special significance for our target population. We performed a longitudinal and prospective study, without memory bias, which gave the data obtained greater credibility. Another strength of the study would be the chosen evaluation instrument. The Mini-CEX was designed, based on (and is used in) real situations, to be a comprehensive yet streamlined tool for evaluating clinical skills and is, therefore, a distinguishing instrument. The Mini-CEX also enables evaluations in different scenarios and with different levels of complexity.³

Because there was no previous sample calculation of the subjects to be evaluated, the final nonprobability sample was considered small, coming from only two hospitals. Statistically, nonrandom selection procedures may not guarantee representativeness; consequently, our findings should not be generalized to a broader population. Despite the need for new studies on the validity and feasibility of the Mini-CEX in the Portuguese language, this study did demonstrate that the instrument tested has a high reliability and internal consistency.

Conclusion

This study found that the performance of residents of orthopedics and traumatology during clinical examinations presented a progressive improvement in all skills, regardless of the year of residency. The research favors the development of other studies to broaden the understanding of the phenomenon studied or to empirically confirm the results obtained.

Acknowledgments

We would like to thank all the residents and teaching physicians that participated in this study and accepted to be observed during evaluation sessions and to fill the questionnaire. This study has been possible thanks to their generosity in sharing their beliefs and opinions about this assessment approach.

Ethical Considerations

The work was approved by the Ethics and Research Committee, under the number 1,326,398, on November 9, 2015, and registered in the Platform Brazil, CAAE protocol number: 49761515.1.0000.5078. All participants included in the study agreed to participate and signed the informed consent form.

• Financial Support
  The present work did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. No funding was obtained for this study.

References

• ¹
  Pitts D, Rowley DI, Sher JL. Assessment of performance in orthopaedic training. J Bone Joint Surg Br 2005;87(09):1187-1191
• ²
  Fromme HB, Karani R, Downing SM. Direct observation in medical education: a review of the literature and evidence for validity. Mt Sinai J Med 2009;76(04):365-371
• ³
  Norcini JJ, Blank LL, Duffy FD, Fortna GS. The mini-CEX: a method for assessing clinical skills. Ann Intern Med 2003;138(06):476-481
• ⁴
  Alves de Lima A, Barrero C, Baratta S, et al. Validity, reliability, feasibility and satisfaction of the Mini-Clinical Evaluation Exercise (Mini-CEX) for cardiology residency training. Med Teach 2007;29(08):785-790
• ⁵
  Weller JM, Jolly B, Misur MP, et al. Mini-clinical evaluation exercise in anaesthesia training. Br J Anaesth 2009;102(05): 633-641
• ⁶
  Abadie Y, Battolla J, Zubieta A, et al. Uso de descriptores durante la implementación de Mini-CEX en la residencia de pediatría. Medicina (B Aires) 2015;75(05):289-296
• ⁷
  Malhotra S, Hatala R, Courneya CA. Internal medicine residents’ perceptions of the Mini-Clinical Evaluation Exercise. Med Teach 2008;30(04):414-419
• ⁸
  Paravicini I, Peterson CK. Introduction, development, and evaluation of the miniclinical evaluation exercise in postgraduate education of chiropractors. J Chiropr Educ 2015;29(01):22-28
• ⁹
  Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA 2009;302(12):1316-1326
• ¹⁰
  Goel A, Singh T. The usefulness of Mini Clinical Evaluation Exercise as a learning tool in different pediatric clinical settings. Int J Appl Basic Med Res 2015;5(Suppl 1):S32-S34
• ¹¹
  Pinney SJ, Mehta S, Pratt DD, et al. Orthopaedic surgeons as educators. Applying the principles of adult education to teaching orthopaedic residents. J Bone Joint Surg Am 2007;89(06): 1385-1392
• ¹²
  Ende J. What if Osler were one of us? Inpatient teaching today. J Gen Intern Med 1997;12(02, Suppl 2):S41-S48

Publication Dates

• Publication in this collection
  13 Jan 2023
• Date of issue
  Nov-Dec 2022

History

• Received
  08 Sept 2021
• Accepted
  20 Jan 2022