Workplace Based Assessment in Obstetrics and Gynecology Clerkship to Detect Learning Gaps

Avaliação Baseada em Cenários de Prática para Detectar Lacunas de Aprendizagem no Internato Médico em Ginecologia e Obstetrícia

ABSTRACT

Background

An outcome-based curriculum helps to communicate expectations of performance to students and clinical teachers. The Mini Clinical Evaluation Exercise (mini-CEX) is a useful tool for workplace-based formative assessment. The objective of this study was to use workplace-based assessment and student feedback to evaluate an Obstetrics and Gynecology (Ob&Gyn) clerkship curriculum.

Methods

A cross-sectional study was conducted with faculty members and medical students in an Ob&Gyn clerkship. The Mini-CEX was introduced into the clerkship assessment system, together with multiple choice question (MCQ) tests. This tool evaluates the history collection, physical examination, clinical judgment, professionalism and humanism, and also gives an overall score at the end of the test. At the end of the rotation, questionnaires were used to collect the students’ perceptions about their skills acquisition during the program. The results of the Mini-CEX, the MCQ test, and questionnaire responses were compared, to determine the extent to which learning objectives were achieved.

Results

three faculty members assessed 84 medical students using the mini-CEX during the four-month clerkship. The scores for the physical examination and clinical judgment skills were lower, compared to those of the interviewing skills. Based on the students’ feedback, ratings for physical examination and counseling preparation were rated as “inadequate”, especially for the topic breaking bad news. The bivariate correlation between the mini-CEX skills and MCQ test scores showed a positive relationship (r = 0.27). Although they assess different skills on the Muller pyramid, there appears to be a relationship between “Knowing” and “Doing”. These findings will help curriculum managers to identify important gaps in the rotation design and delivery. Based on these results, training in the skills workshop during the first month of the rotation was initiated, in addition to previous simulated training during the third year of medical course.

Conclusions

The introduction of the mini-CEX offers critical information to identify and refine important curriculum elements in the clinical years. Based on this, physical examination and communications skills training were initiated in the skills workshop.

Clinical Competence; Program Evaluation; Educational Measurement

RESUMO

Contexto

Currículo Baseado em Desfechos ajuda a comunicar as expectativas de desempenho entre alunos e professores. O Miniexercício de Avaliação Clínica (Miniex) é uma ferramenta útil para avaliação formativa no local de trabalho. O objetivo deste estudo foi usar avaliação em cenários reais de prática e feedback dos alunos para qualificar um programa de internato médico em Ginecologia e Obstetrícia (GO).

Métodos

Foi realizado um estudo transversal com docentes e estudantes do internato médico em GO. O Miniex foi introduzido no sistema de avaliação de estágio adicionado a testes de perguntas de múltipla escolha (MCQ). Esta ferramenta avalia a coleta de histórico, o exame físico, o julgamento clínico, o profissionalismo e o humanismo, além da escala global no final do teste. No final da rotação de quatro meses, questionários foram aplicados para coletar a percepção do aluno sobre a aquisição de habilidades durante o programa. Os resultados do Miniex, o teste de MCQ e as respostas aos questionários foram comparados para determinar a medida em que os objetivos de aprendizagem foram alcançados.

Resultados

Três docentes avaliaram 84 estudantes de Medicina usando Miniex ao longo do estágio de quatro meses. As pontuações médias para exame físico e habilidades de julgamento clínico foram menores em comparação com as habilidades de anamnese e aconselhamento. Com base na percepção dos alunos, o preparo para exame físico e o aconselhamento foram classificados como “inadequados”, especialmente quanto à divulgação de más notícias. A correlação bivariada entre as habilidades do Miniex e os escores do teste MCQ mostrou uma relação positiva (r = 0,27). Apesar de avaliar as diferentes habilidades da pirâmide de Muller, parece haver uma relação entre “knows” e “does”. Essas descobertas ajudaram os coordenadores de internato e docentes a identificar lacunas importantes no programa do estágio. Com base nesses resultados, foi iniciado um treinamento no laboratório de habilidades durante o primeiro mês de rotação para todos os alunos ingressantes, adicionado ao treinamento simulado anterior durante o terceiro ano do curso de Medicina.

Conclusões

A introdução do Miniex oferece informações críticas para identificar e aprimorar elementos curriculares importantes nos anos clínicos.

Competência Clínica; Avaliação de Programas e Projetos de Saúde; Avaliação Educacional

INTRODUCTION

Traditionally, undergraduate clinical curricula have been defined in terms of their duration, and syllabi have been developed based on content specification. Clear expectations of learning and performance were not considered. This has been changing, with a shift towards outcome-based education, which seeks to make learning, performance objectives and educational strategies explicit to students and faculty¹. Such a change makes curriculum evaluation essential to ensure achievement of outcomes, and assessment needs to be part of ongoing curricular evaluation. A system of assessment supports curricular change and promotes the achievement of curricular goals.

In clinical education, it is important to provide structured training and feedback, but there are few teaching methods based on direct observation of students in real life situations, incorporating the perspective of peers and patients, or evaluating clinical outcomes². Workplace-based assessment (WPBA) consists of direct observation of trainee performance in clinical settings, followed by the provision of focused feedback. These methods aim to define individual strengths and weaknesses, and to involve trainees in the reflection process. The results of WPBA should be interpreted in the context of other assessment methods, to obtain a comprehensive evaluation of professional competence. The mini-CEX is a snapshot observation of a clinical encounter, using a standard rating form^3,4, and meets the goals of WPBA. Feedback coupled with educational interventions has been shown to be the best way to improve the quality of learning in the clinical context, both in residency and undergraduate medical education^5,6.

The Medical School of the Federal University of Ceará (UFC) has a six-year curriculum. In the first four years, a discipline-based and community-oriented curriculum is used. The final two years consist of clerkship rotations in Public Health, Pediatrics, Surgery, Internal Medicine and Ob&Gyn. The goal of the medical clerkship is to develop clinical skills in contextualized practice.

A workplace based assessment WPBA was introduced in the Ob&Gyn rotation as part of a system of assessment at UFC. The objective of this study is to describe the use of mini-CEX to evaluate student preparation during an Ob&Gyn clerkship.

METHODS

A cross-sectional study was conducted involving faculty members and undergraduate medical students during the Ob&Gyn clerkship is intended. Students completed 16 weeks of clinical practice in OB&Gyn during the rotation. We used the results of the MCQ test, mini-CEX, and student ratings of their own proficiency to identify areas of strength and weakness in the OB&Gyn program.

Measures

Three faculty members individually evaluated 84 undergraduate medical students in the mini-CEX encounters over an eight-month period. Before using the mini-CEX, faculty members were trained using a Portuguese translation of the form⁷. The examiners rated the student using a 9-point rating scale, in which 1 to 3 were “unsatisfactory”; 4 to 6 were “satisfactory”; and 7 to 9 were “superior”, with subsequent feedback. The faculty members were asked to rate student performance in medical interviewing, physical examination, clinical judgment, counseling skills, and overall clinical competence.

At the end of the four-month Ob&Gyn clerkship, the institution administers a MCQ test intended to assess knowledge. The MCQ test has 20 ‘single best answer questions’ with 4 distracters. The scores range from 0 to 10, with a passing score of 5.

The student questionnaire included statements about their perceptions of whether relevant educational goals prepared them adequately: history taking, practical skills (obstetric and gynecological physical examination), basic procedures, clinical judgment and communication skills. Each statement started with the phrase: “The clerkship in Ob&Gyn helped me to be prepared for...”. The students provided ratings of the extent to which they thought the preparation received during the clerkship was inadequate (1), partially adequate (2) or adequate (3). Higher scores indicated that the students’ felt that the experience in the clerkship supported their preparation to perform that task, and consequently indicated that learning objectives related to that specific domain should have been achieved.

Analysis

The results of students’ performance in the mini-CEX were analyzed to identify areas of strength and weakness in the OB&Gyn program. Data from the mini-CEX and the students’ preparation perception scale were analysed, and mean ± standard deviation (SD) scores were calculated. The data were compared to identify how far the learning objectives had been achieved.

Statistical analyses were conducted using Sigma Plot for Windows Version 12.0 (SyStat Software, San Jose, CA). The level of significance was P < 0.05.

All participants signed an informed consent form. The Research Ethics Board of MEAC/UFC approved the research, under process number 039938/2013.

RESULTS

Three faculty members evaluated 84 students in 252 mini-CEX encounters; most were performed while the students were rotating in the inpatient wards (76.4%). Each student performed the multiple choice test and at least 2 mini-CEX assessments. The average ratings for the Ob&Gyn physical examination (mean = 6.6, SD = 1.5) were lower than those for interviewing skills (mean = 7.2, SD = 1.1) or counseling (mean = 7.4, SD = 1.3). Clinical judgment was another challenging domain (mean = 6.7, SD = 1.5) for the students.

Based on the perception of preparation questionnaire, 4% of the students said they were not prepared to interview patients; 16% to perform physical examinations; and 26% felt inadequately prepared to make clinical judgments at the end of the stage. Although fewer students reported feeling unprepared in history taking, many said they did not prepared to communicate specific topics, such as a cancer diagnosis, terminal illness, i.e. breaking bad news (62% felt their preparation was inadequate). In addition, the responses indicated that the students felt unprepared to perform basic procedures, such as urinary catheterization (69%), suturing (52%), and collecting Pap test/cervical smear samples (43%). The results of the preparation questionnaire are presented in Table 1.

TABLE 1
Scores for student perception (n = 84) on learning domains to Ob&Gyn clerkship

Learning domains		Inappropriate n (%)	Partially adequate n (%)	Adequate n (%)
1. Hystory taking	Anamnesis	4 (4.8)	38 (45.2)	42 (50)
2. Practical skills and basic procedures	Physical examination	16 (19)	44 (52.4)	24 (28.6)
	Collecting Papanicolaou	36 (42.9)	30 (35.7)	18 (21.4)
	Urinary catheterization	58 (69)	16 (19)	10 (11.9)
	Delivery assistance	24 (28.6)	36 (42.9)	24 (28.6)
	Suturing	44 (52.4)	28 (33.3)	12 (14.3)
	Dealing with medical emergencies	30 (35.7)	38 (45.2)	16 (19)
3.Clinical judgment	Clinical knowledge and use of diagnostic tools for judgment	22 (26.2)	54 (64.3)	8 (9.5)
4.Counseling and communication skills	Encourage better health habits	14 (16.7)	32 (38.1)	38 (45.2)
	Prevention diseases education	20 (23.8)	26 (31.0)	38 (45.2)
	Bad news	52 (61.9)	24 (28.6)	8 (9.5)
	Dealing with terminal patients	64 (76.2)	16 (19.0)	4 (4.8)

The results of the knowledge test provided additional information about student peformance. The mean score was 6.1, and 78 students had a score of 5 or more. The bivariate correlation between the mini-CEX skills and MCQ test scores showed a positive relationship (r = 0.27; see Figure 1).

FIGURE 1
Correlation between the Mini Clinical Evaluation Exercise (mini-CEX) mean and multiple-choice question (MCQ) scores

DISCUSSION

The results of the mini-CEX administration and students’ perceptions of their competence were used to identify whether the Ob&Gyn curriculum enabled students to reach the expected outcomes. There was agreement between the mini-CEX results and students’ perception about acquired competence for history taking in Ob&Gyn, which were higher than for the physical examination and clinical judgment. The strengths of the mini-CEX include the opportunity for extensive sampling across patients and settings, and training observation³.

In this study, we used the mini-CEX, MCQ tests and students’ perceptions to identify whether the Ob&Gyn curriculum enabled students to reach the expected outcomes. The reproducibility of the mini-CEX assessment of performance was comparable to that of the written MCQ, even though these measurements assess different domains of competence at different levels of simulations. Other authors reported a deeper approach to learning for the workplace-based assessment (WPBA), when compared to MCQs, with a positive correlation between a successful approach to learning and examination performance^8,9.

Faculty development was critical to implementing the mini-CEX in the clerkship and achieving the desired results. The most challenging issue in the initial phase was to standardize the rating scores among the assessors in this study. The faculty members went through a two-month period of training and discussed the instrument and the meaning of the scoring rubric and domains. This type of professional development improves the consistency of the ratings, which is important to ensure that the results provide a reliable reflection of the students’ ability and training^10,11.

The results of the Mini-CEX were useful for curriculum review in the context of a quality improvement process. It also proved to be a powerful instrument for promoting a change in practices that favors the student-centered teaching model in the primary health care stage¹². Indeed, previous study with medical interns revealed students’ interest in expanding feedback practice after assessments in workplace. It was suggested investing in faculty development to improve the application of WPBA tools¹³.

One important finding was based on students’ comments, and related to the adequacy of training for physical examination skills in OB&Gyn. The students reported a variation in how OB&Gyn techniques or maneuvers are taught. There was not as much variation in the instruction on history taking and counseling skills, based on the students’ feedback. The survey results also pointed to a lack of opportunities to learn more about breaking bad news during the rotation. In addition, the Ob&Gyn physical examination and basic procedures, such as urinary catheterization and performing the Pap test/cervical smear, were shown to be weaknesses in the curriculum. Based on these results, new sessions in the skills workshop were introduced. Sessions in the lab focused on OB&Gyn physical examination, breaking bad news and important procedures including urinary catheterization, suturing, and collecting the Pap test/cervical smear.

While this study supports the use of workplace assessment results and students’ perceptions of their preparation, it is not without limitations. Unfortunately, not all students were able to participate in more than one mini-CEX encounter. While multiple observations are recommended for each student, this may not be feasible for rotations of shorter duration¹¹. Attitude and professionalism were not a focus of this study. Although these domains could be included, additional professional development would be necessary to include these domains in the clerkship assessment. The data collection was transversal, therefore it was not possible to extrapolate the results of the study for different periods. Finally, the students’ responses to the questionnaires were anonymous, therefore it was not possible to look for agreement between examiners’ ratings of skills and individual students’ performances.

However, additional instruments are being included in the evaluation of student performance. Collection of data from other WPBA tools will allow us to evaluate whether the practice provided in the skills workshop improves student performance.

CONCLUSIONS

Despite these limitations, the current investigation showed that the mini-CEX, already recognized as a powerful assessment tool for clnical settings, should be also considered as an approach to determine the coherence between the clinical learning objectives and the acquisition of skills by students.

ACKNOWLEDGEMENTS

The authors thank to FAIMER Brazil and Federal University of Ceará administration.

REFERENCES

• ¹
  Harden RM. Outcome-Based Education: the future is today. Med Teach 2007;29(7):625-9.
• ²
  Miller A, Archer J. Impact of workplace based assessment on doctors’ education and performance: a systematic review. BMJ 2010; 341:c5064.
• ³
  Norcini JJ, Blank LL, Duffy FD, Fontana GS. The mini-CEX: A method for assessing clinical skills. Ann Intern Med 2003; 138(6):476–81.
• ⁴
  Norcini J, McKinley DW. Assessment methods in medical education. Teaching and Teacher Education. 2007; 23(3): 239–50
• ⁵
  Veloski J, Boex JR, Grasberger MJ, Evans A, Wolfson DB. Systematic review of the literature on assessment, feedback and physicians’ clinical performance. BEME Guide No. 7. Med Teach 2006;28(2):117–28.
• ⁶
  Mitchell C, Bhat S, Herbert A, Baker P. Workplace-based assessments of junior doctors: do scores predict training difficulties? Medical Education. 2011; 45(12):1190–1198
• ⁷
  Megale L, Gontijo ED, Motta JAC.Avaliação de Competência Clínica emEstudantes de Medicina pelo Miniexercício Clínico Avaliativo (Miniex). Revista Brasileira de Educação Médica. 2009;33 (2):166 – 175.
• ⁸
  Cobb KA, Brown G, Jaarsma DA, Hammond RA.The educational impact of assessment: a comparison of DOPS and MCQs.MedTeach. 2013 Nov;35(11):e1598-607.
• ⁹
  Cilliers FJ, Schuwirth LW, van der Vleuten CPM. A model of the pre- assessment learning effects of assessment is operational in an under- graduate clinical context. BMC Med Educ. 2012;12:9.
• ¹⁰
  Liao KC, Pu SJ, Liu MS, Yang CW, Kuo HP.Development and implementation of a mini-Clinical Evaluation Exercise (mini-CEX) program to assess the clinical competencies of internal medicine residents: from faculty development to curriculum evaluation.BMC MedEduc 2013; 13:31-37.
• ¹¹
  Gingerich A, Ramlo SE, van der Vleuten CP, Eva KW, Regehr G.Inter-rater variability as mutual disagreement: identifying raters’ divergent points of view. Adv Health SciEducTheoryPract 2016; 1-20.
• ¹²
  Mendes AO, Viveiros LB. O uso do Miniex como ferramenta de feedback continuado para residente de medicina de família e comunidade em estágio na atenção primária à saúde. RevBrasMedFam Comunidade. 2018; 13(40):1-9.
• ¹³
  Sousa AP, Heinisch RH. Estudo sobre a Avaliação Aplicada no Internato em ClínicaMédica da Unisul. Revista Brasileira de Educação Médica. 2012;36 (1):68 – 76.

Publication Dates

• Publication in this collection
  Jan-Mar 2019

History

• Received
  18 Sept 2018
• Accepted
  1 Oct 2018