Workplace Based Assessment in Obstetrics and Gynecology Clerkship to Detect Learning Gaps

43 (1) : 181-186; 2019 I Universidade Federal do Ceará, Fortaleza, Ceará, Brasil. II Universidade Federal do Rio Grande do Norte, Parnamirim, Rio Grande do Norte, Brasil. III Foundation for Advancement of international Medical Education and Research, Philadelphia, Pennsylvania, United States. IV Universidade de São Paulo, Ribeirão Preto, São Paulo, Brasil. Workplace Based Assessment in Obstetrics and Gynecology Clerkship to Detect Learning Gaps Avaliação Baseada em Cenários de Prática para Detectar Lacunas de Aprendizagem no Internato Médico em Ginecologia e Obstetrícia

test, and questionnaire responses were compared, to determine the extent to which learning objectives were achieved.Results: three faculty members assessed 84 medical students using the mini-CEX during the four-month clerkship.The scores for the physical examination and clinical judgment skills were lower, compared to those of the interviewing skills.Based on the students' feedback, ratings for physical examination and counseling preparation were rated as "inadequate", especially for the topic breaking bad news.The bivariate correlation between the mini-CEX skills and MCQ test scores showed a positive relationship (r = 0.27).Although they assess different skills on the Muller pyramid, there appears to be a relationship between "Knowing" and "Doing".These findings will help curriculum managers to identify important gaps in the rotation design and delivery.Based on these results, training in the skills workshop during the first month of the rotation was initiated, in addition to previous simulated training during the third year of medical course.Conclusions: The introduction of the mini-CEX offers critical information to identify and refine important curriculum elements in the clinical years.Based on this, physical examination and communications skills training were initiated in the skills workshop.

INTRODUCTION
Traditionally, undergraduate clinical curricula have been defined in terms of their duration, and syllabi have been developed based on content specification.Clear expectations of learning and performance were not considered.This has been changing, with a shift towards outcome-based education, which seeks to make learning, performance objectives and educational strategies explicit to students and faculty 1 .Such a change makes curriculum evaluation essential to ensure achievement of outcomes, and assessment needs to be part of ongoing curricular evaluation.A system of assessment supports curricular change and promotes the achievement of curricular goals.
In clinical education, it is important to provide structured training and feedback, but there are few teaching methods based on direct observation of students in real life situations, incorporating the perspective of peers and patients, or evaluating clinical outcomes 2 .Workplace-based assessment (WPBA) consists of direct observation of trainee performance in clinical settings, followed by the provision of focused feedback.These methods aim to define individual strengths and weaknesses, and to involve trainees in the reflection process.The results of WPBA should be interpreted in the context of other assessment methods, to obtain a comprehensive evaluation of professional competence.The mini-CEX is a snapshot observation of a clinical encounter, using a standard rating form 3,4 , and meets the goals of WPBA.Feedback coupled with educational interventions has been shown to be the best way to improve the quality of learning in the clinical context, both in residency and undergraduate medical education 5,6 .
The Medical School of the Federal University of Ceará (UFC) has a six-year curriculum.In the first four years, a discipline-based and community-oriented curriculum is used.A workplace based assessment WPBA was introduced in the Ob&Gyn rotation as part of a system of assessment at UFC.The objective of this study is to describe the use of mini-CEX to evaluate student preparation during an Ob&Gyn clerkship.

METHODS
A cross-sectional study was conducted involving faculty members and undergraduate medical students during the Ob&Gyn clerkship is intended.Students completed 16 weeks of clinical practice in OB&Gyn during the rotation.We used the results of the MCQ test, mini-CEX, and student ratings of their own proficiency to identify areas of strength and weakness in the OB&Gyn program.

Measures
Three faculty members individually evaluated 84 undergraduate medical students in the mini-CEX encounters over an eight-month period.Before using the mini-CEX, faculty members were trained using a Portuguese translation of the form 7 .The examiners rated the student using a 9-point rating scale, in which 1 to 3 were "unsatisfactory"; 4 to 6 were "satisfactory"; and 7 to 9 were "superior", with subsequent feedback.The faculty members were asked to rate student performance in medical interviewing, physical examination, clinical judgment, counseling skills, and overall clinical competence.
At the end of the four-month Ob&Gyn clerkship, the institution administers a MCQ test intended to assess knowledge.The MCQ test has 20 'single best answer questions' with 4 distracters.The scores range from 0 to 10, with a passing score of 5.
The student questionnaire included statements about their perceptions of whether relevant educational goals prepared them adequately: history taking, practical skills (obstetric and gynecological physical examination), basic procedures, clinical judgment and communication skills.Each statement started with the phrase: "The clerkship in Ob&Gyn helped me to be prepared for...".The students provided ratings of the extent to which they thought the preparation received during the clerkship was inadequate (1), partially adequate (2) or adequate (3).Higher scores indicated that the students' felt that the experience in the clerkship supported their preparation to perform that task, and consequently indicated that learning objectives related to that specific domain should have been achieved.

Analysis
The results of students' performance in the mini-CEX were analyzed to identify areas of strength and weakness in the OB&Gyn program.Data from the mini-CEX and the students' preparation perception scale were analysed, and mean ± standard deviation (SD) scores were calculated.The data were compared to identify how far the learning objectives had been achieved.
Statistical analyses were conducted using Sigma Plot for Windows Version 12.0 (SyStat Software, San Jose, CA).The level of significance was P < 0.05.
All participants signed an informed consent form.The Research Ethics Board of MEAC/UFC approved the research, under process number 039938/2013.

RESULTS
Three faculty members evaluated 84 students in 252 mini-CEX encounters; most were performed while the students were rotating in the inpatient wards (76.4%).Each student performed the multiple choice test and at least 2 mini-CEX assessments.The average ratings for the Ob&Gyn physical examination (mean = 6.6,SD = 1.5) were lower than those for interviewing skills (mean = 7.2, SD = 1.1) or counseling (mean = 7.4, SD = 1.3).Clinical judgment was another challenging domain (mean = 6.7,SD = 1.5) for the students.
Based on the perception of preparation questionnaire, 4% of the students said they were not prepared to interview patients; 16% to perform physical examinations; and 26% felt inadequately prepared to make clinical judgments at the end of the stage.Although fewer students reported feeling unprepared in history taking, many said they did not prepared to communicate specific topics, such as a cancer diagnosis, terminal illness, i.e. breaking bad news (62% felt their preparation was inadequate).In addition, the responses indicated that the students felt unprepared to perform basic procedures, such as urinary catheterization (69%), suturing (52%), and collecting Pap test/cervical smear samples (43%).The results of the preparation questionnaire are presented in Table 1.
The results of the knowledge test provided additional information about student peformance.The mean score was 6.1, and 78 students had a score of 5 or more.The bivariate correlation between the mini-CEX skills and MCQ test scores showed a positive relationship (r = 0.27; see Figure 1).

DISCUSSION
The results of the mini-CEX administration and students' perceptions of their competence were used to identify whether the Ob&Gyn curriculum enabled students to reach the expected outcomes.There was agreement between the mini-CEX results and students' perception about acquired competence for history taking in Ob&Gyn, which were higher than for the

Figure 1 Correlation between the Mini Clinical Evaluation Exercise (mini-CEX) mean and multiple-choice question (MCQ) scores
physical examination and clinical judgment.The strengths of the mini-CEX include the opportunity for extensive sampling across patients and settings, and training observation 3 .
In this study, we used the mini-CEX, MCQ tests and students' perceptions to identify whether the Ob&Gyn curriculum enabled students to reach the expected outcomes.The reproducibility of the mini-CEX assessment of performance was comparable to that of the written MCQ, even though these measurements assess different domains of competence at dif-ferent levels of simulations.Other authors reported a deeper approach to learning for the workplace-based assessment (WPBA), when compared to MCQs, with a positive correlation between a successful approach to learning and examination performance 8,9 .
Faculty development was critical to implementing the mini-CEX in the clerkship and achieving the desired results.The most challenging issue in the initial phase was to standardize the rating scores among the assessors in this study.The faculty members went through a two-month period of training and discussed the instrument and the meaning of the scoring rubric and domains.This type of professional development improves the consistency of the ratings, which is important to ensure that the results provide a reliable reflection of the students' ability and training 10,11 .
The results of the Mini-CEX were useful for curriculum review in the context of a quality improvement process.It also proved to be a powerful instrument for promoting a change in practices that favors the student-centered teaching model in the primary health care stage 12 .Indeed, previous study with medical interns revealed students' interest in expanding feedback practice after assessments in workplace.It was suggested investing in faculty development to improve the application of WPBA tools 13 .
One important finding was based on students' comments, and related to the adequacy of training for physical examination skills in OB&Gyn.The students reported a variation in how OB&Gyn techniques or maneuvers are taught.There was not as much variation in the instruction on history taking and counseling skills, based on the students' feedback.While this study supports the use of workplace assessment results and students' perceptions of their preparation, it is not without limitations.Unfortunately, not all students were able to participate in more than one mini-CEX encounter.While multiple observations are recommended for each student, this may not be feasible for rotations of shorter duration 11 .Attitude and professionalism were not a focus of this study.Although these domains could be included, additional professional development would be necessary to include these domains in the clerkship assessment.The data collection was transversal, therefore it was not possible to extrapolate the results of the study for different periods.Finally, the students' responses to the questionnaires were anonymous, therefore it was not possible to look for agreement between examiners' ratings of skills and individual students' performances.
However, additional instruments are being included in the evaluation of student performance.Collection of data from other WPBA tools will allow us to evaluate whether the practice provided in the skills workshop improves student performance.

CONCLUSIONS
Despite these limitations, the current investigation showed that the mini-CEX, already recognized as a powerful assessment tool for clnical settings, should be also considered as an approach to determine the coherence between the clinical learning objectives and the acquisition of skills by students.
The final two years consist of clerkship rotations in Public Health, Pediatrics, Surgery, Internal Medicine and Ob&Gyn.The goal of the medical clerkship is to develop clinical skills in contextualized practice.