Results of the Last 5 Years (2018-2022) of the Specialist Title Exam of The Brazilian College of Surgeons

ABSTRACT The article discusses the evolution of the Brazilian College of Surgeons (CBC) specialist title exam, highlighting the importance of evaluating not only theoretical knowledge, but also the practical skills and ethical behavior of candidates. The test was instituted in 1971, initially with only the written phase, and later included the oral practical test, starting with the 13th edition in 1988. In 2022, the assessment process was improved by including the use of simulated stations in the practical test, with the aim of assessing practical and communication skills, as well as clinical reasoning, in order to guarantee excellence in the assessment of surgeons training. The aim of this study is to demonstrate the performance of candidates in the last five years of the Specialist Title Test and to compare the performance results between the different surgical training groups of the candidates. The results obtained by candidates from the various categories enrolled in the test in the 2018 to 2022 editions were analyzed. There was a clear and statistically significant difference between doctors who had completed three years of residency recognized by the Ministry of Education in relation to the other categories of candidates for the Specialist Title..


INTRODUCTION
T he Brazilian College of Surgeons (CBC), founded in 1929, began the selection process for its Title of Specialist in 1971, with only the written phase with multiple-choice questions, validated by RESOLUTION No.

804/1977 of the Brazilian Federal Medical Board (CFM).
From its 13th edition, held in 1988, the oral test was included in the evaluation process as a second phase for candidates approved in the written test.
The public notices of the competitions have been regulated by the Brazilian Medical Association (AMB) since 1991, through an agreement between CBC, AMB, CFM, and the National Commission of Medical Residency (CNRM) of the Ministry of Education (MEC).The CBC's Specialist Title Commission (COTECIG) has been improving its evaluation process every year.In the first (written) phase, specific questions are requested from invited professors, while in the second (oral) phase, clinical cases use imaging exams to interpret findings and intraoperative situations for diagnostic and therapeutic decision-making.
Another important step was taken in the evaluation process of candidates for the CBC Specialist Title in the 2022 edition, with the insertion of a practical test using simulated stations in the second phase, complementing the oral test.

Educational A B S T R A C T A B S T R A C T
The article discusses the evolution of the Brazilian College of Surgeons (CBC) specialist title exam, highlighting the importance of evaluating not only theoretical knowledge, but also the practical skills and ethical behavior of candidates.The test was instituted in 1971, initially with only the written phase, and later included the oral practical test, starting with the 13th edition in 1988.In 2022, the assessment process was improved by including the use of simulated stations in the practical test, with the aim of assessing practical and communication skills, as well as clinical reasoning, in order to guarantee excellence in the assessment of surgeons training.The For the learning objectives that need to be evaluated, the concept of performance was adopted, i.e., the actions effectively manifested, in which the student should have a given performance, defining the level considered acceptable (Gontijo et al., 2013).The best way to evaluate performance is to define the task to be performed in each context and verify the students' ability to appropriately apply their competences to perform the task in that situation (Aguiar & Ribeiro, 2010).Thus, the choice of evaluation methods requires clarity of objectives, as well as knowledge of their psychometric properties to identify which ones should be used at each moment (Wass et al.,   2007).
The implementation of the competency-based model (Epstein & Hundert, 2002) implies measures by evaluation methods (Norcini et al., 2013).A widely used approach to facilitate such measures was proposed by Miller, who grouped similar competencies into broader domains.Figure 1 shows the updated version of Miller's Pyramid (Miller, 1990; Boursicot et al., 2011 indicates what the professionals can do -their clinical skills (Boursicot et al., 2011).This includes psychomotor skills (physical examination maneuvers, medical procedures, etc.) and behavioral skills (communicating with patients and colleagues, complying with patient safety procedures, etc.).On the other hand, "Does" describes the real physician's performance: how one really performs one's functions amid the pressures of a dynamic environment, with its uncertainties and subjectivities, and one's ethical behavior in the face of emotionally complex situations and If the Specialist Title test had only written questions, its validity would be compromised, as it would exclude all practical skills from the evaluation.Thus, the definition of a cognitive (theoretical) test as an assessment of "knowledge, skills, and attitudes" is meaningless.To approximate these objectives, assessments should, at a minimum, include simulation scenarios (such as OSCEs) to assess clinical skills.Of course, this design is more expensive and laborious, requiring abundant inputs and human resources, which demands a logistical study on the evaluations' feasibility.However, considering the relevance of the objectives set, this greater investment is essential, as it is not possible to broadly assess the aptitude of candidates with inferior technical quality resources.

Reliability
Indicates whether the result of the test allows inferences about the candidates' proficiency in the domain being evaluated (Shumway & Harden, 2003) and for this a good sampling of the content in question and a well-standardized correction are essential.
Validity Ability to effectively assess what "should" be assessed; also, describes the power of the test to identify the varying levels of candidates' proficiency.
Trust Related to the precision, accuracy, objectivity, and reproducibility of the assessment instruments used, components that determine the reliability and consistency of the results obtained.

Acceptability
Indicates whether different actors (candidates, evaluators, and managers) agree on the format and content of the evaluation.

Feasibility/Viability
Considers the varied requirements (human and material resources, space conditions, time, support material, with adequate planning, convenient organization, and appropriate cost control) necessary to implement a test (Shumway &  Harden, 2003).

Educational impact
Relates to the consequences the test produces on the learning process (Wass et  al, 2007).In the first phase test, the focus of the evaluation was cognitive knowledge through questions that cover all areas of broad knowledge that compose the matrix of contents of General Surgery.The questions were elaborated and selected by COTECIG members and invited collaborators.
The test was applied by a specialized company to all candidates at the same time and with the same rules.In the first three simulated stations, the evaluators were face-to-face and in the fourth station, 60 online evaluators (members of the CBC with a specialist title) from all over the country were invited.

Statistical analysis
For the analyses, we excluded candidates who were absent from at least one of the exam's phases.
Data distribution was assessed using the Shapiro-Wilk test.Continuous data were compared using the one-way ANOVA test followed by the Bonferroni test.For nonparametric distribution data, we used the Kruskal-Wallis test followed by the Dunn test.Differences between proportions were analyzed using the chi-square test or the Fisher's exact test.We considered differences with an alpha equal to 5% as statistically significant.The analyses were performed using the SPSS (Statistical Package for the Social Sciences) software, version 21.0 (IBM Corp., Armonk, NY, USA) and the GraphPad Prism software, version 9.5.0 (GraphPad Software, San Diego, CA, USA).

RESULTS
The number of candidates registered in the CBC Specialist Title exams from 2018 to 2022 were, respectively, 147, 135, 155, 127, and 215, totaling 779 candidates.Fifty candidates were excluded from the examinations due to absence from at least one of the selection phases.Thus, we studied 729 candidates.
We consolidated the results of the candidates' performances, enrolled in categories A to E in the written tests of the first phase of all editions of the study (2018 to 2022), according to the category of registration, as shown in Table 2.
Table 3 shows the results of failure in the first and second phases, as well as the result of approval in the CBC Specialist Title Test, according to the candidates' registration category.Comparing the results of the candidates by enrollment category, mainly category A (three years of MEC residency) with category D (three years of CBC training), we see that MEC residents (category A) are approved more than 3 times more (73.8%vs. 23.8%).
As for failure, 57.1% of the three-year CBC trainees failed in the first phase (written test) and 19% in the second phase (practical test).In the case of R3 MEC, the failure rate in the first phase is 12.3%, and in the second phase, 13.9%.All these differences were statistically significant.Figure 4 shows the results presented in Table 3.Although we no longer have the two-year Basic Area Programs for MEC residents and the two-year Basic Area Programs for CBC trainees, we can observe that the performance of the R2 MEC was superior to the threeyear CBC trainees (approval of 43.8% vs. 23.8%),with less than half of the failure in the first phase (27% vs. 57.1%) and greater failure in the second phase (29.2% vs. 19%).With the computerization of all the checklists used in the oral and simulated practical tests, there was a deepening of the analysis and cross-referencing of information.Thus, for the first time it was possible for COTECIG to compare the results between the candidates' categories.
In the results presented, the following performances stand out: 1) category A (three years of MEC residency) versus category D (three years of CBC training) with approval more than three times higher (73.8% vs.

23.8%);
2) the extinct two-year Basic Area Programs for MEC residents, whose performance was superior to the three-year CBC trainees; 3) the similar performance of candidates with two and three years of CBC training; and 4) the better performance of category E candidates in relation to two-and three-year CBC trainees.

R E S U M O R E S U M O
aim of this study is to demonstrate the performance of candidates in the last five years of the Specialist Title Test and to compare the performance results between the different surgical training groups of the candidates.The results obtained by candidates from the various categories enrolled in the test in the 2018 to 2022 editions were analyzed.There was a clear and statistically significant difference between doctors who had completed three years of residency recognized by the Ministry of Education in relation to the other categories of candidates for the Specialist Title.. Keywords: Employee Performance Appraisal.General Surgery.Simulation Training.flexibility in the face of diverse demands (Bica & Kornis, 2020).performance in well-defined skills that comprise all these dimensions.Such competencies include theoretical knowledge, psychomotor skills, interpersonal communication, professional ethics, and clinical reasoning applied to medical practice situations (Epstein and Hundert, 2002), with a focus on General Surgery.The notion of competency-based education was associated with the assessment of behavioral goals.
; and Cruess et al., 2016), with examples of what and what forms of assessment should be used at each level."Knows" and "knows how" are domains of cognitive competence (Norcini et al., 2013), the professional's theoretical knowledge.The following domains comprise the competencies related to the practical application of the acquired knowledge."Shows how"

Figure 1 .
Figure 1.Updated version of Miller's Pyramid (1990), with examples of what and what forms of assessment should be used at each level.Adapted from Cruess et al., 2016.
With the clinical simulation, multiple competencies can be tested simultaneously: patient care, medical knowledge, psychomotor skills for procedures, professionalism, and interpersonal skills, among others, allowing both the development and evaluation of individual competencies in similar daily situations and the effective collaboration in teams and the construction of a safety-oriented culture (Pereira Jr, 2021).The objective of this study is to demonstrate the performance of the candidates in the last five years of the Specialist Title Test and to compare the results of the performances between the different groups of surgical training candidates.METHODS In the 2018 to 2022 editions of the CBC Specialist Title test, two phases were held: the first phase, with 100 multiple-choice questions, and the second one, with an oral test (2018 to 2021) and oral and simulated tests in the 2022 edition.All questions and clinical cases used were elaborated based on clinical practice, aiming at problems demanding the application of principles or solutions that require a complex mental process in General Surgery.According to the CBC's notices, candidates could apply in five categories: • A -Have completed the three years of Medical Residency in General Surgery recognized by CNRM/MEC.• B -Have completed the Medical Residency in a Basic Surgical Area (two years) recognized by the CNRM/MEC, plus proof of practice in the specialty of General Surgery, through a statement signed by the Director of the Hospital and/or Head of the Surgery Service, in the period between 2018 and 2022.• C -Have the certificate of completion of the two-year Training in General Surgery issued by the CBC (CBC General Surgery Training Program) plus proof of two years of practice in the specialty Júnior Results of the Last 5 Years (2018-2022) of the Specialist Title Exam of The Brazilian College of Surgeons of General Surgery, through a statement signed by the Director of the Hospital and/or Head of the Surgery Service.• D -Have the certificate of completion of the three-year Training in General Surgery issued by the CBC (CBC General Surgery Training Program).• E -Be registered with the Regional Council of Medicine for at least six years and proof of six years of experience as a general surgeon in Brazil, through a statement signed by the Director of the Hospital and/or Head of the Surgery Service.The proof can be made through documents issued by various Services and Hospitals that, together, reach the minimum of six years required.

Figure 2 .
Figure 2. Presentation (pre-briefing) on the required behavior and explanation about the four simulated stations.

Figure 3 .
Figure 3. Simulated stations of the practical test of the Specialist Title -CBC Notice 2022.Photos 3A: Manual end-to-side anastomosis station; 3B: Face-to-face simulated station with simulated patient; 3C: Videosurgery station; and 3D: Online simulated station.

*
Categories: A (three years of MEC residency), B (two years of MEC residency -Prerequisite), C (two years of CBC Training), D (three years of CBC Training) and E (six years of documented surgical practice, without medical residency or CBC training).Chi-square analysis within all groups significant for statistical difference (p<0.001).Groups identified by the symbols † ‡× are those whose paired comparisons displayed p>0.05.The absence of such symbols indicates p<0.05.

Figure 5
Figure 5 shows the candidates who failed in the second phase of the 2022 edition, with the insertion of the simulated stations associated with the oral test, divided by registration category.Poor performance in the simulated stations was responsible for most failures, both in isolation and associated with poor performance in the oral practical test.

Figure 4 .
Figure 4. Approval/failure of candidates, according to the category of registration* in the 2018 to 2022 exams.*Categories: A (three years of MEC residency), B (two years of MEC residency -Prerequisite), C (two years of CBC Training), D (three years of CBC Training), and E (six years of documented surgical practice, without medical residency or CBC training).Chi-square analysis within all groups was significant for statistical difference (p<0.001).

Figure 5 .
Figure 5. Failure of candidates in the second phase of the 2022 edition, according to the registration category* and the type of practical test.*Categories: A (three years of MEC residency), B (two years of MEC residency -Prerequisite), C (two years of CBC Training), D (three years of CBC Training), and E (six years of documented surgical practice, no medical residency or CBC training).

•
the third year of medical residency conferred greater performance in the approval of MEC residents than of those with only two years, both in the first and second phases, being more than double in both phases; • the third year of CBC training brought practically no gain in relation to the candidates who had two years of training in the two phases of the test and with lower performance than the candidates in category E. The sample of graduates of MEC residency programs and CBC trainees is very small and, therefore, we cannot have further inferences, but the results described above are clear and maintained each year in the period of five years of application of the two phases of the Specialist Title.Even with the much higher performance of candidates with three years of MEC residency (73.8%),COTECIG considers that this percentage should be much higher if these medical residency programs had a large volume of patients treated and operated, with many opportunities for care and surgical procedures for resident physicians, with trained staff, both from a technical and pedagogical point of view, having professional updating programs with periodic Júnior Results of the Last 5 Years (2018-2022) of the Specialist Title Exam of The Brazilian College of Surgeons evaluations, as recommended by Resolution No. 2, of May 17, 2006 of the CNRM, which regulates and clarifies the evaluation procedures of resident physicians working in medical residency programs authorized and offered by institutions accredited by MEC.With the above analyses, we have considerations about the possible educational impacts that these results of the Specialist Title exam can bring to the Medical Residency and CBC Training Programs, even more so now that the CNRM has published Resolution No. 4, of November 1, 2023, which brings several improvements in the evaluation of resident physicians in relation to the previous Resolution (2006), opening space for the collaboration of the Specialty Societies.Thus, in view of these results, the CBC should develop an action plan, together with the MEC Medical Residency Programs and the CBC Training Centers in General Surgery.It would be important for these partnerships to be able to set educational objectives to achieve the necessary competency milestones at each moment of training and, more specifically, at each year of medical residency in General Surgery, through annual updating programs aimed at each of the three years of medical residency, and evaluated following the precepts of the new CNRM resolution (2023).Another important action that could be carried out by the CBC is the training of medical residency preceptors in General Surgery.There are several of these training courses for preceptors in various initiatives, but none is specific to the preceptor of Surgery.CONCLUSION Guaranteeing the population of the various regions of the country a safe and qualified medicalsurgical practice is a demand that can no longer be ignored by the Brazilian College of Surgeons or by any other Medical Specialty Society.COTECIG has sought to improve its techniques for the preparation and review of items, the selection of clinical cases for the oral test, as well as the simulated stations, progressively improving the training of face-to-face and online evaluators.Thus, the commitment to excellence and the constant search for improvement are essential to ensure that obtaining the Title of Specialist by the Brazilian College of Surgeons is recognized as an indication of proficiency and competence in the surgical area.The difference between physicians who completed three years of residency recognized by MEC in relation to the other categories of candidates for the Title of Specialist was evident and statistically significant.There are many possibilities of actions to be carried out by the CBC in partnership with the Medical Residency Programs and, mainly, with the Approved Training Centers so that there is a better quality of training for Brazilian surgeons.

Table 1 -
Psychometric properties and their definitions used by COTECIG.

Table 2 -
Distribution of candidates by registration category*, average grades, standard deviation, and minimum and maximum grades in the first phase (written test) within the grade from zero to ten, from the 2018 to 2022 editions of the CBC Specialist Title Test.
*Categories: A (three years of MEC residency), B (two years of MEC residency -Prerequisite), C (two years of CBC Training), D (three years of CBC Training) and E (six years of documented surgical practice, without medical residency or CBC training).Analysis of variance for all groups is significantfor statistical difference (p<0.001).Groups identified by the symbols † ‡× are those whose two-by-two comparisons have p>0.05.The absence of similar symbols indicates p<0.05

Table 3 -
Results of approval/failure of candidates, according to the category of registration* in the 2018 to 2022 notice.