Procedure-based assessment (PBA) in urology residency: initial experience

Abstract:

Introduction:

Changes in society and in medical practice have demanded improvements in the surgical teaching process in Medical Residency, leading to the emergence of new teaching-learning and assessment models based on competencies. In this process, the Procedure Based Assessment (PBA) stands out as an assessment tool in the workplace, supported by the assessment of competencies and structured feedback.

Objective:

This study aims to present the development and implementation of PBA protocols in an Urology Medical Residency Program.

Method:

This is a prospective, action-research study, carried out from July/2019 to July/2020, involving 10 preceptors and six urology residents. The group consensus methodology was used to create the protocols, in addition to training participants for competency assessment. Six PBA protocols were created, corresponding to the prevalent procedures in the training of the resident/year, followed by their implementation. In addition to the descriptive data analysis, Spearman’s coefficient (r_R) was used for inferential analysis of the correlation between training time and the resident’s performance assessed by the PBA.

Result:

The development of two PBA instruments for each of the three years of training allowed the assessment of all residents. Thirty-one evaluation meetings were held, with an average of five evaluations per resident. There was a positive correlation between longer training time and better resident performance in laparoscopic radical prostatectomy, percutaneous nephrolithotomy, laparoscopic nephrectomy and in the set of the six procedures (r_R = 0.97, 0.55, 0.42 and 0.31, respectively). We report the first use of PBA in Urology Residency in Brazil. The methodology of group consensus associated with a training process proved to be an option for developing this type of instrument. The positive correlation between improved performance in the PBA and training time corroborates studies that resulted in the consolidation of the tool’s validity and reliability.

Conclusion:

The creation of PBA protocols by group consensus is feasible and resulted in the first use of this tool in Urology Residency in Brazil. The PBA may represent a more modern surgical teaching assessment strategy, suitable for training in real scenarios.

Keywords:
Internship and Residency; Competency-Based Education; Urologic Surgical Procedures; Educational Measurement; Urology

Resumo:

Introdução:

Mudanças na sociedade e nas práticas da medicina têm demandado melhorias no processo de ensino cirúrgico nas residências médicas, levando ao surgimento de novos modelos de ensino-aprendizagem e de avaliação baseados em competências. Nesse processo, o Procedure Based Assessment (PBA) se destaca como uma ferramenta de avaliação em ambiente de trabalho, amparada na avaliação de competências e no feedback estruturado.

Objetivo:

Este estudo tem o objetivo de apresentar a elaboração e implantação de protocolos de PBA em um programa de residência médica de urologia.

Método:

Trata-se de estudo prospectivo, do tipo pesquisa-ação, realizado de julho de 2019 a julho de 2020, envolvendo dez preceptores e seis residentes de urologia. Utilizou-se a metodologia de consenso de grupo para a elaboração dos protocolos, além de capacitação dos participantes para avaliação por competência. Elaboraram-se seis protocolos de PBA, correspondentes aos procedimentos prevalentes na formação do residente/ano. Em seguida, esses protocolos foram implantados. Além da análise descritiva dos dados, utilizou-se o coeficiente de Spearman (r_R) para análise inferencial de correlação entre tempo de treinamento e o desempenho do residente avaliado pelo PBA.

Resultado:

A elaboração de dois instrumentos de PBA para cada um dos três anos de formação permitiu a avaliação de todos os residentes. Foram realizados 31 encontros avaliativos, com média de cinco avaliações por residente. Houve correlação positiva entre o maior tempo de treinamento e melhor desempenho do residente na prostatectomia radical laparoscópica, na nefrolitotripsia percutânea, na nefrectomia laparoscópica e no conjunto dos seis procedimentos (r_R = 0,97, 0,55, 0,42 e 0,31, respectivamente). Relatamos a primeira utilização do PBA em residência de urologia no Brasil. A metodologia de consenso de grupo associada a um processo de capacitação mostrou-se como opção para elaboração desse tipo de instrumento. A correlação positiva entre melhora do desempenho no PBA e o tempo de treinamento corrobora estudos que resultaram em consolidação da validade e confiabilidade da ferramenta.

Conclusão:

A elaboração de protocolos de PBA por consenso de grupo é factível e resultou na primeira utilização dessa ferramenta em residência de urologia no Brasil. O PBA pode representar uma estratégia de avaliação de ensino cirúrgico mais moderno e adequado ao treinamento em cenários reais.

Palavras-chave:
Internato e Residência; Educação Baseada em Competências; Procedimentos Cirúrgicos Urológicos; Avaliação Educacional; Urologia

INTRODUCTION

The teaching of surgical techniques in medical residency programs has historically followed Halstead’s model of “see one, do one, teach one”¹. It is a model based on surgical experience, in which proficiency is determined by the number of surgeries performed in key procedures², with little emphasis on details or technical skills³.

Although this teaching-learning model is widely prevalent in Brazil and most of the world, the relationship between proficiency and experience is controversial, reinforced by studies that questioned the usefulness of the number of surgeries performed as a competency marker⁴. Some people require less exposure to a series of challenges to be able to assimilate the necessary skills for good surgical performance, while others require a longer training time⁵.

Some factors have been pressing for changes, such as greater concern about ethics and patient safety, the demand from society for greater efficiency and quality of health services and the limitation of the resident’s workload in some countries^3),(5),(6.

In this scenario, the teaching of the surgical technique has been widely discussed, undergoing significant changes, especially in some developed countries. The teaching model based on the acquisition of competencies is gradually replacing the traditional Halstead’s model. In this new model, students cannot progress and cannot complete their training if they do not demonstrate competency in predefined areas of the curriculum⁷.

The demonstration of competency in the context of surgical training requires the assessment of several attributes, such as knowledge, technical skills, integrity, communication, leadership and teamwork^8),(9. It must, therefore, be inferred from the performance analysis, that is, the way the person performs and behaves in daily practice¹⁰.

One class of competency-based assessment instruments has emerged in the field of specialized medical training: the Workplace-Based Assessment (WBA). Aiming to assess activities routinely performed in the work environment, the WBA offers several advantages, such as the possibility of focusing on a particular area of education or skill, better measurement of its own progression and identification of the resident with low performance, allowing better planning of measures that can correct the course of learning¹¹.

One of the most relevant instruments belonging to this class is the Procedure Based Assessment (PBA), widely used in the United Kingdom, where, since 2007, it has been adopted as the main method for assessing surgical skills¹².

In the PBA, the resident is directly observed during the surgical procedure, being evaluated in relation to a checklist of competencies and receiving immediate feedback.

The PBA is designed to support the development of technical and professional skills, incorporating constructive feedback, allowing the assessment of skills during professional practice¹³. Due to these characteristics, it can fulfill a dual purpose as an assessment method: summative and formative¹⁴.

This study aims to present the initial experience in the creation and implementation of the PBA in a Medical Residency Program in Urology in Brazil.

MATERIALS AND METHODS

The study was approved by the Research Ethics Committee, under CAAE number 16950719.0.0000.5292, Opinion N. 3.532.085.

Between July 2019 and July 2020, a prospective, exploratory, action-research study was developed, involving 10 preceptors and six residents, with two residents from each calendar year, from the Medical Residency Program in Urology at Hospital Universitário Onofre Lopes (PRMUro-HUOL).

The primary objective was the development of six PBA protocols per group consensus. The secondary objectives were team training and performance analysis based on the application of the PBA. For that purpose, the study was developed in two main phases: 1- group training and preparation of PBA protocols; and 2- PBA implementation.

Training of the participants

It occurred in two phases, the first being aimed at the presentation of relevant content for performance assessment and feedback and the second aimed at the supervised application of the PBAs. Both were carried out in-person and integrated the preceptors and residents of PRMUro-HUOL.

The first phase lasted four hours and addressed issues related to student assessment, with emphasis on assessment methods in medical practice, the PBA and the importance of feedback. The methodology used was the video-mediated teaching, by presenting real situations for evaluation, discussion in a conversation circle format, exploring the participants’ previous experiences, as well as a brief dialogue exposition for introductions and closings of the addressed contents.

The PBA protocols were applied at the second phase of the training, under the supervision of two of the authors, who acted as instructors. The objectives of this phase were to promote familiarization with the PBA, resolve doubts and improve the application of the method. In this phase, 10 PBAs were applied during the daily residency activities, as shown in Figure 1.

Figure 1
Schematic drawing of the training process associated with the creation of the PBAs.

Creation of the PBA protocols

The creation of the PBA protocols was carried out using the methodology of group consensus, interspersing rounds of in-person and remote interaction and feedback, which are summarized in Figure 2.

Figure 2
Summary of the phases in the PBA creation process by group consensus.

The steps of this process were the following:

1. Definition, by the authors, of the six most relevant urological surgical procedures in the Residency Program, with two being related to each calendar year. The procedures chosen were: rigid ureterolithotripsy and open prostatectomy, related to the first year of residency; transurethral resection of the prostate and laparoscopic nephrectomy, corresponding to the second year of residency; and percutaneous nephrolithotripsy and laparoscopic radical prostatectomy, related to the third year of residency.

2. Creation of six pilot PBAs by the authors, based on existing protocols abroad. Of the six protocols, four had a model in a foreign language and two were originally developed by the authors.

3. Division of the expert group into six subgroups. Each subgroup developed a PBA model based on the pilot PBA within a one-week period and made it available for full group evaluation. In addition to the specialists, each subgroup had a resident.

4. Through the Google form application, each subgroup was able to give their opinion and suggest changes to the protocols developed by the other subgroups, for a period of one week;

5. Implementation of changes by the authors, resulting in new versions of the protocols, which were presented to the participants.

6. Application of the PBA in practice, under the supervision of two of the authors.

7. Final round of in-person interaction and feedback, with all participants, after the supervised application of the PBAs. Additional adjustments were discussed and included in the protocols, concluding the process.

Main characteristics of the developed PBAs

• Division of the competency checklist into four main domains: preoperative planning, preoperative preparation, intraoperative technique and postoperative care;

• The evaluation options for each item in the competency checklist are: “does not apply”, “insufficient”, “good/regular” and “excellent”;

• After the competency checklist, a “general assessment” grade should be assigned, with the options: “insufficient”, “good/regular” and “excellent”;

• The instrument is finalized with the “Assessment of the Competency Level”, a reference to the assessment of the Intercollegiate Surgical Curriculum Program (ISCP - United Kingdom) called “Global Summary Score”, a progressive scale related to the level of autonomy of procedure performance by the resident.

As an example, the PBA protocol for laparoscopic radical prostatectomy is shown in Figure 3. The other five PBA protocols can be accessed online at https://doi.org/10.6084/m9.figshare.c.5560734.v1.

Figure 3
Laparoscopic radical prostatectomy PBA form (Part 1).

Figure 3
Laparoscopic radical prostatectomy PBA form (Part 2).

PBA implementation

The implementation of the PBAs occurred in two phases: in the training of the participants, during which this activity was supervised by the authors; and after the training, in an unsupervised manner.

Analysis of performance results in the evaluations

In addition to the descriptive analysis of the data, the analysis of the residents’ performance was carried out based on the assessment of the competency checklist. The calculation of the Excellent General Score (ExcGS) was agreed upon, which corresponds to the percentage of items scored as “excellent” among those that were assessed.

In this data analysis, we sought to assess the correlation between the performance measured in the PBA and the training time to verify whether the residents showed a measurable improvement in performance by the PBA over time, that is, as they accumulated more time of training, considering the 148 days elapsed between the first and the last assessment.

Data were analyzed in an Excel 2016 spreadsheet and recorded in scatter plots (Graph 1). The intensity of the correlation between the variables was determined by calculating Spearman’s coefficient.

Graph 1
Scatterplot of the Excellent General Score in different scenarios.

RESULTS

All residents and 77% of the program’s preceptors (n=10) were trained to use the six PBAs, all men. Among the preceptors, 40% had a Master’s Degree, but none reported a previous experience with the PBA.

In total, 31 PBAs were applied, 10 during the training phase and 21 after it. The time elapsed between the first and the last assessment was 148 days. All residents were assessed, with four to six assessments being carried out per resident, an average of 5.2 PBAs/resident.

A positive correlation was demonstrated between training time and PBA performance in the laparoscopic radical prostatectomy, percutaneous nephrolithotripsy, laparoscopic nephrectomy and in the set of the six procedures, with a Spearman’s coefficient of 0.97, 0.55, 0.42 and 0.31, respectively. A trend line was included in the scatter plots of these procedures (Graph 1), for better visualization.

DISCUSSION

The modernization of surgical training with the introduction of the teaching model based on the acquisition of competencies necessitated the development and implementation of mechanisms for monitoring and evaluating the intended competencies¹⁵.

Occupying a central space in this process, WBAs are designed to integrate teaching, learning, assessment and feedback, assessing skills and behaviors, rather than just knowledge¹⁶.

The use of WBAs that address surgical competencies has been shown to be quite limited in medical residencies in Brazil, with no consistent publications on the subject.

In addition to not experiencing an environment of intense debate on the surgical teaching model, other factors can negatively impact the change initiatives in Brazil. The scarcity of assessment tools validated in Portuguese and the difficulty of accessing instruments in other languages stand out.

In this study, we describe the first experience with the use of the PBA - a tool for assessing surgical skills in the workplace - in a Medical Residency Program in Urology in Brazil.

Using a group consensus methodology, all participants had the opportunity to contribute and give their opinion on equal terms, based on several rounds of interaction and feedback, building protocols that were finalized with the group’s approval.

Training is an important step in this process, as it improves the group’s understanding of the teaching-learning process and student assessment concepts, as well as increasing the efficiency and quality of the instrument’s application.

The strategy of creating the instruments per group consensus, including the aggregation of the Service’s residents in the process, together with the training process, proved to be a good option for the introduction of this assessment tool. In addition to the aforementioned benefits, this strategy reflected the concern to create an environment of team integration and engagement around a completely new activity in the routine of PRMUro-HUOL. This is a particularly important point, because the lack of team engagement is identified as one of the main factors of failure in the use of WBAs^16),(17.

When applying PBAs, there was a trend towards improvement in the performance of the PBA in three of the six surgical procedures and in the joint assessment of the six procedures, considering the approximate interval of five months between the first and last assessments.

This finding is in agreement with the literature, which demonstrated that PBA scores are related to surgical training time¹² and that the sequential use of the PBA should demonstrate progression towards competency¹³.

The large-scale use of the PBA in the United Kingdom allowed the validation of this tool in different studies, which also reported the reliability of the method^12),(13. Another evaluated aspect was feasibility and, in this sense, it was concluded that the PBA is viable and valued by students as a means of constructive feedback and training direction¹⁸.

Study limitations

This study was negatively impacted by the COVID-19 pandemic, due to the limited scheduling of elective surgeries in certain periods of the study, resulting in a smaller number of PBA applications than initially planned. This made it difficult to obtain more significant results in relation to the performance analysis.

CONCLUSIONS

The creation and implementation of assessment protocols based on the PBA from a consensus of a group of experts is feasible and is in line with the most modern concepts of the surgical training model based on competency acquisition.

The PBA is an important part of this training model in some countries, and it can be validated in the national territory as other Residency Programs start to adopt them.

REFERENCES

• ¹
  Tennent D, Pitts D, Foote J. Procedure-based assessment in shoulder and elbow surgery. Shoulder Elbow. 2013;5:73-7.
• ²
  Torsney KM, Cocker DM, Slesser AAP. The modern surgeon and competency assessment: are the Workplace-Based Assessments Evidence-Based? World J of Surg. 2015;39:623-33.
• ³
  Hurreiz H. The evolution of surgical training in the UK. Adv Med Educ Pract. 2019;10:163-8.
• ⁴
  Hopkins L, Robinson DBT, Brown C, Egan R, Iorwerth A, Holt M, et al. Trauma and orthopedic surgery curriculum concordance: an operative learning curve trajectory perspective. J Surg Educ. 2019;76(6):1569-78.
• ⁵
  De Siqueira JR, Gouch MJ. Correlation between experience targets and competence for general surgery certification. BJS Open. 2016;103:921-7.
• ⁶
  Beard J, Rowley D, Bussey M, Pitts D. Workplace-based assessment: assessing technical skill throughout the continuum of surgical training. ANZ J Surg. 2009;79:148-53.
• ⁷
  Evgeniou E, Peter L, Tsironi M, Iyer S. Assessments methods in surgical training in the United Kingdom. J Educ Eval Health Prof. 2013;10(2):1-7.
• ⁸
  Bindal T, Wall D, Goodyear HM. Trainee doctors’ views on workplace-based assessments: are they just a tick box exercise? Med Teach. 2011;33:919-27.
• ⁹
  Moorthy K, Munz Y, Sarker SK, Darzi A. Objective assessment of technical skills in surgery. BMJ. 2003;327:1032-7.
• ¹⁰
  Swanwick T, Chana N. Workplace assessment for licensing in general practice. Brit J Gen Pract. 2005:55;461-7.
• ¹¹
  Eardley I, Bussey M, Woodthorpe A, Munsch C, Beard J. Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme. ANZ J Surg . 2013;83:448-53.
• ¹²
  Marriott J, Purdie H, Crossley J, Beard JD. Evaluation of procedure-based assessment for assessing trainees’ skills in the operating theatre. Br J Surg. 2011;98(3):450-7.
• ¹³
  Awad Z, Hayden L, Robson AK. Reliability and validity of Procedure-Based Assessments in otolaryngology training. Laryngoscope. 2015;125:1328-35.
• ¹⁴
  Abdelaal A. Procedure-based assessments: the past, the present and the future. Postgrad Med J. 2020;96:7-8.
• ¹⁵
  Shalhoub J, Marshall DC, Ippolito K. Perspectives on procedure-based assessments: a thematic analysis of semistructured interviews with 10 UK surgical trainees. BMJ Open. 2017;7:1-8.
• ¹⁶
  Ali JM. Getting lost in translation? Workplace based assessments in surgical training. Surgeon. 2013;11(5):286-9.
• ¹⁷
  Hunter AR, Maird EJ, Reed MR. Procedure-based assessments in trauma and orthopaedic training - the trainees’ perspective. Med Teach. 2015;37:444-9.
• ¹⁸
  James K, Cross K, Lucarotti ME, Fowler AL, Cook TA. Undertaking procedure-based assessment is feasible in clinical practice. Ann R Coll Surg Engl. 2009;91:110-2.

• 2
  Evaluated by double blind review process.

• SOURCES OF FUNDING

  The authors declare no sources of funding.

Edited by

Chief Editor: Rosiane Viana Zuza Diniz. Associate Editor: Jorge Carvalho Guedes.

Publication Dates

• Publication in this collection
  13 Dec 2021
• Date of issue
  2021

History

• Received
  21 Aug 2021
• Accepted
  03 Oct 2021