Development, validation and application of clinical simulation scenarios for assessment of stomatherapy specialists

Construcción, validez y aplicación de escenarios de simulación clínica para evaluación de expertos en estomaterapia

ABSTRACT

Objectives:

to build and validate three clinical simulation scenarios and report the application with candidates for the specialist’s degree in stomatherapy.

Methods:

methodological study, building three scenarios and evaluation checklists; content validation with judges, using content validity index and Modified Kappa Coefficient; pre-test and application.

Results:

scenarios built based on nursing care for: 1. insufficiency and venous ulcer; 2. demarcation of intestinal stomia; and 3. Clean intermittent catheterization. In the content validation of the 24 items appreciated, 83%, 80%, and 92% were validated without change. In the pre-test, the objectives and checklists were adjusted. In the application, to standardize the evaluation, actors and evaluators were trained previously, and each candidate passed the three stations.

Conclusions:

scenarios built and with validated content, based on evidence and covering the three areas of stomatherapy. The pre-test allowed for adjustments in the scenarios, and the candidates achieved the expected objectives.

Descriptors:
Educational Assessment; Simulation Training; Patient Simulation; Nursing Education; Validation Studies

RESUMEN

Objetivos:

construir y validar tres escenarios de simulación clínica y relatar aplicación con candidatos al título de experto en estomaterapia.

Métodos:

estudio metodológico, con construcción de tres escenarios y checklists de evaluación; validez de contenido con jueces, usando índice de validez de contenido y Índice de Kappa; pretest y aplicación.

Resultados:

escenarios construidos basado en los cuidados de enfermería para: 1. Insuficiencia y úlcera venosa; 2. Demarcación de estoma intestinal; y 3. Cateterismo intermitente limpio. La validez de contenido, de los 24 ítems apreciados, 83%, 80% y 92% validaron inalterados, respectivamente. El pretest, se ajustaron los objetivos y checklists. La aplicación, para uniformización de la evaluación, actores y evaluadores entrenaron previamente, y cada candidato pasó en las tres estaciones.

Conclusiones:

escenarios construidos y con contenido validado, basados en evidencias y abarcando todas áreas de la estomaterapia. El pretest permitió ajustes en los escenarios, y los candidatos alcanzaron los objetivos esperados.

Descriptores:
Evaluación Educacional; Entrenamiento por Simulación; Simulación de Paciente; Educación en Enfermería; Estudios de Validez

RESUMO

Objetivos:

construir e validar três cenários de simulação clínica e relatar a aplicação com candidatos ao título de especialista em estomaterapia.

Métodos:

estudo metodológico, com construção de três cenários e checklists de avaliação; validação de conteúdo com juízes, usando índice de validade de conteúdo e Coeficiente de Kappa Modificado; pré-teste e aplicação.

Resultados:

cenários construídos com base nos cuidados de enfermagem para: 1. Insuficiência e úlcera venosa; 2. Demarcação de estomia intestinal; e 3. Cateterismo intermitente limpo. Na validação de conteúdo, dos 24 itens apreciados, 83%, 80% e 92% foram validados sem alteração, respectivamente. No pré-teste, ajustaram-se os objetivos e checklists. Na aplicação, para uniformização da avaliação, atores e avaliadores foram treinados previamente, e cada candidato passou nas três estações.

Conclusões:

cenários construídos e com conteúdo validado, baseados em evidências e abrangendo as três áreas da estomaterapia. O pré-teste permitiu ajustes nos cenários, e os candidatos alcançaram os objetivos esperados.

Descritores:
Avaliação Educacional; Treinamento por Simulação; Simulação de Paciente; Educação em Enfermagem; Estudos de Validação

INTRODUCTION

Clinical simulation is an effective teaching-learning strategy used in nursing in several areas^(1-3). One of the advantages is promoting a safe learning environment allowing the participant to practice skills that can be reviewed and reflected in a controlled environment⁽⁴⁾.

Another field for the use of simulation is in the process of summative or formative evaluation. Thus, the simulated environment alters its state of safe learning platform to enable the analysis of the clinical performance of the participants in the affective, cognitive, and psychomotor domains⁽⁵⁾. However, this process’s effectiveness depends on applying validated instruments/checklists and scenarios that explore the specific skills and objectives to be evaluated. That process favors the standardization of the evaluation mechanisms and facilitators that conduct the simulation and prepare the participants to act in the simulated environment, reducing anxiety and psychological stress⁽⁵⁾.

Despite the importance of clinical simulation in evaluation, it is little used in nursing, although validated instruments are available to guide this method⁽⁶⁾. In this context, the opportunity has arisen to introduce such strategy in the titration process of the Brazilian Stomatherapy Association, which provides the certification of high qualification of the professional to act in the care of people with stoma, acute and chronic wounds, fistulas, drains, catheters and anal and urinary incontinence⁽⁷⁾. The evaluation used so far consisted of the performance of specific theoretical tests. Therefore, the simulated environment may be an effective strategy for the evaluation of professionals. It was raised as a hypothesis to be verified in this study if the content of the scenarios and checklists are valid in their content, and if they make it possible to evaluate the candidates’ abilities for the degree test in stomatherapy. Therefore, it is questioned: The evaluation checklists and the scenarios built have relevant content and represent the specific technical skill construct for titling specialists in stomatherapy?

OBJECTIVES

To develop and validate three clinical simulation scenarios and report the application with candidates for the degree of experts in stomatherapy.

METHODS

Ethical Aspects

The Research Ethics Committee of the State University of Campinas approved this study.

Design, location, and period of study

The present research is a methodological study, with a quantitative approach, following the recommendations of Simulation-Based Research Extensions for the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement⁽⁸⁾. The development, content validation, and pre-testing of scenarios and checklists occurred from August to October 2019, and its application, in October 2019, during the Brazilian Stomatherapy Congress held in Foz do Iguaçu, Paraná.

Population and sample - exclusion and inclusion criteria

For validation, judges were invited to meet the following criteria: doctorate/master’s degree in health or the area of simulation, or specialization in stomatherapy; publications, care practice and/or teaching in simulation or stomatherapy. The sample was intentional and composed of five judges, given the availability for response in the estimated time, and followed a previous recommendation that indicates the use of this minimum number for content validation stage when the sample is highly qualified⁽⁹⁾.

In the scenarios’ application stage, participated the candidates who took the test for the specialty degree. Also, there were six advisors in stomatotherapy, titled, and members of the Brazilian Association of Stomatherapy.

Study protocol

The study was conducted in four stages: 1. Development of scenarios and evaluation checklists; 2. Validation of content by judges; 3. Pre-testing; and 4. Application of scenarios.

To support the building of scenarios, a narrative literature review of the international consensus of reference was conducted: Wounds International^(10-11), Wound, Ostomy and Continence Nurses Society⁽¹²⁾ and European Association of Urology Nurses⁽¹³⁾ and integrative or narrative review studies in the three significant areas of stomatherapy (wounds^(14-15), ostomy⁽¹⁶⁾ and incontinence⁽¹⁷⁾).

For the structuring of the scenarios, it was followed the National League for Nursing (NLN)/Jeffries Simulation Theory⁽¹⁸⁾, which contemplates the elements: facilitator, learner (adapted for the candidate), educational practices, simulation design, and expected outcomes. Even with evaluative character, the educational practices of active learning were used once the candidates in each scenario played specialized nurses. The simulation design determined the achievement of general and specific objectives for high fidelity and moderate complexity scenarios. Through patients, clues assisted/directed the candidate’s clinical reasoning. Regarding the expected outcomes, the candidates should use cognitive knowledge to decide according to clinical reasoning to: indicate and apply compressive therapy, perform stoma demarcation and demonstrate clean intermittent catheterization in the simulator. For this purpose, checklists were elaborated for the evaluators to register if the candidate did not act (0 points) if he performed it partially (0.5 points) or if he performed it correctly (1 point).

Later, in the validation of the content of the scenarios and checklists, the judges received via Google Forms^® the constructed materials and characterization tool that contemplated the variables: age, gender, training, working time as a nurse, working time in stomatherapy, working time with clinical simulation and experience with scenario building. In the second part of the form, following the content validation reference in the simulation area^(19-20), each aspect and domain of the scenarios and checklist items were evaluated. It was considered comprehensiveness, clarity, and relevance of psychometry properties and used a 4-point Likert scale, being: 1 - item not very relevant; 2 - item needs considerable revision; 3 - item needs small revision; and 4 - relevant item⁽⁹⁾. Also, the judges should analyze realism, adherence to scientific evidence, objectivity, and complexity of the level of knowledge and skills required of the participant in the scenario.

The pre-test was then conducted with two stomatherapy nurses to test the evaluation’s feasibility, i.e., factors that facilitate or prevent its implementation. There was no need for these nurses’ previous training because they are specialists in stomatherapy, with experience in simulation. The researchers used a hospital outpatient clinic to increase environmental fidelity. The study’s principal researcher, also a stomatother therapist, acted as a standardized patient, i.e., an ordinary person or actor who plays the patient; and another nurse specialized in stomatherapy applied the checklists. In the end, the nurses (acting and evaluator) made their considerations about the scenarios and checklists that were incorporated.

The last stage contemplated the application of scenarios as part of an evaluation to obtain the title. For the evaluation, each candidate went through three scenarios using actors and three evaluators’ pairs, all experts in stomatherapy. Evaluators and actors received scenarios content, checklists, and scripts previously. They were guided about the evaluation dynamics, time and progress of the candidates, impossibility of conversation between evaluators and candidates during the evaluation.

Data analysis and statistics

The results of steps 1, 2, and 4 were presented descriptively and using charts to visualize the findings better. The tabulated content validation data appeared in an Excel^® spreadsheet, and the judges presented a description. Afterward came the calculation of the Content Validity Index (CVI)⁽²¹⁾ and the Modified Kappa Coefficient (MKC)^(22-24). The calculation used in CVI had the sum of “3” and “4” responses from each judge for each item, divided by the sum of the total number of responses; items reviewed had CVI lower than 80% ^(21-22). The MKC, developed for content validity studies mainly with reduced sample size, was calculated according to the formula: = [, em que [, N represents the number of judges, and A the number of judges who evaluated a given item is relevant. Values from 0.40 to 0.59 for the MKC are considered reasonable; from 0.60 to 0.74, good; and above 0.74, excellent^(22-24). The items with CVI ≤ 80% and MKC ≤ 0.74 were reformulated according to considerations made by judges and according to researchers’ consensus.

Step 3 was analyzed descriptively, considering the suggestions and improvements proposed in the pre-test, based on researchers’ consensus.

RESULTS

Scenario development and evaluation checklists

The purpose of the building scenarios was to evaluate candidates’ clinical performance in everyday situations regarding the stomatherapy specialist (Chart 1). The design of the simulations was based on the learning objectives and focused on: Scenario 1: Clinical reasoning in the evaluation and treatment of a patient with venous ulcer; Scenario 2: Preoperative preparation of a patient with the need to make an intestinal ostomy; Scenario 3: Guidance on care for a patient with the need for a clean intermittent catheterization.

Chart 1
Simulation design applying three scenarios after content validation and pre-test

Scenario 1: Wound		Scenario 2: Ostomy	Scenario: Incontinence
Objectives	General: To evaluate the nurse clinical reasoning in evaluation and treatment of patient with venous ulcer. Specifics: To perform data collection related to the patient and injury; To evaluate the characteristics of the ulcer: bed, edema and pain; To teach about topical therapy and apply compressive therapy; To clarify doubts about the maintenance of compressive therapy and exchange frequency.	General: Carry out client-centered care for the preoperative preparation for an intestinal ostomy. Specific: Perform data collection (anamnesis); Perform a physical examination of the abdomen; Perform demarcation of the ostomy bilaterally; Implement interventions based on client needs.	General: Carry out client-centered male care for discharge with clean intermittent catheterization (CIC). Specifics: Carry out data collection/anamnesis (collect information regarding psychological, social and biological aspects); Carry out guidance for the CIC; Promote self-care; Carry out the CIC on the mannequin; Clarify doubts regarding the technique and frequency of the CIC.
Fidelity	High fidelity, with use of standardized patient and moulage for characterization of venous ulcer.	High fidelity, with use of standardized patient.	High fidelity, with standardized patient use and low fidelity mannequin.
Problem resolution	Moderate complexity case, with relevant information for the expert to interpret, make sense of the data and provide an appropriate response, such as: • Carry out guidance on venous ulcer (causes, risk factors, adequacy of treatment). • Explain to the patient the need for compressive therapy as the gold standard of treatment and prevention of recurrence. • Adjust the topical therapy according to the characteristics of the wound. • Perform compressive therapy on the patient. • Inform about the frequency of compressive therapy change and its maintenance at home. • Inform and guide about adverse events such as pain and coverage saturation. • Inform about measures to minimize edema: myolinphokinetics exercises, walking and elevated limbs at rest.	Moderate complexity of the case, with relevant information for the expert to interpret, make sense of the data and provide an appropriate response, such as: • Survey patient's history and abdominal physical examination. • Perform bilaterally stoma demarcation in the abdomen. • Orienting the client about the ostomy and its necessity. • Orienting the client on the care of the ostomy and equipment that are available.	Moderate complexity of the case, with relevant information for the expert to interpret, make sense of the data and provide an appropriate response, such as: • Perform the orientation regarding clean intermittent catheterization. • Explain to the patient the need for catheterization. • Show the equipment that is available for the catheterization. • Demonstrate the CIC technique. • Orient the patient about the self-catheterization. • Inform about the frequency of the catheterization. • Orient on the use of the mycological journal. • Inform and guide about adverse events.
Hints	• Patient's records with the annotation about the time of existence of the ulcer (3 years); previous treatments (papain and essential fatty acid), bad adhesion to the compressive therapy used previously (boot of Unna). • Brachial ankle index (1.0). Products for orientation and adequacy of topical therapy (foam, hydrocolloid, calcium alginate, hydrofibre, hydrogel and silver coverings). • Products for compressive therapy (bandages).	• Patient's records with annotation of the type of surgery proposed and medical evolution (left colectomy due to obstructive sigmoid colon neoplasia). • Report of radiotherapy and postoperative chemotherapy. • Preoperative kit containing different collector equipment, accessories, and adjuvants.	• Patient's records of the type of surgery and level of spinal cord injury and examinations such as urodynamic study, urine I and uroculture. • Mannequin available for the CIC • Kit with different catheters and materials needed to perform the technique. • Voiding dairy available.

• Note: CIC - Clean Intermittent Catheterization.

The simulations were elaborated with high fidelity and the use of a standardized patient (actor), who used a predefined script to facilitate the performance of the venous insufficiency scenario; the moulage technique was used with a venous ulcer›s artistic make-up to increase realism through environmental and psychological fidelity. The scenarios presented moderate complexity, but with relevant information that allowed the candidate to perform the clinical reasoning and offer an adequate response to the objectives.

To help the evaluation verification checklists were composed with 27, 30, and 32 items. Chart 2 demonstrates the expected actions for each scenario. The evaluators should inform if they were partially or not executed, considering the objectives and expected outcomes.

Chart 2
Scenario evaluation checklist after content validation and pre-test

Scenario 1: Wound	Scenario 2: Ostomy	Scenario 3: Incontinence
1. Did you welcome the client? 2. Did you introduce yourself as a specialist nurse? 3. Did you ask about the main complaint? 4. Did you give a brief history of the patient? 5. Did you make a brief history of the wound? 6. Did you inform the need for wound evaluation? 7. Did you accommodate the client on the stretcher for wound evaluation? 8. Did you clean your hands? 9. Did you put on gloves? 10. Wound evaluation:     a. Did you remove the dressing?     b. Did you evaluate edema?     c. Did you ask or evaluate pain?     d. Did you explain to the patient the condition of the wound? 11. Wound treatment:     a. Was topical therapy orientation adequate?     b. Did you guide on adjacent skin care?     c. Did you inform about the need for compressive therapy and its importance? 12. Did you perform compressive therapy correctly?     a. Did you position the foot at a 90º angle?     b. Did the bandage cover the line that goes through the finger root and the tip of the first metatarsal?     c. Was the heel covered?     d. Was the pressure exerted decreasing from the ankle to the thigh? 13. Did you sanitize your hands? 14. Did you help the client get off the stretcher? 15. Did you clarify the client's doubts about the maintaining the compressive therapy at home and about the topical therapy? 16. Did you schedule a return for evaluation? 17. The communication was effective?	1. Did you welcome the client? 2. Did you introduce yourself as a specialist nurse? 3. Did you ask about the main complaint? 4. Did you give a brief history of the patient? 5. Did you give a brief orientation on what an ostomy is? 6. Did you inform about evaluating the abdomen for demarcation? 7. Did you accommodate the client on the gurney for demarcation? 8. Did you clean your hands? 9. Did you explain the importance of demarcation for the confection of the ostomy? 10. Demarcation:     a. Did you divide the abdomen into quadrants?     b. Did you choose and justify the choice of quadrant?     c. Did you locate the rectus abdominus?     d. Did you identify the anatomical accidents?     e. Did you use the ruler or tape measure to evaluate the distances of the umbilical scar and the imaginary line that joins the anterosuperior iliac spine?     f. Did you provisionally demarcate the site of the dorsal decubitus stoma with a circle of ± 2.5 cm?     g. Did you evaluate the client sitting down? Did you identify folds?     h. Did you question the sitting visualization?     i. Did you evaluate the client standing?     j. Did you question about standing view?     k. Did you mark with a dermographic pen?     l. Did you put collector in the demarcated location?     m. Did you confirm with the client the suitability of the site?     n. Did you demarcate bilaterally? 11. Did you clean your hands at the end of the activities? 12. Did you help the client to leave the stretcher? 13. Did you clarify any doubts about the collector equipment? 14. 14. Did you make recommendations about clothing? 15. Did you advise on the right to receive the equipment? 16. Did you advise the client about the daily routines with the ostomy? 17. The communication was effective?	1. Did you welcome the client? 2. Did you introduce yourself as a specialist nurse? 3. Did you ask about the main complaint? 4. Did you give a brief history of the patient? 5. Did you give guidance on what is clean intermittent catheterization (CIC)? 6. Did you evaluate with the client the possibilities of performing the CIC? 7. Did you provide guidance on the benefits of the CIC for the permanent one? 8. Did you provide guidance on the site for the procedure? 9. Did you guide the different types of catheters? 10. Did you demonstrate the CIC technique in the simulator? 11. Did you do the technical orientation of CIC?     a. Have you cleaned the surface where the materials will be placed?     b. Did you separate the adequate catheter?     c. Did you orient about the caliber of the indicated catheter?     d. Did you clean your hands before the procedure?     e. Did you orient on the cleaning of the genital area?     f. Did you orient about cleaning the urethral orifice?     g. Did you orient on the cleaning of the foreskin?     h. Did you orient on how to prepare the catheter?     i. Did you orient about penis positioning?     j. Did you orient on how to introduce the catheter?     k. Did you orient on the interruption of the catheter introduction?     l. Did you orient in situations where the urine stops draining?     m. Oriented the patient or companion on the angle in case of resistance in the catheter introduction.     n. Did you orient on how to remove the catheter after drainage? 12. Did you orient on the use of an external device? 13. Did you advise on the importance of observing the characteristics of urine? 14. Did you advise on the importance of hydration? 15. Did you give guidance on the importance of the urination dairy? 16. Did you orient about the frequency of the catheterization 17. Did you clarify the client's doubts regarding the collector equipment and/or the technique? 18. The communication was effective?

• Note: CIC - Clean Intermittent Catheterization

Validation of content by judges

A committee of five judges reviewed each scenario and checklist. Three accredited stoma specialists evaluated each individually, and two judges, simulation experts, evaluated all three. The judges were predominantly female, with an average age of 43 years. Six judges had doctorates, four had master’s degrees, and one was a specialist. The average time worked as a nurse was 19.5 years, and the average time of attendance in stomatherapy was ten years. As for clinical simulation, the judges had an average of four years of clinical simulation experience, and only two of them had experience with building simulation scenarios.

Table 1 shows the CVI and MKC values for each scenario evaluation items and checklists. In the Wound scenario, judges suggested changes complied in items Objectives, Evaluation, and Information in the chart, such as replacing the verb of the action to “perform” by “evaluate”; withdraw the prescription of the treatment with the Unna boot, because in the chart there was the report of allergy to this topical therapy; include the information to the pre-briefing, justifying the low adherence to the use of the Unna boot. In the specific checklist of this scenario, the only one that presented lower values than the established standard, the judges recommended to include the expected actions during the evaluation of the wound to facilitate the observation, followed by the researchers.

Table 1
Content Validity Index and Modified Kappa Coefficient values for each item evaluated in the scenarios and wound, stoma and incontinence checklists, Campinas, São Paulo, Brazil, 2019

Indicator	Wound		Ostomy		Incontinence
	CVI	MKC	CVI	MKC	CVI	MKC
Indicators relating to scenario
Facilitators	1.00	1.00	1.00	1.00	1.00	1.00
Candidates	1.00	1.00	0.80	0.76	1.00	1.00
Type of knowledgment	1.00	1.00	1.00	1.00	1.00	1.00
Objectives	0.80	0.76	0.80	0.76	1.00	1.00
Fidelity	1.00	1.00	0.80	0.76	1.00	1.00
Problem solutions	1.00	1.00	0.80	0.76	0.80	0.76
Hints	1.00	1.00	1.00	1.00	0.80	0.76
Expected results	1.00	1.00	0.80	0.76	1.00	1.00
Evaluation Criteria	0.80	0.76	0.80	0.76	1.00	1.00
Pre-briefing	1.00	1.00	1.00	1.00	1.00	1.00
Chart information	0.80	0.76	1.00	1.00	1.00	1.00
Script	1.00	1.00	1.00	1.00	1.00	1.00
Scenario structure	1.00	1.00	1.00	1.00	1.00	1.00
Realism	1.00	1.00	1.00	1.00	1.00	1.00
Adherence to scientific evidence	1.00	1.00	1.00	1.00	1.00	1.00
Complexity required	1.00	1.00	1.00	1.00	1.00	1.00
Scope	1.00	1.00	1.00	1.00	1.00	1.00
Objectivity	1.00	1.00	1.00	1.00	1.00	1.00
Relevance	1.00	1.00	1.00	1.00	1.00	1.00
Indicators relating to checklist
Initial Actions	1.00	1.00	1.00	1.00	1.00	1.00
Wound assessment/Demarcation/CIC	0.60	0.42	1.00	1.00	1.00	1.00
Wound treatment/Demarcation/CIC	1.00	1.00	1.00	1.00	1.00	1.00
Final Actions	1.00	1.00	1.00	1.00	1.00	1.00
Relevance of checklist	1.00	1.00	1.00	1.00	1.00	1.00

• Note: CVI - Content Validity Index; MKC - Modified Kappa Coefficient; CIC - Clean Intermittent Catheterization.

In the Ostomy scenario, they suggested making more explicit the candidate role with the description of a clinical case; in the Simulation design (Objectives and Fidelity), they suggested to describe the nursing consultation items in the specific objective and add actors in the Fidelity item; in the Problem resolution item, to add nursing diagnoses that the candidate should identify; and in the Outcomes item, suggested to describe the items which they would employ to evaluate using the checklist. There were no changes in the evaluation instrument.

In the Incontinence scenario, they suggested that, in the item related to problem-solving, should be used “Demonstrate the Clean Intermittent Catheterization (CIC) technique” and not “Perform the CIC technique” because they expected the candidate would use the simulator for demonstration; and in the addition of guidance on self-catheterization, use of the voiding diary and the importance of the CIC. On the Hints topic, they suggested replacing the excretory urography exam with the urodynamic study, which results should appear in the chart. Finally, there were no notes for the checklist of evaluation of this scenario.

Pre-test

Two nurses specialized in stomatherapy participated in the pre-test. The first had 17 years as a nurse and ten years as a stomatotherapist. The second was a nurse for eight years and had five years as a stomatotherapist. Both participated in clinical simulation training but had no experience with building scenarios or using them for evaluation. It happened a change in the wound scenario: removing the materials to perform the dressing because it could induce the candidate to perform it, which was not an objective proposed. As for the wound checklist, to adjust the test time, they suppressed the pulse check in the wound evaluation item; and in the topical therapy item, there was a change to topical therapy orientation as a proper way. They excluded measurement of the ankle circumference to choosing the bandage, and in the final part of the checklist, the items of educational material offering and maintenance of the organized environment were excluded for the test time’s adequacy. In the ostomy checklist, the item of disclosure of test results for the patient from the topic of initial actions was excluded; in the demarcation, they included the description of the bilateral performance; and in the item of final actions, added the guidance on clothing and excluded the organized environment’s maintenance. In the Incontinence scenario, the reformulated briefing included the patient had no indwelling bladder catheter and had been referred to a nurse to undergo further orientation for self-catheterization. Such data were recorded in the patient’s chart.

Scenarios application

The actors and evaluators, specialists in stomatherapy, received the scripts and the checklists one month in advance; and the day before the practical evaluation, a meeting was held for standardization and rehearsals and doubts clarification, along with orientation to not interfere in the scenario and time control.

On the day of the practical evaluation, each evaluator’s pair received three checklists per candidate/stage. After each candidate went through the scenario, the evaluators reached a consensus on the checklist points that presented divergence and delivered a single version with the candidate’s final score in each station. The evaluators were diagonally arranged in the scenario to visualize the actions better and not interfere or communicate during the test. One of them performed the time control of 15 minutes.

They applied the scenarios to the candidates after theoretical evaluation, composed of 60 multiple-choice questions. The candidates toured the three stations in 45 minutes. The candidates were not allowed contact during the practical evaluation, nor with those who were waiting. When they arrived at the door of the scenario room, they had access to the practical station. The evaluator presented the physical structure of the site, devices, available equipment, and the standardized patient (actor). At that moment, they could clarify doubts if they did not know any item exposed in the scenario (pre-briefing).

The briefing was passed in the form of a hospital vignette posted on the office door. The candidate should perform the role of a nurse acting in an outpatient stomatherapy and guide the patient with venous ulcer due to chronic venous insufficiency (scenario 1), make the preoperative guidance of a patient with the need to make an intestinal ostomy (scenario 2) and perform the orientation for a discharged patient who would need CIC due to spinal cord injury (scenario 3). The moulage technique was used in scenario 1 to guarantee the accuracy, which reproduced a venous ulcer in the lower limb of the standardized patient. The wound was remade at each candidate change. In scenario 2, for the ostomy demarcation, a makeup pen was made available to the candidate to replace the dermographic pen so that, at each change of candidate, the patient could remove the demarcation with a makeup solution. In scenario 3, the CIC procedure was performed in the low fidelity simulator available in the environment to prevent the patient from being exposed to an invasive procedure (Chart 1).

Each scenario’s results were measured using the verification checklists, applied by two evaluators to verify if the candidates achieved proficiencies. They could have a maximum score of 27 (scenario 1), 30 (scenario 2), and 32 points (scenario 3), and those candidates who reached an average of 50% in expected actions throughout the three scenarios would be approved.

DISCUSSION

Based on guidelines and scientific evidence and structured according to Jeffries’ theoretical model⁽¹⁸⁾, the three scenarios presented a different approach because they are summative assessment tools⁽⁴⁾. Thus, the facilitator assumed the role of introducing the professional in the scenario and was responsible for conducting and standardizing the assessment, strictly following the script. The candidates were stoma nurses who desired the title and acted individually at each scenario. The main element was the simulation design, where the specificity of the evaluated objectives was defined, determined the high fidelity, and the key points expected to solve the problem. Also, to promote learning high fidelity, and the key points that would be expected to solve the problem, which also permeates the moment of evaluation, it was necessary to design including beginning, middle, and end⁽²⁵⁾.

Another point was the delimitation of the previous time, during, and after the scenario’s application. In this study, there were pre-briefing and briefing moments with site structure and the clinical vignette model, conducting the scenario itself. The candidates performed the actions following hints provided to assist in the decision-making clinical reasoning and finalizing with an evaluation performed via a checklist. It was not applied the final stage of debriefing, common to teaching-learning practices, that allows reflection on what took place, feelings, actions are taken, aspects of improvement for future practices, and type of strategy to resume the weak points identified^(26-28).

Given the number of participants, there was no time for immediate feedback, which consists of information provided on the simulation performance to confirm the expected results⁽²⁹⁾ and improve future performance⁽²⁸⁾. In teaching, it should not only focus on strengths and weaknesses, but also reinforce learning objectives, so it is up to facilitators to promote praise and constructive criticism⁽²⁹⁾. It is an essential activity in evaluation contexts.

The care for patients with venous insufficiency needing to make ostomies and perform CIC, is a frequent activity in ostomy nurses’ clinical practice. Therefore, the attention to choose judges with experience in the area and/or on simulation was essential to create scenarios with more veracity, adding relevance and reaching the required complexity for the candidates’ title of specialist. The instruments used to evaluate these professionals’ clinical competence were validated by a committee of judges with extensive experience and knowledge in the subject. Thus, the recommendations emphasize the need for: checklists to be appreciated by specialists in the field⁽²¹⁾; validation ensures that learning objectives are achieved, and simulated experience accurately portrays the real world⁽³⁰⁾.

There is a lack of validated published scenarios in the literature, which could be replicated or adapted for teaching or evaluation. In recent studies, the validation of scenario content occurred with larger samples of judges^(19-31). There were 11 judges in this study, five per scenario, corroborating the recommended minimum of five judges for content validation⁽⁹⁾. Also, CVI measures were utilized, without the use of binomial tests to assess whether the difference between evaluators measured by CVI was significant as occurred in a previous scenario validation study⁽¹⁹⁾, since the Modified Kappa Coefficient^(22-24), specific for content analysis study^(21-22,32) and used to increase the robustness of results, was employed.

For this study, the objective of inserting clinical simulation as an evaluation method was to analyze the candidate’s clinical competences/skills for the title of specialist. In the nursing area, few specialties apply this evaluation format - one of them is the Brazilian Association of Nursing in Intensive Care, which uses the simulation of procedures and discuss clinical cases⁽³³⁾. Some medical specialties apply the analysis of clinical exams, discussion of cases, and use simulated practical stations or Structured Objective Clinical Examination (OSCE) to evaluate specialists’ expected competencies in intensive care, orthopedics, and rheumatology^(34-36). In this study, the literature recommendation regarding the OSCE was followed, which allows evaluating clinical skills⁽³⁷⁾ and mainly the application of knowledge in practice⁽³⁸⁾.

In the OSCE, the evaluations may be diagnostic, summative and/or formative, and consist of practical stations, generally with a minimum number of eight, which allow the professionals’ evaluation competence of observing their clinical performance in simulated care environments with pre-established checklists⁽³⁸⁾. The main difference between its use and the clinical simulation as an evaluation method is the level of fidelity provided by the simulation, which favors evaluating participants’ critical thinking⁽³⁹⁾.

When used for evaluation, simulation is often considered summative but can be formative⁽⁴⁰⁾. In the present work, only three summative evaluation stations were applied, which cannot be considered an OSCE itself. However, it was similar in the structural components of the stations and the rotation, with a time limit for the performance of the activity of 15 minutes, consistent with that recommended for structured examinations, in which the period for the stations varies from 5 to 20 minutes⁽³⁸⁾.

To contribute to the realism of the scenarios, moulage, and standardized patients were employed, given the technical and behavioral preparation. The first is defined as a makeup technique with effects close to reality⁽⁴¹⁾, and the second refers to actors’ use, professional or not^(30,42). For standardized patients, training is recommended to avoid excesses or situations that may compromise the objectives⁽⁴²⁾.

Another point that deserves attention in this kind of structure is the evaluators. They do not need to go through all the stations and must be different from each scenario to increase reliability. The training of assessors in defining the competencies to be assessed and how they will be identified is a crucial step⁽³⁸⁾. It is also recommended that: the checklists have thoroughly established the expected behaviors and actions so that no interpersonal differences occur; and that, preferably, the examiners are related to the discipline or the course being evaluated⁽³⁸⁾.

Study Limitations

The first limitation was the reduced number of judges in content validation and pre-test participants. This limitation was determined to reduce the possibility of information leakage since the content was used in the practical evaluation to obtain a title and was therefore confidential. Second limitation, a consensus among the researchers was used to discuss the judges’ suggestions, and no new round was held to measure the CVI and MKC. This strategy was adopted given the suggestions pointed out, which were mainly to improve the terms’ clarity and were repeated among the judges. Third: the final application of the checklists occurred with a small sample, and it was not possible to estimate other measurement properties of the instruments, such as reliability, sensitivity, among others. Moreover, finally, because it was the first time that the titration test was held in this format, it was decided to use the consensus between the marks awarded by the evaluators for each candidate.

Contributions to the Area

The use of simulated scenarios in the teaching-learning process has already been a current reality; however, it is still little applied as an evaluation method and can be a useful tool to verify if professionals and students use the knowledge transmitted by different educational strategies daily in clinical practice.

The scenarios built can be replicated both in the context of the stomatherapy specialization courses and in the undergraduate courses. However, for teaching, it is necessary to include the simulation debriefing stage to reflect on what happened, clarify doubts and favor the consolidation of knowledge, and allow a formative evaluation and opportune the application of the theoretical knowledge in the student’s practice.

CONCLUSION

The scenarios were developed on scientific evidence and covered the three significant areas of stomatherapy (wounds, stoma, and incontinence). The relevant and representative contents of the scenarios and checklists were validated and made it possible to evaluate the technical skills for the titration in stomatherapy.

The design of each scenario and the checklists for evaluating nurses’ performance were validated by judges with in-depth knowledge in clinical simulation and stomatherapy, and it was necessary to review the items that had intervalence agreement values and/or CVI less than 0.90. The pre-test phase with specialist nurses was fundamental for adjustments in the checklists and briefings provided to the candidates to achieve the proposed objectives. With the application of the scenarios, it was observed that the chosen design allowed the candidates to demonstrate their knowledge in the area, which was helped by the realism of using standardized patients and moulage techniques.

It should be pointed out that it was impossible to measure the participants’ level of satisfaction and/or perception about the evaluation format applied for a degree test. Therefore, we recommend works that verify the perceptions of the individuals participating in this evaluation method to verify the satisfaction and contributions to the improvement of the scenarios elaborated and their applicability. There is also a suggestion that studies be conducted to assess whether the scenarios and checklists used present similar results inter- and intra-assessors and investigate the other measurement properties of the checklists used to ensure the quality of the results achieved.

REFERENCES

• ¹
  Orique SB, Phillips LJ. The effectiveness of simulation on recognizing and managing clinical deterioration: meta-analyses. West J Nurs Res. 2018;40(4):582-609. doi: 10.1177/0193945917697224
  » https://doi.org/10.1177/0193945917697224
• ²
  Silva JLG, Oliveira-Kumakura ARS, Zanchetta FC, Coutinho VRD, Lima MHM. Clinical simulation for teaching of wound evaluation and treatment. Clin Simul Nurs. 2020;38:5-13. doi: 10.1016/j.ecns.2019.09.003
  » https://doi.org/10.1016/j.ecns.2019.09.003
• ³
  Lin HH. Effectiveness of simulation-based learning on student nurses' self-efficacy and performance while learning fundamental nursing skills. Technol Health Care. 2015;24(suppl1):369-75. doi: 10.3233/THC-151094
  » https://doi.org/10.3233/THC-151094
• ⁴
  Warren JN, Luctkar-Flude M, Godfrey C, Lukewich J. A systematic review of the effectiveness of simulation-based education on satisfaction and learning outcomes in nurse practitioner programs. Nurse Educ Today. 2016;46:99-108. doi:10.1016/j.nedt.2016.08.023
  » https://doi.org/10.1016/j.nedt.2016.08.023
• ⁵
  Sando CR, Coggins RM, Meakim C, Decker S, Lioce L, Borum JC, et al. Standards of Best Practice: Simulation Standard VII: participant assessment and evaluation. Clin Simul Nurs. 2013;9:30-2. doi: 10.1016/j.ecns.2013.04.007
  » https://doi.org/10.1016/j.ecns.2013.04.007
• ⁶
  Lavoie P, Michaud C, Bélisle M, Boyer L, Gosselin É, Grondin M, et al. Learning theories and tools for the assessment of core nursing competencies in simulation: a theoretical review. J Adv Nurs. 2017;74(2): 239-50. doi: 10.1111/jan.13416
  » https://doi.org/10.1111/jan.13416
• ⁷
  Associação Brasileira de Estomaterapia (SOBEST). Critérios para obtenção ou renovação do TiSobest[Internet].2020[cited 2020 Apr 18]. Available from: http://www.sobest.org.br/texto/45
  » http://www.sobest.org.br/texto/45
• ⁸
  Cheng A, Kessler D, Mackinnon R, Chang TP, Nadkarni VM, Hunt EA, et al. Reporting Guidelines for Health Care Simulation Research: Extensions to the CONSORT and STROBE Statements. Simul Healthc. 2016;11(4):238-48. doi: 10.1097/SIH.0000000000000150
  » https://doi.org/10.1097/SIH.0000000000000150
• ⁹
  Lynn MR. Determination and quantification of content validity. Nurs Res [Internet]. 1986[cited 2020 Apr 18];35(6):382-5. Available from: https://www.ncbi.nlm.nih.gov/pubmed/3640358
  » https://www.ncbi.nlm.nih.gov/pubmed/3640358
• ¹⁰
  Lee N. An evaluation on the use of adjustable compression wrapping devices as an alternative to compression bandaging in lower leg wounds. Wounds Int [Internet]. 2018[cited 2020 Jun 01];9(4). Available from: https://www.woundsinternational.com/resources/details/an-evaluation-on-the-use-of-adjustable-compression-wrapping-devices-as-an-alternative-to-compression-bandaging-in-lower-leg-wounds
  » https://www.woundsinternational.com/resources/details/an-evaluation-on-the-use-of-adjustable-compression-wrapping-devices-as-an-alternative-to-compression-bandaging-in-lower-leg-wounds
• ¹¹
  Harding K, Dowsett C, Fias L, Jelnes R, Mosti G, Öien R, et al. Simplifying venous leg ulcer management: consensus recommendations. Wounds Int [Internet]. 2015[cited 2020 Apr 18]. Available from: www.woundsinternational.com
  » www.woundsinternational.com
• ¹²
  Wound, Ostomy and Continence Nurses Society. Guideline Development Task Force WOCN Society Clinical Guideline. J Wound Ostomy Continence Nurs. 2018;45(1):50-8. doi: 10.1097/WON.0000000000000396
  » https://doi.org/10.1097/WON.0000000000000396
• ¹³
  Lauridsen S, Cobussen-Boekhorst H, Eikenboom J, Geng V, Holroyd S, Lester M, et al. European Association of Urology Nurses. Evidence-based Guidelines for Best Practice in Urological Health Care Catheterization Urethral intermittent in adults: dilatation, urethral intermittent in adults. [Internet]. 2017 [cited 2020 Apr 18]. Available from: https://nurses.uroweb.org/guideline/catheterisation-urethral-intermittent-in-adults/
  » https://nurses.uroweb.org/guideline/catheterisation-urethral-intermittent-in-adults/
• ¹⁴
  Ministério da Saúde (BR). Meias elásticas compressivas para insuficiência venosa crônica CEAP 5[Internet]. Brasília: MS; 2019[cited 2020 Apr 18]. Available from: http://conitec.gov.br/images/Consultas/Relatorios/2019/Relatorio_MeiasCompressao_CP24_2019.pdf
  » http://conitec.gov.br/images/Consultas/Relatorios/2019/Relatorio_MeiasCompressao_CP24_2019.pdf
• ¹⁵
  Cardoso LV, Godoy JMP, Godoy MFG, Czorny RCN. Compression therapy: unna boot applied to venous injuries: an integrative review of the literature. Rev Esc Enferm USP. 2018;52:e03394. doi: 10.1590/S1980-220X2017047503394
  » https://doi.org/10.1590/S1980-220X2017047503394
• ¹⁶
  Silva JC, Borsatto AZ, Teixeira ER, Umpiérrez AF. Abdominal demarcation stoma in oncological patients by stomotherapy nurse. Enferm. 2017;6(1):12-8. doi: 10.22235/ech.v6i1.1365
  » https://doi.org/10.22235/ech.v6i1.1365
• ¹⁷
  Lamin E, Newman DK. Clean intermittent catheterization revisited. Int Urol Nephrol. 2016;48:931-9.doi: 10.1007/s11255-016-1236-9
  » https://doi.org/10.1007/s11255-016-1236-9
• ¹⁸
  Jeffries PR, Rogers KJ. Theoretical framework for simulation design. In: Jeffries PR, Rogers KJ, ed. Simulation in nursing education: from conceptualization to evaluation. 2nd ed. New York: National League for Nursing; 2012. 265p.
• ¹⁹
  Andrade PON, Oliveira SC, Morais SCRV, Guedes TG, Melo GP, Linhares FMP. Validation of a clinical simulation setting in the management of postpartum haemorrhage. Rev Bras Enferm. 2019;72(3):624-31. doi: 10.1590/0034-7167-2018-0065
  » https://doi.org/10.1590/0034-7167-2018-0065
• ²⁰
  Eduardo AHA, Mendes AA, Binotto CCS, Tognoli SH, Tucci AMGB. Scenario for a simulation of health services waste: a methodological study. Online Braz J Nurs. 2016;15:611-6. doi: 10.17665/1676-4285.20165672
  » https://doi.org/10.17665/1676-4285.20165672
• ²¹
  Alexandre NMC, Coluci MZO. Content validity in the development and adaptation processes of measurement instruments. Ciênc Saúde Colet. 2011;16(7):3061-8. doi: 10.1590/S1413-81232011000800006
  » https://doi.org/10.1590/S1413-81232011000800006
• ²²
  Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? appraisal and recommendations. Res Nurs and Health. 2007;30:459-67. doi:10.1002/nur.20199
  » https://doi.org/10.1002/nur.20199
• ²³
  Cicchetti DV, Sparrow S. Developing criteria for establishing interrater reliability of specific items: application to assessment of adaptive behavior. Am J Ment Defic [Internet].1981[cited 2020 Apr 18];86:127-37. Available from: https://www.ncbi.nlm.nih.gov/pubmed/7315877
  » https://www.ncbi.nlm.nih.gov/pubmed/7315877
• ²⁴
  Fleiss J. Statistical methods for rates and proportions. New York: John Wiley; 1981.
• ²⁵
  Jeffries P. The NLN Jeffries simulation Theory. New York: National League for Nursing; 2016.
• ²⁶
  Gibbs G. Learning by doing: a guide to teaching and learning methods. Further Education Unit: Oxford Polytechnic; 1988.
• ²⁷
  Eppich W, Cheng A. Promoting excellence and reflective learning in simulation (PEARLS): development and rationale for a blended approach to healthcare simulation debriefing. Simul Healthc. 2015;10(2):106-15. doi: 10.1097/SIH.0000000000000072
  » https://doi.org/10.1097/SIH.0000000000000072
• ²⁸
  Saywer T, Eppich W, Brett-Fleegler M, Grant V, Cheng A. More than one way to debrief: a critical review of healthcare simulation debriefing methods. Simul Healthc. 2016;11(3):209-17. doi: 10.1097/SIH.0000000000000148
  » https://doi.org/10.1097/SIH.0000000000000148
• ²⁹
  Tekian A, Watling CJ, Roberts TE, Steinert Y, Norcini J. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 2017;39(12):1245-9. doi: 10.1080/0142159X.2017.1372564
  » https://doi.org/10.1080/0142159X.2017.1372564
• ³⁰
  Munroe B, Buckley T, Curtis K, Morris R. Designing and implementing simulation as a research tool. Austral Emerg Nurs J. 2016;19:90-105. doi: 10.1016/j.aenj.2016.01.001
  » https://doi.org/10.1016/j.aenj.2016.01.001
• ³¹
  Negri EC, Pereira Jr GA, Cotta Filho CK, Franxon JC, Mazzo A. Construction and validation of simulated scenario for nursing care to colostomy patients. Texto Contexto Enferm. 2019;28:e20180199. doi: 10.1590/1980-265X-TCE-2018-0199
  » https://doi.org/10.1590/1980-265X-TCE-2018-0199
• ³²
  Coluci MZO, Alexandre NMC, Milani D. Construction of measurement instruments in the area of health. Ciênc Saúde Colet. 2015;20(3):925-36. doi: 10.1590/1413-81232015203.04332013
  » https://doi.org/10.1590/1413-81232015203.04332013
• ³³
  Associação Brasileira de Enfermagem e Terapia intensiva. Edital de certificação profissional para obtenção do título de enfermeiro especialista em terapia intensiva adulto ou pediátrica ou neonatal. [Internet].2020[cited 2020 Apr 18]. Available from: http://abenti.org.br/pdf/ENF2020/EDITAL_CIETI_2020_13_12_2019.pdf
  » http://abenti.org.br/pdf/ENF2020/EDITAL_CIETI_2020_13_12_2019.pdf
• ³⁴
  Waldauf P, Rubulotta F, Sitzwohl C, Elbers P, Girbes A, Saha R, et al. Factors associated with success in the oral part of the European Diploma in Intensive Care. J Intensive Care Soc. 2017;18(4):294-99. doi:10.1177/1751143717712623
  » https://doi.org/10.1177/1751143717712623
• ³⁵
  Araújo ROD. Fatores associados à aprovação na prova de títulos de especialistas em ortopedia. [Dissertação]. Belo Horizonte: Universidade José do Rosário Vellano (UNIFENAS); 2018.
• ³⁶
  Criscione-Schreiber LG, Sloane RJ, Hawley J, Jonas BL, O'Rourke KS, Bolster MB. Expert panel consensus on assessment checklists for a rheumatology objective structured clinical examination. Arthritis Care Res. 2015;67(7):898-904. doi: 10.1002/acr.22543
  » https://doi.org/10.1002/acr.22543
• ³⁷
  Harden RM. Revisiting 'Assessment of clinical competence using an objective structured clinical examination (OSCE)'. Medical Education. 2016;50:376-9. doi: 10.1111/medu.12801
  » https://doi.org/10.1111/medu.12801
• ³⁸
  Harden RM, Patrício M, Norman G. The Definitive Guide to the OSCE: The Objective Structured Clinical Examination as a performance assessment. First edition. Elsevier 2016.
• ³⁹
  Brown S, Hoadley T, Kingston CJ. Simulation and the Objective Structured Clinical Examination: a method to evaluate students. Clin Simul Nurs. 2010;6(3):e109. doi:10.1016/j.ecns.2010.03.013
  » https://doi.org/10.1016/j.ecns.2010.03.013
• ⁴⁰
  Kelly MA, Mitchell ML, Henderson A, Jeffrey CA, Groves M, Nulty DD, et al. Adv Simul. 2016;1. doi: 10.1186/s41077-016-0014-1
  » https://doi.org/10.1186/s41077-016-0014-1
• ⁴¹
  Stokes-Parish J, Duvivier R, Jolly B. Expert opinions on the authenticity of moulage in simulation: a Delphi study. Adv Simul. 2019;4:16. doi:10.1186/s41077-019-0103-z
  » https://doi.org/10.1186/s41077-019-0103-z
• ⁴²
  Talwalkar JS, Cyrus KD, Fortin AH. Twelve tips for running an effective session with standardized patients. Med Teach. 2019. doi: 10.1080/0142159X.2019.1607969
  » https://doi.org/10.1080/0142159X.2019.1607969

Edited by

EDITOR IN CHIEF: Antonio José de Almeida Filho

ASSOCIATE EDITOR: Ana Fátima Fernandes

Publication Dates

• Publication in this collection
  24 Mar 2021
• Date of issue
  2021

History

• Received
  08 June 2020
• Accepted
  28 June 2020