Development, validation and application of clinical simulation scenarios for assessment of stomatherapy specialists

Objectives : to build and validate three clinical simulation scenarios and report the application with candidates for the specialist’s degree in stomatherapy. Methods : methodological study, building three scenarios and evaluation checklists; content validation with judges, using content validity index and Modified Kappa Coefficient; pre-test and application. Results : scenarios built based on nursing care for: 1. insufficiency and venous ulcer; 2. demarcation of intestinal stomia; and 3. Clean intermittent catheterization. In the content validation of the 24 items appreciated, 83%, 80%, and 92% were validated without change. In the pre-test, the objectives and checklists were adjusted. In the application, to standardize the evaluation, actors and evaluators were trained previously, and each candidate passed the three stations. Conclusions : scenarios built and with validated content, based on evidence and covering the three areas of stomatherapy. The pre-test allowed for adjustments in the scenarios, and the candidates achieved the expected objectives. validado, basados en evidencias y abarcando todas áreas de la estomaterapia. El pretest permitió ajustes en los escenarios, y los candidatos alcanzaron los objetivos esperados. Descriptores: Evaluación Educacional; Entrenamiento por Simulación; Simulación de Paciente; Educación Validez.


INTRODUCTION
Clinical simulation is an effective teaching-learning strategy used in nursing in several areas (1)(2)(3) . One of the advantages is promoting a safe learning environment allowing the participant to practice skills that can be reviewed and reflected in a controlled environment (4) .
Another field for the use of simulation is in the process of summative or formative evaluation. Thus, the simulated environment alters its state of safe learning platform to enable the analysis of the clinical performance of the participants in the affective, cognitive, and psychomotor domains (5) . However, this process's effectiveness depends on applying validated instruments/checklists and scenarios that explore the specific skills and objectives to be evaluated. That process favors the standardization of the evaluation mechanisms and facilitators that conduct the simulation and prepare the participants to act in the simulated environment, reducing anxiety and psychological stress (5) .
Despite the importance of clinical simulation in evaluation, it is little used in nursing, although validated instruments are available to guide this method (6) . In this context, the opportunity has arisen to introduce such strategy in the titration process of the Brazilian Stomatherapy Association, which provides the certification of high qualification of the professional to act in the care of people with stoma, acute and chronic wounds, fistulas, drains, catheters and anal and urinary incontinence (7) . The evaluation used so far consisted of the performance of specific theoretical tests. Therefore, the simulated environment may be an effective strategy for the evaluation of professionals. It was raised as a hypothesis to be verified in this study if the content of the scenarios and checklists are valid in their content, and if they make it possible to evaluate the candidates' abilities for the degree test in stomatherapy. Therefore, it is questioned: The evaluation checklists and the scenarios built have relevant content and represent the specific technical skill construct for titling specialists in stomatherapy?

OBJECTIVES
To develop and validate three clinical simulation scenarios and report the application with candidates for the degree of experts in stomatherapy.

Ethical Aspects
The Research Ethics Committee of the State University of Campinas approved this study.

Design, location, and period of study
The present research is a methodological study, with a quantitative approach, following the recommendations of Simulation-Based Research Extensions for the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) Statement (8) . The development, content validation, and pre-testing of scenarios and checklists occurred from August to October 2019, and its application, in October 2019, during the Brazilian Stomatherapy Congress held in Foz do Iguaçu, Paraná.

Population and sample -exclusion and inclusion criteria
For validation, judges were invited to meet the following criteria: doctorate/master's degree in health or the area of simulation, or specialization in stomatherapy; publications, care practice and/ or teaching in simulation or stomatherapy. The sample was intentional and composed of five judges, given the availability for response in the estimated time, and followed a previous recommendation that indicates the use of this minimum number for content validation stage when the sample is highly qualified (9) .
In the scenarios' application stage, participated the candidates who took the test for the specialty degree. Also, there were six advisors in stomatotherapy, titled, and members of the Brazilian Association of Stomatherapy.

Study protocol
The study was conducted in four stages: 1. Development of scenarios and evaluation checklists; 2. Validation of content by judges; 3. Pre-testing; and 4. Application of scenarios.
To support the building of scenarios, a narrative literature review of the international consensus of reference was conducted: Wounds International (10)(11) , Wound, Ostomy and Continence Nurses Society (12) and European Association of Urology Nurses (13) and integrative or narrative review studies in the three significant areas of stomatherapy (wounds (14)(15) , ostomy (16) and incontinence (17) ).
For the structuring of the scenarios, it was followed the National League for Nursing (NLN)/Jeffries Simulation Theory (18) , which contemplates the elements: facilitator, learner (adapted for the candidate), educational practices, simulation design, and expected outcomes. Even with evaluative character, the educational practices of active learning were used once the candidates in each scenario played specialized nurses. The simulation design determined the achievement of general and specific objectives for high fidelity and moderate complexity scenarios. Through patients, clues assisted/directed the candidate's clinical reasoning. Regarding the expected outcomes, the candidates should use cognitive knowledge to decide according to clinical reasoning to: indicate and apply compressive therapy, perform stoma demarcation and demonstrate clean intermittent catheterization in the simulator. For this purpose, checklists were elaborated for the evaluators to register if the candidate did not act (0 points) if he performed it partially (0.5 points) or if he performed it correctly (1 point).
Later, in the validation of the content of the scenarios and checklists, the judges received via Google Forms ® the constructed materials and characterization tool that contemplated the variables: age, gender, training, working time as a nurse, working time in stomatherapy, working time with clinical simulation and experience with scenario building. In the second part of the form, following the content validation reference in the simulation area (19)(20) , each aspect and domain of the scenarios and checklist items were evaluated. It was considered comprehensiveness, clarity, and relevance of psychometry properties and used a 4-point Likert scale, being: 1 -item not very relevant; 2 -item needs considerable revision; 3 -item needs small revision; and 4 -relevant item (9) . Also, the judges should analyze realism, adherence to scientific evidence, objectivity, and complexity of the level of knowledge and skills required of the participant in the scenario.
The pre-test was then conducted with two stomatherapy nurses to test the evaluation's feasibility, i.e., factors that facilitate or prevent its implementation. There was no need for these nurses' previous training because they are specialists in stomatherapy, with experience in simulation. The researchers used a hospital outpatient clinic to increase environmental fidelity. The study's principal researcher, also a stomatother therapist, acted as a standardized patient, i.e., an ordinary person or actor who plays the patient; and another nurse specialized in stomatherapy applied the checklists. In the end, the nurses (acting and evaluator) made their considerations about the scenarios and checklists that were incorporated.
The last stage contemplated the application of scenarios as part of an evaluation to obtain the title. For the evaluation, each candidate went through three scenarios using actors and three evaluators' pairs, all experts in stomatherapy. Evaluators and actors received scenarios content, checklists, and scripts previously. They were guided about the evaluation dynamics, time and progress of the candidates, impossibility of conversation between evaluators and candidates during the evaluation.

Data analysis and statistics
The results of steps 1, 2, and 4 were presented descriptively and using charts to visualize the findings better. The tabulated content validation data appeared in an Excel ® spreadsheet, and the judges presented a description. Afterward came the calculation of the Content Validity Index (CVI) (21) and the Modified Kappa Coefficient (MKC) (22)(23)(24) . The calculation used in CVI had the sum of "3" and "4" responses from each judge for each item, divided by the sum of the total number of responses; items reviewed had CVI lower than 80% (21)(22) . The MKC, developed for content validity studies mainly with reduced sample size, was calculated according to the formula: = [, em que [, N represents the number of judges, and A the number of judges who evaluated a given item is relevant. Values from 0.40 to 0.59 for the MKC are considered reasonable; from 0.60 to 0.74, good; and above 0.74, excellent (22)(23)(24) . The items with CVI ≤ 80% and MKC ≤ 0.74 were reformulated according to considerations made by judges and according to researchers' consensus.
Step 3 was analyzed descriptively, considering the suggestions and improvements proposed in the pre-test, based on researchers' consensus.

Scenario development and evaluation checklists
The purpose of the building scenarios was to evaluate candidates' clinical performance in everyday situations regarding the stomatherapy specialist (Chart 1). The design of the simulations was based on the learning objectives and focused on: Scenario 1: Clinical reasoning in the evaluation and treatment of a patient with venous ulcer; Scenario 2: Preoperative preparation of a patient with the need to make an intestinal ostomy; Scenario 3: Guidance on care for a patient with the need for a clean intermittent catheterization.
The simulations were elaborated with high fidelity and the use of a standardized patient (actor), who used a predefined script to facilitate the performance of the venous insufficiency scenario; the moulage technique was used with a venous ulcer›s artistic makeup to increase realism through environmental and psychological fidelity. The scenarios presented moderate complexity, but with relevant information that allowed the candidate to perform the clinical reasoning and offer an adequate response to the objectives.
To help the evaluation verification checklists were composed with 27, 30, and 32 items. Chart 2 demonstrates the expected actions for each scenario. The evaluators should inform if they were partially or not executed, considering the objectives and expected outcomes.

Validation of content by judges
A committee of five judges reviewed each scenario and checklist. Three accredited stoma specialists evaluated each individually, and two judges, simulation experts, evaluated all three. The judges were predominantly female, with an average age of 43 years. Six judges had doctorates, four had master's degrees, and one was a specialist. The average time worked as a nurse was 19.5 years, and the average time of attendance in stomatherapy was ten years. As for clinical simulation, the judges had an average of four years of clinical simulation experience, and only two of them had experience with building simulation scenarios. Table 1 shows the CVI and MKC values for each scenario evaluation items and checklists. In the Wound scenario, judges suggested changes complied in items Objectives, Evaluation, and Information in the chart, such as replacing the verb of the action to "perform" by "evaluate"; withdraw the prescription of the treatment with the Unna boot, because in the chart there was the report of allergy to this topical therapy; include the information to the pre-briefing, justifying the low adherence to the use of the Unna boot. In the specific checklist of this scenario, the only one that presented lower values than the established standard, the judges recommended to include the expected actions during the evaluation of the wound to facilitate the observation, followed by the researchers.
In the Ostomy scenario, they suggested making more explicit the candidate role with the description of a clinical case; in the Simulation design (Objectives and Fidelity), they suggested to describe the nursing consultation items in the specific objective and add actors in the Fidelity item; in the Problem resolution item, to add nursing diagnoses that the candidate should identify; and in the Outcomes item, suggested to describe the items which they would employ to evaluate using the checklist. There were no changes in the evaluation instrument.
In the Incontinence scenario, they suggested that, in the item related to problem-solving, should be used "Demonstrate the Clean Intermittent Catheterization (CIC) technique" and not "Perform the CIC technique" because they expected the candidate would use the simulator for demonstration; and in the addition of guidance on self-catheterization, use of the voiding diary and the importance of the CIC. On the Hints topic, they suggested replacing the excretory urography exam with the urodynamic study, which results should appear in the chart. Finally, there were no notes for the checklist of evaluation of this scenario. Chart 1 -Simulation design applying three scenarios after content validation and pre-test

Pre-test
Two nurses specialized in stomatherapy participated in the pre-test. The first had 17 years as a nurse and ten years as a stomatotherapist. The second was a nurse for eight years and had five years as a stomatotherapist. Both participated in clinical simulation training but had no experience with building scenarios or using them for evaluation. It happened a change in the wound scenario: removing the materials to perform the dressing because it could induce the candidate to perform it, which was not an objective proposed. As for the wound checklist, to adjust the test time, they suppressed the pulse check in the wound evaluation item; and in the topical therapy item, there was a change to topical therapy orientation as a proper way. They excluded measurement of the ankle circumference to choosing the bandage, and in the final part of the checklist, the items of educational material offering and maintenance of the organized environment were excluded for the test time's adequacy. In the ostomy checklist, the item of disclosure of test results for the patient from the topic of initial actions was excluded; in the demarcation, they included the description of the bilateral performance; and in the item of final actions, added the guidance on clothing and excluded the organized environment's maintenance. In the Incontinence scenario, the reformulated briefing included the patient had no  indwelling bladder catheter and had been referred to a nurse to undergo further orientation for self-catheterization. Such data were recorded in the patient's chart.

Scenarios application
The actors and evaluators, specialists in stomatherapy, received the scripts and the checklists one month in advance; and the day before the practical evaluation, a meeting was held for standardization and rehearsals and doubts clarification, along with orientation to not interfere in the scenario and time control.
On the day of the practical evaluation, each evaluator's pair received three checklists per candidate/stage. After each candidate went through the scenario, the evaluators reached a consensus on the checklist points that presented divergence and delivered a single version with the candidate's final score in each station. The evaluators were diagonally arranged in the scenario to visualize the actions better and not interfere or communicate during the test. One of them performed the time control of 15 minutes.
They applied the scenarios to the candidates after theoretical evaluation, composed of 60 multiple-choice questions. The candidates toured the three stations in 45 minutes. The candidates were not allowed contact during the practical evaluation, nor with those who were waiting. When they arrived at the door of the scenario room, they had access to the practical station. The evaluator presented the physical structure of the site, devices, available equipment, and the standardized patient (actor). At that moment, they could clarify doubts if they did not know any item exposed in the scenario (pre-briefing).
The briefing was passed in the form of a hospital vignette posted on the office door. The candidate should perform the role of a nurse acting in an outpatient stomatherapy and guide the patient with venous ulcer due to chronic venous insufficiency (scenario 1), make the preoperative guidance of a patient with the need to make an intestinal ostomy (scenario 2) and perform the orientation for a discharged patient who would need CIC due to spinal cord injury (scenario 3). The moulage technique was used in scenario 1 to guarantee the accuracy, which reproduced a venous ulcer in the lower limb of the standardized patient. The wound was remade at each candidate change. In scenario 2, for the ostomy demarcation, a makeup pen was made available to the candidate to replace the dermographic pen so that, at each change of candidate, the patient could remove the demarcation with a makeup solution. In scenario 3, the CIC procedure was performed in the low fidelity simulator available in the environment to prevent the patient from being exposed to an invasive procedure (Chart 1).
Each scenario's results were measured using the verification checklists, applied by two evaluators to verify if the candidates achieved proficiencies. They could have a maximum score of 27 (scenario 1), 30 (scenario 2), and 32 points (scenario 3), and those candidates who reached an average of 50% in expected actions throughout the three scenarios would be approved.

DISCUSSION
Based on guidelines and scientific evidence and structured according to Jeffries' theoretical model (18) , the three scenarios presented a different approach because they are summative assessment tools (4) . Thus, the facilitator assumed the role of introducing the professional in the scenario and was responsible for conducting and standardizing the assessment, strictly following the script. The candidates were stoma nurses who desired the title and acted individually at each scenario. The main element was the simulation design, where the specificity of the evaluated objectives was defined, determined the high fidelity, and the key points expected to solve the problem. Also, to promote learning high fidelity, and the key points that would be expected to solve the problem, which also permeates the moment of evaluation, it was necessary to design including beginning, middle, and end (25) .
Another point was the delimitation of the previous time, during, and after the scenario's application. In this study, there were pre-briefing and briefing moments with site structure and the clinical vignette model, conducting the scenario itself. The candidates performed the actions following hints provided to assist in the decision-making clinical reasoning and finalizing with an evaluation performed via a checklist. It was not applied the final stage of debriefing, common to teaching-learning practices, that allows reflection on what took place, feelings, actions are taken, aspects of improvement for future practices, and type of strategy to resume the weak points identified (26)(27)(28) .
Given the number of participants, there was no time for immediate feedback, which consists of information provided on the simulation performance to confirm the expected results (29) and improve future performance (28) . In teaching, it should not only focus on strengths and weaknesses, but also reinforce learning objectives, so it is up to facilitators to promote praise and constructive criticism (29) . It is an essential activity in evaluation contexts.
The care for patients with venous insufficiency needing to make ostomies and perform CIC, is a frequent activity in ostomy nurses' clinical practice. Therefore, the attention to choose judges with experience in the area and/or on simulation was essential to create scenarios with more veracity, adding relevance and reaching the required complexity for the candidates' title of specialist. The instruments used to evaluate these professionals' clinical competence were validated by a committee of judges with extensive experience and knowledge in the subject. Thus, the recommendations emphasize the need for: checklists to be appreciated by specialists in the field (21) ; validation ensures that learning objectives are achieved, and simulated experience accurately portrays the real world (30) .
There is a lack of validated published scenarios in the literature, which could be replicated or adapted for teaching or evaluation. In recent studies, the validation of scenario content occurred with larger samples of judges (19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30)(31) . There were 11 judges in this study, five per scenario, corroborating the recommended minimum of five judges for content validation (9) . Also, CVI measures were utilized, without the use of binomial tests to assess whether the difference between evaluators measured by CVI was significant as occurred in a previous scenario validation study (19) , since the Modified Kappa Coefficient (22)(23)(24) , specific for content analysis study (21)(22)32) and used to increase the robustness of results, was employed.
For this study, the objective of inserting clinical simulation as an evaluation method was to analyze the candidate's clinical competences/skills for the title of specialist. In the nursing area, few specialties apply this evaluation format -one of them is the Brazilian Association of Nursing in Intensive Care, which uses the simulation of procedures and discuss clinical cases (33) . Some medical specialties apply the analysis of clinical exams, discussion of cases, and use simulated practical stations or Structured Objective Clinical Examination (OSCE) to evaluate specialists' expected competencies in intensive care, orthopedics, and rheumatology (34)(35)(36) . In this study, the literature recommendation regarding the OSCE was followed, which allows evaluating clinical skills (37) and mainly the application of knowledge in practice (38) .
In the OSCE, the evaluations may be diagnostic, summative and/or formative, and consist of practical stations, generally with a minimum number of eight, which allow the professionals' evaluation competence of observing their clinical performance in simulated care environments with pre-established checklists (38) . The main difference between its use and the clinical simulation as an evaluation method is the level of fidelity provided by the simulation, which favors evaluating participants' critical thinking (39) .
When used for evaluation, simulation is often considered summative but can be formative (40) . In the present work, only three summative evaluation stations were applied, which cannot be considered an OSCE itself. However, it was similar in the structural components of the stations and the rotation, with a time limit for the performance of the activity of 15 minutes, consistent with that recommended for structured examinations, in which the period for the stations varies from 5 to 20 minutes (38) .
To contribute to the realism of the scenarios, moulage, and standardized patients were employed, given the technical and behavioral preparation. The first is defined as a makeup technique with effects close to reality (41) , and the second refers to actors' use, professional or not (30,42) . For standardized patients, training is recommended to avoid excesses or situations that may compromise the objectives (42) .
Another point that deserves attention in this kind of structure is the evaluators. They do not need to go through all the stations and must be different from each scenario to increase reliability. The training of assessors in defining the competencies to be assessed and how they will be identified is a crucial step (38) . It is also recommended that: the checklists have thoroughly established the expected behaviors and actions so that no interpersonal differences occur; and that, preferably, the examiners are related to the discipline or the course being evaluated (38) .

Study Limitations
The first limitation was the reduced number of judges in content validation and pre-test participants. This limitation was determined to reduce the possibility of information leakage since the content was used in the practical evaluation to obtain a title and was therefore confidential. Second limitation, a consensus among the researchers was used to discuss the judges' suggestions, and no new round was held to measure the CVI and MKC. This strategy was adopted given the suggestions pointed out, which were mainly to improve the terms' clarity and were repeated among the judges. Third: the final application of the checklists occurred with a small sample, and it was not possible to estimate other measurement properties of the instruments, such as reliability, sensitivity, among others. Moreover, finally, because it was the first time that the titration test was held in this format, it was decided to use the consensus between the marks awarded by the evaluators for each candidate.

Contributions to the Area
The use of simulated scenarios in the teaching-learning process has already been a current reality; however, it is still little applied as an evaluation method and can be a useful tool to verify if professionals and students use the knowledge transmitted by different educational strategies daily in clinical practice.
The scenarios built can be replicated both in the context of the stomatherapy specialization courses and in the undergraduate courses. However, for teaching, it is necessary to include the simulation debriefing stage to reflect on what happened, clarify doubts and favor the consolidation of knowledge, and allow a formative evaluation and opportune the application of the theoretical knowledge in the student's practice.

CONCLUSION
The scenarios were developed on scientific evidence and covered the three significant areas of stomatherapy (wounds, stoma, and incontinence). The relevant and representative contents of the scenarios and checklists were validated and made it possible to evaluate the technical skills for the titration in stomatherapy.
The design of each scenario and the checklists for evaluating nurses' performance were validated by judges with in-depth knowledge in clinical simulation and stomatherapy, and it was necessary to review the items that had intervalence agreement values and/or CVI less than 0.90. The pre-test phase with specialist nurses was fundamental for adjustments in the checklists and briefings provided to the candidates to achieve the proposed objectives. With the application of the scenarios, it was observed that the chosen design allowed the candidates to demonstrate their knowledge in the area, which was helped by the realism of using standardized patients and moulage techniques.
It should be pointed out that it was impossible to measure the participants' level of satisfaction and/or perception about the evaluation format applied for a degree test. Therefore, we recommend works that verify the perceptions of the individuals participating in this evaluation method to verify the satisfaction and contributions to the improvement of the scenarios elaborated and their applicability. There is also a suggestion that studies be conducted to assess whether the scenarios and checklists used present similar results inter-and intra-assessors and investigate the other measurement properties of the checklists used to ensure the quality of the results achieved.