Assessment of professional competence of nurses in emergencies : created and validated instrument

Objective: To create an instrument to assess the professional competence of nurses in emergencies. Method: Methodological study carried out between 2013 and 2016 in a Brazilian university. We used the Pasquali’s model and Psychometrics as references because they allow subjective phenomena to be measured. The survey covered defi ning steps of the behavior to be measured, the formulation of the items in the instrument, development of instructions for using the instrument and elements necessary to demonstrate validity evidence. Although we have previously performed, in a distinct and consecutive way, the literature review, defi nition of the Core (or Matrix) Competence and the Competence Profi le, and identifi cation of validity with Delphi, essential components to structure a technology, in this step we continued the conclusion of these Theoretical Procedures. Result: We created an instrument for selfand/or hetero-evaluation containing: personal/professional/academic characterization, Competency Scale, fi ctitious cases and actions representing the practice. Conclusion: We created a new softhard technology based on the validity evidence of contents according to experts of the fi ve regions of Brazil. Descriptors: Assessment of Professional Performance; Professional Competence; Nursing in Emergency; Psychometrics; Validity of Tests. RESUMO Objetivo: Criar Instrumento de Avaliação da Competência Profi ssional do Enfermeiro em Emergências. Método: Estudo metodológico realizado entre 2013 e 2016 em universidade brasileira. Usou-se o modelo de Pasquali e a Psicometria como referenciais, permitindo que fenômenos subjetivos sejam medidos. A pesquisa abrangeu etapas de defi nição do comportamento a ser medido, formulação dos itens da ferramenta, desenvolvimento de instruções para uso do Instrumento e de elementos necessários para demonstrar evidências de validade. Embora realizadas previamente de forma distinta e consecutiva revisão da literatura, defi nição da Matriz e do Perfi l de Competências e identifi cação de validade com Delphi, componentes essenciais para estruturar uma tecnologia, deu-se continuidade à fi nalização desses Procedimentos Teóricos nesta etapa. Resultado: Instrumento criado para auto e/ou heteroavaliação contendo: caracterização pessoal/ profi ssional/acadêmica, Escala de Competências, casos fi ctícios e ações representativas da prática. Conclusão: Construiu-se tecnologia leve-dura inédita alicerçada pelas evidências de validade do conteúdo consensuado por experts das cinco regiões brasileiras. Descritores: Avaliação de Desempenho Profi ssional; Competência Profi ssional; Enfermagem em Emergência; Psicometria; Validade dos Testes. RESUMEN Objetivo: Crear Instrumento de Evaluación de la Competencia Profesional del Enfermero en Urgencias. Método: Estudio metodológico realizado entre 2013 y 2016 en universidad brasileña. Se utilizó el modelo de Pasquali y la Psicometría como referenciales, pues permiten que fenómenos subjetivos sean medidos. La investigación abarcó etapas de defi nición del comportamiento a ser medido, la formulación de los ítems de la herramienta, desarrollado de instrucciones para el uso del instrumento y de elementos necesarios para demostrar evidencias de validez. Aunque ya haya sido realizada previamente de manera distinta y consecutiva la revisión de la literatura, la defi nición de la Matriz y del Perfi l de Competencias e identifi cación de validez con Delphi, los componentes esenciales para estructurar una tecnología, se dio continuidad a la fi nalización de esos Procedimientos Teóricos en esta etapa. Resultado: Assessment of professional competence of nurses in emergencies: created and validated instrument Avaliação da competência profi ssional do enfermeiro em emergências: Instrumento criado e validado Evaluación de la competencia profesional del enfermero en urgencias: Instrumento creado y validado


INTRODUCTION
In Brazil, as of 2010, there was a reorganization of the Brazilian Unified Health System (SUS) in five Thematic Health Care Systems, which considered the aspect of justice and efficiency, focusing on the situations of greater vulnerability in this system. Specifically the Health Care to Emergencies was organized in components interconnected to each other, to promote health care to acute and chronic conditions, generating various challenges to professionals in this health care (1) .
Henceforth, one of the challenges of the nurse manager in emergency services is to identify in each nurse their competencies and if such must be developed, in order to adapt the professionals to the new context. Since nurses who work in the emergency room stand out for their technical responsibility in taking care of their patients both for their work and that of the nursing staff, without the expense of the administrative staff, this multiplicity of actions requires the development of specific capacities to the professional exercise in level of excellence. As a result, to have an instrument that identifies existing competencies and those to be acquired becomes very meaningful to the distinction of the uniqueness of the actions for a safe, humane and professional practice and without risk to the patient, the nurse, the institution and the community. To do so, an instrument to evaluate the specific competence of nurses in their field of expertise is necessary.
The routine and systematic assessment of the level of competence of nurses in the workplace represents an important object of interest among educators, health managers and other professionals in different spheres (2) . It enables tracing the competencies along the nurses' professional career, their path, and the achievement of objectives (2) .
A literature research conducted by European researchers related to such theme identified several self-evaluation instruments for the nurse competence, but most focused on a general context (2) . In our study, we found that in Brazil, so far, there is no published study on a technology to create an instrument capable of measuring the professional competence of the nurse in emergencies in a systematic and scientific way, whether to use it in self-or hetero-evaluation or as a support for the nursing manager or the professionals themselves.
In this context, the creation of a specific instrument capable of measuring the professional competence of the nurse working in emergencies was essential, based on the profiles of nurses, patients, the institution, and also of the Brazilian public policy for the health care provided in this area, following a specific methodology. Considering these facts, we had the following question: which procedures of the reference psychometrics are indispensable for creating an instrument to assess the professional competence of nurses in emergency?

OBJECTIVE
To create an instrument to assess the professional competence of nurses in emergencies, considering psychometrics as the theoretical-methodological reference for its development.

Ethical aspects
To prepare the Assessment Instrument of the Professional Competence of Nurses in Emergencies, national and international standards of ethics in research involving humans were met, with approval of the Research Ethics Committee in 01/06/2012.

Study design, location and period
This was a methodological study with psychometric reference, part of a larger survey, carried out between 2013 and 2016 in a public university located in the city of São Paulo, Brazil. Testing of the instrument developed in this research occurred in April 2015 with nurses from a large and of high complexity teaching hospital, reference in the Health Care System, serving primarily patients of the Brazilian Unified Health System (SUS).
Population, sample, inclusion and exclusion criteria Two groups of nurses took part in the testing of the instrument. The groups differed regarding the time of performance in emergencies and specialized training. The first group had seven nurses enrolled in the first and second years of the Interprofessional Residency Program in Emergency Service, being carried out in a university environment in the afternoon shift. The other group consisted of seven nursing assistants and three managers who worked in the hospital emergency service. Testing of the instrument took place during working hours, in the morning, afternoon, and night shifts in all sectors linked to the emergency service. The average time of the first group was 20 minutes, and 30 minutes for the second group.

Study protocol
The methodological research is justified because it is characterized by developing data collection instruments that can assume different purposes (3) . Considering that all processes that were necessary for the preparation of this research were supported by the behaviors of the daily practice of the nurses, psychometrics has proven to be the most appropriate reference for creating the assessment instrument of the professional performance of nurses in emergencies. Psychometrics is a theory measurement technique that allows the interface of psychology with statistics, enabling the empirical observation of phenomena that cannot be directly observed (4)(5) . Moreover, it enables the systematization of a plausible explanation for the meaning of the numeric responses given by the subjects to a series of items that represent the defining behaviors of the competencies (4)(5) .
In addition, within the line established by psychometrics, we seek to create an instrument with descriptive and predictive purposes, of diagnostic classification, whose results may be used to plan interventions (6) . While the descriptive purpose is related to interpret the results of the instrument to understand the aspects that highlight the most the strengths or weaknesses in terms of competencies, the predictive purpose refers to the use of the results of the instrument to anticipate other features of people's behavior with the construct assessed by measurement of competencies.
In turn, the purpose of diagnostic classification is related to the detailed examination of the results of the instrument as one of the information sources to identify people's characteristics that may be interpreted within a taxonomic system, which limits the presented levels of competence, for example, low, medium and high. Finally, the results obtained with the implementation of the instrument can also be used to support decision-making regarding interventions that may be needed in the field of competencies.
For the instrument of this study to be prepared and to meet the aforementioned purposes, the following steps were covered: definition of the behavior to be measured, formulation of the items of the instrument, as well as the development of instructions for using the instrument and the elements necessary to demonstrate validity evidence in psychometrics. When creating the Assessment Instrument of the Professional Competence of Nurses in Emergencies we used the model proposed by Pasquali, which is divided in three lines named Theoretical Procedures, Empirical/Experimental Procedures, and Analytical/Statistical Procedures (4)(5) .
Considering that our study aimed to elaborate a research instrument, we only used as reference the Theoretical Procedures, since they are the only ones needed in this stage of the study. Such procedures were the basis for creating the instrument, since these, in addition to highlighting the theory that justifies the psychological object for which we want to develop a measuring instrument, also highlight the operationalization of the construct in items (4)(5) . To ground this creation and support it scientifically there was need to meet steps in sequence: review of the Brazilian literature, creation of a mini-theory, definition of the construct and the instrument elements relevant to the assessment instrument, as shown in Figure 1.
We developed two integrative reviews in the Brazilian literature review, one regarding professional competence of nurses in the world of work and the other regarding emergencies, in order to identify the existence of scientific studies in the field, and from the results, to define guiding elements for creating the research instrument. From the works identified, only two theses have a theme related specifically to our research, and any of them was related to the development of an assessment instrument of nurses' performance in emergencies.
For creating the mini-theory we considered the empirical evidence obtained in the review of the literature, public policy in emergencies, the purpose of the instrument as well as the knowledge and experience of researchers in management and emergencies. We also considered the concept of nurses' professional competence as the nurses' ability to provide, coordinate and ensure proactive and individual human care, in a planned way, based on scientific and technical knowledge to achieve the best result by working on time, as we propose in this research.
First we chose the Basic Competencies and the Associated Competencies essential to the capable performance of the nurse.  It is worth mentioning that the Basic Competence is the ability of nurses to perform their work based on a set of general and specific knowledge associated with skills and attitudes that result in added value to their professional and social actions, differing them within the reality they work; on the other hand, the Associated Competence is the support required for the development of each Basic Competence for the qualified performance of the nurse. After selecting the two types of competencies, we elaborated the constitutive definition of each one of them, in order to delimit the ideas that characterized them in a Professional Core Competence (or Matrix) (7) . From these concepts we have developed the operational definitions, characterizing the behaviors that demonstrated the proposed competences, named Identifying Questions, to determine a Professional Competence Profile (8) . By elaborating this process, we created the mini-theory which allowed us to ground the development of the construct we wanted to include in the assessment instrument of each professional competence of nurses. From this Core Competence and the Competence Profile, we continue the creation process of the Assessment Instrument, using the Delphi method, according to which expert nurses evaluated, using the five-point Likert scale, agreement or disagreement quantitative data, and, in a proper space for comments and suggestions, the interpretative content of relevance, clarity, and compliance. The comments suggested minor adjustments in writing and the inclusion of a new professional competence. Thus, we created and conceptualized the Professional Growth associated competence, as well as its Identifying Question, which were inserted into the Assistance Performance basic competence. The Delphi method was used in four steps with a national sample of expert nurses on the subject: 1st Step: n = 25; 2nd Step: n = 21; 3rd Step: n = 21; and 4th Step: n = 18. In the last step the questions evaluated with the Likert scale obtained 90% consensus of Percentage Score and 98.61 of Content Validity Index, demonstrating validity evidence.
When defining the construct, each of these Identifying Questions were arranged into items to represent only one action, but without disassociating the central idea of each behavior already described. After arranging them in items, their content was submitted to experts to verify understanding, clarity, and objectivity. Then, the items started to integrate the assessment instrument in the part pertaining to the measurable actions in daily practice of the nurse in emergencies, in level of excellence.
For creating this instrument, it was necessary to define all the elements that would compose it: spreadsheet with the actions to be measured on the daily practice of nurses in emergencies and the levels of competence with the possible alternatives of responses; scale of assessment of the degree of professional competence of nurses in emergencies; personal and professional characterization data of the subjects; and the description of cases representing the reality to verify the competence level of nurses in their daily practice.
The spreadsheet contained 81 items arranged at random in its lines, constituting a bank of items. The degree of competence to be marked was in five columns according to the assessment of what each item represented to the nurse, ranging from five, equal to the extremely competent level, to one, corresponding to not competent at all.
To resolve doubts about what each competence level means, which were presented as response alternatives in the spreadsheet, we developed a scale that differentiated the quality of devotion to the care provided in each action within the five levels. The assessment scale of the levels of professional competence of nurses in emergencies was understood as a set of data that define quantitative and qualitative variables to verify the quality of the actions of nurses in their daily practice in emergencies, as showed in Chart 1.

Level of professional competence
Quality obtained by the performance of nurses in emergencies.

Frequency of action
Frequency with which the actions proposed to the competent practice of nurses in emergencies are carried out.

Autonomy
Relation of independence of nurses and the need for supervision in daily practice of actions in emergencies, considering their complexity.

Complexity of the action
It considers the required knowledge associated with the proper techniques for nurses to take actions in their daily practice, from the basic to the advanced ones.

Results of the actions
Result obtained by the range of the objectives recommended for the excellence for the practice in emergencies.
Another part of the research instrument contemplated data of personal identification and those of the nursing assistant as well as those regarding their education and professional development. These professional data of the sample included variables related to the probability of the nurses giving an answer compatible with a particular item in line with their level of competence at work.
To complement the data to characterize the level of competence at work of the research subjects, we described three fictitious cases about emergency care as a control element. We aimed to verify the ability of the respondent in analyzing each situation and rate it regarding the level of competence of the nurse in the actions described in the respective cases.

Analysis of results
Testing of the instrument with a pilot study had the purpose of identifying possible limitations of the product and the process. Regarding the product, the focus was to verify difficulties as for the language, understanding of the text, and use of the scale of competence level, both in the spreadsheet and in the cases. As for the process, special attention was given to the time taken to fill the assessment instrument and the most suitable place for its application.
With the pilot test we verified the need for making minor changes in the writing of a few items, and in the second group, we deemed necessary for the nurse manager to do the hetero-evaluation of the nursing assistant only after these have done the self-evaluation.

RESULTS
The assessment instrument of the professional competence of nurses in emergencies was divided into five parts. Part A, entitled characterization of nurses to be evaluated, contained data regarding age, sex, marital status, number of children, place, time, working time, and leave of absence. Part B, entitled professional training of nurses, included graduation year, specialization course or graduate program in specialized fields, and emergency programs completed in the last two years. We also verified participation in scientific events, study groups, commissions and committees, and scientific activities already carried out. Parts C, D and E are presented respectively in Charts 2, 3 and 4, totally or in an exemplified way. To choose the level of competence to be indicated in each action, please consider the quality of devotion to the provided care as a reference, i.e., the association of the following variables: frequency, autonomy, complexity, and result of the action, in such a way that it corresponds to the level of competence you deem more appropriate for each action described.
The parameters delimiting the levels of competence of nurses in emergencies, such as "Extremely competent" and "Not competent at all", will serve as a reference for you to reply the three cases exemplified and to the self-evaluation or hetero-evaluation described next: Degree/Level of competence Quality of devotion to the provided care

Extremely competent
Always perform the action described in your daily practice with all required knowledge and use of proper techniques, independently and without any need for supervision when taking managerial and assistance activities at all levels of complexity, achieving the recommended objectives to execute it.

Very competent
Almost always perform the action described in your daily practice with the aimed knowledge and use of proper techniques, independently, although with the need for supervision when taking more complex managerial activities, in such a way to achieve the recommended objectives to execute it.

Competent
Frequently perform the action described in your daily practice with enough knowledge and use of proper techniques, independently and with the need for supervision exclusively when taking more complex managerial activities, achieving the recommended objectives to execute it.

Little competent
Sometimes perform the action described in your daily practice with a certain lack of knowledge, and often without the use of proper techniques, not always independently and therefore with need for supervision in assistance and managerial actions of medium complexity, in order to achieve most of the objectives recommended to execute it.
1 Not competent at all Rarely perform the action described in your daily practice, with a lack of knowledge and not using proper techniques, requiring constant supervision in the actions of low complexity in such a way to achieve the objectives recommended in the plan.
Chart 3 -Example of one of the fictitious cases representing the reality for assessing the level of competence of nurses, considering their own daily practice, São Paulo, Brazil, 2016 Part D -Assessment of level of competence presented by the nurses in the following situations. Mark with an "X" the degree of competence that best depicts the reality of the work of nurses described in each case.
Case 2 -Adult man, mixed-race, aged approximately 30 years was found lying in the street victim of a gunshot wound in the chest. He was attended by rescuers and sent to the Emergency Room (ER). On that late shift on Sunday, as a routine determined by the nurse, only half the nursing staff that should compose the scale in the ER was present. When the rescuers arrived, most of the nursing staff was already attending a young person, ZBN, aged 20 years, victim of a motorcycle fall, and also an older lady, AMS, aged 65 years, with a history of sudden and bounding headache. Immediately, the nurse who was providing care to the old lady, AMS, received the gunshot victim and asked the nursing technician who was taking care of the patient FCD, aged 35 years, who was confused and without a companion, to get the chest drainage material. While the technician was picking up the material, his/her patient fell out of the bed, before the nurse or his/her team could intervene. The husband, who was outside waiting to visit her, heard the screams of his wife and broke into the ER. When he saw her lying on the floor, he started arguing with the nurse, who sought to avoid conflicting behaviors when hearing his complaints.
Nurses' actions in emergencies Competence degree Listen to people with clear interest in meeting their requests.
Use the freedom of action regarding the law and without harm to others.
Consider the risks involved when choosing actions. 27. Assume co-responsibility for the work of the nursing staff in providing care in emergencies.
28. Spontaneously provide help to one or more people at work in a coherent manner.
29. Listen to people without prejudging their ideas and positions.
30. Have energy, perseverance, and moral force in confronting the difficulties present in the everyday life.

Consider the risks involved when choosing actions.
32. Direct efforts to anticipate actions before problems arise.
33. Use other opportunities for your professional development.
34. Use the freedom of action regarding the law in force and without harm to others.

35.
Share with others what a fact or thought means in the context in which it was exposed.
36. Act with self-confidence in unexpected work situations that can really be resolved by your interference.
37. Spontaneously conceive one or more useful ideas to the work completion. Part E -Representative items of actions of nurses in emergencies proposed by Brazilian researchers. Consider your actions on a daily basis to indicate the degree of competence that best reflects your work reality.

DISCUSSION
Creating a measurement instrument for assessing the professional competence of nurses in emergencies was quite a challenge, because this is a new area in Brazilian Nursing, which demands knowledge of other specific areas such as psychometrics.
Although this measurement instrument was not a psychological test, doing the research from the perspective of a psychometric reference allowed us to characterize, describe, diagnose, and predict behaviors with defined standards regarding what was expected (6) . When defining exactly what the construct predicts was extremely relevant to indicate how care should be managed, how the training of nurses should be planned, including how the performance of such professionals should be evaluated in such a way to intervene in time.
When creating the instrument, two processes had to be involved: the creation of the instrument itself and the demonstration of validity evidence. Although distinct, they are interrelated. None was more important than the other, since both had focused on structures that depict the behavior to be measured. Considering it was a long research with several steps, we verified that the existing processes related to the creation of the instrument were the most emphasized at this step of the research.
According to Pasquali, to survey the scientific production carried out on the subject during this process is important, since the research can corroborate and support the construction of the instrument (4)(5) . Identifying evidence in the literature to guide the preparation of the construct and systematize it was not a particularity of our study, but it is a global reality (9)(10)(11)(12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22) . Considering the lack of Brazilian literature focused on the professional competence of nurses in emergencies, we have proposed, based on our extensive experience, Basic and Associated Competencies and Identifying Questions to enable the definition of actions to be measured in the instrument, properly represented in 81 items described in the spreadsheet.
We did not find assessment instruments of the competencies of nurses both in the wide researched Brazilian literature and in international literature. The American nurse Patricia Schwirian can be considered the pioneer in creating an assessment instrument on this issue. After the literature review and consultation with experts, the nurse created a scale of nurses' performance according to the theoretical-methodological procedures of psychometrics (9)(10) .
Considering competence proposals for the nurse to work in emergencies, we verified that in the new millennium (12)(13)(14)(15)(16)(17)(18) trade associations and nursing researchers from other countries identified and elaborated constitutive and operational definitions of competencies. In the United States of America, for instance, the Emergency Nursing Association (ENA) summoned nurses and, based on the "core competencies" created, established ten entry level competencies and 60 items for Nurse Practitioner (NP), and seven competencies and 25 items for Clinical Nurse Specialist (CNS) (14)(15) . In the Australian continent, expected performance standards were defined for the nurse specialist in emergency containing nine areas of competence and 112 items (16) . In Europe, nurse Riitta Meretoja, with collaboration of other authors, have proposed seven competencies and 73 items for nurses working in different sectors, among them, the hospital emergency (17)(18) .
In our study, in addition to the content of the items to assess the construct, we defined the scale of assessment of the degree/ level of professional competence of nurses in emergencies. We also sought, in addition to grading the competence, to describe the quality of devotion to the provided care, and we did not find similar scales for nurse evaluation in all consulted literature, specifically in this area.
In a more general scope, in the United States of America, in 1982, the American Patricia Benner developed and applied the classic model of the Dreyfus brothers and established a clinical competence scale with five levels of abilities: Novice, Advanced Beginner, Competent, Proficient and Expert. In her research she sought to distinguish the differences between such levels. She theorized that the experience in clinical practice is the basis for nurses' performance, since it allows to continuously expand their knowledge ensuring a holistic and competent care to the patient (11) . This scale has been used and approved in several studies. In Finland, nurse Riitta Meretoja, with the collaboration of other authors, proposed an assessment instrument of nurses with a 0-100-value visual analog scale to measure levels of competence, considering the weak, moderate, good, and very good alternatives. She applied it in various sectors of a hospital, including the emergency (17)(18) . In China, researchers have created a five-point scale with options of strongly agree to disagree of the item described in an assessment instrument of nurses' competence in obstetrics (19) .
Considering the literature found in more than one continent, we could identify several components of the assessment instruments of nurses' competencies. In order to allow a more accurate analysis of the replies given by the research subjects in the instrument, other elements were also included:-demographic characterization of respondents, data regarding their professional training and development, and the use of fictional cases. Such indicators are justified because the respondents, when it comes to an assessment, may underestimate or overestimate their ability, and it is necessary to combine information and resolve inconsistencies, using multimethods and multitrainers.
Although empirical evidence about the most effective or safe indicators are still scarce, there is a general consensus in the literature that the competence assessment should use more than one indicator such as self-evaluation, hetero-evaluation, direct observation, Objective Structured Clinical Examination (OSCES), and realistic simulation, among others (21)(22) .
Regarding the behavior of health professionals in the selfevaluation, in studies that more emphatically address this subject, doctors and nurses are the professionals that most perform such evaluation. In a systematic review conducted in 2006, which aimed to determine the precision with which doctors perform self-evaluation compared with external observations, the researcher grouped the results into methods of self-evaluation, external evaluation, comparison between both of them, and the accuracy of the self-evaluation (23) . The researcher identified that self-evaluations were carried out through questionnaires, checklists or research focused on the needs of learning, general clinical skills, knowledge of medical evaluation and critical assessment as well as non-clinical skills. Moreover, the self-evaluation was compared with stable external indicators of measurement such as OSCEs, simulations, training performance, and the ability to explain evidence-based concepts to an interviewer (23) . In New Zealand, researchers used an assessment instrument of the performance of nurses and doctors in a realistic simulation of emergency (24) .
Finally, it is worth mentioning that we are developing new studies for demonstrating validity evidence and accuracy to scientifically analyze the use and interpretation of the results of this assessment instrument of nurses who work in emergencies, to be reported in future publications.

Study limitations
Despite all the care that surrounded the creation of this instrument, there were some limitations because it is a pioneering study in Brazil without the possibility of comparison with equivalent proposals. Such datum has highlighted the need to move forward with the research, and we recommend joining forces between nursing researchers, since this procedure can strengthen them in this area.

Contributions to the nursing field
The survey allowed contributing in the advancement of nursing in creating new technologies regarding professional self-evaluation and/or hetero-evaluation as a starting point for future studies.

CONCLUSION
The Assessment Instrument of the Professional Competence of Nurses in Emergencies was created and developed based on validity evidence of the content of the items, in compliance with all the steps of the theoretical procedures contained in the psychometric reference. We hope this can be a mean for identifying strengths and weaknesses in such a way to improve the existing level of competence and/or the development of new competencies.

FUNDING
This research was funded by the National Council for Scientific and Technological Development (CNPq), process no. 483.449/2013-0.