Evidence of validity of the Competence Scale of Actions of Nurses in Emergencies

ABSTRACT Objective: to evaluate the validity of the Competence Scale of Actions of Nurses in Emergencies based on internal structure, internal consistency, and external criteria. Methods: methodological study to verify new evidence of validity of the Scale, with contents previously validated. The Scale has 81 measurable actions at five levels of competence and can be applied both for self- and/or hetero-evaluation. Results: one hundred and forty seven nursing assistants and 41 managers from the five regions of Brazil participated in the study. They were linked to mobile prehospital emergency service, fixed prehospital emergency service, or hospital emergencies. Dimensionality was evidenced by exploratory factorial analysis of the 81 items, pointing out seven factors that explained 66.5% of the total data variance. Cronbach’s alpha ranged from 0.79 to 0.98. The Kaiser-Meyer-Olkin 0.988 indicated that the correlations between the items were significant. In the external criterion, Pearson’s correlations between hetero-evaluation competence scores and the manager’s subjective classification were significant (p < 0.001), as well as differences in the means of these competencies by criterion group. In addition, scores by characteristics were evaluated to detect statistically different means. Conclusion: through the adopted Statistical Procedures, with multi-methods and multi-informants, different psychometric properties were analyzed. A summary of evidence was generated showing that the Scale is valid and reliable.


Introduction
In the process of evaluating professional competence, several elements are taken into consideration in the creation of a research tool capable of extracting from the respondents the best response to the alternatives offered. The availability of valid tools for assessing the workforce is essential to measure the actual competencies as well as to identify the area of practice to be developed (1) .
In Psychometrics, the concept of validity can be generically defined as the degree to which theoretical-empirical evidence supports inferences and interpretations about people's psychological characteristics. This is done based on behaviors observed/measured by a measuring instrument, always considering the relevance and usefulness of the uses proposed in certain contexts (2)(3)(4)(5) . with two types of compentencies to be considered: Basic Competencies (BC) or Associated Competencies (AC), as well as the number of compentencies in each; then to, they were constitutively defined (7) .
From the eight basic competencies (BCs) and the 32 Associated Competencies (ACs) indicated on the PCM (7) , the operational definition of 56 attitudes/ behaviours was made, represented by Identifying Questions (IQs),which resulted in the Professional CompetenceProfile (PCP) (8) . Then, evidence of validity was verified through the Delphi Technique based on the content of the IQs by nurseswho were experts in the theme, with 90% agreement and 98.61 of the Content Validity Index (9) . These 56 IQs were separated into 81 unique actions, composing the Pilot Instrument (10) . Although the validity of the content of these attitudes/behaviors is an important source, it is not enough. Demonstrating evidence of validity to use a measuring instrument is a continuous and cumulative process of studies that aggregate a set of scientific evidence (3)(4)6,(11)(12)(13) .
In order to continue the processes of construction, application and creation of a body of evidence of validity, we sought sources of information that strengthened it through Empirical/Experimental Procedures and Analytical/Statistical Procedures (6) .
In view of the above, the question raised is: Does the Scale created based on the PCM and the PCP have other sources of evidence of validity? To answer this question, we tried to evaluate evidence of validity of the Competence Scale of Actions of Nurses in Emergencies based on internal structure, internal consistency and external criteria, according to Psychometrics. Thus, the study was based on the hypothesis that the professional competences are distributed in a multifactorial structure, that the hetero-evaluations are correlated, and that the competent performance of the subject in a given item is explained by the direction of the answers given as a function of the latent trait.

Method
This was a methodological study, part of a larger research started in 2013 at a public university located in the city of São Paulo, Brazil. In the present phase, we used Psychometrics, specifically Empirical and Statistical Procedures (6) , as references to check the behavioral representation of the construct.
In the Empirical Procedures, the pilot instrument created (10)  According to Pasquali (2009), the observed score is equal to the true score plus the error (11) . Therefore, the evaluation involving a broader reasoning, combining multimethods and multi-informants, solving inconsistencies, will probably provide more realistic information about the person evaluated.
Then, the institutions were contacted by telephone calls and electronic mails. One hospital did not return the contact and was excluded from the study.

Data was collected between May 2015 and
January 2016 from nurses who agreed to participate in the study by signing the Informed Consent Term (ICT).
Nurses on leave, leave and post-graduate students lato sensu were excluded.
To collect the data, the researcher used a preestablished routine, following specific steps according to the profile of the institution and the sample:  (SEM) (12) .
The CSANE was analyzed for its dimensionality, worked in more than one institution.
In the distribution of answers to the 81 items of the original instrument, it was noticed that the answer "Not competent at all" did not occur in the selfevaluation except for the items: i (item) 6 "Periodically take part in realistic simulations in emergencies" and i 12 "Make nursing diagnosis for the patient according to the theoretical reference adopted by the institution".
This was different from the findings of the heteroevaluation. The answer "extremely competent" was given in 70 of the 81 items the self-evaluation, and in 15 of the 81 items of the hetero-evaluation, indicating the presence of a ceiling effect. This happened when more than 15% of the answers were concentrated at the maximum score of the scale (12,14) .
In the initial EFA with the 81 items, seven factors explained 66.1% of the total variance of the items. In the subsequent analyses, the i17, i15 and i13 were eliminated because their factor loads were lower than 0.4 or their commonalities were lower than 0.5. After these exclusions, another EFA was applied, resulting in seven factors that explained 66.5% of the total     The results of the factorial analysis can be interpreted through the factorial loads. Each of the "factor loads" represents the measure of correlation between the factor derived from the analysis and the original items, and can be interpreted like the Pearson correlation coefficient. Based on the factorial loads, the theoretical reference (7)(8) , the analysis of the content of the items by factor (Table 1)  to  In Table 3, there were no significant correlations between the self-evaluation and the hetero-evaluation, except for F6, Professional excellence, which although significant ( p < 0.001), presented a weak magnitude.
Thus, according to the functionality of the score, nurses      the prognoses (15) . Brazilian studies present numerous nursing care actions, but there is no mention of the SNC (16) , the ND is only identified from the analysis of other nurses' records (15,(17)(18) and although they consider it essential to perform the SNC. The ND is a step of the SNC this process, but nurses find it difficult to do it because of lack of time, high client demand and circulation, insufficient theoretical knowledge and resistance of the nurses themselves to do it (19) . These results raise concerns about what is happening because the SNC is the foundation and professional identity for excellence in the practice.
In the inferential analysis, an attempt was made  When designing the logistic regression model for this scale, it was considered the possibility to measure the degree/level of competence of nurses described by actions and data related to their latent trait. For this, the EFA was carried out through the principal components method, but not principal components analysis (20) . The Research carried out abroad on the validation of an instrument for the evaluation of the competence of nurses had psychometric values close to those of the present study (21)(22)(23) .
In this research, the authors did not find a found that it is difficult to know the degree to which the competence attributed by the nurses in their selfevaluation is related to their actual behavior (24) . In a comparative hospital study were 81 self-evaluations and their respective hetero-evaluations were conducted, it was identified that the level of competence is related to the intensity of the practice of such competence (25) .
In addition to the comparisons between self-and hetero-evaluation, the authors compared the evaluation of managers through two different methods.  (27). In another research, it was affirmed that the nurses must have a solid training in basic areas for action, clinical experience and good perception for early detection of changes that occur in traumatized patients (18) .
As for post-graduation lato sensu, the staff nurses who specialized in the modality of residence considered themselves more competent in the factor "Adaptation With regard to specialization, the specialist staff nurses were considered more competent by managers in the factor "Professional excellence". This Excellence was also pointed out by managers in the case of nurses who provided Advanced Life Support (SAV) in Cardiology, or SAV in Pediatrics, compared to those who did not shared in simulations. Therefore, the nurses who continually update their knowledge have a higher quality in the delivery of care.
In short, the summary measures of the seven factors according to the characteristics showed that the nurses trained after NCG, specialists, nurses who updated their knowledge in courses with realistic simulations, and nurses from the SAMU were evaluated as the most competent.
With all these elements that make up the CSANE, it is possible to have a vision of the set of attributes/traits that each nurse has or needs to develop.
Although all the hypotheses were confirmed, this research had the following limitations: firstly, it was impossible to compare it with other similar scales validated in Brazil, because such scales are absent.
Secondly, since the sample was of national extent and the collection was made by a single researcher, it was not possible to perform the test-retest, because this would be done with the same sample and within a short time. According to the literature, the test-retest is a form of reliability through which consistency is measured. However, the most commonly used measure to verify reliability is the internal consistency (12) .
Considering the above, internal consistency was the measured adopted in this study, which was indicated by the high KMO, showing that the items in the proposed scale measure the construct to which they were designed, and they are interrelated. In addition there were items with maximum mean values of degree of competence. However, care was taken to interpret the data set, seeking meaning to the answers given by the subjects in such a way that they

Conclusion
The statistical procedures performed allowed the conclusion that the proposed CSANE scale, based on the PCM and the PCP, presents evidence of validity based on internal structure, internal consistency and external criteria. The instrument can be considered reliable and valid to measure the real professional competence of staff nurses in Brazilian emergency units.