VALIDATION OF COMPETENCIES ASSESSMENT SCALE IN A UNIVERSITY HOSPITAL NURSING TEAM

VALIDACIÓN DE LA ESCALA DE EVALUACIÓN DE COMPETENCIAS EN UN EQUIPO DE ENFERMERÍA DE UN HOSPITAL UNIVERSITARIO

ABSTRACT

Objective:

to analyze the evidence of validity of an instrument for the evaluation of competencies in a nursing team of a university hospital.

Method:

methodological study to investigate evidence of validity of the scale, with previously validated content, was sent to 978 technical-administrative employees in education who worked in the nursing team in a university hospital, and the non-parametric sample consisted of 143 employees. The scale consisted of 10 competencies and 25 measurable actions in four levels of competencies. The internal consistency of the instrument was analyzed by calculating Cronbach's alpha coefficient and construct validity by confirmatory factor analysis.

Results:

53 nurses, 30 nursing technicians and 60 nursing assistants participated, totaling 143 professionals working in the hospital of a federal university in the state of São Paulo, Brazil. The confirmatory factor analysis of the instrument, based on the structure of 10 competencies, presented good adjustment indexes with χ²/GL=1.706, SRMR=0.071, RMSEA=0.070 and CFI=0.906, conferring construct validity. Cronbach's alpha was 0.924.

Conclusion:

through the statistical procedures adopted, it was concluded that the instrument has evidence of validity based on both the internal structure - construct, and internal consistency.

DESCRIPTORS:
Nursing; Nursing human resources; Public administration; Professional competence; Psychometry; Reproducibility of tests

RESUMEN

Objetivo:

analizar la evidencia de validez de un instrumento para la evaluación de competencias en un equipo de enfermería de un hospital universitario.

Método:

estudio metodológico para investigar evidencias de validez de la escala, con contenido previamente validado, se envió a 978 empleados técnico-administrativos en educación que laboraban en el equipo de enfermería de un hospital universitario, y la muestra no paramétrica estuvo conformada por 143 empleados. La escala constaba de 10 competencias y 25 acciones medibles en cuatro niveles de competencias. La consistencia interna del instrumento se analizó mediante el cálculo del coeficiente alfa de Cronbach y la validez de constructo mediante análisis factorial confirmatorio.

Resultados:

participaron 53 enfermeros, 30 técnicos de enfermería y 60 auxiliares de enfermería, totalizando 143 profesionales que laboran en el hospital de una universidad federal del estado de São Paulo, Brasil. El análisis factorial confirmatorio del instrumento, basado en la estructura de 10 competencias, presentó buenos índices de ajuste con χ²/ GL=1.706, SRMR=0.071, RMSEA=0.070 y CFI=0.906, confiriendo validez de constructo. El alfa de Cronbach fue 0,924.

Conclusión:

a través de los procedimientos estadísticos adoptados, se concluyó que el instrumento tiene evidencia de validez basada tanto en la estructura interna - constructo, como en la consistencia interna.

DESCRIPTORES:
Enfermería; Recursos humanos de enfermería; Administración Pública; Competencia profesional; Psicometría; Reproducibilidad de pruebas

RESUMO

Objetivo:

analisar as evidências de validade de instrumento para avaliação de competências em equipe de enfermagem de hospital universitário.

Método:

estudo metodológico para investigação de evidências de validade da escala, com conteúdo anteriormente validado, foi enviada a 978 servidores técnico-administrativos em educação que atuavam na equipe de enfermagem em hospital universitário, sendo a amostra não paramétrica constituída por 143 servidores. A escala foi composta por 10 competências e 25 ações mensuráveis em quatro níveis de competências. Foram analisadas a consistência interna do instrumento mediante o cálculo do coeficiente alfa de Cronbach e a validade de construto, por meio de análise fatorial confirmatória.

Resultados:

participaram 53 enfermeiros, 30 técnicos de enfermagem e 60 auxiliares de enfermagem, totalizando 143 profissionais atuantes no hospital de uma universidade federal do estado de São Paulo, Brasil. A análise fatorial confirmatória do instrumento, baseado na estrutura de 10 competências, apresentou bons índices de ajustamento com χ²/GL=1,706, SRMR=0,071, RMSEA=0,070 e CFI=0,906, conferindo a validade de construto. O alfa de Cronbach foi de 0,924.

Conclusão:

por meio dos procedimentos estatísticos adotados, concluiu-se que o instrumento possui evidências de validade com base tanto na estrutura interna - construto, quanto na consistência interna.

DESCRITORES:
Enfermagem; Recursos humanos de enfermagem; Administração pública; Competência profissional; Psicometria; Reprodutibilidade dos testes

INTRODUCTION

In order to perform teaching, research, extension and care activities, federal universities have several employees in academic activities, such as higher education professors, and technical-administrative professionals in education (TAE). The Career Plan of Technical-Administrative Positions in Education (PCCTAE) is a set of regulations that regulates the professional development of employees and is based on Law No. 11,091/2005.1 The PCCTAE includes a variety of positions, including those in the health area, such as physicians, physiotherapists, nutritionists, nurses, technicians and nursing assistants, most of them working in university hospitals.

In the midst of the worst health crisis of the century, the work of health professionals, especially that of the nursing team, has been fundamental to cope with Covid-19, a disease caused by the new Coronavirus. On the front line, these professionals are responsible for actions related to bedside care and resource management so that this care is carried out with quality and without risk. The members of the nursing team need the due attention in the development of their skills, as well as to stimulate continuous training. Consequently, the importance of this workforce is highlighted, which responsibly provides not only patient care, but also support in teaching, research and extension tasks in federal universities. Therefore, it is essential to guarantee these professionals with a development plan within the scope of efficient, transparent and effective performance management.

Law No. 11.091/2005 determines that people management must observe, among other guidelines, the evaluation of the performance of servers.1 Article 4 of Decree No. 5,825/2006 provides that the National Plan for the Professional Development of Employees Members of the PCCTAE should include the relationship between the performance of the servers and institutional objectives, with evaluations carried out by the employees themselves, by other employees and by the institution.2

Decree No. 7,133/2010, in turn, regulates general criteria and standards to be observed for the performance evaluations of individual and institutional performance. It also establishes that evaluations should be based on criteria that reflect the competencies of the server in the performance of the activities assigned to it.3

With these regulations, the Federal Government begins the philosophy of management by competency in the career of public servants of Federal Institutions of Higher Education (IFES), associating performance evaluation supported by organizational strategies and qualifying civil servants to meet institutional demands. The regulation directly affected the TAE category and forced institutions to adapt in terms of compliance with government regulations. It is important to note that, despite this regulation, the government leaves each institution free to develop its own evaluation model.4

However, it is clear that there are many skills necessary for technical and administrative servers in education, in their professional practice. Regardless of the position held, there are fundamental skills for these professionals to develop their role at a level of excellence. It is noted that there are few publications of an instrumental or practical nature, because much of the research on competencies deals with conceptual discussions and theoretical propositions.5 For the construction of the competence assessment instrument, a previous integrative review of the literature on Competence Management in the Brazilian Public Administration was carried out in the Scientific Periodicals Electronic Library (SPELL^®) database of the National Association of Graduate Studies and Research in Administration (ANPAD) and in the Theses and Dissertations Catalog of the Coordination for the Improvement of Higher Education Personnel (CAPES), from 2010 to 2019.

It was concluded that, to date, there are no published scientific studies on the development and validation of a reference instrument to evaluate the performance based on competencies for technical-administrative servers in education of federal universities. Therefore, it is justified to develop an instrument that can serve as a basis or orientation for universities to develop their own evaluation form, according to the criteria established in the current legislation, enabling us to meet the institutional objectives, as well as the specificities of each position.

However, the construction and implementation of evaluative instruments are one of the main challenges of people management.6^-7 These evaluations do not always fulfill the role of supporting actions for vocational training and end up summarizing the mere formality to justify career progressions. In addition, the skills assessment instruments should not only measure the stock of knowledge, skills and attitudes, as it is also necessary to evaluate the actual deliveries of the server to the institution, to the extent that they add value, as stated by Dutra, Hipólito, Maillard and Miyahira.7 By means of the evaluation, data are provided that guide the management actions of development in cases of gaps between the expected competencies (or deliveries) and those expressed by servers.

In this study, the instrument used to assess the professional competence of the TAE server of federal universities originated from the professional competence profile, composed of a scale of 10 skills and 25 performance references and validated in its content (Cronbach's alpha of 0.858), within the scope of an article in the publication phase.

Given the above, it is questioned: does the instrument created for TAE servers and validated in its content have other evidence of validity based on the profile of professional competence? To answer this question, we sought to analyze the validity of the instrument for assessing the skills of the nursing team at a university hospital.

METHOD

A methodological study, part of a doctoral thesis defended in 2020 at a public university in the state of São Paulo. In this phase, psychometrics was used according to Pasquali,8 specifically the Empirical and Statistical Procedures, as a reference to analyze the internal consistency and construct validity of the competencies assessment instrument.8

To construct and validate the instrument in order to evaluate competencies performance, it was necessary to define the measurement scale in the instrument proposal. According to Lima,9 measuring means assigning values to characteristics of an object, according to rules that guarantee the validity and reliability of the measure. Thus, the construction of the scale should be guided by mathematical and statistical rigors, as well as scientific-behavioral rigors, especially for this case. It so happens that human behaviors, as in work performance, are not subject to direct measurement, but can be measured through psychometric theory. In this case, measuring by theory means that, when evaluating behavior, there is, in fact, the measurement of the latent psychological structure, that is, of a hypothetical attribute. The function of psychometric theory is to ensure the legitimacy of this action.10

Thus, the development of the evaluation instrument for the professional competence of TAE servers used the psychometric reference and was performed in three stages: Theoretical Procedures, Empirical Procedures and Analytical Procedures.8

In the Theoretical Procedures, the objective was to scientifically base the proposal of the reference instrument for competency assessment, to be composed with the actions necessary for the competent practice of technical-administrative servants in education. The investigation and analysis of systems and instruments for performance evaluation existing in Brazilian federal universities were carried out and, later, the profile of professional competence was developed, aligned with the specificities of the positions of technical-administrative servants in education, the existing studies and legislation on the subject. From 10 basic competencies, there was an operational definition of 25 performance references or actions expected in the practice of work, named as Identifying Competency Issues (items). Subsequently, through the Delphi technique, evidence of content validity of the items was verified by professionals and expert researchers with 0.982 (Content Validity Index) and 0.858 (Cronbach's alpha). Next, the pilot instrument was developed to assess the professional competence of TAE servers. In the Empirical Procedures, the instrument was tested in order to identify possible difficulties related to language, text comprehension, use of the competency level scale and the necessary guidance to complete the questionnaire. The study sample was defined by convenience and five technical-administrative servants of a federal university participated, three of which held middle and two higher-level positions, which did not participate in the subsequent application of the instrument. After the pre-test, the instrument was adapted and prepared for the application stage in the nursing team of a university hospital.

The chosen location is one of the largest and most important hospitals in the state capital. It serves all medical specialties and, in particular, those with high complexity procedures. The inclusion criterion of the participants in the research was to belong to the staff of TAE staff, who were members of the nursing team of the university hospital (auxiliaries, technicians and nurses) and, consequently, all the employees of the other positions were excluded. The instrument was sent to the 978 nursing staff employees by e-mail between September and November 2020, and data collection was performed with those who agreed to participate by signing the Informed Consent Form (ICF) and accessing the questionnaire link, which was elaborated in the Google Forms tool^® (Google Forms). A minimum of five responses were defined as the sample number for each of the 25 items, totaling 125 servers, as defined in the studies of Holland, Marra and Cunha11 and Cualheta, Abbad, Faiad and Borges-Junior.12 The sample was defined by convenience with the respondents who completed the questionnaire by November 4, 2020. Due to the difficulty in returning the questionnaires, it was necessary to send five messages to the servers, requesting participation in the study.

The instrument was composed of personal, academic and professional characterization data of the respondents, by a four-point classification scale (very competent =4; competent =3; not competent =2; not competent =1), adapted from the work of Holland, Marra and Cunha13 and also by a spreadsheet with 25 actions or measurable performance references in the four levels of competencies. This scale allows the measuring of actions or deliveries resulting from the professional exercise of the server and can be used both in self and in heteroevaluations.

Regarding the Analytical Procedures, the entire data set was submitted to statistical analysis by a professional in the area. This step made it possible to determine the internal consistency of the instrument by calculating cronbach's alpha coefficient, as well as construct validity through confirmatory factor analysis (CFA).14

This factor analysis allows the evaluation of the correlations in a large number of variables, defining the factors and grouping the variables with a strong relationship with each other.14 In CFA, the researcher must previously list the number of factors (dimensions) that exist in a set of variables and on which factor each variable will be related, before the results are processed. Thus, it is possible to test how much the variables represent a number of factors, confirming the structure of an instrument. 14 The confirmatory model is much more rigorous and restrictive, so it is strongly indicated for questionnaire validation.15 The CFA was used to evaluate the reasonableness of the competency assessment scale with 25 items, conceptually distributed in 10 factors or competencies: Problem solving (2 items), Teamwork (2 items), Self-development (3 items), Productivity and quality at work (3 items), Creativity and innovation (2 items), Communication (3 items), Decision making (3 items), Leadership (2 items), Systemic Vision (2 items) and User Focus (3 items).

The indicators of adequacy of the instrument adjustment to the sample data were obtained by means of the chi-square ratio and degrees of freedom (χ2/GL) tests,16 by the Goodness of fit index (GFI) and Adjusted for Degrees of Freedom (AGFI),16^-17 by Standardized Root Mean Square Residual (SRMR),14^,16 by the Root Mean Square Error of Approximation (RMSEA)18 and by Bentler's Fit Comparative Index (CFI).19^-20

The calculations were performed with the aid of the statistical software R version 4.0.3.21

RESULTS

A study was conducted with 60 nursing assistants, 53 nurses and 30 nursing technicians. The profile of these employees could be characterized as: female gender (77.6%), aged 41 years (56.6%), with more than 11 years at the service of the institution (56.6%) and specialization school level (38.5%) and graduates in the last 10 years (30%). In addition, most are nursing assistants (42%), without a leadership position (93.7%), from various fields of training and specialization, with a predominance of the health area.

In order to verify the average degree of agreement of the respondents in relation to each item and the distribution of responses around the mean, the statistical measures of position and scale were calculated, according to Table 1.

Table 1 -
Descriptive position and scale statistics. São Paulo, SP, Brazil, 2020. (n=143)

Expected Action/Performance	Minimal	Maximum	Average	Standard deviation	Median	Third quartile	First quartile
Troubleshooting
1.1 I act immediately in problem solving in the work sector.	1	4	3.566	0.612	4	3	4
1.2 I contribute to activities that are beyond my direct attributions in problem solving, as long as they are within the scope of my position.	1	4	3.385	0.711	4	3	4
Teamwork
2.1 I relate cordially with colleagues/subordinates/superiors/users, in order to keep the work environment pleasant and productive, respecting differences.	1	4	3.657	0.629	4	3	4
2.2 I collaborate with colleagues of the work team in performing joint tasks, seeking alternatives and contributing to the achievement of institutional objectives.	1	4	3.678	0.577	4	3	4
Self-development
3.1 I set goals for the self-development of work skills necessary through different learning strategies.	1	4	3.266	0.702	3	3	4
3.2 I participate in internal and external professional training actions of the institution with a focus on meeting institutional needs.	1	4	3.105	0.820	3	3	4
3.3 I use feedback from performance reviews to improve my own development.	1	4	3.210	0.863	3	3	4
Productivity and quality at work
4.1 I develop the activities with the expected quality, aiming at achieving established goals.	1	4	3.615	0.616	4	3	4
4.2 I correctly use methods and techniques related to the activities I perform.	1	4	3.699	0.582	4	3,5	4
4.3 I rationalize the use of resources to carry out the work.	1	4	3.594	0.631	4	3	4
Creativity and innovation
5.1 I propose new ideas/solutions in the development of activities to face and overcome adversities and pressures at work.	1	4	3.280	0.764	3	3	4
5.2 I participate in the implementation of changes and innovations proposed in the labor sector.	1	4	3.350	0.771	4	3	4
Communication
6.1 I express ideas with clarity and objectivity.	1	4	3.524	0.670	4	3	4
6.2 I actively listen to the various opinions of the work team.	1	4	3.545	0.636	4	3	4
6.3 I share knowledge and information that may be relevant to the improvement of team activities.	1	4	3.615	0.604	4	3	4
Decision making
7.1 I set possible alternatives for certain situations, from the analysis of the context in which I am inserted.	1	4	3.524	0.579	4	3	4
7.2 I choose the most appropriate alternative to face the problem.	1	4	3.517	0.637	4	3	4
7.3 I follow the effectiveness of my decision.	1	4	3.448	0.678	4	3	4
Leadership
8.1 Promote the union of the work team around common goals, supporting and stimulating the development of people.	1	4	3.350	0.744	3	3	4
8.2 I encourage the team to carry out work, highlighting the importance of each one's contribution to achieving institutional objectives.	1	4	3.287	0.766	3	3	4
Systemic vision
9.1 I have a global view of the institution and identify how my work, directly or indirectly, impacts the development of the work of the other areas/sectors of the university.	1	4	3.371	0.678	3	3	4
9.2 I dialogue with other areas/sectors/organs of the university, identifying opportunities for the development of integrated actions.	1	4	2.930	0.917	3	2	4
User focus
10.1 I meet the needs of users, linked to my professional performance, with promptness and cordiality.	1	4	3.748	0.537	4	4	4
10.2 I anticipate users' needs, linked to my professional performance, developing solutions and respecting current standards.	1	4	3.650	0.608	4	3	4
10.3 I continuously seek to improve the quality of the services provided.	1	4	3.748	0.550	4	4	4

For the analysis of the internal consistency of the instrument, Cronbach's alpha coefficient was used, which evaluates how much a set of items evaluates a single factor (in this case, each competency). The Cronbach's alpha of the instrument was 0.924.

It was also tried to analyze how each item influenced the alpha value in relation to every instrument. From this perspective, Cronbach's alpha coefficient was also calculated in the absence of each of the items, according to Table 2.

Table 2 -
Cronbach's Alpha excluding each item. São Paulo, SP, Brazil, 2020. (n=143)

Performance benchmarks (item)	Cronbach's Alpha without the item
1.1 I act immediately on problem solving in the work sector.	0.922
1.2 I contribute to activities that are beyond my direct attributions in troubleshooting, as long as aligned with the scope of my position.	0.921
2.1 I relate cordially with colleagues/subordinates/superiors/users, in order to keep the work environment pleasant and productive, respecting the differences.	0.922
2.2 I collaborate with work colleagues in performing joint tasks, seeking alternatives and contributing to the achievement of institutional objectives.	0.921
3.2 I participate in internal and external professional training actions to the institution with a focus on meeting institutional needs.	0.923
3.3 I use feedback from performance reviews to improve my own development.	0.925
4.1 I develop the activities with the expected quality, aiming at achieving established goals.	0.919
4.2 I correctly use methods and techniques related to the activities I perform.	0.920
4.3 I rationalize the use of resources to carry out the work.	0.921
5.1 I propose new ideas/solutions in the development of activities to face and overcome adversities and pressures at work.	0.923
5.2 I participate in the implementation of changes and innovations proposed in the labor sector.	0.921
6.1 I express ideas with clarity and objectivity.	0.921
6.2 I actively listen to the various opinions of the work team.	0.919
6.3 I share knowledge and information that may be relevant to the improvement of team activities.	0.919
7.1 I set possible alternatives for certain situations, from the analysis of the context in which I am inserted.	0.921
7.2 I choose the most appropriate alternative to face the problem.	0.919
7.3 I follow the effectiveness of my decision.	0.919
8.1 Promote the union of the work team around common goals, supporting and stimulating the development of people.	0.922
8.2 I encourage the team to carry out the work, highlighting the importance of each one's contribution to achieving institutional objectives.	0.921
9.1 I have a global view of the institution and identify how my work, directly or indirectly, impacts the development of the work of the other areas/sectors of the university.	0.922
9.2 I dialogue with other areas/sectors/organs of the university, identifying opportunities for the development of integrated actions.	0.925
10.1 I meet the needs of users, linked to my professional performance, with promptness and cordiality.	0.919
10.2 I anticipate users' needs, linked to my professional performance, developing solutions and respecting current standards.	0.919
10.3 I continuously seek to improve the quality of the services provided.	0.920

In the CFA, the quality indexes of the adjustment were calculated, according to Table 3.

Table 3 -
Estimates of the quality indices of the adjustment of the confirmatory factor analysis. São Paulo, SP, Brazil, 2020. (n=143)

Indexes	Calculated values	Reference values	Authors' references
χ2/GL	1.706	Between 1 and 3	Kline16
GFI	0.813	0.85	Kline;16 O’Rourke e Hatcher17
AGFI	0.736	0.80	Kline;16 O’Rourke e Hatcher17
SRMR	0.071	Less than 0.10	Hair Junior et al.;14 Kline16
RMSEA	0.070	acceptable up to 0.08	Browne e Cudeck18
CFI	0.906	>= 0,90	Bentler;19 Hu e Bentler20

It is noted that the ratio between chi-square (χ2) and degrees of freedom (GL) is within the values proposed by Kline.16 The results of GFI and AGFI are close to those referenced by Kline and O'Rourke and Hatcher.17 The SRMR of 0.071 is in line with the values mentioned by Hair Junior, William, Babin and Anderson14 and Kline, and the RMSEA of 0.070 is within the acceptable parameter of up to 0.08 proposed by Browne and Cudeck.18 Finally, the CFI of 0.906 is in accordance with the limit of 0.90, indicating adequate adjustment.19^-20

Figure 1 represents the confirmatory factor analysis model.

Figure 1-
Confirmatory factor analysis model. São Paulo, SP, Brazil, 2020. (n=143)

All CFA parameters showed a significance level of 5%, p<0.05. Questions 1.1, 2.1, 3.1, 4.1, 4.2, 5.2, 6.2, 7.2, 8.2, 9.2, 10.1 and 10.2 had the highest correlation with their respective factors.

On the other hand, 55.95% of the variance in question 1.1 is explained by factor 1; the percentage of 64.16% of variance in question 2.1 is explained by factor 2; 64.48% of variance in question 3.1 is explained by factor 3; 71.74% of variance in question 4.2 is explained by factor 4; 58.37% of the variance in question 5.2 is explained by factor 5; 65.28% of variance in question 6.2 is explained by factor 6; 65.77% of variance in question 7.2 is explained by factor 7; 89.11% of the variance in question 8.2 is explained by factor 8; 59.9% of the variance in question 9.2 is explained by factor 9; and that of 72.08% of the variance in question 10.2 is explained by factor 10.

It is possible to observe factorial correlations above 0.5 for most factors. Regarding the items, it was found that the correlations presented revealed acceptable values, since 70% of them are above 0.5.

DISCUSSION

The predominance of females in the nursing team was already expected and refers to the traditional participation of women in this area of health, especially from the hospital organization, which enabled female work outside the domestic scope, considering the natures of care, typical of women at the time.21

In the distribution of the answers given to the 25 items of the instrument, it was noticed that most of the servers consider themselves competent or very competent in relation to actions in the practice of work. Thus, the averages of all items were between 2.930 and 3.748. The high value attributed to self-assessment can be related to schooling (specialization) and working time (over 10 years), as found in the study by Holanda, Marra and Cunha.11

The maximum score (4.0) was observed in all items, as well as the minimum (1.0). The highest averages (3.748) were identified in the references of the following performances: "meeting the needs of users, linked to my professional performance, with promptness and cordiality" and "I continuously seek to improve the quality of the services provided". These references are part of the competence "user focus", which demonstrates that the servers assess that they exercise this competence adequately and sufficiently, that is, that they practice patient care, meeting their needs. The lowest average (2.930) was found in the performance reference "establish the dialogue with other areas/sectors/organs of the university, identifying opportunities for the development of integrated actions", which signals possible difficulties of the servers in the relationship and interaction with other sectors of the university hospital, impairing the systemic vision and integration with the other university areas.

Cronbach's alpha of the instrument was 0.924, demonstrating satisfactory internal consistency, which was observed in other studies.11^,23^-24 In addition, Cronbach's alpha coefficient was analyzed in the absence of each of the items, as identified in the studies by Góes, Cubero, Nogueira and Fernandez25 and Nogueira and Cunha.26 There were very small changes in the alpha value if any item was deleted. Thus, no statement of the instrument was deleted.

The CFA of the competencies assessment instrument, proposed in this study and based on the structure of 10 competencies, presented good adjustment rates. It was verified that, in the factorial model, there were good estimates in the parameters adopted (χ²/GL, SRMR, RMSEA and CFI). The GFI and AGFI values are close to those referenced in the literature.16^-17 The adjustment of the instrument by the CFA, although in some indexes, provides the instrument with construct validity, since its measurement structure is based, in view of the coefficients found.14

There were also satisfactory factorial correlations, indicating adjustment of the factorial model, with good estimates of the adopted indices, revealing a consistent and stable measurement, as identified in the studies by Noronha, Pinto and Ottati27 and Stacciarini and Pace,28 of whom used CFA to validate evaluation scales.

It is important to highlight some international competency assessment scale validation studies of nursing teams. The work of Kleib and Nagle,29 for example, was based on the validation of the Canadian scale of evaluation of nursing computer skills.. Moghaddam, Jame, Rafiei, Sarem, Ghamchili and Shafii30 studied a management skill assessment scale of leading nurses in Iran. Additionally, in South Korea, researchers worked on the validation of a scale for the evaluation of basic nursing competencies for nursing education graduates.31

Finally, a Spanish study on the development and psychometric validation of a questionnaire to assess competence in evidence-based nursing practice is mentioned.32

It is important to highlight that the international studies mentioned above refer to nursing professionals and, of course, without specifically mentioning technical-administrative servants in education or Brazilian federal public servants, focuses on this work. However, all emphasize the importance of studying competency assessment scales in nursing.

After identifying evidence of validity, it is understood that the proposed instrument is comprehensive, with the potential to be widely used to assess the competencies of TAE servers of Brazilian federal universities, including nursing team professionals, who are included in several studies on competencies, in various areas of activity, such as care, management, emergency and others, corroborating the importance of the category. As an example, there is a study on the competencies of nurses in the prevention of falls of hospitalized children33 and also on the validity of the content of the profile of competencies for the education of generalist nurses in the areas of health care, administration and management.34

It is important to highlight the relevance of the validated instrument, which is based on federal regulations1^-3 and on the competencies or effective deliveries made by the nursing team at work. Even more importantly, the instrument allows the identification and evaluation of competencies in order to ensure a development plan for these professionals in the field of performance management, enabling the improvement of patient care and clinical and management practices, according to national studies published.11^,13^,26

CONCLUSION

The statistical procedures performed allowed to conclude that the proposed instrument, based on the profile of professional competence of TAE servers, presents evidence of validity based on the internal structure - construct, and excellent internal consistency of the items, and can be considered reliable and valid for use in the evaluation of the professional competence of TAE servers working in Brazilian federal universities.

The great theoretical-practical contribution of the proposed competencies assessment instrument is the possibility of assessing the real contributions of employees at work and providing support for management actions resulting from this evaluation, especially those related to valorization and professional development.

It is also noteworthy that the proposal is unprecedented, since, in the literature, there are no similar studies, considering the idea of relating observable attitudes/behaviors with competences of TAE servers. In addition, the proposed profile of competences makes a relevant contribution to lawmakers, as it demonstrates the possibility of complying with legal precepts through the concept of competences.

In this study, only the self-assessment of the university hospital's employees was performed. In this evaluation, most of these servers self-evaluated themselves well, which characterizes a limitation of the research. Moreover, it is not possible to compare the validated scale with others, given the lack of similar scales in Brazil.

Thus, competency assessment is a topic that deserves attention in new research, especially empirical and focused on tools and practices with real impacts on people management in federal universities and, especially, on the care provided by nursing professionals. Another relevant point is the possibility of using the competency profile in several areas of people management, such as selection, allocation, movement, functional progression and career development of employees.

As potentialities, future studies can be cited on the adaptation of the competencies profile according to the institutional objectives of universities, with the inclusion of other specific competencies per position and also on the validation of the scale by servers of other positions of the technical-administrative career in education

ACKNOWLEDGMENT

To the employees of the nursing team of the University Hospital of the Universidade Federal de São Paulo, participants of this research.

REFERENCES

• 1. Brasil. Lei n. 11.091 de 12 de janeiro de 2005. Dispõe sobre a reestruturação do Plano de Carreira dos Cargos Técnico-Administrativos em Educação, no âmbito das Instituições Federais de Ensino vinculadas ao Ministério da Educação, e dá outras providências [Internet]. Brasília; 2005 [cited 2021 Feb 25]. Available from: Available from: http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2005/lei/l11091.htm
  » http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2005/lei/l11091.htm
• 2. Brasil. Decreto n. 5.825 de 29 de junho de 2006. Estabelece as diretrizes para elaboração do Plano de Desenvolvimento dos Integrantes do Plano de Carreira dos Cargos Técnico-Administrativos em Educação, instituído pela Lei no 11.091, de 12 de janeiro de 2005 [Internet]. Brasília; 2006 [cited 2021 Feb 25]. Available from: Available from: http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2006/decreto/d5825.htm
  » http://www.planalto.gov.br/ccivil_03/_ato2004-2006/2006/decreto/d5825.htm
• 3. Brasil. Decreto n. 7.133 de 19 de março de 2010. Regulamenta os critérios e procedimentos gerais a serem observados para a realização das avaliações de desempenho individual e institucional e o pagamento das gratificações de desempenho de que tratam as Leis nos 9.657, de 3 de junho de 1998, 10.484, de 3 de julho de 2002, entre outras [Internet]. Brasília; 2010 [cited 2021 Feb 25]. Available from: Available from: http://www.planalto.gov.br/ccivil_03/_Ato2007-2010/2010/Decreto/D7133
  » http://www.planalto.gov.br/ccivil_03/_Ato2007-2010/2010/Decreto/D7133
• 4. Baptista ALX, Sanabio MT. Avaliação de desempenho de técnico-administrativos em educação no âmbito das instituições federal de ensino superior. Rev GUAL [Internet]. 2014 [cited 2021 Feb 28];7(3):182-205. Available from: Available from: http://doi.org/10.5007/1983-4535.2014v7n3p182
  » http://doi.org/10.5007/1983-4535.2014v7n3p182
• 5. Mello SPT, Melo PA, Mello Filho RT. Competências gerenciais evidenciadas e desejadas dos integrantes do fórum de gestão de pessoas da rede federal de educação profissional, ciência e tecnológica. RACE [Internet]. 2016 [cited 2021 Feb 27];15(1):349-74. Available from: Available from: https://doi.org/10.18593/race.v15i1.6937
  » https://doi.org/10.18593/race.v15i1.6937
• 6. Fernandes BHR, Hipólito JAM. Dimensões de avaliação de pessoas e o conceito de competências. In: Dutra JS, Fleury MTL, Ruas R, organizers. Competências: conceitos, métodos e experiências. São Paulo, SP(BR): Atlas; 2013. p. 151-71.
• 7. Dutra JS, Hipólito JAM, Maillard NAPGD, Miyahira NN. Gestão de pessoas em empresas e organizações públicas. São Paulo, SP(BR): Atlas ; 2019.
• 8. Pasquali L, organizer. Instrumentação psicológica: fundamentos e práticas. Porto Alegre, RS(BR): Artmed; 2010.
• 9. Lima DVM. Research design: a contribution to the author. Online Braz J Nur [Internet]. 2011 Oct 17 [cited 2021 Mar 1];10(2). Available from: Available from: https://doi.org/10.5935/1676-4285.20113648
  » https://doi.org/10.5935/1676-4285.20113648
• 10. Pasquali L, organizer. Teoria e métodos de medida em ciências do comportamento [Internet]. Brasília, DF(BR): INEP; 1996 [cited 2021 Feb 08]. 432 p. Available from: Available from: https://www.faecpr.edu.br/site/documentos/teoria_metodos_ciencias_comportamento.pdf
  » https://www.faecpr.edu.br/site/documentos/teoria_metodos_ciencias_comportamento.pdf
• 11. Holanda FL, Marra CC, Cunha ICKO. Evidence of validity of the Competence Scale of Actions of Nurses in Emergencies. Rev Lat Am Enfermagem [Internet]. 2019 Mar 10 [cited 2021 Feb 08];27:e3128. Available from: Available from: https://doi.org/10.1590/1518-8345.2814.3128
  » https://doi.org/10.1590/1518-8345.2814.3128
• 12. Cualheta L, Abbad G, Faiad C, Borges C Jr. Competências Empreendedoras: Construção de uma Escala de Avaliação. Rev Empreendedorismo Gest Pequenas Empresas [Internet]. 2020 [cited Feb 25];9(2):158-80. Available from: http://doi.org/10.14211/regepe.v9i2.1621
  » http://doi.org/10.14211/regepe.v9i2.1621
• 13. Holanda FL, Marra CC, Cunha ICKO. Avaliação da competência profissional do enfermeiro em emergências: Instrumento criado e validado. Rev Bras Enferm [Internet]. 2018 [cited 2021 Mar 01];71(4):1865-74. Available from: Available from: https://www.scielo.br/j/reben/a/nmVz78DqD9FzRjy4vfQ5rMS/?format=pdf⟨=pt#:~:text=A%20escala%20de%20avalia%C3%A7%C3%A3o%20dos,em%20emerg%C3%AAncias%2C%20conforme%20Quadro%201
  » https://www.scielo.br/j/reben/a/nmVz78DqD9FzRjy4vfQ5rMS/?format=pdf⟨=pt#:~:text=A%20escala%20de%20avalia%C3%A7%C3%A3o%20dos,em%20emerg%C3%AAncias%2C%20conforme%20Quadro%201
• 14. Hair JF Jr, William B, Babin B, Anderson RE. Análise multivariada de dados. 6th ed. Porto Alegre, RS(BR): Bookman; 2009.
• 15. Polit DF. Assessing measurement in health: beyond reliability and validity. Int J Nurs Stud [Internet]. 2015 [cited 2021 Mar 01];52(11):1746-53. Available from: Available from: https://doi.org/10.1016/j.ijnurstu.2015.07.002
  » https://doi.org/10.1016/j.ijnurstu.2015.07.002
• 16. Kline RB. Principles and Practice of Structural Equation Modeling [Internet]. New York, NY(US): The Guilford Press; 2016 [cited 2021 Mar 02]. Available from: Available from: https://www.semanticscholar.org/paper/Principles-and-Practice-of-Structural-Equation-Kline-Little/13ad6c2d86679ca7d6f5ad3aa69615afdde33d93
  » https://www.semanticscholar.org/paper/Principles-and-Practice-of-Structural-Equation-Kline-Little/13ad6c2d86679ca7d6f5ad3aa69615afdde33d93
• 17 O’Rourke N, Hatcher L. A step-by-step approach to using SAS system for Factor Analysis and Structural Equation Modeling [Internet]. 2nd ed. Cary, NC(US): SAS Institute Inc.; 2013 [cited 2021 Mar 02]. Available from: Available from: https://support.sas.com/publishing/pubcat/chaps/61314.pdf
  » https://support.sas.com/publishing/pubcat/chaps/61314.pdf
• 18. Browne MW, Cudeck R. Alternative ways of assessing model fit. Sociol Methods Res [Internet]. 1992 [cited 2021 Mar 02];21(2):230-58. Available from: Available from: https://doi.org/10.1177/0049124192021002005
  » https://doi.org/10.1177/0049124192021002005
• 19. Bentler PM. Comparative Fit Indexes in Structural Models. Psychol Bull [Internet]. 1990 Mar [cited 2021 Feb 20];107(2):238-46. Available from: Available from: http://doi.org/10.1037/0033-2909.107.2.238
  » http://doi.org/10.1037/0033-2909.107.2.238
• 20. Hu L-T, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equ Model [Internet]. 1999 [cited 2021 Feb 20];6(1):1-55. Available from: Available from: http://doi.org/10.1080/10705519909540118
  » http://doi.org/10.1080/10705519909540118
• 21. R CT. R: A language and environment for statistical computing. Version 4.0.3 [software]. 2020 [cited 2020 Sep 10]. Available from: Available from: http://www.R-project.org/
  » http://www.R-project.org/
• 22. Ribeiro AC, Ramos LHD, Mandú ENT. Perfil sociodemográfico e profissional de enfermeiros de um hospital público de Cuiabá - MT. Cienc Cuid Saude [Internet]. 2014 [cited 2021 Mar 3];13(4):625-33. Available from: Available from: https://periodicos.uem.br/ojs/index.php/CiencCuidSaude/article/view/20480
  » https://periodicos.uem.br/ojs/index.php/CiencCuidSaude/article/view/20480
• 23. Coelho FA Jr, Borges-Andrade JE, Seidl J, Pereira AC. Validação Psicométrica de Medida de Auto Avaliação de Desempenho no Trabalho. Anais do 34º Encontro da Anpad [Internet]. 2010 Sep 25-29; Rio de Janeiro. Rio de Janeiro: ANPAD; 2010 [cited 2021 Mar 09]. Available from: Available from: http://www.anpad.org.br/diversos/down_zips/53/eor1449.pdf
  » http://www.anpad.org.br/diversos/down_zips/53/eor1449.pdf
• 24. Dias-Trindade S, Moreira JA, Nunes CS. Escala de autoavaliação de competências digitais de professores. Procedimentos de construção e validação. Texto Livre [Internet]. 2019 [cited 2021 Mar 09];12(2):152-71. Available from: Available from: https://doi.org/10.17851/1983-3652.12.2.152-171
  » https://doi.org/10.17851/1983-3652.12.2.152-171
• 25. Goes LF, Cubero J, Nogueira KSC, Fernandez C. Análise da confiabilidade de um questionário para identificar concepções sobre radicais livres e antioxidantes usando alfa de Cronbach. Anais do 10º Congreso Internacional sobre Investigación en Didácticas de las Ciencias [Internet]. 2017 Sep 5-8; Sevilla: España, Rev Investig Exp Didácticas; 2017 [cited 2020 Sep 30]. Available from: Available from: https://dialnet.unirioja.es/servlet/articulo?codigo=6690623
  » https://dialnet.unirioja.es/servlet/articulo?codigo=6690623
• 26. Nogueira VO, Cunha ICKO. Perfil de competências de Enfermeiros gestores do Ensino Superior Paulista: análise da confiabilidade. Res Soc Dev [Internet]. 2020 [cited 2021 Mar 03];9(8):e888984044. Available from: Available from: http://doi.org/10.33448/rsd-v9i8.4044
  » http://doi.org/10.33448/rsd-v9i8.4044
• 27. Noronha APP, Pinto LP, Ottati F. Análise fatorial confirmatória da Escala de Aconselhamento Profissional. Arq Bras Psicol [Internet]. 2016 [cited 2021 Mar 03];68(1):62-71. Available from: Available from: http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1809-52672016000100006&lng=pt
  » http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=S1809-52672016000100006&lng=pt
• 28. Stacciarini TSG, Pace AE. Confirmatory factor analysis of the Appraisal of Self-Care Agency Scale - Revised. Rev Latino-Am Enfermagem [Internet]. 2017 Jan 30 [cited 2021 Mar 04];25:e2856. Available from: Available from: https://doi.org/10.1590/1518-8345.1378.2856
  » https://doi.org/10.1590/1518-8345.1378.2856
• 29. Kleib M, Nagle L. Psychometric properties of the Canadian nurse informatics competency assessment scale. Comput Inform Nurs [Internet]. 2018 Jul [cited 2021 Oct 26];36(7):359-65. Available from: Available from: https://doi.org/10.1097/CIN.0000000000000437
  » https://doi.org/10.1097/CIN.0000000000000437
• 30. Moghaddam NM, Jame SZB, Rafiei S, Sarem AA, Ghamchili A, Shafii M. Managerial competencies of head nurses: a model and assessment tool. Br J Nurs [Internet]. 2019 Jan 10 [cited 2021 Oct 26];28(1):30-7. Available from: Available from: https://doi.org/10.12968/bjon.2019.28.1.30
  » https://doi.org/10.12968/bjon.2019.28.1.30
• 31. Ko Y, Yu S. Core nursing competency assessment tool for graduates of outcome-based nursing education in South Korea: a validation study. Jpn J Nurs Sci [Internet]. 2019 Apr [cited 2021 Oct 27];16(2):155-71. Available from: Available from: https://doi.org/10.1111/jjns.12223
  » https://doi.org/10.1111/jjns.12223
• 32. Ruzafa-Martínez M, Fernández-Salazar S, Leal-Costa C, Ramos-Morcillo AJ. Questionnaire to Evaluate the Competency in Evidence-Based Practice of Registered Nurses (EBP-COQ Prof©): development and psychometric validation. Worldviews Evid Based Nurs [Internet]. 2020 Oct [cited 2021 Oct 27];17(5):366-75. Available from: Available from: https://doi.org/10.1111/wvn.12464
  » https://doi.org/10.1111/wvn.12464
• 33. Gurgel SS, Ferreira MKM, Sandoval LJS, Araújo PR, Galvão MTG, Lima FET. Nursing Competences in the Prevention of Falls in Children in Light of the Galway Consensus. Texto Contexto Enferm [Internet]. 2017 [cited 2021 Mar 03];26(4):e03140016. Available from: Available from: https://doi.org/10.1590/0104-070720170003140016
  » https://doi.org/10.1590/0104-070720170003140016
• 34. Cioffi ACS, Ribeiro MRR, Ormonde JC Jr. Validation of the competence profile proposal for the training of nurses. Texto Contexto Enferm [Internet]. 2019 [cited 2021 Mar 04];28:e20170384. Available from: Available from: https://doi.org/10.1590/1980-265X-TCE-2017-0384
  » https://doi.org/10.1590/1980-265X-TCE-2017-0384

NOTES

• ORIGIN OF THE ARTICLE

  Extracted from the thesis - Construction and Validation of reference instrument for assessment of competencies of technical-administrative servants in education of Brazilian Federal Universities, presented to the Graduate Program in Nursing of the Universidade Federal de São Paulo, in 2020

• APPROVAL OF ETHICS COMMITTEE IN RESEARCH

  Approved by the Ethics Committee in Research of the Universidade Federal de São Paulo, opinions No. 1,971,309 and No. 4,052,678, Certificate of Presentation for Ethical Appreciation 64209117.0.0000.5505

Edited by

EDITORS

Associated Editors: Natália Gonçalves, Monica Motta Lino. Editor-in-chief: Roberta Costa

Publication Dates

• Publication in this collection
  09 May 2022
• Date of issue
  2022

History

• Received
  18 June 2021
• Accepted
  25 Nov 2021