Development and Validation of an Instrument to Evaluate General Competences in Health Area Courses

43 (1 Supl. 1) : 598-605; 2019 I Complexo Hospital de Clínicas, Curitiba, Paraná, Brazil. II Faculdades Pequeno Príncipe, Curitiba, Paraná, Brazil. III Universidade Estadual de Maringá, Maringá, Paraná, Brazil. Development and Validation of an Instrument to Evaluate General Competences in Health Area Courses Desenvolvimento e Validação de Instrumento para Avaliação das Competências Gerais nos Cursos da Área da Saúde


INTRODUCTION
Health professions, in the world context, have shared challenges regarding the quality of vocational training and the expansion of the practice fields caused as a consequence of changes in work processes. In Brazil, the public policies of the health system are added to these challenges.
In this sense, there are new needs for overcoming and adapting the training models proposed for the health area, a fact that meets the challenges that add to professional competency, defined as the ability to mobilize, articulate and put into action values, knowledge and skills that necessary for the efficient and effective performance of activities required by the nature of the work 1 .
Competencies have been widely discussed in the literature with different concepts. One of the definitions of Competency is the ability to perform and make the opportunity, which includes reconstructive questioning as an innovative basis for the process of development of the capable historical subject. Knowledge is an important tool of competency, but it does not ensure a competent action, since it is the ability and courage to innovate, associated with other cognitive and behavioral skills that shape "the profile of the competent individual" 2 . This concept has been well accepted and used to instrumentalize individuals with adequate knowledge so that they can reflect on their critical reality 3 4 . The National Curriculum Guidelines (NCG) in its 2014 issue, bring in its scope, the definition of competency to be assumed by medical education.
In Brazil, the NCG approved between 2001 and 2004 brought to the discussion general competencies for 11 of the 14 courses defined as part of the health area by the National Health Council Resolution N. 287, of 1998. The NCG were defined based on a context of interests and guidelines for the Unified Health System, which focuses on the public, democratic character and the search for universality and comprehensiveness of care 5 . Therefore, they should be an expression of the commitment that the change in education would have in positioning the University in its social role in the fields of health and education.
They are divided into two major blocks: six General Competencies that include Health Care, Decision-Making, Communication, Leadership, Administration and Management, and Continuing Education, which are common to the Nursing, Pharmacy, Physical Therapy, Speech Therapy, Veterinary Medicine, Nutrition, Dentistry, Psychology, Occupational Therapy and Biomedical Sciences courses; and the General Competencies for the Medical Course, until recently equal to the abovementioned courses, which were reformulated into three blocks in 2014: Health Care, Health Management and Health Education. The Biological Sciences, Social Work and Physical Education courses have a differentiated NCG structure, not including general competencies and the Specific Competencies of each course.
The NCG empowering role in the changes in health undergraduate courses has brought important contributions to the implementation of competency-based curricula, changes in practice, and the use of active methodologies. However, Feuerwerker (2000) 5 states that these changes are still incipient, emphasizing the existence of inaccuracies in the text of the Guidelines, due to the existence of disputes about future changes that allowed gaps in the adopted guidelines, not favoring the overcoming of the traditional approaches.
Therefore, the assessment of competencies has occupied a prominent place in this discussion. In this context, we increasingly seek to discover, explore and draw positive lessons from evaluations practices, seeking to overcome complex situations associated with teaching that require innovative conceptions of student assessment and maturity 6 .
Therefore, in search of expanding the intricacies of competencies and their insertions in the health education process, the following objectives of this study stand out: to develop an instrument to support teachers in the evaluation of the General Competencies of undergraduate students from health courses; test the reliability of the instrument with teachers and students from health courses; and validate the instrument on General Competencies.

METHOD
The present was a methodological study, which refers to the investigation of the methods of obtaining, organizing and analyzing the data, discussing the development, validation and evaluation of the research instruments and techniques, aiming at the construction of an instrument that would be reliable, accurate and usable for other researchers to apply 7 .
For the instrument validation process, we used the model proposed by Raymundo (2009) 8 and Anastasi (1988) 9 , which indicate content validity, criterion validity and construct validity.
The content validity, understood as the systematic examination of the instrument's content, determining its scope in relation to the object to be measured 10,11,12 , was followed by the criterion validity, understood as the instrument's ability to function as a future predictor, of an operationally independent variable called criterion 13 , because there is no similar validated instrument that aims to evaluate the General Competencies.
To assess the statistical significance of the difference between two sample means for a single dependent variable, we chose the t-Test and also the Chi-square Test, a nonparametric procedure used to test hypotheses about the overall proportion 14 . We also used Pearson's Linear Correlation Coefficient to measure the linear association between paired quantitative values between sample variables 15,16 .
According to Callegari-Jaques (2003) 17 , the linear correlation coefficient can be evaluated as follows: And after implementing the described statistical tests, the Akaike Information Criterion (AIC) was assessed, which helps to recognize the importance of a remodeling, in some cases, of the proposed initial model 18 .
The Construct validity was performed to identify and group different measures of some attribute and differentiate For reliability, defined as the degree of consistency between multiple measures of a variable and that indicated the extent to which it can be stably reproduced 20,21 , the Cronbach's alpha coefficient was used.
The study used the classification proposed by Freitas and Rodrigues (2005) 22 , which presents the following scale for analysis of Cronbach's alpha coefficient: After assessing Cronbach's alpha coefficient, the AIC was also used to ensure that the versions proposed in this study were reliable.
For a better visualization, Table 1 shows a validation scheme and the techniques/procedures used in each of the aspects considered in this validation process of the instruments to assess General Competencies. The data collection site for validation was a Higher Education Institution (HEI) in Curitiba, state of Paraná, which has courses in the health area of Nursing, Biomedical Sciences, Pharmacy, Psychology and Medicine.

Scheme of validation and reliability/ consistency of the assessment instrument for General Competencies of health courses
A convenience or accidental sampling was used, which selects the available people as study participants 14 .
The inclusion criteria used were: • for teachers: having the highest workload of the semester in the period of the studied course; have graduated from the same course to which the instrument is being applied. • for students: being part of the evaluated period; The following were excluded from this study: • All teachers and students who started and did not complete the instrument.
The instrument used was created using the Lime Survey Software, a free environment chosen to apply the online research instrument.
In order to comply with the Guidelines and Regulatory Standards for Research Involving Human Beings, established by Resolution 466/12 of the National Health Council, this study was approved by the Research Ethics Committee under number 826,770.

RESULTS AND DISCUSSION
For the content validity, a survey was performed of the NCGs of all courses in the health area, as well as a research in the Virtual Health Library that includes the LILACS, IBECS, MED-LINE, Cochrane Library and SciELO databases to support the construction of the instruments. After that, a literature review was performed for each of the General Competencies (Health Care, Decision-Making, Communication, Leadership, Administration and Management and Continuing Education); at the time of search for information, for some General Competencies, it was necessary to add the keywords "in the health area", "and health area" or "General Competencies", aiming to refine the search, due to the large number of publications related to the topic, totaling 93 evaluated articles.
After the alignment with the NCGs of the 11 courses that have the General Competencies with the search theoretical framework, it was possible to develop Version 1 of the instrument, for the Evaluation of General Competencies in the Health Area, in the option directed to teachers and in the mirror option, adapted to the students (self-evaluation).
Subsequently, a survey of twenty-two experts from the assessed areas was carried out, who were chosen from the Lattes curriculum, publications in the teaching area, those working in the coordination of courses, with a master's or doctoral degree in the area. Two experts from each course were invited to contribute to the proposed instrument. After the returns, Version 2 of the instrument was developed, maintaining both options (teacher/student). This version was sent to the survey participants through the Lime Survey Software, after sending an invitation by email, which contained a personal presentation, a brief project report, objectives, deadlines for the feedback and the expected role in the study, as well as a link to answer the instrument. Each teacher participated as a respondent for only one course and period. A total of 51 instruments were retuned by students and 50 instruments by teachers. One instrument from the students that did not have the answers for the teacher's version was discarded, totaling 50 instruments for each group, all with the corresponding mirror.
The collected data were organized and submitted to statistical analysis carried out with three classifications of different groups: an evaluation called Teacher, which considered only the evaluations made by the teachers; that of the Student, which considered only the students' evaluations and a third, called the Teachers and Students together, which contemplates the results of the combined Teachers' and Students' evaluations.
The results for Criterion and Construct Validity were described and discussed together, following the statistical analyses used for the two validation steps.
The factorial analysis verified the adequacy of the items to the dimensions, as well as measured the degree of effect of each item in this dimension. This analysis proposed the grouping of the six initial dimensions: Health Care, Decision-Making, Communication, Leadership, Administration and Management, and Continuing Education into three dimensions: Health Care, Health Management, and Health Education. The proposal meets the new NCGs approved for Medical education in 2014 that bring exactly the three described dimensions, with no damage to the information to be evaluated.
The criteria used for grouping the information were the chi-square test, the p value of each item and the AIC that allowed identifying the improvements in the model.
As a result, we obtained Version 3 of the instrument, directed to the teacher and the student's version, which had four questions disregarded, totaling 44 questions (original model with 48), but distributed into three dimensions, instead of six, as it initially did.
The exploratory factorial analysis, both in teacher's and student's data, indicated that this is the most appropriate model, and can be applied to teachers and students, alone or together. In all questions, p had a significant value -p < 0.05.
Pearson's correlation coefficient (r) for this version shows that there is a strong correlation between all dimensions, none of which has negative values.
A strong association is observed between the Health Care and the Health Management dimension: (r = 0.74) in the stu-dents' evaluation and a very strong association (r = 0.92) between the same dimensions in the teacher's evaluation. When we evaluate teachers and students together, the association remains strong, as well (r = 0.88). The other associations between the dimensions are very close, confirming the association of all dimensions.
Regarding the chi-square test, it showed p <0.001, demonstrating its acceptability for all versions of the instrument.
Regarding reliability, the results were safe and reliable, with the lowest Cronbach's alpha value of 0.758 in the Health Care Dimension for students, considered a strong value in the evaluation by Freitas and Rodrigues; regarding the other values, they are all strong or very strong, which we desire for all evaluations. Thus, the instruments of this version indicate safety for their use in these groups, separately or together.   as a whole, the AIC value for Version 3 was 8849 and for version 2, it was 9854; the lower the value in the comparison, the higher the indication. Thus, the results suggest that the courses that have the six General Competencies could realign them in three dimensions, because statistically, the work showed that the six General Competencies can be directed into three, reducing the number of dimensions without changing their essence.
Based on the NCGs, the competencies are critical to health courses. When assessing the student, it is important to include several aspects that are part of their education, and not only their understanding during the course, which eventually includes the way they interact and their skills 23 .
Thus, it is expected that the students will be able to apply their knowledge in the most diverse situations, no matter how challenging they might be, justifying their competency, whenever necessary. In this sense, one of the main challenges is to evaluate the training of this student through their competencies.
For that purpose, it is imperative the student's empowerment to be a protagonist in the training process, knowing how and where to seek information and having the capacity to interpret its quality. It is up to the course to encourage these skills of autonomy, criticism and reflection in future doctors.
Since the implementation of NCGs in 2001, the aim has been to change Brazilian education with a focus on what the population really needs. Not understanding the social view of the courses and the lack of research in Education hinder the curriculum development. It is necessary to learn how to connect knowledge to skills, emotions, reasoning and values to consolidate the competency for the good of the patient. This can be developed from early academic immersion in practice focused on person-centered teaching, demonstrating that ethics is much more than the memorization of the code of medical ethics, coming closer to what is advocated by the NCGs 24 .
Clearly, the focus of graduation must be to go beyond the technical environment into a fair education.
Competencies can assess quality during the professional training process. According to the French Labor Code, competency development must be part of the teaching of theory, with personal and professional experience 25 .
Within the core of the objectives of changes in the new guidelines is the focus on learning by competency, plus a holistic view of the human being, considering them as a whole, as well as their social context. One way to achieve this is to unite the multiprofessional team with students, teachers and patients. It is essential to apply humanization and ethics in training with the same emphasis as disciplinary research, fo-cusing not only on healing, but also on the care of the individual. For that purpose, it is interesting to reinforce the insertion of a humanistic axis in curricular matrices in Medicine, developing adequate skills for each undergraduate period 26 .
As a result, professionalism has changed due to the social contexts and economic relations of the professions over time.
It is important to know what the values, rules and norms of the profession are, and what is the contribution to society in the development of these concepts. Professionalism goes far beyond the Code of Ethics and for that, it is important that both teachers and students be aware of professionalism as the basis of the social contract to legitimize their profession, and it is important to ensure proper training from the correct understanding of this topic and respect its principles, such as always considering first the patient's well-being and autonomy and the professional's commitment to the competency, confidence and quality of care with their professional responsibilities 27 .
Returning to the assessed subjects allows the possibility of contextualizing learning, associating theory with practice in health. However, it is common to question why knowledge occurs separately if the objective is to aim for an integrated professional practice. Thus, attention is already given to curriculum integration, which allows new strategies for teaching / learning and academic assessment. For that, multidisciplinarity promotes a more integrated learning 28 .
A model that encompasses professionalism in the curricular matrix of the courses is yet to be observed and, if desired, it should be adapted to the local teaching context in the different realities of education 29 .
The acquisition of knowledge currently highlights technical and scientific skills. Thus, professionalism emerges to renew the teaching-learning process in a fair manner. Although several studies are being conducted to assess competencies, they still show low reliability, and there are few objective instruments that evaluate only some aspects of professionalism. Thus, there is a scarcity in the literature regarding the aspect of professionalism assessment methods and use of instruments, which is the current focus on the topic according to the author, in addition to the importance of incorporating the topic in the curricula. In this type of assessment, it is relevant to include students, health professionals and patients. Since no single instrument addresses all aspects of a person's performance, it is important to emphasize their expertise, the validity of the instruments, and the appropriate sampling of participants 30 .

CONCLUSIONS
Thus, the study brought, in version 3, a valid model for the assessment of General Competencies, adequate to be used with teachers, students and both, directed to the NCGs of ten health courses that subdivide their General Competencies into six dimensions, in addition to Medicine, now subdivided into three dimensions. Version 3 is ideal to be used with the eleven health courses (Nursing, Pharmacy, Physical Therapy, Speech Therapy, Medicine, Veterinary Medicine, Nutrition, Dentistry, Psychology, Occupational Therapy and Biomedical Sciences).
This instrument aims to minimize the challenges faced in the evaluation of the General Competencies, considering the difficulty in implementing them at the HEIs.
The implementation of this instrument may contribute for the HEIs to obtain subsidies to evaluate the comprehensive training process of health students based on the General Competencies expected from graduated students. The model allows identifying weaknesses in the process and correcting them before the end of the course and can be used from the beginning to the end of undergraduate school.
The study suggests the consolidation of clear and objective evaluation models for both teacher and student, helping to demystify the evaluation process.

ORGANIZATION OF FUNDING SOURCES
The entire study was developed with the authors' own financial resources.