Pathways for graduation evaluation in Dentistry: logical model building and validation criteria

Abstract

The need for universal access to health care and the failure of the pedagogical model centered on only the transmission of knowledge has led to changes in the training of health professionals. The aim of this study was to provide a new alternative for evaluation of Dentistry courses based on the National Curriculum Guidelines (NCG) of the area. For this, a logical model was formulated on the need for training in the oral health pathway which allowed for the construction of a criteria matrix, validated by Delphi consensus technique and modified by the participation of 33 “experts.” The matrix dimensions presented as a pedagogical approach the profile of graduates, the teaching-service integration and orientation of health care. The detailing of these dimensions into sub-dimensions and of measurable criteria allowed for deepening structural elements of the NCG unexplored in other studies evaluating undergraduate courses in Dentistry. The final instrument proposed in this study is differentiated alternative assessment training, for both dentists and other professionals, considering that the NCG of all healthcare courses provide for the training focused on the health needs of the population, integrated to Unified Health System (SUS) and based on student-centered learning.

Keywords
Dental education; Educational assessment; Human resources in health

Resumo

A universalização do acesso à saúde e a falência do modelo pedagógico centrado na mera transmissão de conhecimento tem levado a mudanças na formação de profissionais de saúde. O objetivo deste estudo foi proporcionar uma nova alternativa para avaliação de cursos de Odontologia, com base nas Diretrizes Curriculares Nacionais (DCN) da área. Para isto, foi formulado um modelo lógico sobre o percurso necessário para a formação em saúde bucal, o que permitiu a construção de uma matriz de critérios, validada por meio da técnica de consenso Delfos, modificado com a participação de 33 “experts”. A matriz apresentou como dimensões a abordagem pedagógica, o perfil do egresso, a integração ensino-serviço e a orientação do cuidado em saúde. O detalhamento destas dimensões em subdimensões e critérios mensuráveis permitiu aprofundar elementos estruturantes das DCN, pouco explorados em outros estudos, sobre avaliação de cursos de graduação em Odontologia. O instrumento final proposto neste estudo representa alternativa diferenciada de avaliação da formação, tanto de cirurgiões-dentistas como de outros profissionais, considerando que as DCN de todos os cursos da área da saúde preveem a formação com foco nas necessidades de saúde da população, integrada ao SUS e baseada na aprendizagem centrada no estudante.

Palavras-chave
Educação em odontologia; Avaliação educacional; Recursos humanos em saúde

Introduction

Several aspects contribute to the current findings, although emerging/developing, that are changing in the training of health professionals. Among them the need for universal access to health care may be cited or discredited on a purely assistant model and focused on procedures, as well as the finding that the pedagogical model centered only on the transmission of knowledge has failed.

In discussing today the differences between human resources and health workers, a number of questions related to how the education of health professionals has overcome the tensions in its structure involving work, the worker and society in favor of developments in the health system, the flexibility of labor relations and the development of social policies¹. In view of these paradigm shifts, consequent demands for investigations have been proposed, with the perspective of elucidating phenomena employment/ work and education¹. In research with students of Dentistry, Medicine and Psychology, an imaginary liberal-privatized operation became evident, articulated by its role in the public sphere simply for greater professional experience².

The reference that differs this movement and incorporates an expanded dimension in health is translated in the National Curriculum Guidelines (NCG), instituted in undergraduate courses from 2001³. By analyzing the difficulties of dentists in public health services, Costa and Araújo⁴ suggest the need for changes in curricula, forming health professionals who reflect on their practice and develop competencies of learning, know-how and behaviour.

The NCG reinforce the approach between Education and Health, with emphasis on promotion, prevention, recovery and rehabilitation, and training of professionals capable of working with quality, efficiency and resolution, consistent with the terms of the construction of the SUS, which projects this system as a training order for human resources in health⁵. For both, it points to the challenge of promoting autonomous and permanent intellectual and professional development in the student, competence that does not end at the completion of course⁴.

With the establishment of the NCG and curriculum changes taking place in higher education institutions, it was necessary to develop evaluation processes which suggest an analysis of how the guidelines were met, with regard to the quality, relevance and suitability of courses to the development needs of country⁶. This review is benchmarked on the orientation of authorization procedures and recognition of courses. However, in addition to aspects related to the regulatory process, studies are needed to further investigate essential elements in the everyday development of these institutions.

In 2006, the Teixeira National Institute for Educational Studies (INEP) published a study on the trajectory of undergraduate courses in health. It sought to stimulate reflection and discussion on evaluating the training of upper-level courses in health from the Census of Higher Education, the Teacher's Registry, the Register of Institutions and Courses, the National Survey of Student Performance (ENADE) and Audit Reports of Evaluation Commission (CPA)⁶. It found a significant increase in the availability of Dentistry courses throughout the country, mainly in the private sector.

In 2006, the Ministry of Health and of Education investigated the adoption of the National Curriculum Guidelines in the context of the Pedagogical Projects Course (PPC)⁷ based on the reports of the evaluation processes of these PPC courses, conducted by INEP in the period 2002-2006. According to this study, the NCG for Dentistry courses signified an important step forward by establishing the principles and rationale for the formation of the dental surgeon.

However, even after more than ten years since the establishment of NCG and the beginning of the curriculum changes in undergraduate Dentistry, several questions remain about the true effectiveness of the guidelines in the training of dentists. Therefore, the need to think of criteria that can verify how the training of health professionals has responded to the recommendations by the NCG emerges, even in the face of subjectivities and scope of factors for proper training to meet the health needs of the population. Consistent with this need, the objective of this study was to propose the construction of a logical model for the formation of dentists based on National Curriculum Guidelines and from this, the construction of an evaluation criteria matrix and its validation, providing a new alternative for evaluation of undergraduate courses in dentistry.

Methodology

In evaluation studies, considering the complexity of the object, it is necessary, notwithstanding the methodological rigor, to use flexibility, triangulation of research approaches, and complementarity of quantitative and qualitative techniques⁸.

This study consisted of a pre-evaluation or assessment-ability, which arise as a possibility to verify the extent to which an intervention can be evaluated⁹. It provides a favorable environment to the extent that builds understanding among stakeholders about the nature and goals of the evaluation objective, seeks agreement between the interests of the assessment and potential users of the study, thereby increasing the possibilities of using results¹⁰. For this study, the pre-evaluation consisted of the following steps: construction of the logical model, the development of an evaluation criteria matrix, and validation of the criteria matrix.

Construction of the Logical model

The representation of the research objective, which in the case of this study is to train dentists based on the NCG, their movements, and their relationships, is likely to be modeled with reference to a building theory¹¹.

The construction of the model began with research and reading documents related to the objective, from NCG³,⁴,¹², publications from the Ministries of Health and Education⁶,⁷,¹³,¹⁴, the Brazilian Association of Dental Education¹⁵, and other studies and publications on the assessment of undergraduate education and dentistry¹⁶–²².

The construction of a theoretical / logical model needs to be convincing and present plausibility of established associations¹¹. In this perspective, based on the literature and research objectives, several possible elements of the model were raised so that this could become a ‘conceptual schematic representation of a thing or real situation or perceived as such’²³. In this case, it could as closely as possible represent the training of dentists according to what is recommended in the NCG.

The logical model therefore presented the need for training as a structuring element linked to the shifting paradigm facing health; the health needs of the population (Figure 1).

Figure 1
Logical Model for dental surgeons based on NCG.

Development of matrix Criteria

Based on the logical model constructed, a matrix of evaluation criteria was developed covering the main issues to be addressed from the theoretical framework. These were considered relevant to the formation of a “objective image” assessment of Dentistry courses based on the NCG. For the construction of the matrix, each structural element highlighted in the logical model was transformed into a dimension of evaluation. The matrix was then built on four dimensions: Profile of Graduated, Health Care Orientation, Service-Learning Integration and Pedagogical Approach. Still based on the logical model, sub-dimensions describing evaluation criteria that express the expected objective image standard were proposed for each of these dimensions.

The matrix drawn from the logical model, with dimensions, sub-dimensions, criteria and their respective scores initially proposed by researchers is expressed in Chart 1.

Chart 1
Matrix of criteria and scoring system proposed for assessment of courses in dentistry in northeastern Brazil, from the dimensions and sub-dimensions based on NCG, referred by the “experts.”

Dimension	sub-dimension	Criteria	Proposal score
Profile of graduate	Generalist	In the collective field, develops programs of health promotion and management of services, at the same time in the individual field performs the prevention, diagnosis, planning, and dental treatment of major dental problems.	10
	Humanist	Offers quality care articulating the technological advances with reception, improving care and the working environment, building solidarity exchanges and committed to the production of health in relation to users and other professionals.	10
	Autonomous	Makes decisions in both clinical procedures as in management situations and collective work, safely and with skill.	10
	Critical / reflective	Discusses and assesses situations of individual or collective nature and proposes alternatives for solution from their scientific knowledge and reflection	8
	Understanding of social reality	Identifies the social context of the professional practice, respecting the characteristics of the population and seeking appropriate solutions to this reality.	8
Orientation of health care	Epidemiological approach	Curricular structure organized from the epidemiological reality of the region.	10
	Health promotion	Understanding the social determination of health care and the development of comprehensive strategies for the expansion of healthy choices based on the living conditions of individuals and the population.	10
	Prevention of disease	Use of individual and collective technologies for the prevention of dental caries and periodontal diseases.	6
	Early diagnosis	Sensitivity in the use of technologies that enable a comprehensive view of the individual and subsidize early diagnosis for all oral problems.	8
	Dental treatment	Clinical activities focused on solving the most prevalent oral health problems of the population.	6
	Oral health team	Development activities with ancillary staff (auxiliary oral health technician or dental care) enabling teamwork.	8
	Interdisciplinarianism	Development of teaching activities with students from other courses or professional healthcare providing completeness of health.	8
Integration of education service	Insertion of students in SUS	Students involved with health services from the early semesters.	5
	Activities of the Interns	Clinical activities in individual and collective actions in areas outside the scope of IES, articulated with the SUS, formalized by agreement and supervised by teachers from all areas.	10
	Theoretical reference of the conceptual frameworks of SUS	Understanding the breadth and complexity of the SUS, its principles and guidelines.	5
	Experiences in SUS	Expertise in all levels of care in the health system, including the flow of network services and skills at each level.	10
	Reference and counter-reference	Course clinics linked to SUS, with developing institutional mechanisms of reference and counter-reference.	5
	Planning and evaluation of services	Effective participation in the planning and evaluation of health services, allowing for joint actions and partnerships.	5
Pedagogical approach	Active Methodologies	Educational process based on problem solving in small groups with active student participation.	10
	Role of the teacher	Facilitator of learning, knowledge manager and organizer of activities that promote student learning.	10
	Teacher development	Constant promotion of faculty training activities in the pedagogical area and the integration of the contents of several knowledge areas.	10
	Learning scenarios	Learning based on multiple sources of knowledge such as libraries, community agencies for data processing, sanitation and environment institutions, schools, kindergartens, social spaces.	10
	Integrated curriculum	Competencies and skills organized into modules with increasing complexity throughout the training process, articulating biological sciences, health, humanities and social sciences with dentistry.	10
	Evaluation procedure	Based on skills and abilities provided, performed in order to systematically monitor and contribute to student learning.	10
	Curricular flexibility	Mechanisms capable of providing flexibility to meet the curriculum according to the development of vocations, specific interests and potential of the student.	10
	Teaching-researchextension	Involvement of research and extension as curricular strategies that provide full educational experience.	10

Validation of the criteria matrix by consensus among “experts” (modified Delphi method)

To validate the evaluation criteria of dentistry courses proposed by the authors, an adaptation of the matrix technique of consensus among “experts” (Delphi method) was used.

The Delphi method is a technique for formal consensus among “experts” on a particular topic using their theoretical knowledge, experience and creativity converted to scientific information through a methodologically structured application²⁴,²⁵. In health, this technique has been used to obtain consensus for various purposes, such as technology assessment, selection of quality of service indicators, and the development of curricula for medical education²⁵.

The Delphi consensus consists of a structured process in rounds of consultations with experts in order to obtain a collective and qualified opinion on certain issues until a consensus is reached²⁶,²⁷.

For this study, we opted for an adaptation called “modified Delphi,” which is the inclusion of at least one round of face-to-face discussion between specialists²⁵,²⁷ through association with the consensus conference technique recommended by Souza et al.²⁸. In short, a first round with the application of the traditional Delphi technique took place in the study, while the second and third rounds corresponded to the implementation of the consensus conference.

For the first round sixty experts were invited consisting of thirty PhD professors with active inclusion in discussions on education in dentistry with the Brazilian Association of Dental Schools (ABENO), and thirty doctorate professors with membership in the Working Group on Oral Health of the Association of Collective Health (ABRASCO). To identify these participants the understanding was that “experts” are people connected to organizations able to provide conclusive values about a particular problem with utmost competency, make objective predictions about the effects, applicability, feasibility and relevance that can contribute to solving this problem, and suggesting recommendations about what to do to perfect them²⁹.

The ideal number of participants for achieving Delphi consensus is variable. While Okoli and Pawlowski³⁰ indicate that between 10 and 18 “experts” are sufficient for method development, Valdes and Marín²⁵ suggest a minimum of seven “experts,” noting that the consensus error is remarkably decreased by every expert added. However, studies recommend prudence in inviting more members to the panel of “experts” than provided because usually some quit or don't attend²⁵,²⁷. Bloor et al.³¹ reported that more important than the size of a Delphi group is the balance of the shares represented by the range of views, knowledge and interests placed.

Individual messages were sent to the “experts” via email, including the Free and Informed Consent Form (ICF), an explanatory text containing the study objectives and method of building consensus by Delphi method, as well as instructions for completing the instruments research. The logical model and the matrix of criteria were presented accompanied by the following question: “What are relevant criteria for the training of dentists based on prescribing to the national curriculum guidelines for Dentistry?” Each criterion in the matrix, as well as other studies³²,³³ included a score from 0 to 10, where 0 meant the exclusion criteria and 10 meant the maximum importance for the evaluation of courses based on the NCG. We asked the “experts” to judge each criterion and to attribute a score 0-10 following the same logic, but also to describe the reasons for their persistence or change in the criterion score. In the matrix, criteria averaging less than 7 (seven) and standard deviation value of less than or equal to 3 (three) were maintained²⁸.

From the results flagged in the first round of the Delphi, the second round was developed in the consensus conference, for which four experts who were not participants in the first round were invited; doctors and professors with active inclusion in discussions on education in dentistry along ABENO, with extensive academic achievement in public health and publications in the area of vocational training of dentists, and those who had previously received the matrix with the criteria and their scores scaled in the Delphi consensus.

Just as in traditional Delphi, there is no consensus regarding the optimal number of participants in consensus conferences. A third variation of 3 to 15 experts has been observed in recent studies³², ³⁴–³⁷, which conclude that representativeness should be more evaluated for quality by a panel of experts than by the number of participantes³⁸, as occurred in this study.

A consensus conference was developed at an in-person meeting in which experts reviewed and discussed about the dimensions, sub-dimensions and proposed evaluation criteria for the matrix. From the debate, the consensus of this group was recorded.

The experts at the consensus conference were invited to individually analyze the resulting matrix of their work, which consisted in the third round of the Modified Delphi.

Ethical Aspects

The project was submitted to the Ethics Committee at Hospital Onofre Lopes and registered in the Brazil Platform, following all the guidelines of Resolution 466/12³⁹ of the National Health Council's Project and was approved.

Results

The logical model in Figure 1 is the result of the preliminary construction by researchers, plus the suggestions forwarded by the “experts” in the first round of the Delphi consensus. To ensure the configuration of the proposed image-goal, the following components were entered into the model with aspects related to training: pedagogical approach, the graduate profile, teaching-service integration and orientation of health care, all organized to establish relationships and routes that were relevant to the objective. The logical model obtained allowed for a better understanding of the structure and evaluation elements of the issues involved and the possible relationships between these, guiding the construction of a proposed evaluation criteria matrix for courses of Dentistry.

This matrix was sent to “experts” who were able to judge the consistency of the criteria with the corresponding dimension and also indicate their suggestions for improving the matrix, either by exclusion criteria, appointment of new criteria, new dimensions, with respective scores or any other reformulations they deemed necessary.

The consulting of the ‘experts’ was performed by Delphi during the months of July to August 2013. Of the sixty invited experts, a total of thirty-three professors responded to the matrix, a number deemed satisfactory by other studies with the same characteristics²⁵,³⁰.

The scores given by the “experts” to each criterion matrix were entered into a spreadsheet and analyzed descriptively using the mean and standard deviation. The higher the average, the higher the importance given to the criteria. The standard deviation verified the estimate of the degree of agreement being inversely proportional to this, regardless of the importance attributed to criteria²⁸.

The results are shown in Table 1, which indicates the proposed dimensions and sub-dimensions in the matrix, the average and standard deviation and its final score to be considered after the evaluation by “experts.”

Table 1
Mean, standard deviation and final score of the sub-dimensions, according to “experts” participating in the first round of the Delphi Method by dimension.

Dimenson/sub-dimension		Average	DP	Pontuação Final
Profile of graduate
	Generalist	9.9	0.4	10
	Humanist	9.6	1.1	10
	Autonomous	9.3	2.0	9
	Critical / reflective	8.4	2.0	8
	Understanding social reality	8.9	1.0	9
Orientations of health care
	Epidemiological approach	9.1	2.0	9
	Health Promotion	9.9	0.4	10
	Prevention of disease	6.9*	1.8	-
	Diagnostic	8.1	1.1	8
	Dental Treatment	7.5	1.4	8
	Oral health team	8.1	1.3	8
	Multiprofissionality	8.0	1.1	8
Integration of Education Service
	Insertion of students in SUS	6.6*	2.4	-
	Internship activities	9.8	0.7	10
	Theoretical reference of SUS	6.6*	2.0	-
	Experiences in SUS	9.1	2.4	9
	Reference and counter-reference	6.2*	1.7	-
	Planning and evaluation of services	6.0*	2.1	-
Pedagogical approach
	Active methodologies	9.8	0.9	10
	Role of the teacher	9.6	1.3	10
	Teacher development	9.6	1.3	10
	Learning scenarios	9.8	0.9	10
	Integrated curriculum	9.6	1.3	10
	Procedural evaluation of student learning	9.8	0.9	10
	Curricular flexibility	9.7	1.0	10
	Teaching-research-extension	9.7	1.0	10

• ^*
  Average < 7,0

Among the twenty-five initial criteria proposed, five criteria were considered non-essential to the evaluation of training dentists and were therefore excluded from the matrix: disease prevention (Dimension of orientation of health care); inclusion of students in the SUS, SUS theoretical framework, reference and counter-reference, and planning and evaluation of services (Dimension of teaching-service integration).

In the second round corresponding to the consensus conference, the four experts confirmed the dimensions and sub-dimensions validated in the previous step and reconsidered part of the sub-dimensions defining criteria. The analyzes obtained from experts in the third round confirmed the criteria discussed throughout the actual meeting, setting the final matrix of evaluation criteria for Dentistry courses according to the NCG, shown in Chart 2.

Chart 2
Sub-dimensions and criteria validated from the third round of the Delphi, Modified by dimension.

sub-dimension	Criteria
Profile of graduate
Generalist	In the collective field, develops programs of health promotion and management of services and at the same time, in the individual field performs the prevention, diagnosis, planning, and dental treatment of major dental problems.
Humanist	Offers quality care with receptive qualified listening, an amplified view of the subject and their care needs, coordinating with the technological improvement of care and conditions of workplace advances, building solidarity exchanges and committed to the production of health in relation with users and other professionals, based on ethical principles.
Autonomous	Makes decisions in both clinical procedures as in management situations and collective work, safely and skillfully.
Critical / reflective	Discusses and assesses situations of individual nature and / or collective and proposes alternative solutions from their scientific knowledge and reflection.
Able to understand social reality	Identifies the social context in which the professional practices, respecting the characteristics of the population and seeking appropriate solutions to this reality.
Orientation of health care
Epidemiologial approach	Curriculum structure organized from the epidemiological reality of the region and the country, based on the health needs of the population.
Health Promotion	Understanding the social determination of health care and the development of comprehensive strategies for the expansion of healthy choices based on the living conditions of individuals and the population.
Diagnostic	Use of technologies and comprehensive view of the individual to support the diagnosis of all oral diseases.
Dental Treatment	Solutions of the most prevalent oral problems and needs of the subject, with an integrated approach.
Oral health team	Development activities with ancillary staff (auxiliary oral health technician or dental care) enabling teamwork.
Multiprofissionality	Development of guidance activities for multidisciplinary and interdisciplinary care of students, professionals or technicians from the same area (dental technicians) and / or other areas of health (nursing technicians, community workers, etc.), providing the completeness of health actions.
Integration of education service
Internship Activities	Clinical activities in individual and collective actions in areas outside the scope of IES, articulated with the SUS, formalized by agreement and supervised by teachers from all areas, from the first semesters of the course.
Experiences in SUS	Developed activities at all levels of the health care system, including the network fow, planning and evaluation of services and skills at each level, allowing us to understand the breadth and complexity of the NHS, its principles and guidelines.
SUS pedagogical approach
Active methodologies	Educational process based on new teaching strategies that enable reflection and encourage creativity in problem solving and teamwork.
Role of the teacher	Facilitator of learning, knowledge manager and organizer of activities that promote student learning.
Teacher development	Constant promotion of faculty training activities in the pedagogical area and the integration of content from several knowledge areas.
Learning scenarios	Learning based on multiple sources of knowledge such as libraries, virtual environments, community planning bodies, management and monitoring, schools, kindergartens, and social spaces, outside of the health units.
Integrated curriculum	Competencies and skills organized into units of learning with increasing complexity throughout the training process, articulating biological, health, human and social sciences with dentistry.
Procedural evaluation of student learning	Based on competencies and skills providing a formative and participatory nature (including self-assessment), held in order to systematically monitor and contribute to student learning.
Curricular flexibility	Flexibility, including workload, the fulfillment of the curriculum so that students can create their own paths of development in vocations, specific interests and potential.
Teaching-research-extension	Involvement of research and extension as curricular strategies that provide full educational experience.

Discussion

In a study of pre-evaluation, steps were developed such as a detailed description of the intervention, evaluative questions relevant to the process, building the logical model and the consensus on the items to be evaluated to provide clarification and reflection on boundaries and possibilities of the proposed evaluation¹⁰.

Among the three types of models ranked by Champagne et al.⁴⁰, the logical operating model is the closest to the model obtained in this study, since it represents the way the goals of the intervention can be achieved through resource and processes. In the constructed model, the detail of the image-goal allowed for a list of the aspects perceived in the NCG that are priorities for training better to the health needs of the population. In this perspective, Lampert¹⁶ states that the “train of thought” for those who will move toward the evaluation of institutions that purport to train health professionals begins with the health needs of society.

The logical model enabled the construction of the matrix of criteria for evaluating the training of dentists in the four dimensions proposed. There was consensus and similarity in value judgment criteria, and most of these were considered important to assess in view of the “experts” by Delphi consensus technique. One of the biggest challenges for the application of this technique was asking stakeholders to strengthen the principles of NCG, especially focused on the SUS as a privileged setting for learning and student centered teaching approaches.

The Delphi consensus facilitates access to a broad group and simultaneously a geographically dispersed population and allows the exchange of information between a large number of important people²⁴. Because there are no universally accepted and standardized requirements for the Delphi technique, a large variability can be found in investigations that use the method. Authors²⁵ argue that this variability does not consist in the failure of developing the method, it can be considered an opportunity due to its adaptability to conditions in which the study is situated. In a systematic review of the literature for the selection of quality indicators in health using the Delphi technique²⁷, most of the 80 studies analyzed describe modifications of the original method in which at least one physical meeting between the participants was included, just as occurred in the present study for the validation which describe sub-dimension criteria.

The option for the association of consensus techniques occurred with the view to reach consensus on all the criteria in the first round, for the possibility of adding the benefits related to classroom discussion of the criteria already submitted to initial individual consensus²⁸, which allowed to clarify possible misunderstandings, while still respecting individual opinions in the third round²⁷.

The dimensions “profile of graduate” and “pedagogical approach” had all the criteria considered relevant to the evaluation of training dentists, the latter dimension to that obtained by the average in all high scores of relevance criteria from the experts.

Although currently coexisting, teaching trends change over time and suggest a better way for education, including health professions, of questioning or criticising pedagogies, because the value of the student to know and to institute a change in reality and of itself in the development of human skills and techniques⁴¹.

In the dimension “orientation of health care,” the only criterion excluded was “disease prevention.” The prevention of diseases, mainly dental caries, is already a practice worked at extensively in dentistry training which may have influenced the lack of association of this criterion to the changes required in the curriculum aligned with current requirements of the NCG. Furthermore, the presence of the criterion “health promotion” may have led to the understanding that preventive practices should be incorporated from the strengthening of people's capacity to adopt the various factors that influence health.

Most of the criteria considered non-essential belonged to “teaching-service integration” dimension. Contrary to what may initially seem to be, it is believed that there was a devaluation of this magnitude by the “experts” as to its relevance in the evaluation of the training of health professionals. This finding may be explained by the fact that an overlap of interpretations between the proposed criteria in the initial matrix for this dimension was observed. It is noticed that the four criteria excluded (inclusion of students in the SUS, SUS theoretical framework, reference and counter-reference, and planning and evaluation services) are included in the evaluation criteria of “experiences in SUS” and “internship activities.”

The need for assessment systems with quantitative and qualitative indicators to identify strengths and weaknesses and guide the redirection of institutional planning are closely related to the mission of the institution formed by actions of teaching, research and extension based on the relationship between state and society and political institutes¹⁷. Public policies in the field of evaluation are potential inducers of transformation aligned to fulfill the NCG and constitutional guidelines of the SUS in the graduation of the health professional, reinforcing social responsibility and response to contemporary health demands of the population⁴².

The Federal Constitution of 1988 brings the notion of the state as an evaluator, putting the government as responsible for approval of the entry and permanence of courses in the education system, based on the evaluation process conducted by them⁵,⁴³. Currently this regulation is carried out by the National System of Higher Rating (SINAES), proposed in 2002 and established as a state policy in 2004, which results from various initiatives introduced for higher education evaluation since the late 70's⁴³.

In the instrument for assessing undergraduate courses used by SINAES, the evaluative dimensions are didactic-pedagogic organization, teaching faculty and infrastructure. Still herein, compliance with the NCG is characterized as a legal and regulatory mandatory requirement, however, to be an essentially regulatory item, not part of the calculation concept of course evaluation¹³.

The indicators and criteria recommended in the assessment tool of undergraduate INEP courses allow to affirm values that ensure adherence to the PPC of NCG⁴³, but also make valid judgments on the proposed dimensions. Nevertheless, despite the efforts of joint action between Ministries of Health and Education to include the NCG and SUS standards in higher education evaluation in health care based on SINAES⁷, for specific evaluation of courses in the area of health, and in this case dentistry courses, attention to the aspects inherent to the training peculiarities in this area of work is necessary. Therefore, the matrix of criteria proposed in this study brings the relative issues of health care guidance in their current perspective and the dimension of teaching-service integration facing the SUS, a condition stipulated in its own NCG and indispensable to the professional profile required by it.

Pinheiro et al.¹⁸ discussed the production of academic papers on the training of dentists in Brazil between 1992 and 2005. Studies found by the authors dealt with the profile, the social role of the dentist, the characteristic of work related to training, and evaluating curricular internship, curriculum structure, analysis, trends and contradictions, all related to NCG. Furthermore, Casotti et al.¹⁹ have presented results of the production of knowledge about dental education between the years 1995 and 2006 based on 52 dissertations and 20 theses from the database of the Coordination of Improvement of Higher Education Personnel. They concluded that production in this field is still in its infancy, but with multiple issues, such as disciplinary innovative experiences, teacher training, curriculum and graduate profile.

Among the few studies that purport to work with validated criteria for evaluating dentistry courses, Zilbovicius et al.²⁰ aimed to analyze the trend of changes in Brazilian dental education in the face of the need for NCG implementation. Quantitative data was gathered from the assessment tool used in the ABENO workshops in 2005 and 2006. Indicators of this instrument are based on radial analysis⁴⁴ consisting of three axes: theoretical orientation, pedagogical approach and practice scenarios. At the end of the radial assessment, most of the courses showed an developing degree of curricular innovation²¹.

In the study titled “The Adhesion of Undergraduate Courses in Nursing, Medicine and Dentistry to National Curriculum Guidelines”⁷, quantitative indices were created following dimensions and criteria of INEP reports that were more associated with prescribing to the NCG, and the evaluation reports and PPC courses which achieved lesser and greater adherence to the NCG were also qualitatively assessed. For courses of Dentistry, the results indicate that the barriers to NCG implementation are primarily related to the integration of teaching and service in the SUS, the difficulty of integration and curricular modification and difficulty in developing innovative, pedagogical projects different from the traditional teaching model. They concluded based on the pedagogical projects analyzed that the vocational training of dentists is still largely concentrated on the diagnosis and treatment of disease⁷.

In both studies cited⁷,²¹, important results could be obtained on the trend of changing the training of dentists since the implementation of NCG. However, many shortcomings remain on how the guidelines can be effected into curricula and educational practices, and which can be assessed by a greater number of ways through more sensitive and specific valid indicators⁴⁵, and verified by accessible, legitimate and close sources to the everyday training spots. It is also essential to understand the public health services and the world of work as central elements of a new pedagogical approach, capable to contribute to the new health care professional acting with the capacity of understanding reality and intervening on it⁴⁶.

The final matrix proposed in this study brings dimensions, sub-dimensions and is capable of being used to conduct evaluative studies of dentistry courses based on proposals of the NCG criteria. To do so, these criteria can be understood as cognitive categories²⁴, constructed from the NCG guidelines for training of health professionals and may be used in evaluating them.

The dimensions shown in this evaluative proposal had a different purpose as they deepened structural elements unexplored in the scientific literature. Still, one can glimpse the possibility of the matrix to be used, with appropriate adaptations, to other courses in the area of health. There is also possibility to use different methodological approaches in studies which intend to use the validated criteria herein.

The criteria presented here have duly validated scores for evaluation in quantitative studies, showing the sensitivity provided on a scale of 1 to 10. Still, the improvement of these criteria could be accomplished through consensus conference, where subjective reflections on aspects mentioned as inherent to the evaluation of the object in question were made possible since the attempt to insert subjective elements of the different actors in the construction of criteria enables increased validity of the parameters analyzed⁴⁷.

Faced with the need of evaluative culture of health education implementation based on new paradigms and training needs of the health system and the population, and consisting of a process that results in renewals for improving training activities, limitations are explicit as to the applicability of criteria built here. An example is the very purpose and evaluative understanding of the definition of competent authorities for these actions in higher education institutions (HEIs). Self-assessment was thought of as an important strategy since it allows the HEI self-awareness that favors the construction of a participatory and inclusive culture of assessment based on self-evaluative management of this process, enriching the planning and subsidizing the construction of internal development policies and offering higher quality education and public education policies⁴⁸.

This leads itself to the reflection on the importance of careful planning and methodological evaluative research on health in general, including in the educational field. Difficulties on the availability of theoretical guidance for methodological construction of evaluation studies in health that are not limited to programs and services are currently found in the literature.

The final instrument proposed in this study, legitimized by various participants who comprised the steps of construction and validation, is a differentiated alternative assessment of training dentists and other health professionals, to enable an assessment by a larger number of aspects, through valid, sensitive and specific indicators, and that can be checked by accessible, legitimate and closer everyday spots of the training sources, allowing for better approximation between evaluation methods of health education and the reality of the effectiveness of NCG.

Referências

• ¹
  Assunção AA, Belisário SA, Campos FE, D’Ávila LS. Recursos humanos e trabalho em saúde: os desafios de uma agenda de pesquisa. Cad Saude Publica 2007; 23(Supl. 2):S193-S201.
• ²
  Ceccim RB, Armani TB, Oliveira DLLC, Bilibio LF, Moraes M, Santos ND. Imaginários da formação em saúde no Brasil e os horizontes da regulação em saúde suplementar. Cien Saude Colet 2008; 13(5):1567-1578.
• ³
  Brasil. Ministério da Educação (MS). Conselho Nacional de Educação. Câmara de Educação Superior. Parecer nº 583 de 04 de abril de 2001. Aprova a orientação para as diretrizes curriculares dos cursos de graduação. Diário Oficial União 2001; 29 out.
• ⁴
  Costa ICC, Araújo MNT. Definição do perfil de competências em saúde coletiva a partir da experiência de cirurgiões-dentistas atuantes no serviço público. Cien Saude Colet 2011; 16(Supl. 1):1181-1189.
• ⁵
  Brasil. Constituição da República Federativa do Brasil de 1988. Diário Oficial da União 1988; 5 out.
• ⁶
  Haddad AE, Pierantoni CR, Ristoff D, Xavier IM, Giolo J, Silva LB, organizadores. A trajetória dos cursos de graduação na área da saúde: 1991-2004 Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira; 2006.
• ⁷
  Brasil. Ministério da Saúde (MS). Ministério da Educação (ME). A aderência dos cursos de graduação em enfermagem, medicina e odontologia às diretrizes curriculares nacionais Brasília: MS; 2006.
• ⁸
  Januzzi PM. Avaliação de programas sociais no Brasil: repensando práticas e metodologias das pesquisas avaliativas. Planejamento e políticas públicas 2011; (36):251-275.
• ⁹
  Bezerra LCA, Alves CKA, Reis YAC, Samico I, Felisberto E, Carvalho ALB, Silva GS. Identificação e caracterização dos elementos constituintes de uma intervenção: pré-avaliação da política Participa SUS. Cien Saude Colet 2012; 17(4):883-900.
• ¹⁰
  Mendes MFM, Cazarin G, Bezerra LCA, Dubeux LS. Avaliabilidade ou pré-avaliação de um programa. In: Samico A, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em Saúde: bases conceituais e operacionais Rio de Janeiro: MedBook; 2010. p. 57-64.
• ¹¹
  Medina MG, Silva GAP, Aquino R, Hartz ZMA. Uso de modelos teóricos na avaliação em saúde: aspectos conceituais e operacionais. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102.
• ¹²
  Brasil. Ministério da Educação. Conselho Nacional de Educação. Câmara de Educação Superior. Resolução 3, de 19 de fevereiro de 2002. Institui Diretrizes Curriculares Nacionais do Curso de Graduação em Odontologia. Diário Oficial União 2002; 4 mar.
• ¹³
  Brasil. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira – Inep. Diretoria de Avaliação da Educação Superior – Daes. Sistema Nacional de Avaliação da Educação Superior – Sinaes. Instrumento de Avaliação de Cursos de Graduação presencial e a distância. Brasília: maio 2012. [acessado 2013 jun 13]. Disponível em: http://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2012/instrumento_com_ alteracoes_maio_12.pdf
  » http://download.inep.gov.br/educacao_superior/avaliacao_cursos_graduacao/instrumentos/2012/instrumento_com_ alteracoes_maio_12.pdf
• ¹⁴
  Brasil. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. SINAES – Sistema Nacional de Avaliação da Educação Superior: da concepção à regulamentação 5ª ed. revista e ampliada. Brasília: Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira; 2009.
• ¹⁵
  Morita MC, Kriger L, Carvalho ACP, Haddad AE. Implantação das diretrizes curriculares nacionais em odontologia. Maringá: Dental Press, Abeno, Opas, MS; 2007.
• ¹⁶
  Lampert JB. Avaliação institucional nos cursos de graduação da área da saúde: avaliar o quê e para quê? Cadernos ABEM 2009; 5:45-55.
• ¹⁷
  Braid LMC, Machado, MFAS, Aranha AC. Estado da arte das pesquisas sobre currículo em cursos de formação de profissionais da área da saúde: um levantamento a partir de artigos publicados entre 2005 e 2011. Interface (Botucatu) 2012; 16(42):679-692.
• ¹⁸
  Pinheiro FMC, Nobrega-Therrien SM, Almeida MEL, Almeida MI. A formação do cirurgião-dentista no Brasil: contribuições de estudos para a prática da profissão. RGO 2009; 57(1):99-106.
• ¹⁹
  Casotti E, Ribeiro VMB, Gouvêa MV. Educação em odontologia no Brasil: produção do conhecimento no período 1995-2006. Hist Cien. saude-Manguinhos 2009; 16(4):999-1010.
• ²⁰
  Zilbovicius C, Araujo ME, Botazzo C, Frias AC, Junqueira SR, Junqueira CR. A paradigma shift in predoctoral dental curricula in Brazil: evaluating the process of change. J Dent Educ 2011; 75(4):557-564.
• ²¹
  Araujo ME. Palavras e silêncios na educação superior em odontologia. Cien Saude Colet 2006; 11(1):179-182.
• ²²
  Carvalho ACP, Kriger L. Educação Odontológica São Paulo: Artes Médicas; 2006.
• ²³
  Bunge M. Os conceitos de modelo. In: Bunge M. Teoria e realidade São Paulo: Perspectiva; 1974. p. 11-40.
• ²⁴
  Esher A, Santos EM, Magarinos-Torres R, AzeredoTB. Construindo critérios de julgamento em avaliação: especialistas e satisfação dos usuários com a dispensação do tratamento do HIV/AIDS. Cien Saude Colet 2012; 17(1):203-214.
• ²⁵
  Valdés MG, Marín MS. Empleo del método Delphi en investigaciones sobre salud publicadas en revistas científicas cubanas. Rev Cubana de Información en Ciencias de la Salud 2013; 24(2):133-144.
• ²⁶
  Fonsêca GS, Junqueira SR. Programa de Educação pelo Trabalho para a Saúde da Universidade de São Paulo (Campus Capital): o olhar dos tutores. Cien Saude Colet 2014; 19(4):1151-1162.
• ²⁷
  Boulkedid R, Abdoul H, Loustau M, Sibony O, Alberti C. Using and Reporting the Delphi Method for Selecting Healthcare Quality Indicators: A Systematic Review. Plos One 2011; 6:e20476
• ²⁸
  Souza LEPF, Silva LMV, Hartz ZMA. Conferência de consenso sobre a imagem-objetivo da descentralização da atenção à saúde no Brasil. In: Hartz ZMA, Silva LMV, organizadores. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador, Rio de Janeiro: EDUFBA, Fiocruz; 2005. p. 65-102.
• ²⁹
  Crespo T. Respuestas a 16 preguntas sobre el empleo de “expertos” en investigación pedagógica. Lima: San Marcos; 2007.
• ³⁰
  Okoli C, Pawlowski S. The Delphi method as a research tool: an example, design considerations and applications. Information & Management 2004; 42:15-29.
• ³¹
  Bloor M, Sampson H, Baker S, Dahlgren K. Useful but no Oracle: Reflections on the use of a Delphi Group in a multi-methods policy research study. Qualitative Research 2014; DOI: 10.1177/1468794113504103.
• ³²
  Higa EFR, Hafner MLMB, Tonhom SFR, Taipeiro EF, Moreira HM, Guimarães APC, Pinheiro OL. Indicadores de avaliação em gestão e saúde coletiva na formação médica. Rev Bras Educ Médica 2013; 37(1):52-59.
• ³³
  Gontijo ED, Alvim C, Megale L, Melo JRC, Lima MECC. Matriz de competências essenciais para a formação e avaliação de desempenho de estudantes de Medicina. Rev Bras Educ Médica 2013; 37(4):526-539.
• ³⁴
  Costa EAM, Costa EA, Graziano KU, Padoveze MC. Reprocessamento de produtos médicos: uma proposta de modelo regulatório para hospitais brasileiros. Rev Esc Enferm USP 2011; 45(6):1459-1465.
• ³⁵
  Silva GS, Samico I, Dubeux LS, Felisberto E. Redes de atenção às urgências e emergências: pré-avaliação das Unidades de Pronto Atendimento (UPAs) em uma região metropolitana do Brasil. Rev. Bras. Saúde Matern. Infant. 2012; 12(4):445-458.
• ³⁶
  Tonhom SFR, Higa EFR, Pinheiro OL, Hafner MLMB, Moreira HM, Taipeiro EF, Amaral APCG. Indicadores de Avaliação do Cuidado Individual: Subsídios para a Formação Médica Orientada por Competência. Rev Bras Educ Médica 2014; 38(3):331-336.
• ³⁷
  Reis YAC, Cesse EAP, Carvalho EF. Consensos sobre o papel do gestor estadual na regionalização da assistência à saúde no Sistema Único de Saúde (SUS). Rev. Bras. Saúde Matern. Infant. 2010; 10(Supl. 1):S157-S172.
• ³⁸
  Powell C. The Delphi technique: myths and realities. J Adv Nurs 2003; 41(4):376-382.
• ³⁹
  Brasil. Ministério da Saúde (MS). Conselho Nacional de Saúde. Resolução n° 466, de 12 de dezembro de 2012. Diário Oficial da União 2013; 13 jun.
• ⁴⁰
  Champagne F, Brousselle A, Hartz Z, Contandriopoulos AP. Modelizar as intervenções. In: Brousselle A, Champagne F, Contandriopoulos AP, Hartz Z, organizadores. Avaliação: conceitos e métodos. Rio de Janeiro: Editora Fiocruz; 2011. p.61-74.
• ⁴¹
  Pereira ALF. As tendências pedagógicas e a prática educativa nas ciências da saúde. Cad Saude Publica 2003; 19(5):1527-1534.
• ⁴²
  Ceccim RB, Feuerwerker LCM. Mudança na graduação das profissões de saúde sob o eixo da integralidade. Cad Saude Publica 2004; 20(5):1400-1410.
• ⁴³
  Ristoff D, Giolo J. O Sinaes como sistema. Rev Bras Pós-Grad 2006; 3(6):193-213.
• ⁴⁴
  Campos FE, Ferreira JR, Feuerwerker LCM, Sena RR, Campos JJB, Cordeiro H, Cordoni Júnior L. Caminhos para aproximar a formação de profissionais de saúde das necessidades da atenção básica. Rev Bras Educ Médica 2001; 25(2):53-59.
• ⁴⁵
  Alves CKA, Natal S, Felisberto E, Samico I. Interpretação e análise das informações: o uso de matrizes, critérios, indicadores e padrões. In: Samico A, Felisberto E, Figueiró AC, Frias PG, organizadores. Avaliação em Saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook; 2010. p.89-107.
• ⁴⁶
  Werneck MAF, Senna MIB, Drumond MM, Lucas SD. Nem tudo é estágio: contribuições para o debate. Cien Saude Colet 2010; 15(1):221-231.
• ⁴⁷
  Minayo MCS. Construção de indicadores qualitativos para avaliação de mudanças. Rev Bras Educ Médica 2009; 33(Supl. 1):83-91.
• ⁴⁸
  Polidori MM, Marinho-Araújo CM, Barreyro GB. Sinaes: perspectivas e desafios na avaliação da educação superior brasileira. Ensaio: aval pol pub educ 2006; 14(53):425-436.

Publication Dates

• Publication in this collection
  July 2015

History

• Received
  08 Mar 2014
• Reviewed
  14 Nov 2014
• Accepted
  16 Nov 2014