The meanings of evaluation practice in nursing education

Abstracts

El presente trabajo analizó los sentidos de la evaluación en la formación de enfermeros, orientada por la competencia, utilizando el marco teórico de análisis de las prácticas discursivas y de la producción de sentidos en lo cotidiano. Los datos fueron recolectados en un grupo focal, compuesto por siete profesores de un curso de Enfermería del municipio de Marília, en el estado de San Pablo, Brasil. En el análisis de los datos, se encontró que la mayoría de los repertorios lingüísticos se refieren a la evaluación tradicional y a la noción de competencia basada en la matriz constructivista francesa. Sin embargo, también fue observada la presencia de repertorios que producen sentidos de renovación de la práctica evaluativa basada en la evaluación democrática y en la matriz dialógica de competencia.

educación basada en competencias; educación en enfermería; evaluación educacional; curriculum; enfermería

O presente trabalho analisou os sentidos da avaliação na formação de enfermeiros, orientada por competência, utilizando o referencial de análise das práticas discursivas e produção de sentidos no cotidiano. Os dados foram coletados em grupo focal, composto por sete professores de um curso de enfermagem do município de Marília, SP. Na análise dos dados, foi reconhecido que a maioria dos repertórios lingüísticos reporta-se à avaliação tradicional e à noção de competência baseada na matriz construtivista francesa. Porém, também, observou-se a presença de repertórios que produzem sentidos de renovação da prática avaliativa baseada na avaliação democrática e na matriz dialógica de competência.

educação baseada em competências; educação em enfermagem; avaliação educacional; currículo; enfermagem

ARTIGO ORIGINAL

The meanings of evaluation practice in nursing education¹

Maria Cristina Martinez Capel Laluna^I; Clarice Aparecida Ferraz^II

^IRN, Ph.D. in Nursing, Faculty at Faculdade de Medicina de Marília, Brazil, e-mail: [e-mail]

^IIAssociate Professor, Escola de Enfermagem de Ribeirão Preto, da Universidade de São Paulo, Centro Colaborador da OMS para o Desenvolvimento da Pesquisa em Enfermagem, Brazil, e-mail: [e-mail]

ABSTRACT

This study analyzed the meanings of evaluation practice in competence-based nursing education through discursive practices and production of meanings in daily routine. Data were collected with a focal group composed of seven professors from a nursing program in Marília, SP, Brazil. It could be acknowledged, during data analysis, that most of the linguistic repertories refer to the traditional mode of evaluation and to the competence notion based on the French constructivist framework. However, repertories producing meanings related to the innovation of the evaluation method, based on democratic evaluation and on the dialogical competence framework, are also observed.

Descriptors: competence-based education; education nursing; educational measurement; curriculum; nursing

INTRODUCTION

The current changing world of work, knowledge production and formulation of educational public policies focused on the Brazilian Single Health System (SUS) are pushing higher education institutions to modify their pedagogical projects. In this context, a nursing program in Marília, SP, Brazil, implemented an integrated competence-oriented curriculum in 1998, in which nursing is understood as a historically determined social practice that is ethically and politically committed to the populations' health according to the rationale proposed by the SUS⁽¹⁾ model.

Competence-oriented education can be based on distinct frameworks - conductivism, functionalism, French constructivism and the Australian model. The first consists of a set of independent tasks. The second includes, in addition to tasks, detailing functions, disregarding attributes (knowledge, attitudes and abilities) that underly work practices. The third, of French origin, seeks the construction of competences that include the relation of functions and tasks with the environment, including work experiences with education. It becomes a way to adapt personal capacities to tasks that fix content and prioritize results⁽²⁾. The Australian dialogical notion of competence proposes the integration of attributes and different social constructions that legitimate them. It seeks to combine a complex net of attributes in action, considering the context and incorporating ethics and values as elements of performance^(2-3).

In order to preserve coherence with its educational goal, this program curriculum seeks to adopt a dialogical notion and develop the following areas of competence: care to individual and collective needs, work organization and management and scientific investigation. Evaluation aims to favor the teaching-learning process in order to transform professional practice. It involves both formative and summative criteria, among which standards of competences are used to compare the performance of each student. With a view to a careful and reflective follow-up of learning development, this evaluation diminishes competition among students and promotes dialog among those involved. In addition to instruments used in student performance evaluation, there are specific forms in which a descriptive synthesis of students' development is registered. Based on these notes, Sufficient and Insufficient concepts are applied^(2-3).

Evaluation is thematic and complex and it is the result of relations established in its process. Up to the 1990s, the concept of evaluation included measure, description and value judgment, marked by the traditional framework, giving priority to the epistemological positivist orientation. At the end of the century, it was based on the constructivist framework, in which evaluation was oriented by subjective epistemology, which implies negotiation and requires a more democratic attitude⁽⁵⁾. At the beginning of this millennium, evaluation started to include the concept of empowerment, and different perspectives focused on development and learning⁽⁶⁾ are shared. In this context, evaluation is included under the epistemological orientation, an individual-society bond. It requires understanding human activity, the subject's practical action, which implies analyzing the meaning and objective of this conscious action⁽⁵⁾.

A democratic evaluation framework can be built on the basis of the epistemological orientations of the evaluation frameworks exposed so far. Even though they are distinct, contradictory and competitive, they can be complementary. That is, a democratic evaluation framework based on social and dialogical construction, where participation, autonomy, negotiation, inclusion and commitment to everyone's learning and to integral education is allowed. Changing evaluation methods requires commitment from institution and professors towards a new attitude, that is, professors change their previous role based on surveillance and judgement to one that aims to educate and create a new culture jointly with students, professors, school and community ⁽⁷⁾.

It is verified in the Brazilian nursing^(8-9) literature that the production and discussion regarding nurses' competence-oriented education, as well as the analysis on evaluation practices of nurses' competence-oriented education, is focused on the verification of content and abilities in curricula, instead of appreciating competence in an integrated curriculum.

Evaluation in this nursing program has been a critical issue and its practice is not in agreement with the evaluation method proposed in its curricular project. This article is an excerpt of a doctoral dissertation⁽¹⁰⁾ that aimed to analyze the meaning of evaluation in the competence-oriented educational routine, in the perspective of a group of professors from a nursing program in Marilia, SP, Brazil.

METHOD

This study adopted the constructionist approach, in which knowledge is the result of social construction. In the analysis of discursive practices, the focus is on the role of the language used in social interaction and is directed to the ways "people produce meanings and engage in daily social relations"⁽¹¹⁾. Discursive practices contain as constitutive elements "the dynamic (which are statements oriented by voices), speech genres (which Bakhtin considers somewhat fixed genres of statements) and content, linguistic repertories", which are set in movement in the process of "dialogical inter-animation, that is, in the interpersonal dimension, that of relation with the other, whether physically present or not" ⁽¹¹⁾.

The analysis, focused on searching for meaning, is carried out through immersion in the set of collected information, seeking to expose meanings without forcing them into pre-established categories. It is from the confrontation between meanings, constructed during investigation and previous familiarization acquired through literature and theoretical references, that categories of analysis are co-constructed⁽¹²⁾.

Data collection was carried out through a focal group and initiated the discussion about how evaluation has been experienced in the daily routine of academic education. The project was evaluated by the Research Ethics Committee from the Medical School at Marília, and participants signed a free and informed consent term. The group was composed of seven professors from a nursing program in Marilia, SP, Brazil, including a health service professional who works as a faculty collaborator. Subjects' identities were preserved through the use of the letter O for observer, P for coordinator and P, with an ordinal and sequential number, for the remaining participants.

Dialogical maps were adopted for data organization. It is a strategy used to "systematize the analysis process of discursive practices in search of formal aspects of the linguistic construction, repertories used in this construction and dialog implicit in the production of meanings"⁽¹²⁾. The analysis began from linguistic repertories that favor connection between the use given by subjects in the study and the theoretical framework used during investigation and the researcher's interpretation.

RESULTS AND DISCUSSION

One of the themes analyzed was Evaluation practice: reference standard, object, instrument and intervenient activities in determining Insufficient result. To better understand the co-construction theme, a synthesis of the dialog is presented, highlighting the linguistic repertories that produced meanings regarding the evaluation method, followed by respective analysis.

Participants talk about the object, reference standard, activities and evaluation instruments. Thus, P7 talks about the practice of performance evaluation, indicating that it is very hard to do it, because professors are still stuck to the traditional idea of evaluation. P7 explains that this difficulty is because, in addition to observing the student's knowledge, one has to observe the student's attitudes. P7 stresses the need to have some criteria and some standards to facilitate comparison between students' performance, pointing out that the standard is not very clear to professors. P7 also highlights that, when one has to present the evaluation result, the subjectivity component has to be considered as well, because what one believes to be sufficient, is not necessarily sufficient for another. The participant also argues that professors do not have the habit and practice of including such a component in their evaluation process. For this reason, P7 appoints that professors have difficulties in saying that a student's performance is insufficient...

For P1, on the other hand, the insufficient concept seems to be [...] ready in the Cognitive Evaluation Exercise (CEE). P6 disagrees and questions that, if the proposal is to stimulate students to connect dimensions (cognitive, attitudinal and ability) how can progression or retention be based on the cognitive evaluation only?

P3 considers the discourse of P7, puts in discussion the existent standard performance and says that professors present difficulties because they compare students. P3 argues that, when one considers performance as standard, one has to understand that students have different ways to achieve their performances. For P6, the great difficulty is related to the criteria used by professors to indicate which performance [...] is necessary for students to get [...] their degrees at the end of the nursing program. P6 argues that, until a consensus is reached, although he believes that this may not be possible, as this situation also contains subjectivity, the issue will not be solved. P2 says that the difficulty is due to performance interpretation.

When P7 talks about the evaluation activity per se and its instrument, he appoints that the cognitive evaluation is still based on the traditional evaluation method, and associates it to the traditional test, which causes a lot of stress to students and professors, besides the effort spent elaborating the CEE. P7 states that there is a certain effort to elaborate an integrated evaluation when one seeks connection between different areas of knowledge.

P5 points out that professors need to reach a consensus regarding what and how to evaluate and also explains that a written account is needed because, during the supervision carried out in the fourth year, evaluation is performed through pure observation. Then, when one has to fill out the instrument, P5 reports not being able to evaluate. He suggests that the instrument should indicate how much initiative one has to have [...] detailed, quantified, so as to help during the evaluation process. When P3 talks about the difficulty P5 reports, he points out that the work proposal is to observe the performance, follow the students' work, together so as to develop individual and collective care and management as written and how I believe a nurse should be trained. He reports that the standard reference is the bible, the work contract the professor maintains with the student, in which the school's beliefs [...] on how to train nurses, students in this kind of performance need to be clarified. He also adds that professors needs to create oportunities so students can develop [...] those performances and, when students do not accomplish it, it should be constructed with them.

In his reply, P5 points out that his difficulty was to register and understand that the experienced situation fits in the three activities (from care to collective and individual needs, work process management and organization). However, P4 questions how one can use a situation like this [...] with the understanding of performance one expects. There used to be a partnership (academy-health service), you were not alone. For P6, one of the ways of working in these situations is the reflective portfolio, because it is an instrument that can support [...] students' evaluation and also favors students' self-evaluation, so they can think about what has been presented and construct, find meaning. P6 points out, though, that professors have not managed yet to adequately use it in order to facilitate students' work.

When P1 asked P3 why nurses, who work in the health services, do not have access to the portfolio, P3 explains that this is a limitation imposed in the fourth year, because it is an activity from the pedagogical cycle and nurses do not participate in it. He states that, during the movement of the cycle, students present a report, we problematize this report, work on a provisional synthesis and then a new synthesis. He highlights that this systematized reflection of supervision should be constructed with nurses who are active in the health service.

P4 is experienced in the Systematized Unit (Educational Unit in which problem cases are used to provide stimulus to learn content) and he talks about his reflection jointly with students on the portfolio and they report it's so boring to organize the UPP portfolio (Unit of professional practices in which learning is acquired during professional practice). He points out that he takes note in the systematized unit in a very easy way; there is no nomination, organization or evaluation of the instrument.

P6 acknowledges students' difficulties regarding the portfolio and believes these difficulties are related to the way the school and professors present it to students since its implementation. The normative requirement is met when the student hands in the portfolio, which implies getting sufficient concept. P3 adds that students prepare it only to hand it in. They do not work on it to construct knowledge.

P5 talks about the professional practice simulated evaluation (PPSE) carried out in the UPP4 (Unit of professional practice in the fourth year). P5 informs that the students' perception regarding this activity is that they are required to present a certain practice, although another one is performed in this scenario. He argues that aspects related to the physical exam are asked during these situations (referring to problem-cases used during professional practice simulated evaluations). These aspects should already have been required in previous years, since they are already in the fourth year. P1 says that students perform individual care evaluation on a daily basis in the basic health network, however, care in hospitals is extremely focused on complaints. Still discussing this issue, P3 explains that the performances were constructed with nurses and professors from the program and asks why nurses' practice in hospitals differs from the basic health network, since professors are re-thinking and reconstructing this practice and working with nurses and students during the program.

Most of the linguistic repertories suggest that professors face difficulties when they have to attribute an insufficient concept to students' performance due to subjectivity, object integrality, and evaluation instruments and activities. This meaning refers to the traditional evaluation and to the competence notion based on the French constructivist framework. Also, repertories are present that produce meaning in the attempt to innovate the evaluative practice, based on the democratic framework and dialogical competence.

In the competence-oriented curriculum, professors' lack of clarity affects the object to be evaluated, the form of implementation and the interpretation of students' production, because professors are responsible for providing criteria to establish the decision shared by subjects who participate in the learning-teaching process.

The competence standard, essential dimension for the evaluation process, becomes a criterion that permits analyzing students' development. Thus, it should be consistent with what has been proposed by professional practice and with its definition. Competence is not directly observable but inferred by performance. Performance consists of indicators that favor integrated analysis of attributes (cognitive, attitudinal, and abilities) in action, considering the context and relations established when based on the dialogical notion of competence. This notion favors the construction of meanings of professional practice committed with projects to transform social reality, supported on ethical values. Yet, it proposes integration between practice and theory, as well as meaningful learning through active teaching-learning methods, to promote critical and reflective training. Thus, students can mobilize their attributes in different ways in daily situations of professional practice⁽³⁾.

The role of educational institutions, when based on the French constructivist framework, is to construct knowledge, even though this framework is directed to social construction and to the relation of individual and collective capacities. Given the existent dichotomy between work field and education in the exploration and analysis of competences, the role of work institutions is to develop and use competence^(3,13). Thus, during education, performance consists of a collection of attributes and might favor "a disconnected development of the cognitive, psychomotor and attitudinal domains and reduce practice to simple implementation of theory"⁽³⁾. Evaluation, focused on the verification of attributes, reduces and fragments competence⁽³⁾.

Professors, in the search for a new evaluation method, face a strong internalization of traditional evaluation, represented by the constant need for objectivity of criteria to express the result of the evaluation. The traditional evaluation has mainly used reference to standards in order to compare students' performance in relation to the group average when they perform the same activity and, in turn, proposes to verify knowledge or lack of knowledge and presents results in the form of grades or concepts. The use of results is restricted to students who need measures to achieve the expected, promoting competition, which reveals an evaluation more committed to selection than to learning and education.

Professors need to understand that subjectivity is always present during evaluation. It is through dialog between professors and students and the use of an understandable and welcoming language that consensus can be reached. This way, students will understand that their relation is democratic and committed with learning. On the contrary, difficulty in managing subjectivity and lack of clarity are reproduced and interfere in the pedagogical relation between students and professors.

The evaluation of students' performance as a whole is a challenge professors are faced with. Even though there is understanding of its importance, many professors have difficulty to include the attitudinal attribute in the evaluation process. Professors tend to evaluate attitudinal attributes in the same way cognitive attributes are evaluated. That is, they demand objectivity and the use of grades or concepts, tied to approval or disapproval. This situation represents a distortion of the school system because it "works as a menacing power" for students⁽⁷⁾.

In performance evaluation, professors still seek information, using proofs to confirm learning of content, traditionally focused on the cognitive component. To change this method of evaluation, diversified activities to express performance can be adopted, though, in case there is no understanding of the evaluation goal, chances of changing will be at risk.

The main focus should be on evaluation methods that lead to better understanding of the needs of each and all students. There is no point in innovating if the institution maintains formal mechanisms of control, causing stress to students and making it difficult to express their learning⁽⁷⁾. There are formal evaluation rules that have lead to bureaucracy and control, more than favored changes in evaluation practice and commitment to learning. This does not mean saying that rules are not necessary. In fact, what is argued is that they should be collectively constructed and coherent with the goals of the proposed education and evaluation.

Taking into account the competence dialogical approach, through activities during evaluation, one seeks to combine attributes and tasks, theory and practice, guided by professional practice in the real world or simulated problems, through observation and systematized analysis. Expression activities that favor identification of previous capacities are also used, as well as learning-teaching activities and written exercises elaborated from a problem case, translated from experience in professional practice and integration of several areas of knowledge. Longitudinal and multiple evaluations favor integrated evaluation and these evaluations include values and attitudes, an issue that has much concerned professors⁽¹⁴⁾.

If the evaluation is based on the traditional perspective, records are used as a way to present how much students have learned. However, if it is focused on learning commitment, it is carried out in the process, and there is interaction between quantity and quality, permitting careful consideration of learning and also dialog with students. For these records to have an educative proposition, professors also need to learn how to observe in a systematized way so their perspective is meaningful. The portfolio is among the possibilities of reflective recording, and it can be developed both by professors and students, because it leads to a critical and careful consideration about the learning process and one's learning capacity⁽⁷⁾. When it is used for evaluative purposes, it becomes an instrument of dialog-reflection-action, and favors the follow-up of personal-professional development through self-evaluation and co-evaluation. It also favors evaluation of professional practice in context, evidencing education products and processes. Thus, it cannot be produced "at the end of evaluation periods, but it has to be continually (re)elaborated in action" and shared for critical analysis of its practice⁽¹⁵⁾.

The competence areas in the studied curriculum present a different professional practice, which is still in process. However, if this process does not favor the inclusion of nurses inserted in the work field for continued reflection on professional and evaluative practice, it certainly puts the expected change proposal at risk. Thus, it is necessary to use managers' and professors' political strategies to favor partnership between academia and health services.

FINAL CONSIDERATIONS

It is evidenced that meanings produced on evaluation are closer to the classificatory and selective rationale and to the French constructivist competence framework. This is because it focuses on attributes, favoring an analysis disconnected and out of context of these attributes⁽³⁾. Even though in a smaller proportion, the production of meanings regarding evaluation is observed, focused on inclusion and commitment to education, based on the competence dialogical framework. This is observed every time performance is analyzed from the combination of attributes mobilized in action, in certain contexts, expressing the relation between professional and work⁽³⁾. Although the competence standard was collectively designed, it is necessary to be socially validated, which requires time and trust to guarantee the legitimacy of the process. Cultural internalization of traditional evaluation is a relevant aspect. The dualities objectivity-subjectivity and quantity-quality need to be identified and elaborated upon, so that changing the method of evaluation favors the formation of critical and reflective nurses, who politically interact with changes in health care under an SUS perspective.

Recebido em: 24.7.2007

Aprovado em: 23.12.2008

• 1. Chirelli MQ, organizador. Introdução ao Curso de Enfermagem. Marília (SP): Faculdade de Medicina de Marília; 2000.
• 2. Ramos MN. A pedagogia das competências: autonomia ou adaptação? São Paulo: Cortez; 2001.
• 3. Lima VV. Competência: distintas abordagens e implicações na formação de profissionais de saúde. Interface Comun Saúde Educ 2005 março-agosto; 9(17):369-79.
• ⁴
  Faculdade de Medicina de Marília - FAMEMA. Manual de avaliação do estudante: cursos de medicina e enfermagem. Marília (SP): Faculdade de Medicina de Marília; 2006.
• 5. Franco MLPB. Pressupostos epistemológicos da avaliação educacional. In: Sousa CP, organizador. Avaliação do rendimento escolar. 13Ş ed. Campinas (SP): Papirus; 2005.
• 6. Penna Firme T, Putten JV, Blackburn R. Avaliação de docentes e do ensino. In: Sousa ECBM, organizadora. Avaliação de docentes e do ensino: Curso de Especialização em Avaliação a Distância. Brasília (DF): Universidade de Brasília; 1998. v. 5.
• 7. Vasconcellos CS. Avaliação da aprendizagem: práticas de mudança: por uma práxis transformadora. 7Ş ed. São Paulo: Libertad; 2005.
• 8. Silva KL, Sena RR. A educação de enfermagem: buscando a formação crítico-reflexiva e as competências profissionais. Rev Latino-Am Enfermagem 2006; 14(5):755-61.
• 9. Sadala MLA, Stefanelli MC. Avaliação do ensino de relacionamento enfermeira-paciente. Rev Latino-am Enfermagem 1996; 4(número especial):139-51.
• 10. Laluna MCMC. Os sentidos da avaliação na formação de enfermeiros orientada por competência. [tese]. Ribeirão Preto (SP): Escola de Enfermagem de Ribeirão Preto/USP; 2007.
• 11. Spink MJ. Linguagem e produção de sentidos no cotidiano. Porto Alegre (RS): EDIPUCRS: 2004. 87 p. (Coleção debates contemporâneos em psicologia social, 1).
• 12. Spink MJ, Lima H. Rigor e visibilidade: a explicitação dos passos da interpretação. In: Spink MJ, organizadora. Práticas discursivas e produção de sentidos no cotidiano: aproximações teóricas e metodológicas. 3Ş ed. São Paulo (SP): Cortez; 2004. p. 93-122.
• 13. Zarifian P. O modelo de competência e suas conseqüências sobre os ofícios profissionais. Anais do Seminário Internacional Mercado de Trabalho: transformações e monitoramento de ocupações 1999; Rio de Janeiro; Brasil.
• 14. Hager P, Gonczi A, Athanasou J. General issues about assessment of competence. Asses Eval High Educ 1994; 19(1):3-16.
• 15. Sá-Chaves I. Portfólios reflexivos: estratégia de formação e de supervisão. Aveiro (PT): Universidade de Aveiro; 2000. (Cadernos didácticos: série supervisão, 1).

1

Paper extracted from Doctoral Dissertation

Publication Dates

• Publication in this collection
  14 Apr 2009
• Date of issue
  Feb 2009

History

• Received
  24 July 2007
• Accepted
  23 Dec 2008