Capturing the meaning of context for a meaningful evaluation

Captura el contexto para una evaluación significativa

Captando o significado do contexto para uma avaliação significativa

REPORTS AND CONTRIBUTIONS

Capturing the meaning of context for a meaningful evaluation^*

Captando o significado do contexto para uma avaliação significativa

Captura el contexto para una evaluación significativa

Thereza Penna Firme^I; Vathsala Stone^II; Ana Carolina Letichevsky^III; Angela Cristina Dannemann^IV

^ICesgranrio Foundation. E-mail: [e-mail]

^IIUniversity of Buffalo, State University of New York. E-mail: [e-mail]

^IIICesgranrio Foundation. E-mail: [e-mail]

^IVD&S Associate Engineers. E-mail: [e-mail]

ABSTRACT

That an individual evaluee is a unique composition of attributes and needs is commonly understood. What is less obvious is that programs, too, have complex personalities of their own, like the individuals that compose them. They draw their needs from their own contexts, and draw meaning from an evaluation, operating in these unique contexts. What is meaningful to one program may not be so for another. Evaluation is influenced by the program's context. Reciprocally, it can influence the context too. The challenge is to fully discover the context with its needs, so results are meaningful to the program in its sphere of influence. Three types of evaluation contexts from the Brazilian experience will be illustrated and discussed in this paper - a high visibility program within a political context; a low visibility program with an unclear context; and a program context with two simultaneously active components.

Keywords: Meaningful Evaluation. Program Context. Evaluation-Context Relationship.

RESUMO

O ato de um indivíduo avaliado ser uma composição única de atributos e necessidades é facilmente entendido. O menos óbvio é que os programas também têm personalidades complexas por si mesmos, como os indivíduos que os compõem. Eles extraem as suas necessidades de seus próprios contextos, e daí encontram sentido na avaliação que opere nesses contextos singulares. O que é significativo para um programa pode não ser para outro. A avaliação é influenciada pelo contexto do programa. Reciprocamente, ela pode influenciar o contexto também. O desafio é descobrir por completo o contexto com suas necessidades. Assim os resultados serão significativos para o programa em sua esfera de influência. Três tipos de contextos de avaliação da experiência brasileira são ilustrados e discutidos neste trabalho - um programa de grande visibilidade dentro de um contexto político, um programa de baixa visibilidade em um contexto difuso, e um programa cujo contexto contem simultaneamente os dois componentes.

Palavras-chave: Avaliação significativa. Contexto do programa. Relação avaliação contexto

RESUMEN

La evaluación de un individuo se la entiende, comúnmente, como una composición única de atributos y necesidades. Lo menos obvio es que los programas también tienen personalidades complejas por sí propias, así como los individuos que los componen. Ellos sacan sus necesidades de sus propios contextos, elaboran de ese modo el significado para una evaluación que opere en estos contextos únicos. Lo que es significativo para un programa puede no serlo para el otro. El contexto del programa influye en la evaluación. Y viceversa, la evaluación también puede influenciar el contexto. El desafío es descubrir por completo el contexto con sus necesidades, así los resultados serán significativos para el programa en su esfera de influencia. En este trabajo se ilustrarán y discutirán tres tipos de contextos de evaluación de la experiencia brasileña: un programa de gran visibilidad dentro de un contexto político, un programa de poca visibilidad en un contexto inespecífico, y un programa cuyo contexto contenga simultáneamente los dos componentes.

Palabras clave: Evaluación significativa. Contexto del Programa. Relación Evaluación-Contexto

Introduction

Although evaluation as a distinct discipline has celebrated a century of development, its practice in Brazil has only seen three decades of accomplishments. In its efforts to overcome this time lag, Brazilian evaluation practice followed an accelerated rhythm in detriment of results utilization. This has also impaired knowledge acquisition and learning, achieved only over time and through proper maturation. Practice has been faced with the challenges of conducting a true evaluation (STUFFLEBEAM; SHINKFIELD, 1985), while dealing with unqualified professionals, scarce technical literature in Portuguese and a rather adverse institutional environment. Testing and adapting acclaimed foreign methodologies to the Brazilian context have added to the challenge.

The approach has been to create daring solutions through inclusion (MERTENS, 2003) of stakeholders; dialogue (HOUSE, 1993) between them; and appreciation (PRESKILL; CATSAMBAS, 2006) as a means to improve persons and organizations. What lessons have been learned by these different types of relationships between evaluation and its context? How can these lessons be systematized for formulating evaluation policies and strengthening theory? To what extent is the study of context an integral part of an evaluation process (STUFFLEBEAM; MADAUS; KELLAGHAN, 2000)? To what extent is it a pre-requisite for conducting an evaluation? How can the challenges help develop an evaluation culture?

Responses to such questions are crucial for the practicing professional evaluator. This paper attempts to discuss some of these in light of the Brazilian experience through three distinct external evaluation situations: Evaluation One: the initial phase of Education for the Table Program (Educação à Mesa); Evaluation Twoelementary and high school learning in a renowned school in the city of Rio de Janeiro; and Evaluation Three-an in service education program for adolescents and adults of a Brazilian Mining Company.

Evaluation One corresponds to the first of the abovementioned types, i.e: a highly political program context, receiving extensive media coverage. It was a Federal Government education program implemented in partnership with an important private foundation of national influence. With a focus upon eliminating hunger, interventions occurred in very needy localities through education for nutrition and food consumption, in order to promote: changes in eating habits at school and at home; better understanding of basic nutrition concepts; and the practice of buying easily accessible and available food, as well as the means for accessing information.

In Evaluation Two, the purpose was to judge to what extent students attending a renowned high school demonstrated learning acquisition compatible with the national curricular standards Parâmetros Curriculares Nacionais (PCNs). This case represents the second context type, i.e., not completely visible, but which has to be unveiled. Specifically, the school was concerned about its students' performance in the national exams, an institutional evaluation exclusively based upon student performance on cognitive knowledge tests. Thus, the evaluation team considered that the school evaluation context should be expanded subtly, doing this by introducing a discussion forum between evaluators and teachers. Finally, Evaluation Three , the third context type, had two simultaneously relevant contextual situations. One of the largest adolescent and adult education programs in Brazil being conducted, on one hand, by a Brazilian mining giant, among the largest in the world in its market, and on the other, by a private foundation of national relevance. The task at hand for evaluators was to create an evaluation context that would integrate ideas and expectations from each party (STAKE, 2004), in order to reach meaningful outcomes for both of them, as well as to society at large.

Evaluation One: the case of the Brazilian Program for Sound and Healthy Eating: Education for the Table

Education for the Table Program, implemented by Roberto Marinho Foundation, was part of a nationwide hunger elimination (Fome Zero) effort launched by President Lula during his first presidential term. This program sought to promote sound nutritional eating habits through mobilization and intervention in communities, by training individuals to become specialists with audiovisual instructional material, which would then intervene in their communities, to multiply the initial effort. Roberto Marinho Foundation was chosen for its expertise in multimedia and television education. Program's content comprised both access to food and food quality - it sought to provide continuous knowledge about proper eating habits and sound ways to select and consume food items. Its vision, in the long run, was to fight malnutrition, obesity, diabetes, hypertension and other health problems generated or aggravated by inadequate eating habits.

The Program's initial phase, for which this external evaluation was commissioned, was aimed at ten impoverished districts, throughout several Brazilian states, that had just been completed in time for this study, after a six month implementation period. The Program was entering its expansion stage, preparing to gradually reach 200 districts, towards a final goal of 1,000 districts, nationwide.

This initial phase consisted mostly of activities in preparation for project implementation - development of audiovisual instructional material and specialists' training. Program staff invited volunteers from the community to be trained, such as community leaders, health agents, and school principals and lunch supervisors. Experts in health and nutrition trained the individuals in sound and healthy eating habits, as well as in associated socio-psychological concepts and citizenship rights.

The projected intervention in communities and neighborhoods included one-onone and group interaction in homes, schools, health centers and community centers, to encourage, teach, demonstrate and pass on knowledge about sound and healthy eating habits, as well as the importance of acting as role models. This phase was also the startup of community mobilization, by employing two to three individuals in each community to motivate people to attend Thematic Days, local radio shows and other organized group activities.

Outputs for this initial phase included a Food and Nutrition Education activity with instructional videos, a soap opera aired on the local radio (Radionovela), and the training of 100 specialists in each of the districts. Specialists, thus trained, would become agents of change in their communities, in charge of attendance to educational activities, as well as of visiting locations related with health and nutrition, such as health centers, schools, and food courts at shopping malls and residential neighborhoods.

The immediate (short term) outcomes anticipated were to have trained specialists, whose direct intervention in their community would then lead to further change in the school and health system (mid-term outcomes) by means of acceptance, support and cooperation. Community at large outcomes were then expected to occur, starting with nutritional awareness and understanding of basic eating and health concepts, which would then lead to interest in seeking for more information and growth in attendance of project activities. The change in eating and health habits would then become visible within the communities (long term outcomes), in daily eating activities and health related practices. These mid to long term changes would ultimately lead to more visible effects in the community health and welfare in a broader context. By then, the Program anticipates expansion to a full national context.

Evaluation challenges and responses

Evaluation context in this case was in many ways similar, yet different from the other two cases described earlier. Similar challenges called for and reinforced practices found effective earlier.

A major similarity among the three contexts was regarding Brazilian reality, with no significant changes from 1986 to the time of the study, especially regarding the lack of properly trained evaluation personnel, direly needed for Program expansion. No graduate programs exclusively in evaluation existed until then, and those trained abroad were not enough. Learning from our previous experience, and given the current scope, we proposed that evaluation capacity building become part of the external evaluation process. Evaluators involvement should come early on, unlike the other two cases, so that the situation encountered, combined with strong expectation of useful results, would reinforce our proposed concomitant role for evaluation. Empowerment (FETTERMAN, 2002) was the useful method in our capacity building and mentoring role, as in the other two cases. It is a practical and effective way of training program participants, to support them in appropriate decision making, as well as to build evaluator credibility.

New challenges were posed to our evaluation team, but we were happy not to have some of the challenges the other cases had to face. First of all, evaluation was commissioned at the beginning of the program - in fact, during the preparatory "Initial Phase", with minimal community intervention. Unlike the other two cases, we were able to propose continuous data collection, in order to verify program evolution and sequential outcomes. In other words, baseline data could be obtained. Furthermore, introduction of a logic model for proposed operations was made feasible right at the beginning, for use by project management. Our intention was that the logic model would become a planning tool, lending direction both to program operation and evaluation - internal, as well as external.

Second, the hiring party (the Foundation) had better defined specific objectives and directions for the Program, and presented concrete expectations in terms of evaluation concerns and overarching indicators. This gave the needed direction to the evaluation team, yet also the needed flexibility to develop evaluative questions and indicators by negotiating with stakeholders in a responsive approach. After discussing with Program sponsors, and based on project documents, we arrived at five evaluative questions and derived more specific and relevant sets of indicators from sponsor-identified concerns. Four of the questions related to identification of program impact respectively on: a) family eating habits; b) reduction of food waste by target community's members; c) people's sound and healthy eating habits in and out of their homes; and d) prevention of health problems caused or aggravated by unhealthy eating. A fifth evaluative question referred to program merit: the extent to which program activities were consistent with its mission. Table 1 below shows categories and indicators used.

Work started by collecting data for a SWOT analysis, in order to encourage and maintain an appreciative inquiry standpoint (PRESKILL; COGHLAN, 2003) in all participants, and to keep them focused upon identifying and building on strengths while striving to improve weaknesses. Flexibility of naturalistic methods was preserved by combining quantitative (checklists) and qualitative (interviews and focus groups) techniques. However, the large scope of data collection from communities, compared to small staff size, called for better use of quantitative data and more judicious use of qualitative methods.

Key participants and sources of data for evaluation came from schools, families (residents), and community in general. At least 12 interviews (using 5 broad questions) were conducted in each district, with school principals, lunch supervisors, elementary schoolteachers, community leaders, mobilizing agents, trainers, health agents and local politicaladministrative representatives. About 800-1,000 questionnaires/checklists were administered in homes and schools under supervision of a trained researcher, who filled them out in personal contact with the interviewee. Sample was determined over previously mapped areas, considering statistical significance testing requirements. In addition, demographic data was also compiled.

Data was collected at two different times and results were interpreted by means of a SWOT analysis. For this purpose, quantitative data was organized into percentages of answers relative to each impact and merit indicator, after checking against qualitative data (from field interviews) that had been triangulated among team members and validated by Program participants.

Results showed both strengths and weaknesses. There were differences between districts in terms of data in general, which confirmed quantitative gains, but also gave deeper insights into what was happening, desirable as well as undesirable outcomes. Nevertheless, improvement occurred in all five categories, more regarding the first two (interest and comprehension), which reflected the immediate changes that could be anticipated to occur initially.

Evaluation Two: the case of Elementary and High School Learning

With an apparently objective purpose, i.e., demonstration of learning acquisition compatible with national curricular standards, this project had, however, several undisclosed and strongly influential variables, that needed to be cautiously unveiled throughout the evaluation process.

To what extent do school pupils show evidence of learning, coherent with school syllabus, and in turn, reflecting the National Curriculum Standards requirements?

Students from Elementary, Middle and High School were tested for all the following subjects: Portuguese (test and essay), Math, Social Studies, Science, History, Geography, Biology, Physics and Chemistry.

• Specialist teachers were in charge of testing for each of their subject areas;

• Tests were administered by interns from an undergraduate psychology course;

• Several meetings between school teachers and the evaluation team occurred to prepare a propitious environment for evaluation;

• Students were previously informed about the evaluation, and the meaning of the tests that would be administered, and regarding the need for course, teacher and student improvement;

• Dialogue between school teachers and exam authors was promoted and stimulated, for interpreting of results achieved;

• Evaluation addressed the whole school and not each individual student.

Main results

While presenting the answer for the evaluative question, one must point out that the results obtained were just a partial reflection of the school's learning process. Other variables comprising this process, such as attendance, class schedules, teacher educational level, classroom techniques, and school management, as well as several relevant others were never assessed along this evaluative process. Thus, this study has essentially raised hypotheses, since it would be premature, and, certainly, inappropriate to mention its results as positive conclusions or as direct effects from clearly identified causes. Notwithstanding what has been considered, the partial assessment undertaken is especially relevant to, and has allowed multiple, as well as diverse, reflections and analyses by, school officers.

Broadly speaking, students showed a very reasonable test score, especially when considering some relevant issues, such as: not having previous specific preparation for the test, being tested for a broader context than what was offered in the classroom and being surprised with question formats different from the usual. Here, students were quite bold and committed to the process, and teachers, as well, were sufficiently openminded, critical and professional in accepting and cooperating fully with the evaluative process.

Outcomes suggest that proper reflection over collected data, interpretations and expressed hypotheses, followed by preliminary actions to correct weak points or to consolidate strengths, will certainly contribute to raise school excellence. However, this may only be carried out by taking into account the very high commitment level of students and teaching staff, duly stimulated by school Principals' and Headmaster's competent leadership.

Evaluation Three: the case of Adolescent and Adult Education Program in a Brazilian Mining Company: Basic Education for Workers

The Basic Education for Workers program was aimed at adult workers, either employees or service renderers, at Vale - a Brazilian mining giant. Its educational approach focused on promoting the completion of basic education in the workplace to develop broad competencies and promote citizenship values and attitudes. The program had the following goals:

• Raise the number of workers graduating from High School, in areas surrounding company operational sites;

• Enhance educational processes with specific competencies, required by company operational procedures;

• Allow all company employees' access to Elementary and High School education. Also include all service renderers acting directly in company sites.

• Above all, the program focused upon social inclusion and appreciation of workers, by recognizing human capital as the company's most valuable asset, demonstrated in the opening statement of the Program's educational material, reproduced below:

"The Basic Education program is an educational proposal designed for Vale's adult workers, focused towards the work environment, to develop competencies and to stimulate citizenship values. It is mainly focused upon:

• Raise the amount of qualified high school level professionals at regional markets where Vale operates;

• Add to the educational process, competencies required by Vale's operational procedures;

• Allow access to formal education (i.e. Middle and High School level), for employees and service renderers directly operating in Vale's sites.

A program that also is a social inclusion proposal, for it recognizes human capital as the company's most valuable asset. Program activities are proof of this purpose, when their expressed intention is that results overcome material profit and achieve, above all, the personal and professional satisfaction of the workers as well as their full commitment to the company's mission, in a full demonstration of professionalism and citizenship."

Program specific goals were:

• Provide an opportunity to students/Vale workers for the completion of their basic education, in a shorter term, with learning outcomes compatible with the continuation of studies;

• Ensure in-service and continuous distance education programs to committed professionals (teachers, supervisors and program coordinators);

• Carry out pedagogical monitoring of all "teleclasses";

• Support the improvement of learning and assessment processes implemented in all "teleclasses";

• Allow for all students/workers attending the program certification in Middle and High School Level Graduation.

Evaluative questions

What personal changes are observed in the students, as a consequence of the program?

What professional changes are observed in the students, as a consequence of the program?

Indicators

The following indicators, under the corresponding evaluative question, refer either to the desired impact in terms of students' personal change (Evaluative Question 1), or to students' change in professional performance in their area of expertise (Evaluative Question 2).

Main results

In summary, the answers encountered to the evaluative questions reveal that results show evidence of impact of the Basic Education Program on both personal and professional aspects of student attendees.

• the personal aspect showed marked evidence of change in: self-confidence; respect for other; self-esteem; intrapersonal relationship (within family); self-judgement; interest in information; interpersonal relationship; and oral communication.

• the professional aspect, showed proof of change in: motivation for work, persistence in performing tasks; satisfaction at work; work initiative; amount and quality of work produced; reading comprehension; and relationship with supervisor and peers. Although in a lesser way, but still noticeable, impact was also observed over autonomy at task completion, problem solving skills, reading ability, ability at calculating numbers, perception of value by the company and written communication.

Mention must be made that change observed in personal aspects were more important than in the professional aspects. This makes sense, when one understands that the individual is, in general, more sensitive to personal change and, in due time, this influence will show on professional performance.

Several program characteristics were certainly responsible for this impact. The most impressive ones, which speak of program merit, are listed below:

• flexible class schedule;

• teachers' capability, commitment and dedication;

• the NOVO TELECURSO methodology (television education developed by the Foundation), which integrates diversified didactic resources and strategies (telelesson, group dynamics, books, texts, frequent content revision, etc.);

• close relationships between student and teacher;

• course modules containing two subject matters per module;

• student group comprising peers where there was no educational discrimination;

• coordinators' leadership;

• teleclass locations;

• students attending only by choice;

• family support;

• social inclusion, with classes comprising students/workers from both genders and diversified ethnic origins, age levels and socio-economic backgrounds;

• program being initiated and supported by the company.

Conclusion

The three cases described here have, in fact, been chosen as representatives of Brazilian program evaluation culture, to offer the best possible view of the relevance of context towards achieving meaningful evaluations. The chosen cases do not, by any means, exhaust the diversity of contexts and of evaluation approaches pertinent to them.

A point to be made is that employing capable work teams with different backgrounds and the combination of diversified methodologies to make the evaluation process and outcomes more meaningful are the most positive factors towards evaluation success. Brazil's lack of duly trained evaluation professionals is a gap to be filled, a context that has not yet been properly evaluated. Open mindedness from an adequately trained evaluation team is crucial for approaching program participants and for ensuring proper application of methodology. An open minded practice can only be achieved by evaluators with diversified personal and professional backgrounds, under competent coordination, that will avoid losing track of the evaluation's purpose. Learning from ongoing experience, will improve approaches to similar contexts in the future and to promote deeper changes. A dynamic evaluation process that feeds from the context, allows for better understanding of the relevance of context, and, finally learns to employ context to its own benefit.

Recebido em: 26/02/2010

Aceito para publicação em:10/03/2010

• FETTERMANN, D. M. Foundation of empowerment evaluation Thousand Oaks, CA: SAGE Publications, 2002.
• HOUSE, E. R. Professional evaluation: social impact and political consequences. Thousand Oaks, CA: Sage Publications, 1993.
• MERTENS, D. M. The inclusive view of evaluation: visions for the new millennium. In: DONALDSON, S. I.; SCRIVEN, M. (Ed.). Evaluating social programs and problems: visions for the New Millennium. Hillsdale, NJ: Lawrence Elbaum Associate, 2003.
• PRESKILL, H. S.; CATSAMBAS, T. T. Reframing evaluation through appreciative inquiry Thousand Oaks, CA: Sage Publications, 2006.
• PRESKILL, H. S.; COGHLAN, A. Using appreciative inquiry in evaluation. New Directions For Evaluation, San Francisco, n. 100, p. 41-52, 2003.
• STAKE, R. E. Standards-based & responsive evaluation. Thousand Oaks, CA: SagePublications, 2004.
• STUFFLEBEAM, D. L,; MADAUS, G. F.; KELLAGHAN, T. Evaluation models Boston: Kluwer Academic Publishers, 2000.
• STUFFLEBEAM, D. L.; SHINKFIELD, A. J. Systematic evaluation Boston: Kluwer Nujhoff , 1985.

*

Paper Presented at the Evaluation 2009: Conference of the American Evaluation Association: Context and Evaluation, Orlando, Florida, USA, Nov. 2009.

Publication Dates

• Publication in this collection
  28 Mar 2011
• Date of issue
  Dec 2010

History

• Received
  26 Feb 2010
• Accepted
  10 Mar 2010