Abstracts
The text analyzes the data obtained from a study into the use of Quality Indicators in Education - Indique -, a participative self-evaluation tool for schools. The study aimed to look for information on how to use the material, its limits and its results, and consisted in interviews and focus groups with technicians from Departments of Education, principals, teachers, employees, students and family members. The analysis, conducted according with the quality standards of evaluation processes and with the contemporary challenges to evaluation proposed by specialists, indicated that the use of Indique meets the usefulness criterion and can maintain an acceptable level of accuracy, thus managing to involve stakeholders in change processes. Evaluation and the actions triggered by it occur within the tension fields that are related to evaluation models, to the sharing of knowledge by teachers (especially knowledge of an educational nature) and to a reluctance to participate, among others. Indique is considered simple and feasible by most respondents. However, this feasibility has its limitations: it comes up against the dependence of schools on managing bodies and even of Departments of Education on the external agencies that encourage its use
quality of education; educational policies; educational evaluation; school
O texto analisa os dados obtidos por estudo sobre o uso dos Indicadores da Qualidade na Educação - Indique -, instrumento participativo de autoavaliação voltado para escolas. O estudo teve como objetivo buscar informações sobre as formas de utilização do material, seus limites e resultados, e consistiu na realização de entrevistas e grupos focais com técnicos de Secretarias de Educação, diretores, professores, funcionários, alunos e familiares. A análise, realizada à luz dos padrões de qualidade de processos avaliativos e dos desafios contemporâneos para a avaliação propostos por especialistas, mostrou que o uso do Indique cumpre o critério de utilidade e pode manter um nível aceitável de precisão, conseguindo assim envolver stakeholders em processos de mudanças. A avaliação e as ações por ela desencadeadas ocorrem dentro de campos de tensão relacionados a modelos de avaliação, compartilhamento de saberes dominados pelos professores (sobretudo os de caráter pedagógico) e resistências à participação, dentre outros. O Indique é considerado simples e factível pela maior parte dos entrevistados. Contudo, tal factibilidade tem limites: esbarra na dependência das escolas em relação aos órgãos dirigentes e mesmo das Secretarias de Educação em relação a instâncias externas que estimulem seu uso
qualidade do ensino; políticas educacionais; avaliação da educação; escola
OTHER ISSUES
An interpretation of the uses of quality indicators in education
Vanda Mendes RibeiroI; Joana Borges Buarque de GusmãoII
IPh.D. student in Education at the School of Education, University of São Paulo, holding a scholarship awarded by the Committee for Post-Graduate Courses in Higher Education (CAPES - Coordenadoria de Aperfeiçoamento do Pessoal de Nível Superior) vandaribeiro@usp.br
IIM.A. student in Sociology at the School of Education, University of São Paulo, holding a scholarship awarded by the Committee for Post-Graduate Courses in Higher Education (CAPES) joanabg@usp.br
ABSTRACTY
The text analyzes the data obtained from a study into the use of Quality Indicators in Education - Indique -, a participative self-evaluation tool for schools. The study aimed to look for information on how to use the material, its limits and its results, and consisted in interviews and focus groups with technicians from Departments of Education, principals, teachers, employees, students and family members. The analysis, conducted according with the quality standards of evaluation processes and with the contemporary challenges to evaluation proposed by specialists, indicated that the use of Indique meets the usefulness criterion and can maintain an acceptable level of accuracy, thus managing to involve stakeholders in change processes. Evaluation and the actions triggered by it occur within the tension fields that are related to evaluation models, to the sharing of knowledge by teachers (especially knowledge of an educational nature) and to a reluctance to participate, among others. Indique is considered simple and feasible by most respondents. However, this feasibility has its limitations: it comes up against the dependence of schools on managing bodies and even of Departments of Education on the external agencies that encourage its use.
Keywords: quality of education - educational policies - educational evaluation - school
This article aims to present and analyze information from an exploratory study
Below, we present the design and working methodology of Indique. We go into detail about the methodology of the study, describing the data and data-analysis. We then comment on several fields of tension in assessment processes as identified by Indique. Finally, we reach some conclusions.
INDIQUE: A PROPOSAL IN SELF-ASSESSMENT FOR SCHOOLS
Indique is the result of a project coordinated by Ação Educativa, the United Nations Children's Fund (UNICEF), the United Nations Development Program (UNDP), Brazil's National Institute for Educational Studies and Research (INEP), and the Ministry of Education (MEC). Published in 2004, the material is a participatory methodological proposal for a system of indicators through which the community assesses the situation of a range of aspects of a school, identifies priorities, sets up an action plan and executes and monitors activities intended to enhance quality in education. The instrument was based on Brazilian educational legislation and upon discussion and negotiation among participants in a large technical group made up of reference institutions in the field of education, including UNDIME (the National Union for Municipal Educational Leaders), CONSED (the National Council of Education Secretaries), the National Campaign for the Right to Education, and other specialists.
Indique puts forward indicators grouped into a set of dimensions, understood as "fundamental elements that the school must take into consideration in reflecting upon its quality" (Ação Educativa et al., 2007, p.5). The proposed dimensions are: the educational environment, pedagogical practice and evaluation, learning to read and write, democratic school management, the training of and working conditions for professionals in the school, the school's physical setting and students' access to the school and their stay within it. The indicators, seen as signs of quality, are assessed by the school community through the collective discussion of questions relating to situations, actions and practices that may or may not occur in the school. The assessment given to the questions is expressed through ascribing colors: green - a fully consolidated situation, attitude or practice; yellow - a less well consolidated situation, attitude or practice; red - a non-existent situation, attitude or practice. The school community is understood in its broadest meaning, including mothers and fathers, teachers, directors, students, school staff, managers, local NGO representatives, representatives from councils concerned with education or children's rights, and other school-related institutions.
After the assessment - carried out in heterogeneous groups containing representatives from all the segments of the school community - everyone meets in a plenary session to present and discuss each group's results. They then establish priorities and draft an action plan. The instrument also suggests how to organize the monitoring of the execution of the action plan. The assessment and planning process is completed in one day's work.
The creators initially expected the instrument to be self-applicable, which is why they drafted it in simple, easily-understood language. The material's basic premise is that participatory assessment and planning enable the school to enhance quality according to criteria and priorities established by the school community on the basis of indicators. It is intended to be an instrument enabling the school to reflect, propose and act in the pursuit of "quality in education", translated by the mosaic of dimensions and indicators
THE EXPLORATORY STUDY: METHODOLOGY
The first step of the study was the systematization of information on the dissemination of Indique, available at Ação Educativa, which showed that the material was used by programs of the Ministry of Education, Education Secretariats, NGOs and corporate institutions in partnership with schools and education networks. The focus of the study was defined as information-gathering about the use of the instrument by the Education Secretariats, both at state and at municipal levels.
The next stage was to construct a set of assessment indicators divided into four groups: dissemination, evaluation, planning and results attained (understood as changes). Descriptors were created for each indicator. The indicators and descriptors below provided guidance for the design of the research instruments: scripts for interviews and focus groups with education managers and members of the school communities, the sources of the study.
Table 1 - Click to enlarge
Interviews were carried out by telephone and e-mail from November 2007 to March 2008. We contacted fourteen Education Secretariats which had, according to information available, used Indique. It was usually extremely difficult to find the right people in each agency, and several telephone calls were necessary. In the end, interviews were held with the representatives of nine secretariats, three State-level secretariats (Amazonas, Bahia and Rio de Janeiro) and six municipal-level secretariats (Guarulhos and Suzano in the State of São Paulo, Ibitiara and São Félix in the State of Bahia, Ituiutaba in the State of Minas Gerais, Londrina in the State of Paraná), as well as two schools in Londrina. Based on these interviews, we selected three places to visit that had more consolidated experience in the use of the instrument: Bahia, Ituiutaba and Suzano. The selection criteria were: continued use of Indique in the education network; participation in actions to foster the use of the material/autonomous use; municipal/state networks; and other motives for use.
The visits took place in February and March 2008, in which three interviews were conducted, five focus groups were carried out, and one debate was held: three interviews in Salvador, three focus groups in Suzano, two focus groups and one debate in Ituiutaba, in which 83 Indique users took part. Of this total, fifty people took part in the debate in Ituiutaba, the audience being made up of three secretariat specialists, one educational consultant for the municipality and 46 representatives from the school management teams, made up of directors, vice-directors and supervisors
This article will take into consideration the information concerning the assessment and planning processes and the results attributed to the use of Indique, according to the perceptions of the users who are taking part. It should be stressed that the analysis is based on these perceptions, since the information has not been verified.
PRESENTATION AND ANALYSIS OF INFORMATION ON THE USE OF INDIQUE
Simplicity
Fernandes (2007) includes simplicity as one of the challenges for the field of evaluation. He sees this attribute as one of the factors on which stakeholder engagement depends. Most of the secretariat specialists, principals and teachers assessed the Indique methodology as simple, easy to understand and easy to apply. They also said that the several segments of the community were able to understand the methodology, the indicators, and the questions.
Some interviewees and focus group participants, however, said the material was long and tiring. Opinions on the simplicity of the instrument are not unanimous.
It's very heavy, there are a lot of questions, it's complex and repetitive. Simplify the language. The material is well-made. We used it. We took it seriously. Difficult to explain to the parents. Extensive material - we realized that. The introduction is clear. (Teacher, Suzano)
Difficulty in understanding questions and indicators, especially on the part of the students' relatives, was reported in interviews and focus groups. A teacher from the town of Suzano states that some parents were reluctant to assess more pedagogical questions: "our main difficulty in the questions was with the parents, they said it was the teachers who should know".
Some accounts showed there was a need to clarify the contents of the questions so that discussion and evaluation could take place. They also showed that this clarification was usually done by the teachers, the segment with most knowledge of daily life in the schools. This issue will be discussed below.
Some teachers and directors reported complexity because the teachers need to explain certain content to the relatives. Specialists from Suzano, however, state that this perception is based on the need built into the material to share knowledge (and power) with family members: "The use of Indique highlights the fact that teachers hold power because they have more knowledge of pedagogical procedures; and leads to the sharing of this power with parents and staff" (Municipal Education Secretariat specialist, Suzano).
Accuracy/precision
Accuracy/precision relates to the reliability of assessment results. Vianna states that "precision standards demand that information gathered be technically suitable, and that conclusions arrived at be logically related to the data" (2000, p.120-121). Assessments carried out through Indique are not exempt from bias. In Londrina and Ituiutaba, teachers stated their role was to clarify things for the parents. Secretariat specialists and teachers point out the delicate nature of this practice: on the one hand, it brings teachers and parents together, allowing the latter to better understand the educational processes in the school and promoting negotiation; on the other hand it may lead to inducing answers in the assessment. When attempting to "clarify" certain topics, many teachers try to make clear what they themselves think about the situation being evaluated.
Fernandes thinks that self-assessment is more prone to bias. However, its major benefit is greater authenticity and depth in the data.
It is true that internal assessment, when carried out by people who are directly involved in the program, may describe more authentically and more profoundly what is actually happening. However, it is also true that assessment in these conditions may run the risk of being too partial and biased. (s/d, p. 6)
It should be considered that by bringing together the several segments of the school community to address the same issues and to negotiate, Indique's application methodology may work against excessive bias in the evaluation. Through negotiation the material meets the demand for the triangulation of opinions. Fernandes sees triangulation as one of the criteria leading to accuracy in evaluation processes: "Participation in evaluation by at least the major players in a given project assures a diversity of points of view as to the project's merit and value, enabling a more exact view of the real situations one intends to assess" (Fernandes, s/d, p. 6).
A specialist from Suzano also had a positive view of teachers' explaining things to parents. These explanation, in his view, demonstrate Indique's capacity to impinge upon the teachers' need to share knowledge (and power) with family members. "The use of Indique highlights the fact that teachers hold power because they have more knowledge of pedagogical procedures; and leads to the sharing of this power with parents and staff".
It is important to point out that the explanation of questions by teachers was not consistent throughout all schools. There were also cases where, in the words of one teacher, "the teachers did not have the patience to explain very much. And if you don't clarify things, you answer for the other person". In other words, not clarifying the content of the material may also, in the opinion of a Londrina school director, lead to influencing the respondent.
Some interviewees mentioned Indique's capacity to make conflicts explicit, which reaffirms the project's choice to triangulate opinions through negotiation, an assessment technique that Fernandes says increases the accuracy of the evaluation. One supervisor from Ituiutaba stated that with the use of Indique "conflict appeared in the plenary. Certain problems became apparent, problems that we normally ignore". In Suzano, specialists spoke of the material's ability to "bring conflict to the surface".
Also related to accuracy/precision is the issue of the material's relevance. If the system is suitable, more accurate assessments can be reached. Most of the interviewees, especially the secretariat specialists, considered the set of indicators and their respective questions to be relevant and suitable to assessing quality in education.
Participants in the interviews and focus groups also spoke of the instrument's capacity to reveal the school's problems, which enhances the quality and accuracy of the evaluation.
It has laid everything bare; in the past we used to hide the school's problems. [...] When it started [use of Indique] [the pedagogical consultant] said we would be 'probing wounds'. It is painful work, bringing to light stuff we had swept under the carpet. The secretariat ran a great risk, because the people involved might not have been able to cope with whatever appeared. (Supervisor, Ituiutaba)
Usefulness
The "usefulness" standard has to do with how far the assessments meet the needs of those involved and thus engage them in improving the life of the institutions. Some aspects of usefulness are described below: engagement of the school community, review/drafting the political pedagogical plan, relations between secretariats and schools and improvement of school infrastructure.
Engagement of the school community in planning the changes
The study showed that the stage of drawing up plans was not common to all schools. The main reason, as far as could be seen, was that the several secretariats varied in their policy of using Indique, some of them prioritized planning as a working stage while others focused on the work of assessments. In Ituiutaba, all schools in the network drew up their own plans. In Londrina, most of the schools drew up their plans collectively as a result of the assessment that was carried out. In Ibitiara, this happened with the Reading and Learning to Read and Write dimensions. In Suzano, action plans were not made, although actions following the Indique evaluation were reported in the focus groups held in the schools and in the Education Secretariat.
In Ituiutaba and Londrina, reference was made to the fact that meetings were held between managers for several schools to exchange experiences about the evaluation and planning processes. "Based on the Indique results sent to the secretariat by the schools, a survey of problems was completed and this material was discussed with directors, both those of schools taking part and those from schools that had not applied Indique" (Londrina Municipal Education Secretariat specialist). The meeting in Ituiutaba led to a compilation of the evaluations and plans by the institutions, which was the basis for definition of priorities, strategies and general guidelines issued by the Education Secretariat for the following years.
Enhancement of family participation
Some reports by participants from the interviews and focus groups show that Indique can improve family-member participation in the school. In Suzano, teachers said that the instrument develops "the family's ability to understand pedagogical issues" and "[that] over the school year, family members improved the way in which they followed things in this area". This perception is shared by teachers and directors in Ituiutaba and Londrina: "Previously, what they did was bring their kids to school, leave them there, and that was that. We've seen a change in the parents' attitudes (Supervisor, Ituiutaba). "Indique encouraged the parents and they have more trust in the school and confidence in the chances of the school improving" (Director, Londrina). This change in family participation impacted teachers' attitudes, as the following statements show.
As a teacher, I am now more concerned to explain and talk with parents about how I teach. I like to justify myself to the parents. It's an exercise. They quickly begin to take an interest. It strengthens the partnership with the parents. Parents are following their children's learning process. They can give them better support at home. (Teacher, Suzano)
Parents still do not participate as much as they should, but Indique has enhanced their interest in school matters, not only about within-school issues, but also about how education works at the local and national levels. School meets the needs of parents and children better now, and there have been improvements in discipline and mutual respect. (Director, Suzano)
Changes in the way the staff works
Taking part in the processes triggered by Indique has also brought changes in the way the staff work. After a focus group in Suzano, one of the school cooks came up to the researchers to say that thanks to Indique she now better understood her own role (responsible for the school meals) as part of the education work carried out by the school. "I have learned that my work here in the canteen is also part of the children's education". In Ituiutaba, a supervisor mentioned "non-teaching staff talking about pedagogical issues and making good suggestions".
Strengthening school boards
Interviewees and focus group participants referred to Indique's contribution to school boards. In Ituiutaba, a Secretariat specialist reported how they held lectures and workshops with MEC specialists in order to strengthen and structure the collegiate bodies. Another important result within the municipality was the opening of the Casa dos Conselhos, or Board House (a space for meetings of the Municipal Education Board, FUNDEF - the Fund for the Maintenance and Development of Primary Education and Enhancement of the Teaching Profession, the School Transportation body, the School Meal body, and School Boards, which aims to promote more autonomy, liaison and interaction between the several collegiate bodies). In Suzano, a school director stated that after applying Indique, "the school board worked better. It enhanced participants' interest and the number of those enrolled for election to the board increased". In Bahia, a director said that she began to consult the collegiate to apply financial resources, define priorities, address disciplinary issues among students, and misconduct by teachers.
Student participation
Little mention was made of the possible influence of Indique on student participation. The only reference was in Ituiutaba, where a school director included the strengthening of the student association among results achieved by the material. Also in Ituiutaba, management teams mentioned taking actions to improve discipline, and draft an ethics code, which may lead to better relations among the actors within the school setting, including students.
Relations with the local community
When the processes and results of Indique were presented in Ituiutaba it could be seen that the material has served as an instrument to bring the school closer to the local community, since such figures as the neighborhood association presidents, police officers, employees of the local health clinic and corporate representatives took part in the assessment in some schools. A supervisor stated that using Indique had meant "huge growth; it brought the community into the school, [...] a moment for integrating the school, the community, and the neighborhood association, addressing education as something that involves everyone". Strengthening democratic management
Participants in the interviews and focus groups also reported impacts on school management, making it more democratic. A director from Salvador stated that "having used Indique, I understand that a single situation or practice can be seen in a range of ways by people from different segments of the school community". She also says that using the material meant that "identification of the problems was no longer the work of the management team only" and that "she now increasingly shared responsibility for managing the school". A specialist from the Education Secretariat of Bahia confirmed the director's opinion, saying there had been an impact on the management of the schools.
Collective identification of priorities was cited as not only guiding the management of schools but of the network as a whole. According to a consultant who provided service to the City Hall of Ituiutaba about using Indique, the town education secretary sees the material as an "investment guide", insofar as it enables priorities to be identified by the school communities themselves and directs investment of resources in the field of education. In Suzano, a town education secretariat employee stated that "needs become visible and urgent. We can see how the secretariat's priorities are not the same as the community's".
A director in Londrina stated that community engagement lasted for as long as there was an outside team present (Ação Educativa) following the process (in the first year of use): "the mobilization didn't last and petered out because of problems in following up the process". This shows the need to further investigate the continuity of using Indique and what impact this has on schools. According to what the specialist in Bahia says, for example, use was not continued in many schools "because of the appointment of new school managers, replacing the old ones". It would be interesting to study whether these disruptions prevent the consolidation of changes.
Changes in pedagogical practice
The commitment of teachers and other players in the school environment to enhancing pedagogical practice was also mentioned. In Ituiutaba, members of the management team mentioned that they had started having weekly meetings again, as one result of the instrument. Directors in Londrina stated that Indique "enhanced the process of on-going training for staff and teachers". In Ibitiara, one of the changes detected by the specialist interviewed after using the Teaching and Learning of Reading and Writing dimension was "the strengthening of the teacher/student bond in learning to read and write". Directors and teachers in Suzano also mentioned the reconstruction of the meaning of evaluation that using Indique had caused, going from a record based on scores and on criticism, to an assessment based on dialog. From this point of view, Indique was helping establish a more developmental logic in evaluation.
Review/formulation of the political-pedagogical project
One important result of the study is how Indique enables a review or formulation of the political-pedagogical project, as statements from Bahia and Suzano show.
The PPP, or political-pedagogical project was also restructured. The community answered the questions contained in Indique and these answers were used in reformulating the PPP. We put up syntheses in the form of murals. We created a participatory commission to systematize the results and then reconstruct the PPP. (Director, Salvador)
We began including a lot of what appeared in the assessments in our planning and in the PPP. Once again, this year, we want to hear the community as a whole, not just the representatives. And change the PPP, since it has been changing. (Director, Suzano)
In Ibitiara, the conclusion was that the solution to the problems identified in the Indique evaluation had to be thought of within the political-pedagogical project, since Indique had been used prior to formulating the project.
Relations between secretariats and schools
The information gathered in the study allows us to state that Indique can bring about changes in relations between the Education Secretariat and schools. There were statements made in Ituiutaba and Londrina about rapport between these bodies.
When we started to introduce Indique the managers and the schools were worried about "censorship" and expressing the problems, it was "fear of the red"! But as the study progressed we were able to show that items in red would not be censored, but rather they would be taken and worked on. This new relationship allowed a more open dialog and gave the schools greater freedom to say what they want and how they want it. (Ituiutaba Municipal Education Secretariat specialist)
Yes, the relationship became closer because the Secretariat was there for us, and the consultants helped us apply Indique. This closeness still exists. (Director, Londrina)
In Ituiutaba, members of school management teams reported that "the secretariat has changed its interaction with the schools. The secretariat has been more open to us, given us more support and interacted more with us"; "before this we were afraid to ask anything of the secretariat"; and "the secretariat has managed to get to know the schools better". In Ibitiara, the Municipal Education Secretariat specialist stated that the agency's relationship with the schools had changed: "a closer relationship, with transmission of information and a more relevant dialog about what each school has and what it needs, helping define the network's priorities". The specialist also stated that "there has been a change in the way the secretariat looks at schools, going so far as to say what the schools can do for themselves and what they depend on the agency for". A director of an education establishment in Suzano said that "they listen to schools more, and there's more respect for our autonomy". Only the director of a school in Londrina said that there had been no kind of change.
One director said that "Indique gave us more independence". According to her, "the Education Secretariat was unable to meet the demands that came after the use of Indique. Managers have a huge role when working with the community. Are the secretariats up to it?" A point of view that is different from most of the statements but that shows that the relations between the Education Secretariat and the schools may be subject to new types of conflicts after the Indique assessment process.
The demarcation of roles and responsibilities
Reports showed that Indique helped better demarcate what is the responsibility of the Education Secretariats and what is the responsibility of the schools. The specialist from Ituiutaba stated that "before this, they thought that the secretariat had to provide them with everything. But what about all the things that the schools can do for themselves?" A school director from the same town said that "the main thing about Indique was that teachers spoke and the secretariat listened. But they [the teachers] also listened, and realized that there was a lot that they could influence too. And this led to greater commitment to the management". Specialists from Suzano said that "schools used to come to us with all their problems, but afterwards they began to understand that everybody can do their own share".
The Indique self-assessment corroborates Simons' view that self-assessment tends to reinforce professionalism (in the sense of encouraging responsible professionals). For this author, self-assessment is "the best way to enhance the quality of education provided to the children. [...] It is a suitable relationship between responsibility and professionalism. Self-assessment enables one to question to whom and for what reason one is responsible" (1993,
p. 163).
Enhancements of schools' infrastructure
According to interviewees and to the focus groups, Indique has enabled improvements to be made to schools' infrastructure. In Londrina, the secretariat specialist mentioned some results: "a roof over the sports courts and the adoption of the Palavras Andantes reading project, which led to more books being taken out of the library and the collection". The director of a school in Suzano spoke of the construction of an access ramp for disabled people, of a teachers' room and a playground as the results of using the material.
In Ituiutaba they mentioned building an infants' school, and of bringing computers into the schools, and of improvements to the physical and material structure overall. One director said that the teaching materials available "improved a lot", and another spoke of setting up reading corners in the classrooms.
Feasibility
According to Vianna,
Feasibility standards [weight the evaluation] in material and cost-benefit terms. Assessment demands a defined, operational, practical and parsimonious plan with regard to financial resources. In other words, the feasibility standard means being realistic, prudent, politically viable and parsimonious, as Stufflebeam and Madaus put it. (2000, p.120-121)
Ribeiro and Pimenta analyzed the potential of the methodological proposal of Indique to meet the quality standards of assessment and the challenges put forward by specialists for the field of evaluation, without reference to information about its application. They state, with regard to the feasibility of the methodology, that
We may consider that the proposal is potentially feasible: it enables assessment of a primary school or an infant education institution, with participation by the community, in a relatively short timeframe. The fact that the material needed to conduct this self-assessment (photocopies, colored pencils, paper, pens) is used on a daily business by infant schools and education institutions, and is in any case not expensive, bears out this potential. (2009, p. 13)
Indique user reports from this study showed that the issue of the feasibility of the material is more complex. Although it was found that there is no financial or material difficulty in using the material, there were some queries as to the political feasibility and the time needed. Among reasons listed by Education Secretariats for not continuing the use of Indique are: changes in the team and in the secretariat's focus; difficulty in setting aside time in the schedule and in the dynamic of the school itself for introducing Indique; the Secretariat team is too small to do the job; the appointment of new school managers replacing the old ones; lack of outside pressure and follow-up; not sufficiently disseminated to promote its use; lack of feedback and guidance for the task; and the school/secretariat bureaucracy is too time-consuming.
Finally, it is worth saying that the creators of the instrument expected to achieve a self-applicable material focused on schools and that the schools themselves would be interested and thus apply it independently. In practice this expectation was not met. Quite the contrary: schools were heavily dependent upon the Secretariats and the latter were sometimes dependent upon other agencies to promote the use of Indique. For example, in São Félix (Bahia) the municipal education manager stated that the secretariat did not follow up the introduction of Indique because of "lack of external demand".
However, in places where Indique showed better results there was intense support from the Education Secretariat, although in some cases this support was later withdrawn for political reasons. The feasibility of Indique may, therefore, be limited by discontinuities in the administration, and by a lack of follow-up, incentive, feedback and support for schools and Education Secretariats. In this case it is a limit to the initial expectations of the creators of the material with regard to self-application and therefore to feasibility.
TENSIONS IN THE SELF-ASSESSMENT PROCESSES
The interviews and focus groups led to the identification of a set of tensions underlying the use of Indique. The first has to do with assessment models and pathways. Reports by specialists from the Education Secretariats of Amazonas, Ituiutaba, Londrina and Suzano show that a discussion of several focuses of evaluation is permeating the decisions taken by those responsible for the educational policies in these places.
Indique is a good instrument for assessment, diagnosis and follow-up. [...] The assessment enabled by Indique has the dimensions of the school built-in, which allows a broad view of education, contrasting with closed formats that have cold questions. (Municipal Education Secretariat specialist, Ituiutaba)
Before Indique, schools only worried about data, not the school process or quality in education. (State Education Secretariat specialist, Amazonas)
We are more used to the type of assessment that is ready-made, focusing on the student. When we opted for indicators, the idea was to assess all the segments of the comunity, and see what problems could be solved in the school environment and which were solvable in the Secretariat, in any case to allow the community to evaluate the school. (State Education Secretariat specialist, Suzano)
The perception is that there are several pathways: assessment focused on the students' performance, based on large-scale testing and/or assessment turning its gaze toward the teaching and learning process. The introductory text of the publication that accompanied the launch of Indique in Suzano helps to show how the debate is seen in that network:
Assessment has taken on a leading role in Brazilian education. An analysis of recent educational policies shows a definite option to set up a National Assessment System at the expense of the National Education System-that is to say, evaluations of the several levels of education play the role of guiding municipal and state educational policies, setting up "rankings" to justify even the allocation of funding to these institutions. Within this context, and linked to the experience many of us have had in our own school careers, assessment was associated with the idea of punishment, bringing in a competitive, coercive and centralizing logic in the assessment processes. A logic that renders void the primary meaning of evaluation, which is to analyze a given moment and discern how close to or how far from the object or process in question it is. And this is precisely the meaning we wish to recover. (Suzano, 2006, pág. 5)
As Monica Thurler puts it, a current debate
...opposes the advocates of a drive to have external calibration of the effectiveness of the pedagogical plan and those who wish to gear maximum efforts to introduce on a local level [in school establishments] the indispensable competencies of self-assessment and self-regulation. This is not a new debate: it has been going on for decades between advocates of an orientation focused on effects, and those who defend the priority of the process. It bears witness to the very strong-and persistent- belief of those who still believe that quality is assured on the one hand by regular verification of the performance of students and, on the other, by external control of students' competency. (2002, p. 64)
In Brazil, institutional assessment of basic education and the organization of systems of evaluation of students' performance by means of large-scale testing are both recent, with the latter perspective rather than the former broadening its scope (Sousa, 1999; Oliveira, 2006). Although Indique was not designed to counteract external evaluation, this is how some of its users have identified the instrument. Large-scale testing has been seen as a "top-down model" that "blames the school" and "punishes and oversees", and Indique is seen as something that may be built up within the school and that recovers the importance of assessing the process. "It is a type of assessment to begin a dialog with people. While SARESP and Prova Brasil try to come up with an evaluation of the quality of education based on students' answers, Indique takes the opposite direction. Thinking through the problems and considering how they can be solved" (Municipal Education Secretariat specialist, Suzano).
A second point of tension that was found was that using Indique demands a certain level of knowledge sharing by teachers ("power sharing" in the opinion of Suzano specialists) about so-called "pedagogical" issues. This issue is linked to some teachers' reluctance to allow family members to participate more, and to their complaints that they have to spend time explaining pedagogical processes to family members, which is exhausting (this situation was explained in a focus group in Suzano as being tedious). For Simons, one of the positive aspects of self-assessment is that it lays the foundations for "the public knowledge of the educational problematic" (1993, p. 163). The study points out that the scope of this "public knowledge" is a process pervaded by conflicts, where the role of the teacher and the way in which the teacher understands the relationship between the school and the community are called into debate. It can also be observed that while the teacher's need to explain to family members leads to sharing of knowledge/power, it also confers another power on the teachers: the possibility of imposing their own opinions during discussions, which may interfere in the accuracy of the evaluation.
The third point has to do with teachers' resistance to evaluation, reported by specialists from Suzano and Bahia. The resistance is related to the concepts of models of assessment in force in society, to the difficulty of feeling evaluated, to unfamiliarity with the sharing of schools' internal issues with family members, and to the issue of prejudice against the participation of the community.
According to the specialists, resistance was gradually broken down by the perception that the aim of Indique hinged upon self-assessment and on the proposal of solutions to the problems identified, as well as to the information that there would be no punishment or retaliations.
Another theory explaining the resistance is that Indique might cause the school problems. In Ituiutaba, one of the participants reported that during the evaluation she felt afraid about "whether I could touch on some points". Some sectors therefore, predicting that the assessment will explain quality aspects of their own work, may be reluctant to trigger the use of Indique.
It was the Suzano education secretariat that most clearly verbalized the existence of resistance to the use of Indique. One reason for this may be the fact that specialists from this municipality are highly self-reflective and critical. Another reason may be the way in which the material was introduced into this network. There was no process of signing-up; all the schools were called on to participate. Nor were activities carried out in order to train school teams in using the material. We feel the stage of introducing the Indique proposal to schools is extremely important, since this is when people can be mobilized, and where schools are attracted to participate. These points of tension show the importance of building the meaning of the evaluation with the schools. According to Thurler,
...in order for the teachers to voluntarily take part in the survey and data analysis, and for them to accept the results of the assessment and change their practice as a result of it, they must absolutely be convinced that what they do will not be turned against them. (2002, p. 76)
The Education Secretariats must actively construct the meaning of self-assessment by Indique in schools, highlighting the goal of mobilizing the school community into involvement in enhancing quality. The study showed that there is a clear relationship between the possibility of constructing meanings, the systematic use of the material, follow-up by secretariats and the perception by the school community of the results achieved.
CONCLUSION
The present article analyzed the results of a study of the use made of Quality Indicators by Education Secretariats, benchmarking against standards of quality in evaluation processes and current challenges to evaluation put forward by specialists. The analyses carried out enable us to state that, from the users' perspective, Indique leads to assessments with degrees of accuracy that can generate changes (greater engagement and responsibility on the part of the school community in the processes, creation/review of the political-pedagogical project, improvements to infrastructure, enhancements to management of schools and of the network). However, these assessments come up against constraints, such as the leading questions that may be posed by those who hold greater power in the daily life of schools.
It is clear also that the community's participation in the school, above all with regard to evaluation, is a process that needs to be constructed and that involves a change in the school culture. Using Indique is one step: the school community cannot be expected to be able to evaluate the school - this capacity must be constructed. Nonetheless, it can be stated that the material manages to involve stakeholders in the evaluation process, leading to negotiations and activities that could have a positive impact on the enhancement of institutional life and people's lives. Most of the interviewees and participants in focus groups stated that the material is simple to use, although some thought the indicator system was long, tiring and that some questions were hard to understand, especially for family members. People thought it was highly practical (owing to the proposed methodology), although its feasibility is limited above all by the fact that schools and secretariats depend on other agencies to promote its use.
Indique users' perceptions suggest important results: the posing of problems; the discovery of an educational meaning in evaluation; sharing of power/knowledge, opinions and values; making conflict explicit; negotiation; strengthening the mechanism of dialog, communication and management; improving teacher/student relations; consistently setting aside a time for reflection upon pedagogical practice; strengthening processes identified by specialists as important factors for promoting the school's independence, such as participation in creating or reviewing the political-pedagogical plan (Azanha, undated) and boosting the family's participation in the school and on the school boards.
Most of the interviewees acknowledged that Indique is legitimate to assess the quality of education, above all policy managers. This underscores the instrument's potential usefulness
When used throughout an education network, Indique may lead to an exchange of knowledge between schools; may bring secretariats and institutions closer together in a trusting relationship; and may strengthen responsibility for education among schools and secretariats, clearly identifying the roles each one must play. These results, however, occur in contexts pervaded by tensions: tensions inherent to the evaluation universe (questioning the purpose, resistance, fear, upset); tensions from the sphere of political relations within schools (acceptance of and resistance toward the capacity of different segments of the school community to take part in assessment, centralization/sharing of knowledge/power); and tensions inherent to the relationship between schools and Education Secretariat (sharing/centralization of power, disputes between educational models, and accountability).
BIBLIOGRAPHICAL REFERENCES
- AÇÃO EDUCATIVA. Projeto indicadores da qualidade na educação: balanço de resultados. (The quality indicators in education project-taking stock of results). São Paulo: Ação Educativa, 2008. Mimeo
- AÇÃO EDUCATIVA et al. (Coord.). Indicadores da qualidade na educação (Quality indicators in education). 3. ed. São Paulo: Ação Educativa, 2007.
- AZANHA, J. M. Proposta pedagógica e autonomia da escola (The pedagogical proposal and the autonomy of the school). [s/d]. Available at: <http://www.smec.salvador.ba.gov.br/site/documentos/espaco-virtual/espaco-escola/coord-pedagogico/artigos/proposta%20pedagogica%20e%20autonomia%20da%20escola.pdf>. Viewed on 8 April 2009.
- FERNANDES, D. Avaliação de programas e projetos. (Assessment of programs and projects) Lisbon, [s/d]. Mimeo. [The text of a class taught as guest lecturer, FEUSP, May 2009]
- ______. Limitações e potencialidades da avaliação educacional (Constraints and potentials of educational assessment) Lisbon: 2007. mimeo. [The text of a class taught as guest lecturer, FEUSP, May 2009]
- FITZPATRICK, J.; CHRISTIE, C.; MARK, M. Evaluation in action: interviews with expert evaluators. Thousand Oaks: Sage, 2009.
- OLIVEIRA, R. P. Estado e política educacional no Brasil: desafios do século XXI. (The Brazilian State and educational policy: twenty-first century challenges). 2006. Thesis (Livre Docência) - FEUSP, São Paulo.
- RIBEIRO, V. M.; PIMENTA, C. O. Análise de uma proposta de avaliação institucional para a escola. (The analysis of a proposal for the institutional evaluation of schools). In: ENCONTRO ANPAE, 2009, São Paulo. Text presented.. São Paulo: ANPAE, 2009.
- RIBEIRO, V. M.; RIBEIRO, V. M; GUSMÃO, J. B. Indicadores da qualidade para a mobilização da escola. (Quality indicators for mobilizing schools). Cadernos de Pesquisa, São Paulo, v. 35, n. 124, p. 227-251, January/April 2005.
- SIMONS, H. Avaliação e reforma das escolas. (Assessment and reform of schools). In: ESTRELA, A.; NÓVOA, A. (Org.). Avaliações em educação: novas perspectivas. (Assessments in education: new perspectives). Porto: Porto, 1993. p. 155-170.
- SOUSA, S. Z. L. Avaliação institucional: elementos para discussão. (Institutional evaluation: points for discussion). In: SEMINÁRIO O ENSINO MUNICIPAL E A EDUCAÇÃO BRASILEIRA (SEMINAR: MUNICIPAL EDUCATION AND EDUCATION IN BRAZIL) 2000, São Paulo. Annals... São Paulo: Secretaria Municipal de Educação de São Paulo, 1999. p. 83-91
- SUZANO. Prefeitura Municipal. Avaliação institucional dos indicadores da qualidade na educação. (Institutional assessment of education quality indicators) Suzano: Xamã, 2006.
- THURLER, M. Da avaliação dos professores à avaliação dos estabelecimentos escolares. (From assessing teachers to evaluating school establishments). In: PERRENOUD, P.; THURLER, M. (Org.). As Competências para ensinar no século XXI: a formação dos professores e o desafio da avaliação. (Competencies for teaching in the 21st Century: teacher training and the challenge of assessment). Porto Alegre: Artmed, 2002. p.61-87.
- VIANNA, H. Avaliação educacional: teoria, planejamento, modelos. (Educational assessment: theory, planning, models). São Paulo: Ibrasa, 2000.
Publication Dates
-
Publication in this collection
25 Apr 2011 -
Date of issue
Dec 2010
History
-
Accepted
Feb 2010 -
Received
Oct 2009
