Teaching and researching quality indicators: a partnership with middle school students^*

Ensinar e pesquisar indicadores de qualidade: uma parceria com estudantes do ensino fundamental

Enseñando e investigando indicadores de calidad: una asociación con estudiantes de la escuela secundaria

Abstract

Discussions about the quality of public education have generated concerns that are at the base of public policies such as SAEB – Sistema de Avaliação da Educação Básica [Elementary and Middle School Evaluation System], meant to work with quality standards defined by educators themselves. Some scholars accuse this system of having merely regulatory purposes. This study broadens the discussion by investigating students’ opinions on Educational Quality. Our aims are: to identify what 8^th graders understand by educational quality; to investigate the prevailing educational quality concepts in school discourses; to discuss how students understand the notion of educational quality. The methodology articulates documental study and field work (questionnaire and focus groups). The collected data were analyzed with two text analysis software: Analyse Lexicale par Contexte d’un Ensemble de Segment de Texte (ALCEST) and Statistical Package for the Social Sciences (SPSS). The study involved 227 students from four different public schools (two city schools and two state schools) and members of the Education Master’s Program of a private university located in the city of São Paulo’s eastern area. The results highlight the importance of: a) participating in school dynamics and b) developing a good relationship between the students and other school members, as well as the need to: a) restore respect and trust between students and educators, b) have school activities that can make people feel valued and c) educate the teachers/researchers to encourage them to listen to their students.

Student; Self-evaluation; Elementary and Middle School; indicator

Resumo

A discussão sobre a qualidade da educação oferecida pelo sistema público tem sido objeto de preocupação traduzida em políticas públicas, como o Sistema de Avaliação da Educação Básica (SAEB), que prevê um padrão de qualidade definido pelos próprios educadores. Tal sistema recebe críticas de autores por seus propósitos meramente reguladores. Este estudo amplia essa discussão, investigando o que pensam estudantes sobre Educação Escolar de Qualidade. Os objetivos são identificar o que os alunos concluintes do Ensino Fundamental entendem por educação de qualidade, investigar quais são os conceitos de qualidade da educação que aparecem com maior frequência nos discursos das escolas e discutir a noção de educação de qualidade na visão dos alunos. A metodologia articula estudos de natureza documental e trabalho de campo – aplicação de questionário e de grupo focal em estudantes. Para o tratamento dos dados, recorreu-se às análises de conteúdo utilizando-se os software Analyse Lexicale par Contexte d’un Ensemble de Segment de Texte (ALCEST) e Statistical Package for the Social Sciences (SPSS). Participaram da pesquisa 227 estudantes de quatro escolas públicas – duas da rede estadual e duas da rede municipal – e o programa de mestrado em educação de uma universidade privada da zona leste da cidade de São Paulo. Os resultados destacam o resgate de respeito e confiança entre alunos e educadores, a realização de atividades em que se sintam valorizados no espaço escolar, a participação da dinâmica da escola e da boa interação entre alunos e demais sujeitos da escola, além da demanda pela formação do educador-pesquisador para a promoção da escuta dos estudantes.

Aluno; Autoavaliação; Ensino Fundamental; indicador

Resumen

Las discusiones sobre la calidad de la educación pública han generado inquietudes que se traducen en políticas públicas como el SAEB - Sistema de Avaliação da Educação Básica (Sistema de Evaluación de Educación Primaria e Intermedia), que pretende trabajar con estándares de calidad definidos por los propios educadores. Algunos estudiosos acusan a este sistema de tener simplemente propósitos reguladores. Este estudio amplía la discusión al investigar las opiniones de los estudiantes sobre la educación de calidad. Nuestros objetivos son: identificar qué entienden los alumnos de octavo grado por una educación de calidad; investigar qué conceptos de educación de calidad prevalecen más en los discursos escolares; para discutir cómo los estudiantes perciben la noción de educación de calidad. Nuestra metodología articula el estudio documental y el trabajo de campo (cuestionarios y grupos focales). Los datos obtenidos se analizaron utilizando dos softwares de análisis de contenido: Analyse Lexicale par Contexte d’un Ensemble de Segment de Texte (ALCEST) y Statistical Package for the Social Sciences (SPSS). El estudio involucró a 227 estudiantes de cuatro escuelas públicas diferentes (dos escuelas de la ciudad y dos escuelas estatales) y miembros del Programa de Maestría en Educación de una universidad privada ubicada en el área este de la ciudad de São Paulo. Los resultados resaltan la importancia de participar en la dinámica escolar y de una buena relación entre los estudiantes y otros miembros de la escuela, así como la necesidad de restaurar el respeto y la confianza entre estudiantes y educadores, para tener actividades en los terrenos de la escuela en las que la gente pueda sentirse valorado y para educar a los profesores / investigadores para promover la escucha de sus alumnos.

Estudiante; Autoevaluación; Escuela primaria y secundaria; Indicador

1 Introduction

This paper presents the results of a research on how public school students from the city of São Paulo define quality indicators for the concept of Educational Quality. It is part of “Institutional self-evaluation: building indicators to enable exchanges with external evaluation mechanisms”, a study conducted between 2011–2013 as a partnership between a public (city) school and the Education Master’s Programme of a university in the city of São Paulo’s eastern region, with funding by Fundação de Amparo à Pesquisa do Estado de São Paulo (Fapesp) [the State of São Paulo’s Research Support Foundation].

Results from the aforementioned research indicated the need to involve students in order to broaden the discussion on building quality indicators for Brazil’s Basic Education Stage – which corresponds roughly to U.S.’s Elementary and Middle School. Therefore, we proposed that the study be continued as a follow-up investigative action: “Institutional Self-evaluation: Elementary and Middle School students’ understanding of educational quality”. We continued to rely on the partnership between the same public schools and Education Master’s Programme, as well as FAPESP’s funding (by means of Processo n. 15/11305-2).

The goals of the new investigation are: identifying what public-school 8^th graders understand by Educational Quality; investigating which educational quality indicators stand out in school discourses; exploring educational quality indicators based on students’ thoughts (taking into account their respective school culture contexts).

2 Participation: absences and presences

The low quality of the education offered to Brazilian children and teenagers, especially by public school systems, has consistently worried many sectors of our society. Public policies and educational programs are two examples of governmental actions to address that problem. Sistema de Avaliação da Educação Básica (SAEB) [Elementary and Middle School Evaluation System] was created at the early 90’s, when defining quality standards for Brazilian education and strategies to reach those standards became a priority for government administrators and education professionals.

SAEB¹ is a large-scale multidimensional evaluation organized by Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira (INEP) [Anísio Teixeira National Institute for Education Studies and Research] to analyze Brazilian Basic Education and to identify factors that interfere with student performance.

SAEB uses a set of questionnaires to gather information about different aspects of school life and the socioeconomic level, the social and cultural capital of students and educators. The questionnaires reach different school segments: they are answered by students, school principals and Portuguese and Mathematics teachers.

SAEB aims to gather information on management as well as the socioeconomic and cultural profiles of teachers and school principals. For their part, students take different questionnaires: 5^th graders answer 44 questions; 8^th graders answer 47 questions and 12^th graders answer 119 questions. The questions fall into two main categories: socioeconomic profiling and relationships with their teachers and fellow students.

Despite providing researchers with context information about the students and their schools, these questionnaires are not without fault; according to Minayo (2009, p. 53), they should close in on “the cultural universe of the evaluated actors, so as to express the voices, feelings, thoughts and practices that make up the researched universe”. She points out that those cultural universe indicators may be considered internally valid when all research participants have a similar understanding of the conceptual categories employed by data-gathering instruments. That can be accomplished by a process that ensures theoretic and contextual grounding, which does not happen with the current instruments.

The benefits of external evaluations are a controversial topic among Education scholars. While Horta Neto (2010) believes they can be an important instrument to evaluate educational policies, Lapa and Neiva (1996), Domingues and Oliveira (2012) and Freitas (2012) draw attention to underlying aspects of external evaluations, especially those concerning syllabuses and the introduction of market logic in Education. Other authors, such as Stecher (2002), mention the “neutral effects” these evaluations have on schools and classrooms.

Although Freitas (2012) admits that regulating what is taught and learned in schools is important and may even be considered a governmental duty, he sees external evaluations as part of an effort to privatize public school systems led by “corporate reformers” that want to force business management standards on education:

[...] we call them corporate reformers of the US education system, a term coined by American researcher Diane Ravitch (2011b). The terminology refers to a coalition between politicians, the media, businessmen, educational businesses and institutes, as well as private foundations and researchers that support the idea that private initiative is organized in such a way as to offer a better answer to “fix” American education than what is suggested by professional educators (Freitas, 2012, p. 380).

Despite their many critics, external evaluation results have been taken as quality standards and are used to design educational policies and government programs. These results often offer diagnostics that do not match educators’ perceptions. Thurler (1998) justifies this shortcoming of external evaluation systems by claiming that they are unable to “capture a reality that is constantly shifting”. Along the same lines, Sousa (1997) and Morosini et al. (2016) argue that we must regard school organization critically and establish “[…] a systematic practice of evaluating the many subjects and components of that organization, such as the performances of the teachers and other staff” (SOUSA, 1997, p. 126).

Such systematic evaluation is commonly referred to as Institutional Self-evaluation and it differs from external evaluation. According to Brazil’s 1996 Law on Educational Directives and Foundations and to the National Education Council’s Resolution n. 4/2010 (BRASIL, 2010) self-evaluation is to be a mobilizing and linking element of the political-pedagogic project. In a context of democratic management, self-evaluation implies building relationships that favor coexistence and collegiate decision-making as means of lessening the authoritarian aspects of school relationships.

Although self-evaluation or internal evaluation has gained traction on educational debates (especially regarding the definition of indicators to broaden the notion of educational quality), it does not currently include listening to the students, once more neglecting what that group has to say on different aspects of the school as an educational institution.

Therefore, it becomes increasingly important to establish a complementarity between internal and external evaluations. According to Bolívar (2006), internal evaluations can be a starting point for external evaluations, providing specific information about the evaluated institutions. In turn, the external approach adds credibility, bolsters self-evaluation by providing analysis from different points of view and prevents the loss of perspective that might result from an exclusively internal focus. Bolívar highlights the importance of making sure that one evaluation model doesn’t eclipse the other; both evaluations must work in a mutually beneficial tandem if we are to achieve excellence in education.

Leite’s research (2012) also supports the coexistence of internal and external evaluation models, demonstrating that the tension generated by two competing evaluation paradigms is ultimately positive. She believes that the resulting conflict contributes to ensure the legitimate standing of groups struggling to develop different perspectives on social and political experience.

This study takes into account the underlying assumptions, principles and notions that ground and organize the current Institutional Self-evaluation structures, according to Law 10.861/04 (BRASIL, 2004) that institutes Sistema Nacional de Avaliação da Educação Superior (Sinaes) [National Higher Education Evaluation System]. The aforementioned law regulates self-evaluation procedures in Higher Education Institutions, thus providing opportunities to establish collaborative participation in planning, administrative and pedagogic management.

Concerning the methodological aspects of the evaluation process, Technical Assessment n. 65-CGACGIES/DAES/INEP/MEC (BRASI, 2014) subsidizes the discussion and definition of the axes and dimensions that structure the many aspects of data collection and analysis, always considering the specificities of each researched school’s context.

Dias Sobrinho (2005), Gatti (2007), Freitas (2009) and Sordi and Souza (2009) lay the foundations of Institutional Self-evaluation and reshape that notion to articulate a) democratic management, b) participation by different subjects and c) “agreed-upon quality” (Bondiolli, 2004). Bondiolli’s concept posits that to discuss the definition of quality we cannot disregard the thoughts and wishes of the many subjects involved in Education. This approach is then extrapolated and becomes the search for a consensus on aspects that permeate the construction of the educational quality concept in schools. Such an approach proposes to listen to and to consider adequately and efficiently the opinions of all involved, especially those historically excluded from the role of subjects of the educational process, even inside schools: the students.

3 Paths travelled

Since the planning stages of this research, we agreed to collect data by means of a) literature review and b) field work (questionnaires and focus groups with 8^th graders) to explore the indicators associated with the concept of educational quality.

The (still ongoing) research takes place at four public schools in the city of São Paulo: two of them located in low-class, outskirt neighborhoods and two in more central areas – one in Vila Carrão (eastern region) and another in Jabaquara (southern region). Those schools were chosen because teachers and management staff were interested in participating in the study.

The research team is composed of public school teachers and management staff (two teachers per school and the principal, when s/he was available), as well as teachers and grad students from the partner university.

Although the researched schools are located in different areas and belong to different school systems (city and state systems), they share certain characteristics: most students of all four schools are socially vulnerable and all schools have projects to support students who are struggling with learning and behavior difficulties. Those similarities characterize public schools as a conflictual environment, where students’ frustration and lack of motivation lead to aggressive behavior, indiscipline and violence.

4 Instruments and procedures

As for the instruments and procedures, this study heeds Garcia’s (1981) cautions about how reality is always more complex than can be detected by any investigative process. Still in accordance with the same author, we had no intention of using methods that provide reality snapshots, capturing domesticated situations and docile characters. Instead, we were interested in discovering processes that would encourage students to reflect upon the concept of educational quality.

Despite their considerable differences, the questionnaire and the focus group worked complementarily towards that goal. The questionnaire was prepared to gather information using different categories, as synthetized in Table 1.

Table 1
Questionnaire details.

Category	Investigative question	Variables	Questionnaire
Profile	What is the demographic profile of research subjects?	Age	Age: ____ years
		Gender	( ) Female ( ) Male
		School	School name
		Race	You consider yourself: White; Black; Mixed race; Asian – Eastern heritage; Native – indigenous heritage.
		School type (public/private)	What type(s) of school have you studied at? I’ve always studied at public schools; Mostly at public schools; Mostly at private schools.
		City	Where did you study? I’ve always studied in São Paulo; I’ve mostly studied in São Paulo; I’ve mostly studied in cities other than São Paulo.
		Connection to current school	Have you always studied at your current school? ( ) Yes ( ) No If you used to go to another school, do you know why you changed schools?
Preschool	Can Preschool have different quality indicators from Elementary and Middle School?	Preschool experience	Did you attend Preschool? ( ) Yes ( ) No
		Memory •School grounds •Teachers •Peers •Management staff (principal/coordinators) •Staff •Meals •Activities	If you attended Preschool, what do you remember about: School grounds; Teachers; Peers; Management (principal and coordinators); Staff; Meals; Activities.
Educational quality: concept	How do research subjects understand and experience the concept of educational quality?	Educational quality concept	These days, there is much talk about the school education offered to children, teenagers and adults. People often say that school education must be of good quality. What is educational quality to you? Considering what you wrote on educational quality, does your school qualify? Justify your answer.
		Regarding the quality of your school, rate the following aspects from 1 to 5 (Lickert scale).
		Indicators	Terrible	Bad	Regular	Good	Excellent
		Classrooms	(1)	(2)	(3)	(4)	(5)
		School yard	(1)	(2)	(3)	(4)	(5)
		Bathrooms	(1)	(2)	(3)	(4)	(5)
		What is learned	(1)	(2)	(3)	(4)	(5)
		Teacher performance	(1)	(2)	(3)	(4)	(5)
		Relationships between teachers and students	(1)	(2)	(3)	(4)	(5)
		Relationship between the principal and the students	(1)	(2)	(3)	(4)	(5)
		Relationship between coordinators and students	(1)	(2)	(3)	(4)	(5)
		Meal quality	(1)	(2)	(3)	(4)	(5)
		Staff dedication	(1)	(2)	(3)	(4)	(5)
		Relationships between fellow students	(1)	(2)	(3)	(4)	(5)
		School relationship with the parents	(1)	(2)	(3)	(4)	(5)
		Participation in your school’s students’ union	(1)	(2)	(3)	(4)	(5)
		Participation in the school council	(1)	(2)	(3)	(4)	(5)
Comments		Other comments

• Source: Prepared by the authors, 2016.

The questionnaire’s goals were: a) to broaden our understanding of the educational quality concept; b) to provide a better grasp of students’ needs and c) to explore students’ opinion of their daily school experience: what changes would they like to see? What difficulties did they face?

We will not be discussing the questionnaire in depth. Its methodological purpose was to help structure the other data collection instrument: questionnaire answers were used to prepare the script for focus group discussions, and this paper focuses on analyzing data gathered by the latter.

After discussing Interviews and Dialogue Circles as possible data collection instruments, the group of researches decided on Focus Groups. Once that was decided, the group was instructed to do research and to form a study group to debate theoretical and methodological aspects of focus groups. This instrument seemed to be a good fit for our research goals, especially because focus groups enable interaction and mediation between the researched subjects and researchers of an investigative and formative process. We followed the focus groups methodological directives proposed by Morgan (1988) and Wholey et al. (1994).

The students were split into groups of five and a total of seven focus group meetings were recorded². The research team also took comprehensive notes during meetings, documenting what was said and nonverbal cues (facial expressions denoting approval/disapproval, displays of joy and anger etc.). After each meeting, the people responsible for it would take the time to exchange thoughts and impressions, thus enhancing the usefulness of the collected data.

Table 2 synthetizes the discussion script used with the focus groups.

Table 2
Focus groups discussion script.

Category	Investigative question	Focus group discussion script
Educational quality: concept What changes would they like to see? What difficulties do they face?	How do the subjects understand and experience the concept of Educational quality? Listening to the students as a political action.	In your opinion, does your school have the means to listen to what students have to say?
	How do they interpret manifestations organized by their fellow students?	What are your thoughts on the following image:
	What do they think about students’ role in school decision-making spheres and mechanisms?	Do you believe that it is important for students to contribute to discussions and decision-making in their schools?
	What are their thoughts on school-family ties?	Do you believe that family participation is important for schools?
	What are school dynamics like?	What are organization and maintenance like at your school?
	Material conditions: what should be changed or kept as is?	In your opinion, are the material resources available sufficient?
	Regarding the students’ relationship with the classroom: what are their thoughts on that place?	What does the following image tell you?
		Do you believe that there are attitudes that can make it easier to learn?
	Regarding relationships between the various school subjects: what are their thoughts?	How do you see the relationship between people at your school?
	Regarding school projects: what are their thoughts?	What are your thoughts on the school’s projects?

• Source: Prepared by the authors, 2017.

The majority of focus group participants were 9^th grade students chosen among those who had answered the questionnaire the year before as 8^th graders³.

Analysis involved data organization and careful study of students’ answers. In accordance with the principle of methodological diversity, the main themes in the textual corpus (questionnaire answers and focus groups notes) were determined and explored using two different procedures: a) content analysis, adapted from Bardin (1977) and b) software data processing with Analyse Lexicale par Contexte d’un Ensemble de Segments de Texte (ALCEST). ALCEST is considered exploratory; it performs statistic text analysis (word frequency and succession) and was used for a wide-ranging exploration of the students’ manifestations. The subjects’ profile data were also software-generated by Statistical Package for the Social Sciences (SPSS).

Qualitative research guidelines were observed in the analysis of students’ participation and behavior during focus group meetings. In an adaptation of Bardin’s (1977) content analysis method, there were three analytical stages: 1) pre-analysis; 2) analytical description and 3) interpretation, inference or discussion of results. Pre-analysis consisted of free-floating reading procedures: an intuitive reading, open to ideas, reflections, hypothesis etc. – something akin to an individual brainstorming session. For the analytical description stage, we chose to work with thematic content analysis, a type of content analysis that seeks to discern the nodes in a communicational exchange. For the final stage (referential interpretation) we organized the interpretative analytical categories in order to determine which indicators the respondents associated with the concept of educational quality.

It is important to clarify that due to ALCEST’s technical specificities (the software works with vocabulary distribution-based laws) it was impossible to group the answers by school, since the Initial Context Units (ICU)⁴ require a minimum word-count. The initial corpus segmentation by participating school generated Elementary Context Units (ECU)⁵ below the ideal 60% consideration. Therefore, data analysis of the written answers considered the information total of each group/school, thus characterized: Subj1=School1(shift1-morn); Subj2=School1(shift2-aftn); Subj3=School2(shift1-morn); Subj4=School2(shift2-aftn); Subj5=School3(shift1-morn); Subj6=School3(shift2-aftn); Subj7=School4(shift1-morn).

5 Analysis: what data reveal

Our research respondents are 227 public school (city and state) 8^th graders, 48% female and 51% male, 87% of which attended Preschool. The subjects were mostly aged 13-14 years (73%), with a percentage of 15-year-olds (14%). Regarding race, 34% view themselves as white, 22% as black and 38% as mixed race. Nearly 70% of the respondents have studied mostly at public schools.

Data obtained from the seven focus groups were compiled and processed according to analytic guidelines. The results allowed the research team to analyze meanings implicit in the discussion script and the importance granted by students to each investigated category, posing challenges about the different indicators associated with educational quality.

We have been able to verify the existence of different positions and discourses in the four investigated groupings. This was possible because each class proposed by ALCEST has a discourse of its own that reveals the particular interests of each grouping.

As per software requirements, we analyzed the textual production from all four investigated schools. Focus groups production makes up 65% of the ECUs and is therefore a homogeneous enough corpus to ground analysis and conclusions. It should be noted that ECUs do not correspond to the number of research subjects, since ECUs are extracts from the written narratives as explained in footnote n. 5.

Figure 1 displays information provided by ALCEST. The dendrogram-shaped image displays the connections between classes and their representativeness (%) in the evaluated corpus. Despite being articulated, classes indicate differences between their contents, suggesting the presence of four different meaning clusters in students’ thoughts on Educational Quality. We have explored and organized each class by decreasing word frequency of its vocabulary, as well as the significance rate of each word’s connection to its class (chi-squared⁶).

Figure 1
Dendrogram of the evaluated corpus.

Based on a Decreasing Hierarchical Classification, a general analysis of the corpus by ALCEST has identified four classes of meanings that organize the contents of respondents’ answers: 1) professional competence: to go beyond copying; 2) coexistence: the school as a psycho-social environment; 3) dialogue: trust in the partnership between students and educators; and 4) respect: to consider and to include everyone.

In order to contextualize the ECUs, we present below written narrative excerpts to illustrate each class.

Class 1 – Professional Competence: to go beyond copying

Class 1 has 121 ECU – 21% of all ECUs comprised in the Decreasing Hierarchical Classification set. The most significant words in the cluster are:

• “text” – text (30); texts (2);

• “draw” – drawing [verb]⁷ (3); to draw (3); drawing [noun] (23);

• “copy” – copy (6); copying (8); to copy (16); copies (1);

• “blackboard” – blackboard (23);

• “assign” – assigns (26); assigned (2); assign [present tense, 3^rd person plural] (4); assigning (8); to assign (10); should assign [subjunctive mood] (1);

• “book” – book (18); books (1).

This is a city School (1), morning shift student group. Conversation excerpts:

Unité textuelle n. 181 Khi2 = 29 Individual n. 1 *school_1 *city_2 *morn_1

She’ll show us a cubist drawing, and will assign us one too. Then she’s going to assign a text on cubism, is gonna have us copy it! Then she’ll change to another subject or assign more drawing, just one more drawing.

Unité textuelle n. 217 Khi2 = 21 Individual n. 1 *school_1 *city_2 *morn_1

This teacher, then, one left and another one started, and, like, I noticed. I usually get A+, A. My report card got worse, like, my grades on that subject got worse, I don’t even know anymore, because I don’t do anything anymore, I don’t even notice it, I no longer feel like doing what she assigns to us, because it’s not convenient to me.

Class 2 – Coexistence: the school as a psycho-social environment

Class 2 comprises 46 ECU – 8% of the set’s total. The most significant words in the cluster are: “use”, “noise”, “multi-purpose”, “yard”, “precari”, “presentat”, “supply”, “computing” and “laboratory”. This is a city School (2), afternoon shift student group. Conversation excerpts:

Unité textuelle n. 346 Khi2 = 37 Individual n. 2 *City school_2 *aftn:

So, presentation organization here [is] very precarious, because [there’s] a lot of noise, you just can’t do it. So we needed, I don’t know, to have somewhere quiet;

Unité textuelle n. 358 Khi2 = 26 Individual n. 2 *City school_2 *aftn:

Oh, that desk is dirty, that desk is broken, I don’t like it. And the same person [who uses it] makes a point of messing it up. School facilities… at least here at [school name] we have a lot of research spaces, like the video room, we can do different activities and I think that’s very important because [it’s] a diversified way of learning.

Class 3 – Dialogue: trust in the partnership between students and educators

Class 3 comprises 339 ECU – 59% of the set’s total. The most significant words in the cluster are: “parent”, “talk”, “student” and “meeting”. This is a state School (5 and 6), morning shift student group. Conversation excerpts:

Unité textuelle n. 627 Khi2 = 11 Individual n. 5 *State school_5 *morn:

Sometimes, that gets in the way. And this? I think it happens all the time, but how can you choose your words? On both sides, students and teachers, because they can be wrong too, it’s not always the student who’s wrong, they gotta listen to the student and the teacher too.

Unité textuelle n. 814 Khi2 = 11 Individual n. 6 *State school_6: *aftn:

There’s the good and the bad. So you think that if parents and students were at the PTA it would be better, more democratic? Yes. Then the student could defend himself if he didn’t do it, try to convince his parents.

Class 4 – respect: to consider and to include everyone

Class two comprises 67 ECU – 12% of the set’s total. The most significant words in the cluster are: “clean”, “lady”, “bathroom”, “meeting”, “dirt”, “consider”, “door”, “staff”, “kitchen”, “interfer”, “coordinator” and “respect”. This is a city School (3 and 4) and state School (7), morning shift student group. Conversation excerpts:

Unité textuelle n. 500 Khi2 = 14 Individual n. 3 *City school *aftn:

He means everyone, teachers, principals, everyone. No exceptions. But there’s one teacher that supports us, [that] listens, he himself says that we should be listened to and that our voice too is important at the school.

Unité textuelle n. 858 Khi2 = 28 Individual n. 6 *State school_7 *aftn_2:

Definitely! I think there should be cameras at the school to avoid that. In the morning shift, the students help clean the school. And if there was a day for the afternoon [shift] to clean desks, would the students take good care [of them]? Many wouldn’t do it!

6 What the students say

According to Figari (1996, p. 110), the concept of indicator originally “refers to an element that displays another, that tells you where to find that other element, that points to it”. In other words, an indicator works as a sign or index; when it comes to knowledge, it is derived from experience. Citing Barbier (1985), Figari adds that indicators can be considered representative of the investigated reality; that they group information into categories of axiological interest and become indirectly significant of a given set of ends and values, thus acting in the vicinity of the notion of criteria. In that sense, the indicators associated with educational quality allow us to measure availability, access and use of goods, services and knowledge. They also enable us to gauge the intensity of processes and the meaning of possible changes in the researched schools.

As this study is part of a research on institutional self-evaluation with a formative perspective, its goal is to explore educational quality indicators based on students’ opinions (properly situated in the school culture context).

The theoretical-methodological tradition most suitable to meet that challenge is the psycho-social approach, which favors a broadened understanding of a) the knowledge systems shared by the social subjects that coexist at school; b) the symbolic processes of the involved groups and c) the relationships they establish with the social object that is school education.

Students’ opinions on Educational Quality have elements that either express a local, normative discourse, often incompatible with daily experience, or restrict themselves and reinforce beliefs that are so deep-seated as to no longer be questioned.

Our analysis of the corpus generated by the focus group meetings indicates that the set of indicators associated with Educational Quality in students’ discourse suggests a prevailing perception of their condition as users/receptors of formal knowledge, pointing at a highly consolidated representation: a general profile of students that see themselves more as consumers than as protagonists of the work done at schools.

Furthermore, the same students highlight the need to invest in their autonomy by building better relationships with other school members, which would translate as mindful observation, availability, support and control over their formative process. Students expect teachers to encourage their participation.

By comparing results of the four researched schools we have found a stark contrast between expectation and reality of students-school staff relationships: on one hand, students are concerned and denounce the absence of dialogue and trust on the educators’ part. On the other hand, they mention a lack of respect between the different school segments, thus revealing an understanding of co-responsibility for the everyday school experience. This acknowledgement of mutual accountability suggests the possibility to understand educational quality as something more complex and layered than simply “holding teachers responsible for everything that is wrong in education” (BAUER et al., 2017, p. 963).

Throughout our research corpus, students place an equally high value on teachers – the word “teacher” (533) ranks second in frequency, right after “people/us”⁸ (576). That is evidence of the negative impacts of certain school dynamics pertaining to the classroom (for example respondents don’t see copying as favorable for their formative process) and dynamics defined by the management staff for other school activities (break hours and usage of shared spaces, such as the multimedia room).

7 Final thoughts

Our results attest to the possibility of associating technical legitimacy and an evaluative and ethical participative process, so as to broaden and better understand which quality indicators emerge from Basic Education psycho-social processes.

This study takes a formative approach and acknowledges that each educational institution has its own unique principles and culture. Given the difficulties posed by our research subject – giving voice to students means dealing with student participation as a social concept, culturally constructed and perpetually changing –, it was necessary to understand how the subjects see themselves as protagonists and also how others see them, considering the social and subjective dimensions of the meanings and significances activated in both subjective and intersubjective social interaction spaces, which carry the seeds for transformative action.

In order to achieve that, this investigative study has also sought to unveil meanings attributed to being a Basic Education teacher. Finally, we invested on in-service teacher education to perfect the processes that guide the choices and expectations of these professionals when faced with contemporary demands, which in turn also require different indicators to appraise Educational Quality.

In consequence, we state the need for further studies on the formation of researchers that work as public school (city and state) Basic Education teachers in the city of São Paulo. Such a formative process requires the participation of Basic Education teachers in formulating and executing studies that reveal and question underlying values, working with the thoughts and feelings of public school students as empirical data.

To that end, methodological studies and data analysis contribute complementarily to teacher formation. Therefore, one of the results of this research is to show that ties must be formed between Basic Education teachers and their students, thus contributing to the production of knowledge on Basic Education.

References

• BARBIER, J. M. L’évaluation en formation Paris: PUF, 1985.
• BARDIN, L. Análise de conteúdo Lisboa: Edições 70, 1977.
• BAUER, A.; CASSETARI, N.; OLIVEIRA, R. P. Políticas docentes e qualidade da educação: uma revisão de literatura e indicações de política. Ensaio: Avaliação e Políticas Públicas em Educação, v. 25, n. 97, p. 943-70, out./dez. 2017.
• BRASIL. Lei Nº 10.861, de 14 de abril de 2004. Institui o Sistema Nacional de Avaliação da Educação Superior – SINAES e dá outras providências. Diário Oficial da União 15 abr. 2004.
• BRASIL. Ministério da Educação. Conselho Nacional de Educação. Resolução Nº 4, de 13 de julho de 2010. Define diretrizes curriculares nacionais gerais para a educação básica. Disponível em: <http://portal.mec.gov.br/dmdocuments/rceb004_10.pdf Acesso em: 15 fev. 2017.
  » http://portal.mec.gov.br/dmdocuments/rceb004_10.pdf
• BRASIL. Ministério da Educação. Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira – INEP. Nota técnica INEP/DAES/CONAES Nº65: Roteiro para Relatório de Autoavaliação Institucional de 9 de outubro de 2014. Disponível em:<http://wp.ufpel.edu.br/cpa/files/2016/09/nota_tecnica_n65_roteiro_relatorio_de_autoavaliacao_institucional.pdf Acesso em: 12 out. 2017.
  » http://wp.ufpel.edu.br/cpa/files/2016/09/nota_tecnica_n65_roteiro_relatorio_de_autoavaliacao_institucional.pdf
• BRASIL. Ministério da Educação MEC. Portaria Nº 482, de 7 de junho de 2013 que dispõe sobre o Sistema de Avaliação da Educação Básica – SAEB. Disponível em: <http://www.adur-rj.org.br/4poli/gruposadur/gtpe/portaria_482_7_6_13.htm> Acesso em: 10 ago. 2017.
  » http://www.adur-rj.org.br/4poli/gruposadur/gtpe/portaria_482_7_6_13.htm
• BOLIVAR, A. Um olhar actual sobre a mudança educativa: onde situar os esforços de melhoria? In: LEITE, C.; LOPES, A. (orgs.). Educação, currículo e formação de identidades. Porto: Edições ASA, 2006.
• BONDIOLLI, A. O projeto pedagógico da creche e a sua avaliação: a qualidade negociada. Campinas: Autores Associados, 2004.
• DIAS SOBRINHO, J. Dilemas da educação superior no mundo globalizado: Sociedade do conhecimento ou economia do conhecimento? São Paulo: Casa do Psicólogo, 2005.
• DOMINGUES, M. R. S.; OLIVEIRA, N. C. M. A avaliação externa na educação básica e suas implicações. In: ENCONTRO INTERNACIONAL DE EDUCAÇÃO COMPARADA, 5., Belém, 2012. Anais... Cidade: Editora, 2012.
• FIGARI, G. Avaliar: que referencial? Porto: Porto, 1996.
• FREITAS, L. C. Avaliação educacional: caminhando pela contramão. Petrópolis: Vozes, 2009.
• FREITAS, L. C. Construir outro sistema avaliativo para São Paulo. Revista APASE, v. 1, p. 28-29, 2012.
• GARCIA, R. M. A base de uma administração autodeterminada: o diagnóstico emancipador. Educação e Avaliação, n. 2, p. 43-67, jan. 1981.
• GATTI, B. A. Avaliação e qualidade da educação. São Paulo: Fundação Carlos Chagas, 2007.
• HORTA NETO, J. L. Avaliação externa de escolas e sistemas: questões presentes no debate sobre o tema. Revista Brasileira de Estudos Pedagógicos, v. 91, n. 227, p. 84-104, jan./abr. 2010.
• LAPA, J. S.; NEIVA, C. C. Avaliação em educação: comentários sobre desempenho e qualidade. Ensaio: Avaliação e Políticas Públicas em Educação, v. 4, n. 12, p. 149-58, jul./set. 1996.
• LEITE, D. Desafios para inovação pedagógica na universidade do século 21. Revista FAEEBA, v. 21, p. 29-39, 2012.
• MINAYO, M. C. S. Construção de indicadores qualitativos para avaliação de mudanças. Revista Basileira Educação Médica, v. 33, suppl. 1, p. 83-91, 2009. https://doi.org/10.1590/S0100-55022009000500009
  » https://doi.org/10.1590/S0100-55022009000500009
• MORGAN, D. L. Focus group as qualitative research. Newbury Park: Sage, 1988. (Qualitative Research Methods Series 16)
• MOROSINI, M. C. et al. A qualidade da educação superior e o complexo exercício de propor índicadores. Revista Brasileira de Educação, v. 21, n. 64, p. 13-37, jan./mar. 2016. https://doi.org/10.1590/S1413-24782016216402
  » https://doi.org/10.1590/S1413-24782016216402
• SORDI, M. R. L.; SOUZA, E. (orgs.). A avaliação institucional como instância mediadora da escola pública: a Rede Municipal de Educação de Campinas como espaço de aprendizagem. Campinas: Millenium, 2009.
• SOUSA, S. M. Z. L. Avaliação escolar e democratização: o direito de errar. In: AQUINO, J. G. (org.). Erro e fracasso na escola: alternativas teóricas e práticas. São Paulo: Summus, 1997. p. 125-38.
• STECHER, B. M. Consequences of large-scale, high-stakes testing on school and classroom practice. In: HAMILTON, L; STECHER, B. M.; KLEIN, S. P. (ed.). Making sense of test-based accountability en education. Santa Monica, CA: Rand, 2002. P. 79-100.
• THURLER, M. G. A eficácia das escolas não se mede: ela se constrói, negocia-se, pratica-se e se vive. Ideias, n. 30, p. 175-192, 1998.
• WHOLEY, J. S.; HATRY, H. P.; NEWCOMER, K. E. Handbook of practical program evaluation. San Francisco: Jossey Bass, 1994.

• 1
  SAEB was first applied in 1990 to a sample of public schools in urban areas with 1, 3, 5 and 7 grade students. In 2005, SAEB was restructured by the Department of Education (by means of Portaria Ministerial n. 931, March 21, 2005 to comprise two evaluations: Avaliação Nacional da Educação Básica (ANEB) [National Basic Education Evaluation] and Avaliação Nacional do Rendimento Escolar (ANRESC) [National School Performance Evaluation], also known as Brazil Exam. In the 2013 edition, Avaliação Nacional da Alfabetização (ANA) [National Literacy Evaluation], and Pacto Nacional pela Alfabetização na Idade Certa (PNAIC) [National Pact for Age-appropriate Literacy] were also incorporated by SAEB by means of Portaria n. 482, June 7 (BRASIL, 2013).
• 2
  Part-time schools are the Brazilian norm. Therefore, in three of the researched schools there were two focus groups, one for each school’s morning students and another for the afternoon students; in the remaining school, we talked only to the morning students.
• 3
  During our research, schools were implementing a change in Brazil’s educational system that increased the number of Basic Education years from eight to nine. One of the researched schools did not have 8 grade students in 2016, so we had their 7 graders answer the questionnaire and the ones chosen to participate in the focus group meetings the following year were then 8 graders.
• 4
  Initial Context Units (ICU) are defined according to the researcher and to the nature of his/her textual data. As this study works with data gathered via questionnaire, each answered item is considered an ICU. The ICU set makes up our corpus, which is processed by the software.
• 5
  Elementary Context Units (ECU) are small segments of the respondents’ answers to a specific item, usually about 3 lines long, sized by the program in relation to the total size of the corpus, respecting punctuation and the segments’ location in the analyzed text.
• 6
  Chi-squared values correspond to the quantitative relation between expected phenomenon distribution and the research’s findings (analysis of focus groups records).
• 7
  Since different terms in Portuguese may be translated to English homographs, we have provided some contextual information between brackets.
• 8
  The word in question is “gente”, which translates to “people” but is most often used in the colloquial expression “a gente”, which means “us”.

• *
  This study is part of “Institutional Self-evaluation: Elementary and Middle students’ understanding of quality education”, a research funded by Fapesp (Processo n° 2015/11305-2).

Publication Dates

• Publication in this collection
  Oct-Dec 2018

History

• Received
  29 Nov 2017
• Accepted
  30 July 2018