System for the Evaluation of Environmental Education Projects focused on Water Resources Management (SAPEA-Água)

Sistema de evaluación para proyectos de educación ambiental para gestión de recursos hídricos (SAPEA-Água)

Abstract

Given the increased risks of water crises faced by several countries, environmental education emerges as an instrument to help finding solutions to water-related issues, along with local communities and different sectors of society. However, studies have indicated the methodological vulnerability of environmental education programs/projects focused on water resources management. Thus, the current study developed the System for the Evaluation of Environmental Education Programs and Projects focused on Water Resources Management (SAPEA-Água). It adopted a qualitative-quantitative approach, based on the combination of bibliographic research and content analysis of documents substantiating environmental education, in order to evaluate the contextualization, interdisciplinarity, participation, sustainability, communication and self-assessment level of environmental education programs/projects. SAPEA-Água can substantiate the evaluation process, as well as increase the number, quality and effectiveness of environmental education programs/projects implemented in the water resources management context.

Keywords:
Environmental education; Integrated water resources management; Indicators; Evaluation; Programs/Projects

Resumen

En el contexto del aumento de los riesgos de las crisis hídricas en muchos de los países, la educación ambiental se presenta como un instrumento con capacidad de contribuir en la búsqueda de soluciones a la problemática hídrica junto a la comunidad local y a los diversos sectores de la sociedad. Sin embargo, estudios indican la fragilidad metodológica de los programas/ proyectos de educación ambiental direccionados a la gestión de los recursos hídricos. Según esa perspectiva, elaboramos el Sistema de Evaluación de Programas y Proyectos de Educación Ambiental para los Recursos Hídricos (SAPEA-Agua), con base en una pesquisa cuali-cuantitativa que combinó la investigación bibliográfica y la análisis de contenido de documentos que orientan a la educación ambiental, con la finalidad de evaluar el nivel de contextualización, interdisciplinaridad, participación, sostenibilidad, comunicación y autoevaluación de los programas/ proyectos de educación ambiental. Consideramos que el SAPEA-Agua puede apoyar y ampliar la cantidad, la calidad y la efectividad de programas/ proyectos de educación ambiental en el ámbito de la gestión de las aguas.

Palabras-clave:
Educación ambiental; Gestión integrada de recursos hídricos; Indicadores; Evaluación; Programas/ Proyectos

Resumo

No contexto de aumento dos riscos de crises hídricas em muitos países, a educação ambiental apresenta-se como um instrumento capaz de contribuir na busca de soluções para a problemática hídrica junto à comunidade local e aos diversos setores da sociedade. No entanto, estudos apontam para a fragilidade metodológica dos programas/projetos de educação ambiental voltados para a gestão de recursos hídricos. Nessa perspectiva, elaboramos o Sistema de Avaliação de Programas e Projetos de Educação Ambiental para Gestão de Recursos Hídricos (SAPEA-Água), com base em uma pesquisa quali-quantitativa que combinou pesquisa bibliográfica e análise de conteúdo de documentos balizadores da educação ambiental, com a finalidade de avaliar o nível de contextualização, interdisciplinaridade, participação, sustentabilidade, comunicação e autoavaliação dos programas/projetos de educação ambiental. Consideramos que o SAPEA-Água pode subsidiar o processo de avaliação e ampliar a quantidade, a qualidade e a efetividade de programas/projetos de educação ambiental no âmbito da gestão das águas.

Palavras-chave:
Educação ambiental; Gestão integrada de recursos hídricos; Indicadores; Avaliação; Programas/Projetos

Introduction

Despite the knowledge acquired over the civilizational process associated with water resources management, tensions between available water resources and social demands have grown at global level (MADANI, 2014). The use of fresh water increased by 50% from 1960 to 2010, due to the expansion of agricultural activities in several countries, worldwide, in order to meet the needs of a growing urban population. Consequently, the pressure on public water supply systems, as well as on underground reserves, has gradually increased, as addressed by Taylor (2014).

Predictions concerning irregularities in rainfall pattern - which were confirmed by the Intergovernmental Panel on Climate Change (IPCC), AR6 (2021) - tend to put even more pressure on the water issue. Changes in rainfall patterns have been observed in some Brazilian regions since 2012; they have caused water crises in the semi-arid and metropolitan regions of Rio de Janeiro and São Paulo states, in 2015; in the Federal District, in 2017; and in Curitiba City, in 2020 and 2021. Pumping water from groundwater sources to Curitiba’s public supply reservoirs was one of the solutions found to manage the water crisis faced by the city.

According to the National Water and Sanitation Agency, unfavorable water balance, in association with low investments in water infrastructure, mainly in water production systems, as well as with below-average rainfall periods, have generated high vulnerability and led to water crisis periods in several Brazilian regions in recent years (ANA, 2019). This scenario remained unchanged in 2021, and it ended up affecting the interconnected electric power generation system in the country.

The federal government has issued a water emergency alert in five states - Minas Gerais, Goiás, Mato Grosso do Sul, São Paulo and Paraná - from June to September 2021, to help conserving the water of power plants’ reservoirs in Paraná River basin. In addition, reduced inflows in the summer (2020/2021), in Southeastern Brazil, have evidenced a trend towards relative loss in system’s regularization capacity (DUTRA; FIGER, 2021).

Jacobi et al. (2015) have emphasized that this water crisis scenario requires adequate conditions for good water governance - such as inclusion, participation, an open and transparent approach - since accountability is a way to properly provide information to the public and to encourage social control (EMPINOTI; JACOBI; FRACALANZA, 2016). CNRH Resolution n. 98/2009 has set the principles, fundamentals and guidelines for environmental education in the National Water Resources Management System (SINGREH - Sistema Nacional de Gerenciamento de Recursos Hídricos). However, according to Loureiro (2011; 2013), the production and implementation of environmental education programs/projects in river basin committees remains incipient.

Moreover, several studies have pointed out the methodological vulnerability of environmental education programs/projects aimed at water resources management, both in formal and non-formal education processes (GUANABARA et al., 2008; VEIGA; BRANCO, 2009; INEA, 2013; NEFFA et al., 2014, XIII DIÁLOGO INTERBACIAS DE EDUCAÇÃO AMBIENTAL EM RECURSOS HÍDRICOS DO ESTADO DE SÃO PAULO, 2015; CHACON-PEREIRA et al. 2016). The Sustainable Development Goals of the Global Agenda 2030 and the MonitoraEA¹ system used to control and evaluate public policies focused on environmental education in Brazil stands out in this context. This system was proposed by the National Articulation of Public Policies on Environmental Education (ANPPEA - Articulação Nacional de Políticas Públicas de Educação Ambiental).

The current study proposes and introduces the methodology adopted to develop the System for the Evaluation of Environmental Education Programs and Projects focused on integrated water resources management (SAPEA-Água). The System highlights some dimensions capable of evaluating programs/projects of this nature, in order to contribute to the construction of environmental education programs/projects focused on encouraging social participation and transparency in the dissemination of results deriving from the applied investments.

Materials and Methods

The bibliographic research carried out to build the SAPEA-Água System comprised books, bibliographic databases, scientific journals, as well as dissertations and theses’ databases. Documentary research was also carried out in the following documents:

• - International agreements on environmental education;

• - Legislation, such as the National Water Resources Policy - PNRH (1997) and the National Environmental Education Policy - Pnea (1999);

• - CNRH Resolutions, such as Resolution n. 98/2009, which sets the principles and fundamentals for environmental education at river basin scope;

• - Other references associated with integrated water resources management and environmental education.

The herein adopted methodology comprised bibliographic review on the Web of Science, Periodicals Capes and Google platforms.

The Brazilian Periodicals Capes platform was mainly consulted for scientific studies conducted in the national context, as well as to search for perspectives from the academic community (dissertations and thesis), notably, from scholars focused on investigating the topic in question, whereas documents from governmental institutions and non-governmental organizations (NGOs) were selected/obtained in the Google platform; thus it mainly contributed to gray literature reviews. Meanwhile, the Web of Science platform, which is a research tool into global citation core databases, such as the Science Citation Index (SCI), supported the retrieval of high-quality theoretical and scientific articles at international level. Environmental education, water resources, water, integrated water resources management, water resources management, evaluation, indicators, socio-environmental indicators, environmental education programs and environmental education projects were the keywords used in the herein conducted search. They were subjected to multiple combinations; heir equivalents in Portuguese language were also used.

Basic environmental education assumptions found in international and national documents of reference in this knowledge field were used to answer the question about elements of environmental education that should be included in water management programs/projects in order to guarantee ecosystems’ integrity. The selection criterion applied to these documents was based on their international and national acknowledgement.

The Belgrade Charter (1975), the Tbilisi Declaration (1977) and the Treaty on Environmental Education (1992) are documents widely acknowledged in both national and international contexts. On the other hand, legal landmarks of reference in environmental education were selected within the national scenario, namely: the Brazilian legislation of reference in the field - Pnea and CNRH Resolution n. 98/2009 -, which is the legal reference landmark guiding environmental education in the integrated water resources management context.

The content analysis method was used to assess these documents. This method comprises a set of communication analysis techniques, based on both systematic procedures and message content descriptions aimed at finding indicators (either qualitative or not) to enable inferring knowledge associated with message production/reception conditions (BARDIN, 1995).

Texts and expressions in the original contents, which communicated basic environmental education assumptions, were clipped during the analysis of these documents; these excerpts were inserted in Excel spreadsheet. Box 1 presents the analysis category “Contextualization”, as example of such systematization process. Subsequently, these textual records were grouped based on the addressed content in order to generate the analysis categories. Therefore, the herein selected reference documents on environmental education aimed at defining the analysis categories that associated concepts and that could be used to set classifications. Thus, working with categories means grouping elements, ideas or expressions capable of covering the analyzed issue (GOMES, 1994). Therefore, analysis categories are qualitative variables.

After defining the analysis categories, the theoretical framework was prepared to substantiate each of the listed topics/classification groups. In order to do so, other reference documents on environmental education, such as scientific journals, books and gray literature documents, were incorporated to the study.

The following research stage comprised the preparation of a set of environmental education indicators for integrated water resources management and their association with each analyzed category and their evaluation parameters.

Box 1
Example of content analysis systematization - analysis category “Contextualization”

Analysis category	Reference document	Extracted excerpts
Contextualization Promoting solutions to socio-environmental issues at local, national and international levels	Belgrade Charter (1975)	Item F - Basic Guidelines for EE Programs 5: “Environmental education should examine major environmental issues from a world point of view, while paying due regard to regional differences”.
		Item F - Basic Guidelines for EE Programs 8: “Environmental education should promote the value and necessity of local, national and international cooperation in the solution of environmental problems”.
	Tbilisi Declaration (1977)	“By taking a global approach...”
	Treaty on EE (1992)	“This requires individual and collective responsibility at local, national and planetary level”.
		EE principles: 3. “Environmental education is both individual and collective. It aims to develop local and global citizenship with respect for self-determination and the sovereignty of nations”.
		Action Plan 6: “Promote and support training for environmental conservation, preservation and management, as part of the exercise of local and planetary citizenship”.
		Action Plan 14: “Promote the creation and strengthening of national, regional and international networks for joint action between organizations of the South, North, East and West with a planetary perspective (e.g. foreign debt, human rights, peace, global warming, population, contaminated products.)”.
	PNEA (BRASIL,1999)	Basic principle of EE VII: “the articulated approach to local, regional, national and global environmental issues”
		EE’s fundamental goals V: “Encouraging cooperation among different regions in the country, at micro and macro-regional levels, in order to build an environmentally balanced society, founded on the principles of freedom, equality, solidarity, democracy, social justice, responsibility and sustainability”.
	CNRH Resolution n. 98/2009	Article 3 reaffirms the basic principle of EE in PNEA VII: “the articulated approach to local, regional, national and global environmental issues”.
		Article 3 XI - the river basin (Law n. 9,433, from 1997, article 1, item V) and the river basin region (CNRH Resolution n. 32, from October 15, 2003), which encompasses a basin, group of basins or sub-contiguous basins showing homogeneous or similar natural, social and economic features, as water resources planning and management units.

• Source: The authors, 2022.

Sapea-Água was validated in compliance with the Delphi technique, which relies on a panel of experts. The Delphi technique admits the superiority of experienced people’s judgment over the judgment of an individual or small group of individuals (LINSTONE; TUROFF, 2002).

According to Minayo (2009), the Delphi method can be featured as “strategy to structure the improvement of research instruments or analysis results”. According to the aforementioned author, researchers, experts on the subject matter or people with practical experiences in a specific topic are invited to read, criticize and suggest contributions to a given material. The purpose of this process is to refine, adapt and give quality to the previously prepared material. This technique can be used in multiple ways, as described by Minayo in the excerpt below:

A research group or researcher prepares a given text presenting a set of indicators it deems to be fundamental for a given evaluation and shares its preliminary version with the best experts on the subject. Upon receiving their feedback, it is up to the researcher or research group to incorporate the experts’ suggestions and to summarize their contributions. This technique plays fundamental role in validating both instruments and research results. The basis of the Delphi technique lies on acknowledging the superiority of experienced people’s judgment over the judgment of an individual or small group of individuals (MINAYO, 2009, p. 88).

It is worth emphasizing that SAPEA-Água was validated in compliance with the basis of the Delphi methodology and that it was subjected to the analysis of two experts in integrated water resources management and environmental education, namely: Carlos Frederico Loureiro - PhD in Social Sciences and professor at the Education School of Federal University of Rio de Janeiro (UFRJ - Universidade Federal do Rio de Janeiro), author of significant scientific production associated with environmental education in the water resources management context, including aspects associated with both the legislation and evaluation of educational processes -; and Fátima de Lourdes Casarin - expert in Water Resources Management, with strong practical skills at the formal municipal schooling and at the non-formal context, mainly at river basin committees level, consultant for social mobilization and environmental education projects in the water resources and sanitation fields. From 2007 to 2014, she worked as manager at State Institute of the Environment (Inea/RJ) and project coordinator at State Secretary for the Environment and Sustainability (SEAS/RJ). Where she was in charge of the Water Agenda at Schooling Programs - Environmental Education for Water Management, carried out in 22 municipalities and 110 schools in partnership with river basin committees. This program was selected among the six best water management practices in Brazil in the Government Category of the 2012 ANA Award. These professionals have read, criticized and elaborated suggestions on the preliminary version of SAPEA-Água, based on their theoretical-critical knowledge and practical experiences, in order to help expanding and legitimizing this evaluation system.

Thus, the process to build the SAPEA-Água system has followed a qualitative-quantitative methodological approach, which was justified by the application of the classic content analysis (BARDIN, 1995) and Delphi (LINSTONE; TUROFF, 2002) methods, whose stages are shown in Figure 1.

Figure 1
Methodological proposal procedures for the elaboration of SAPEA-Água

The herein adopted methodological approach has pointed out the elements used to evaluate the contents of environmental education programs / projects focused on water resources conservation and management processes. It took into account their theoretical concepts in order to promote minimum quality, promotion and clarity standards for the evaluation processes adopted in environmental education (CARLETON-HUG, 2010; TAO, 2012; STERN et al., 2014).

After the SAPEA-Água construction process was over, values were assigned to indicators in order to provide a numerical measure to work as basis for the necessary analyses and inferences of environmental education programs / projects focused on water resources management. This procedure enabled quantitatively measuring and qualitatively analyzing system’s responses in order to generate inferences to help managers monitoring and evaluating the quality of the program / project.

The sum of the attributed scores is expected to objectively contribute to monitor and evaluate the environmental education programs / projects focused on water resources management.

Defining the analysis categories

As previously reported, the herein adopted analysis categories - Contextualization, Interdisciplinarity, Participation, Communication, Program / Project Sustainability and Self-assessment - are associated with basic environmental education assumptions addressed in the main analyzed documents. Box 2 presents the correlation process among these categories, environmental education assumptions and reference documents.

Box 2
Association among analysis categories, environmental education assumptions and reference documents on environmental education

LISTED CATEGORIES	Environmental Education Assumptions	Belgrade Charter (1975)	Tbilisi Declaration (1977)	Treaty on EE (1992)	National EE Policy (1999)	CNRH Resolution n. 98/2009
Contextualization	Promoting solutions for socio-environmental issues at local, national and global levels.	X		X	X	X
Interdisciplinarity	Adopting the interdisciplinary methodological approach to interpret the environment in its entirety.	X	X	X	X	X
Participation	Including the community in decision-making processes aimed at enabling balance in the local environment through participatory and democratic processes.	X	X	X	X	X
Communication	Democratization of environmental information (social networks and socialization of different knowledge types - popular, technical and scientific).		X	X	X	X
Program / Project Sustainability	Sustainability of the educational process based on training multipliers, as well as on partnership formation, integration to public policies and fundraising.	X	X	X	X	X
Self-assessment	Procedural assessment (elaborating action plans)	X		X	X	X

• Source: The authors, 2022.

The analysis applied to Box 2 has shown that assumptions defining analysis categories such as “Interdisciplinarity”, “Participation” and “Program / Project Sustainability” appeared in all analyzed documents. The Tbilisi Declaration does not provide on the basic environmental education assumptions supporting analysis categories “Contextualization” and “Self-assessment”. The Belgrade Charter (1975), in its turn, does not expose the basic environmental education assumption referring to category “Communication”.

The fact that the adopted analysis categories are not exclusively associated with Brazilian documents increases their application scope, since the herein referenced international documents were addressed in forums that counted on the participation of several countries, before their practical application. Based on the globalization logic, this analysis has indirectly contributed to the universalization of principles guiding environmental education.

The role played by environmental education in integrated water resources management in Brazil was legally acknowledged by CNRH Resolution n. 98/2009, which sets the principles and fundamentals of environmental education at river basins’ scope.

SAPEA-Água presentation

SAPEA-Água is a system used to evaluate environmental education programs / projects focused on integrated water resources management. Its elaboration process was guided by the following question: what environmental education elements should be included in water management programs / projects?

The innovative nature of this system derives from its goal to overcome lack of environmental education evaluation processes, mainly of those focused on water resources. As pointed out by one of the experts consulted during the system elaboration process, SAPEA-Água is an excellent and rare tool that can be feasible and functional, when it is suited before the execution of a given project.

The SAPEA-Água construction process was based on three pillars, namely: critical environmental education, integrated water resources management and socio-environmental indicators.

SAPEA-Água is an evaluation system towards critical environmental education trends and based on the basic assumptions of this education, which were observed in international and national reference documents associated with this field, such as the Belgrade Charter (1975), the Tbilisi Declaration (1977), the Treaty on Environmental Education for Sustainable Societies and Global Responsibility (1992), as well as Pnea and CNRH Resolution n. 98/2009.

The critical environmental education trend - designed and spread by authors such as Philippe Pomier Layrargues, Mauro Guimarães, Carlos Frederico Loureiro and José Silva Quintas - is committed to transform the current social order by taking into consideration its relationship with the environment and by transmitting it through popular, emancipatory and transformative environmental education currents, as well as through education focused on public environmental management.

The environmental education trend defines the political-ideological orientation of public policies and environmental education programs/projects, including the ones focused on integrated water resources management. It is worth emphasizing that the critical trend in Brazil guides public policies in this fields, such as Pnea (1999), the National Environmental Education Program (ProNEA) (2014) and, in the current case, SAPEA-Água.

Critical/emancipatory environmental education programs instrumentalize different sectors of society in order to participate in river basin committees, as well as in committees focused on reasoning about consumption as structural cause of water crisis, on observing and investigating local impacts to find integrated solutions capable of articulating school community, users, organized civil society and public power. They are also committed to the collective elaboration of declarations, treaties or public policies aimed at contributing to the conservation and integrated management of water resources.

The integrated water resources management is another SAPEA-Água’s pillar. The expression “integrated water resources management” is associated with some historical-conceptual landmarks, such as the Dublin Declaration (1992), chapter 18 of Agenda 21 and the concept established by the Global Water Partnership - GWP.

Based on these documents, the river basin was acknowledged as privileged territorial space for the integrated management of water resources. In order to do so, it must be considered a complex system (MORIN, 1990) comprising physical, social, economic, political and cultural environments. The interrelation among these different environments results in a system with low long-term predictability (GONDOLO, 2000).

The integrated water resources management in Brazil takes PNRH as legal-institutional framework. This legislation defines river basins as territorial units for the implementation of this policy and underpins the creation of the National Water Resources Management System (SINGREH - Sistema Nacional de Gerenciamento de Recursos Hídricos).

Thus, the integrated management of water resources provides subsidies to evaluate environmental education programs/projects in the water management context. In addition, it indicates ways to formulate criteria, indicators and parameters to be used to evaluate these programs/projects.

The last pillar supporting SAPEA-Água lies on socio-environmental indicators, which are quantitative and/or qualitative indicators that enable assessing whether the goals of a given program/project are carried out well or whether they have been achieved (MINAYO, 2009).

The use of indicators is recommended in several scientific studies as a way to evaluate progress towards sustainability or sustainable development. Thus, indicators are a promising tool to support decision-making processes (DAHL, 2012; MOLDAN et al., 2012; NICHOLSON et al., 2012) in the governance and management of water resources (PIRES et al., 2017).

The use of indicator matrices to evaluate programs/projects in the environmental education field remains significantly incipient, although some initiatives often point towards a critical environmental education trend (TRAJBER; MENDONÇA, 2006; GUANABARA et al., 2008; GUANABARA et al., 2009; LUZ; TONSO, 2015).

The proposed system (SAPEA-Água) comprises evaluation parameters and indicators for each category analysis topic: Contextualization, Interdisciplinary, Participation, Communication, Program/Project Sustainability and Self-Assessment. Altogether, the system comprises 6 analysis categories, 15 evaluation parameters and 43 indicators. Box 3 presents the SAPEA-Água.

It is essential emphasizing that the SAPEA-Água’s design was evaluation-oriented, although it can also be applied to guide the proposition of environmental education programs/projects focused on the integrated water resources management, as well as work as guidance for reference terms in public calls or for the elaboration of new propositions.

Box 3
System for the evaluation of environmental education programs and projects focused on water resources management - SAPEA-Água

CONTEXTUALIZATI ON
EVALUATION PARAMETERS	INDICATORS
1. Identification of multiple water uses and traditional knowledge.	1. It presents the environmental and socioeconomic diagnosis of water resources and multiple water uses.
	2. It reports the environmental history of water resources in the area covered by the program/project and specifies the traditional knowledge used in production practices associated with water.
2. Identification of local socio-environmental issues.	3. It presents the map of social actors and conflicts involving multiple water uses.
	4. It reports the history of conflicts involving multiple water uses, mediations, negotiations, as well as political alliances and ruptures.
	5. It reports socio-environmental issues indicated by local social actors, by the river basin committee or by the water resources management body.
3. Actions contributing to the process to cope with local socio-environmental issues.	6. At least 90% of the established goals focus on coping with local socio-environmental issues associated with water resources.
INTERDISCIPLINARITY
EVALUATION PARAMETERS	INDICATORS
4. Formation of an interdisciplinary team (articulation between the technical and social dimensions).	7. It presents balance in the number of professionals designated per knowledge field (Exact and Earth Sciences, Biological Science, Health Science, Agrarian Science, Applied Social Science, Humanities, Engineering, Linguistics, Letters and Arts) who form the program/project’s team, in line with the local context.
	8. It includes the number and profile of local social actors participating in the program/project’s team.
	9. It enables the team to understand the program/project in the critical environmental education context and to interact with common knowledge.
5. Adoption of educational practices in order to integrate scientific and common knowledge.	10. It concerns production and consumption practices associated with water resources in educational activities, as well as integrates and strengthens different scientific knowledge fields and traditional knowledge linked to them.
	11. It presents the number, description, performance time, attendance list, profile and contact of participants, as well as photographic records of educational activities carried out based on the interdisciplinary methodological approach.
	12. It presents materials produced in interdisciplinary educational activities.
	13. The number and profile of individuals participating in interdisciplinary educational activities are in compliance with the established goals.
PARTICIPATION
EVALUATION PARAMETERS	INDICATORS
6. Educational actions counting on the participation of local social actors in vulnerability contexts.	14. It reports the demands pointed out by, and negotiated with, local social actors in vulnerability contexts associated with water conservation and management processes.
	15. It presents the number, profile description, attendance list and contacts of participants, as well as the photographic record of meetings held to propose the educational actions of the program/project.
7. Training and instrumentalization of local social actors in vulnerability contexts in order to participate in decision-making processes.	16. It presents the number, profile description, performances time, attendance list and contacts of participants, as well as the photographic record of training and instrumentalization activities focused on local social actors to enable them to make qualified interventions in decision-making and social control processes associated with water resources.
	17. It presents materials produced in the training, instrumentalization and social control activities focused on local communities.
	18. The number and profile of individuals participating in the training, instrumentalization and social control activities are in compliance with the planned goals.
COMMUNICATION
EVALUATION PARAMETERS	INDICATORS
8. Promotion of information about program/project’s actions and about the conservation and management of local water resources.	19. It describes communication actions capable of indicating the action type, the means of communication, their promotion times and purpose, as well as the target audience’s profile.
	20. It presents textual, digital and photographic materials produced in communication actions.
	21. The size and profile of the target audience reached by communication actions are in compliance with the planned goals.
9. Edu-communication actions focused on increasing local population’s participation in decision-making and social control processes.	22. It presents the number, profile description, performance time and contacts of participants, as well as the photographic record of edu-communication actions implemented in the local water resources management context.
	23. It presents textual and digital materials produced in edu-communication actions.
	24. The size and profile of the target audience reached by edu-communication actions are in compliance with the planned goals.
10. Social networks’ formation or strengthening.	25. It presents the number, profile description, performance time and contacts of participants, as well as the photographic record of the social networks interconnecting institutions and social practices concerning water resources.
	26. It presents materials produced by social networks and virtual social network homepages.
	27. The size and profile of the target audience reached through social networks are in compliance with the planned goals.
PROGRAM/PROJECT’S SUSTAINABILITY
EVALUATION PARAMETERS	INDICATORS
11. Multipliers’ training.	28. It presents the number, profile description, performance time and contacts of participants, as well as the photographic record of training activities applied to multipliers in order to keep the program/project alive after its conclusion.
	29. It presents materials produced in training activities applied to multipliers.
	30. The size and profile of the target audience reached in training activities applied to multipliers are in compliance with the planned goals.
12. Partnership formation and fundraising.	31. It presents the number, profile description, performance time and contacts of participants, as well as the photographic record of activities carried out in the River Basin Committees integrating community leaders, non-governmental organizations, the private sector and/or public bodies involved in water resources management, in order to establish partnerships to enable continuing the educational process after the end of the program/project.
	32. It presents the partnership agreements signed by the involved parties.
	33. The number and profile of partner institutions are in compliance with the planned goals.
13. Integration with public policies.	34. It presents the number, description, performance time, profile and contact of participants, as well as the photographic record of educational activities articulating the civil society, the private and public sectors, and non-governmental organizations involved in the water topic in order to formulate documents and to fulfill collective commitments to water resources sustainability.
	35. It presents documents and collective commitments signed by the involved parties.
	36. The number and scope of documents and collective commitments are in compliance with the planned goals.
SELF-ASSESSMENT
EVALUATION PARAMETERS	INDICATORS
14. Procedural Assessment.	37. It accounts for at least 80% of the action plan.
15. Results assessment.	38. It presents physical, chemical and biological indicators associated with water resources as a way to evaluating and monitoring the results (e.g., improvements in water quality, increase in the number/size of green areas, replanted seedlings, recovered lands, launching conservation areas, increase in water availability/security, increase in the population of living beings, and maintained or restored ecosystem services, among others).
	39. It presents socioeconomic indicators associated with water resources as a way to evaluate and monitor the results (e.g., job and income generation, decent quality of life for the local population, increased population and watershed resilience, preparation to face the risks of climate change, adequacy of basic sanitation systems, reducing the number of waterborne diseases affecting local populations, among others).
	40. It achieves at least 80% of established goals.
	41. It presents water conservation and management demands pointed out by local social actors and met by the program/project, as well as the justification for the non-met ones.
	42. It shows beneficiaries’ degree of satisfaction with the program/project (attaching the survey and the number of respondents per social group or institution).
	43. It presents criticism of the program/project and suggestions for its improvement and continuity.

• Source: The authors, 2022.

SAPEA-Água application

Regarding qualitative aspects of SAPEA-Água, it is important emphasizing that this evaluation system dialogues with the critical/emancipatory environmental education trend (LAYRARGUES; LIMA, 2014).

Critical/emancipatory environmental education programs instrumentalize different sectors of society to participate in river basin committees, to reflect about consumerist practice as structural cause of water crisis, to observe and investigate local impacts to find integrated solutions capable of articulating the school community, users, organized civil society and public authorities. They are also committed to the participatory elaboration of declarations, treaties or public policies focused on the conservation and integrated management of water resources.

Conservative environmental education programs implemented in the water resources context aim at raising awareness about the global environment, climate change and water crisis; reducing domestic water needs/consumption; implementation of rainwater reuse systems; as well as observation of, and investigation about, impacts on local water resources without necessarily finding integrated solutions. In other words, they are initiatives that, overall, do not pursuit social, political and economic articulation to propose individuals’ participation in the solution of impacts on aquatic ecosystems.

The application of SAPEA-Água implies assigning scores to each analysis category. These scores result from the sum of scores attributed to each evaluation parameter and corresponding indicator(s). They are attributed by experts, based on their perception about the degree of completion of theoretical and practical concepts guiding each parameter and their corresponding indicator(s) - the attributed score are “+1” (whenever present) or “0”/null (whenever lacking).

Scores attributed to each analysis category receive a different weight at the time to calculate the total score for the program/project. Analysis category “Participation” was attributed weight 2, since critical environmental education is mainly based on social actors’ participation in socio-environmental vulnerability contexts.

The weight 2 attributed to analysis category “Participation” is also justified by experts’ observations. Expert A has emphasized the importance of instrumentalizing and encouraging community inclusion in decision-making processes by prioritizing the most vulnerable social groups, since they were excluded from public participation. Expert B, in his turn, has stated that reference texts supporting environmental education programs/projects focused on water resources management should take into consideration the need of promoting society’s effective citizen participation in river basin committees.

Despite the suggestion to assign weight 2 to analysis category “Participation”, it is convenient to leave open the possibility of differentiating weights in attributions suggested by appraisers of each program/project. Thus, we agree with expert A’s opinion, according to whom, weights should be assigned according to the project’s purpose or framework. It is important reflecting about the assignment of different weights to each analysis category, based on the program/project’s scope. It is so because a differentiated scoring system can lead to heterogeneous SAPEA-Água use, and it would make it hard to compare different proposals. Thus, the SAPEA-Água score should be standardized for all programs/projects and each means of verification.

Box 4 summarizes the total number of analysis categories, evaluation parameters and indicators proposed by the SAPEA-Água evaluation system, as well as values suggested to evaluate environmental education programs/projects in the water resources context.

The funding institution or the supervisory body in charge of the program/project is the one accounting for assessing whether it presents the indicators, or not, based on the sum indicating the level the methodological approach of the analyzed program/project is at, in comparison to basic assumptions of environmental education focused on water management.

Box 4
Total number of analysis categories, evaluation parameters and indicators, in addition to values suggested for SAPEA-Água quantification

ANALYSIS CATEGORY	N. OF PARAMETERS	N. OF INDICATORS	WEIGHT	TOTAL SUM
Contextualization	3	6	1	+6
Interdisciplinarity	2	7	1	+7
Participation	2	5	2	+5 x (2) = +10
Communication	3	9	1	+9
Program/Project’s Sustainability	3	9	1	+9
Self-assessment	2	7	1	+7
Total	15	43		+48

• Source: The authors, 2022.

Another aspect inherent to the sum of Sapea-Água scores concerns the classification of programs/projects in quality ranges, based on the achieved score: “EXCELLENT”, “GOOD”, “REGULAR” or “INSUFFICIENT. Thus, a classification for environmental education programs/projects focused on water resources management was herein proposed, as shown in Box 5. This classification enables managers to compare environmental education programs/projects focused on integrated water resources management.

Box 5
Classification of environmental education and water resources programs/projects, based on the sum achieved in SAPEA-Água

TOTAL SUM	CLASSIFICATION	COLOR
From 37 to 48 points	EXCELLENT	GREEN
From 25 to 36 points	GOOD	BLUE
From 13 to 24 points	REGULAR	YELLOW
From 0 to 12 points	INSUFFICIENT	RED

• Source: The authors, 2022.

From this perspective, the methodological proposal called SAPEA-Água helps improving the theoretical and practical approach of environmental education programs aimed at water resources conservation and management processes.

Conclusions

The main contribution of the current study lied on its description of SAPEA-Água, which is an innovative system developed to evaluate environmental education programs and projects focused on integrated water resources management. Each SAPEA-Água development stage was herein presented. SAPEA-Água is an instrument capable of supporting these programs/projects’ monitoring and evaluation.

SAPEA-Água can also subsidize reference terms for environmental education programs and projects as a way to level proposals supported by both basin agents and the Brazilian integrated water resources management system. SAPEA-Água application indirectly improves the training provided to environmental education professionals and encourages the development of a critical viewpoint in training processes. Defining a standard evaluation system for environmental education programs and projects can help optimizing and improving the management of these initiatives.

The description of the evolutionary SAPEA-Água development stages enables the technical-scientific community to criticize and propose further evolutionary stages for the system. Some stages were already consolidated in the book about SAPEA-Água (CHACON-PEREIRA et al. 2020), which which included a didactic and illustrative chapter about the application of this system to a real case. In addition, other initiatives are already underway, such as the preparation of a manual to support the System for Integrated Water Resources Management.

Acknowledgement

The authors are grateful to the Post-graduation Program in Environmental Science (PPGMA - Programa de Pós-graduação em Meio Ambiente) of Rio de Janeiro State University (UERJ - Universidade do Estado do Rio de Janeiro), to the research board of the Pontifical Catholic University of Paraná (PUCPR - Pontifícia Universidade Católica do Paraná), to the Research Support Foundation of Rio de Janeiro State (FAPERJ - Fundação de Amparo à Pesquisa do Estado do Rio de Janeiro) and to the Military School of Rio de Janeiro (CMRJ - Colégio Militar do Rio de Janeiro), for supporting the current research. We are also thankful to the experts that took part in SAPEA-Água’s evaluation and consolidation processes.

References

• ANA - AGÊNCIA NACIONAL DE ÁGUA. Conjuntura dos Recursos Hídricos no Brasil 2019: Informe anual. Brasília: ANA, 2019.
• BARDIN, L. Análise de Conteúdo. Lisboa: Edições 70, 1995.
• CARLETON-HUG, A.; HUG, J. W. Challenges and opportunities for evaluating environmental education programs. Evaluation and Program Planning 33, p. 159-164, 2010. Disponível em: http://dx.doi.org/10.1016/j.evalprogplan.2009.07.005.
  » http://dx.doi.org/10.1016/j.evalprogplan.2009.07.005.
• CHACON-PEREIRA, A; NEFFA, E.; SILVA, L. P.; BARBOSA, G.; FORMIGA-JOHNSSON, R. M. Educação ambiental e gestão participativa nos comitês de bacia hidrográfica do estado do Rio de Janeiro. Revista Interface, n. 12, p. 70-83, 2016.
• CHACON-PEREIRA, A.; NEFFA, E.; PIMENTEL DA SILVA, L. P. SAPEA-Água: educação ambiental e gestão das águas. Curitiba: Appris, 1. ed., 2020.
• DAHL, A. L. Achievementsand gaps in indicators for sustainability. Ecological Indicators, n. 17, p. 14-19, 2012. Disponível em: http://dx.doi.org/10.1016/j.ecolind.2011.04.032.
  » http://dx.doi.org/10.1016/j.ecolind.2011.04.032.
• DUTRA, J.; FIGER, V. Emergência hídrica e restrições de oferta. Revista Conjuntura Econômica. p. 44-46, Julho/2021.
• EMPINOTTI, V. L.; JACOBI, P. R; FRACALANZA, A. P. Transparência e a governança das águas. Estudos avançados [online], v. 30, n. 88, p. 63-75, 2016.
• GOMES, R. A análise de dados em pesquisa qualitativa. In: MINAYO, M. C. de S. (org.). Pesquisa Social: teoria, método e criatividade. Petrópolis: Vozes, 1994.
• GONDOLO, G. C. F. Desafios de um sistema complexo à gestão ambiental: Bacia do Guarapiranga, Região Metropolitana de São Paulo. São Paulo: Annablume; Fapesp, 2000.
• GUANABARA, R. et al. Educação ambiental e gestão de bacias hidrográficas: análise de algumas experiências. In: IV ENANPPAS - Encontro da Associação Nacional de Pesquisa e Pós-Graduação em Ambiente e Sociedade, 2008, Brasília. Anais do IV Encontro da ANPPAS (CD-ROM e internet). Brasília: Anppas, 2008.
• GUANABARA, R.; GAMA, T.; EIGENHEER, E. M. Contribuições para a construção de uma matriz para avaliação de projetos de educação ambiental. Educação e Pesquisa. São Paulo, v. 35, n. 2, p. 399-411, maio/ago. 2009.
• INEA - INSTITUTO ESTADUAL DO AMBIENTE. Elaboração do Plano Estadual de Recursos Hídricos do Estado do Rio de Janeiro. R4 - Relatório gestão de recursos hídricos. Elaboração Fundação Coppetec. Rio de Janeiro, 2013.
• JACOBI, P. R; CIBIM, J.; LEÃO, R. S. Crise hídrica na Macrometrópole Paulista e respostas da sociedade civil. Estudos avançados [online], v. 29, n. 84, p. 27-42, 2015.
• LAYRARGUES, P. P.; LIMA, G. F. C. As macrotendências político-pedagógicas da educação ambiental brasileira. Ambiente & Sociedade, v. 17, n. 1, p. 23-40, 2014.
• LINSTONE, H. A.; TUROFF, M. (ed.). The Delphi Method Techniques and Applications. 2002. Disponível em: https://web.njit.edu/~turoff/pubs/delphibook/index.html#toc Acesso em: maio 2017.
  » https://web.njit.edu/~turoff/pubs/delphibook/index.html#toc
• LOUREIRO, C. F. B.; GOMES, G. F. A. Resolução nº 98/2009 do Conselho Nacional de Recursos Hídricos e a educação ambiental para a gestão das águas. Diálogo, São Paulo, n. 8, 2011.
• LOUREIRO, C. F. B. Avaliação de processos de educação ambiental na gestão das águas. In: PAULA JR, F. de; MODAELLI, S. (org.). Política de águas e Educação Ambiental: processos dialógicos e formativos em planejamento e gestão de recursos hídricos. Ministério do Meio Ambiente. Secretaria de Recursos Hídricos e Ambiente Urbano. Brasília: MMA, 2013.
• LUZ, W. C.; TONSO, S. Construção de indicadores e parâmetros de educação ambiental crítica. VIII EPEA - Encontro Pesquisa em Educação Ambiental. Rio de Janeiro, 19 a 22 de julho de 2015.
• MADANI, K. Water management in Iran: what is causing the looming crisis? Journal of Environmental Studies and Sciences 4 (4), 315-328, 2014. Disponível em: http://dx.doi.org/10.1007/978-90-481-2776-4_8.
  » http://dx.doi.org/10.1007/978-90-481-2776-4_8.
• MINAYO, M. C. S. Construção de Indicadores Qualitativos para Avaliação de Mudanças. Revista Brasileira de Educação Médica, Rio de Janeiro, n. 33, Supl.1, p. 83-92, 2009.
• MOLDAN, B.; JANOUŠKOVÁ, S.; HÁK, T. How to understand and measure environmental sustainability: indicators and targets. Ecological Indicators, 17:4-13, 2012. http://dx.doi.org/10.1016/j.ecolind.2011.04.033.
  » http://dx.doi.org/10.1016/j.ecolind.2011.04.033.
• MORIN, Edgar. Introdução ao pensamento complexo. 2. ed. Lisboa: Instituto Piaget, 1990.
• NEFFA, E. et al. Saberes e práticas de educação ambiental da rede estadual de ensino do Rio de Janeiro. Rio de Janeiro: Imos, 2014.
• NICHOLSON, E. et al. Making robust policy decisions using global biodiversity indicators. PLoS ONE, v. 7, n. 7, 2012. Disponível em: http://dx.doi.org/10.1371/journal.pone.0041128.
  » http://dx.doi.org/10.1371/journal.pone.0041128.
• PIRES, A. et al. Sustainability Assessment of indicators for integrated water resources management. Science of the Total Environment, n. 578, p. 139-147, 2017. Disponível em: http://dx.doi.org/10.1016/j.scitotenv.2016.10.217.
  » http://dx.doi.org/10.1016/j.scitotenv.2016.10.217.
• STERN, M. J., POWELL, R. B.; HILL, D. Environmental education program evaluation in the new millennium: what do we measure and what have we learned? Environmental Education Research, v. 20, n. 5, p. 581-611, 2014. Disponível em: http://dx.doi.org/10.1080/13504622.2013.838749.
  » http://dx.doi.org/10.1080/13504622.2013.838749.
• TAO, Z. Education Programs on Environment. Procedia Environmental Sciences, n. 12, p. 349-353, 2012. Disponível em: http://dx.doi.org/10.1016/j.proenv.2012.01.288.
  » http://dx.doi.org/10.1016/j.proenv.2012.01.288.
• TAYLOR, R. When wells run dry. Nature, v. 516, p. 179-180, 2014. Disponível em: http://dx.doi.org/10.1038/516179a.
  » http://dx.doi.org/10.1038/516179a.
• TRAJBER, R.; MENDONÇA, P. (org.). Educação na diversidade: o que fazem as escolas que dizem que fazem educação ambiental. Brasília: Secretaria de Educação Continuada, Alfabetização e Diversidade, 2006.
• VEIGA, B. A.; BRANCO, M. As diretrizes de Educação Ambiental no Sistema Nacional de Gerenciamento dos Recursos Hídricos (SINGREH). A trajetória da Resolução CNRH n.º 98/2009. Textos para se pensar a educação ambiental. Coleção Especial Água, v. 1, 2009. OG/PNEA - DEA/MMA - CGEA/MEC SRHU/MMA, 2009.
• XIII DIÁLOGO INTERBACIAS DE EDUCAÇÃO AMBIENTAL EM RECURSOS HÍDRICOS DO ESTADO DE SÃO PAULO, 2015. Apresentações do diálogo e encontro formativo disponíveis para download. Disponível em: http://dialogointerbacias.org/apresentacoes-do-dialogo-e-encontro-formativo-disponiveis-para-download/. Acesso em: jan. 2018.
  » http://dialogointerbacias.org/apresentacoes-do-dialogo-e-encontro-formativo-disponiveis-para-download

• 1
  - Available at http://www.monitoraea.org.br (accessed in November 2021).

Publication Dates

• Publication in this collection
  15 July 2022
• Date of issue
  2022

History

• Received
  07 June 2021
• Accepted
  07 Dec 2021