Development of questionnaire to indicate positions in relation to Sustainable Societies based on correlated concepts

Abstract

Presents the process of preparing the Composite Index of Social and Environmental Concepts (ICCSA): a multi-response model questionnaire that addresses the themes of Environmental Education, Sustainability and Environment from the perspective of the transition to Sustainable Societies. During the elaboration process, two internal consistency and validity analysis phases were conducted with participation of experts. The final version of the ICCSA presents 43 statements, arranged in six dimensions, suggesting sufficiency in terms of internal reliability and content and discriminant validity. The results show that the instrument can be used to indicate stances in relation to the transition to Sustainable Societies. It is recommended that future studies be carried out, with larger populations, to complement the reliability and validity analyzes with non-specialist participants. ICCSA is available for any researchers who might wish to use it.

Keywords:
ICCSA; Questionnaire Validity; Cronbach’s alpha; Sustainable Societies; Environmental Education

Resumo

Apresenta-se o processo de elaboração do Índice Composto de Concepções Socioambientais (ICCSA): um questionário modelo multi-resposta que aborda os temas da Educação Ambiental, Sustentabilidade e Meio Ambiente sob a ótica da transição para Sociedades Sustentáveis. Durante o processo de elaboração foram realizadas duas fases de análises de consistência interna e validade, com a participação de especialistas. A versão final do ICCSA apresenta 43 afirmações, dispostas em seis dimensões, sugerindo suficiência quanto à confiabilidade interna e validades de conteúdo e discriminante. Os resultados mostram que o instrumento pode ser utilizado para indicar os posicionamentos em relação à transição para Sociedades Sustentáveis. Recomenda-se a realização de estudos futuros, com populações maiores, para complementar as análises de confiabilidade e validade com participantes não especialistas. O ICCSA encontra-se disponível para pesquisadoras que quiserem utilizá-lo.

Palavras-chave:
ICCSA; Validade do Questionário; Alfa de Cronbach; Sociedades Sustentáveis; Educação Ambiental

Resumen

Se presenta el proceso de elaboración del Índice Compuesto de Conceptos Sociales y Ambientales (ICCSA): un modelo de cuestionario de respuestas múltiples acerca de los temas de Educación Ambiental, Sostenibilidad y Medio Ambiente desde la perspectiva de la transición a Sociedades Sostenibles. Durante el proceso de elaboración, se llevaron a cabo dos fases de análisis de consistencia interna y validez, con la participación de expertos. La versión final del ICCSA presenta 43 declaraciones, ordenadas en seis dimensiones, sugiriendo suficiencia en términos de confiabilidad interna y validez discriminante y de contenido. Los resultados muestran que el instrumento se puede utilizar para indicar posiciones en relación a la transición a Sociedades Sostenibles. Se recomienda que se realicen estudios futuros, con poblaciones más grandes, para complementar los análisis de confiabilidad y validez con participantes no especialistas. El ICCSA está disponible para las investigadoras que deseen utilizarlo.

Palabras-clave:
ICCSA; Validez del cuestionario; Alfa de Cronbach; Sociedades Sostenibles; Educación ambiental

Introduction

The challenges imposed by the socioenvironmental crises require actions that break away from the practices established in the relationships between western, or westernized, societies and nature - the logic of modernity’s four engines: science, technique, industry and economic interest (MORIN, CIURANA & MOTTA, 2003). In order to construct another rationality, coherent with a possible alternative future to the socioenvironmental crisis, it becomes necessary to question the logics inherited from the modern positivist project (the “hyper-economization of the world”), the rationality of modernity, and the excessively objective and utilitarian thoughts (LEFF, 2006). Thus, the very scientific community should renew itself based on understandings that oppose knowledge fragmentation (MORIN, 2005). The constitution of a new socio-ecological paradigm goes through the political articulation of diverse knowledge in different fields, in a constant negotiation of premises and methodologies to understand the ongoing phenomena (GARRIDO-PEÑA, 2007). Therefore, the focus of this study is a reflection on the development and use of a quantitative questionnaire to indicate the stance of the social agents’ conceptions in relation to Sustainable Societies, specifically in terms of the Environmental Education (EE) practices, about the relationship with the Environment and Sustainability, as well as their interactions.

Given the search for integrative perspectives, it is opportune to reflect on the theoretical-methodological limitations about the available approaches to those who produce science, particularly about the potentialities and limitations of using quantitative instruments in the assessment of complex psychological constructs. Faced with these markers, the production of quantitative indicators is conceived in a historical context of mechanical scientific production, oriented towards and by progress, potentiating globalization by the Planetary Iron Age (MORIN; CIURANA; MOTTA, 2003).

It is proposed to break away from this tradition, which perpetuates separation of the natural scientific know-how from the moral know-how (SANTOS, 2008). It is not a question of discarding the use of systematic assessments to understand a given phenomenon, but rather of explaining the processes and decisions involved in the construction of these indicators, as well as the potential social and ecological implications of their results. Faced with this counter-hegemonic perspective towards quantitative assessments, an apparent antithesis is recognized between the power of a broad cross-sectional perspective and the ability to deepen its reflection before each respondent; however, “(...) it is in this complementarity and antagonism dialectic that complexity is found” (MORIN, 2007, p. 64)

Far from producing consummate and definite knowledge, the aim is to propose an instrument with simplified application and interpretation, although with sufficient power to support new reflections regarding the adoption of practices that exert an impact on the environment. The quantitative approach enables fast analyses, such as in descriptive statistics based on standardized indicators (LANKSHEAR; KNOBEL, 2008). Given these reasons to adopt instruments of this nature, it is necessary to reflect on their validity - whether the instrument can respond to what it was designed to measure (accuracy) - and reliability - stability /consistency( of the instrument in different applications (FIELD, 2009; HAIR-JR; PAGE; BRUNSVELD, 2020). Such reflections aim at knowing and reducing the errors inherent to the measurement of complex constructs to acceptable levels (HAIR-JR; PAGE; BRUNSVELD, 2020).

For an instrument to be considered valid, its results should be in line with the construct it is intended to measure. Consequently, it becomes necessary to delimit the concepts to be assessed, starting from a discussion of their meanings. The debates about “Sustainability” and “Sustainable Development” indicators marginalize dimensions other than the ecological, social and economic ones - such as the psychosocial dimension (PAULISTA; VARVAKIS; MONTIBELLER-FILHO, 2008). The very instruments that build these indicators carry with them the authors’ conceptions about these concepts (GUIMARÃES; FEICHAS, 2009). Added to the intrinsic difficulties of working with complex concepts is the polysemy that these terms carry with them, many times treated as consensus when, in fact, they present structuring divergences (IRVING, 2014; LOUREIRO, 2014; SARTORI; DA SILVA; DE SOUZA CAMPOS, 2014).

In this sense, criticism of the Sustainable Development concept, especially the supremacy of the economic dimension in relation to the others and the incompatibility of unlimited economic growth on a finite planet, strengthens an alternative and counter-hegemonic conception: the construction of Sustainable Societies (IRVING, 2014). The fact that this concept is written in the plural reflects the intention to accept the diversity of possible relationships and concepts of “societies”. Here, sustainability is not only presented as an adjective applied to a given type of development, static and definite, but as a value to be constructed and incorporated by different nations and communities. In this perspective:

The building of sustainable communities and societies should be based on the reassertion of their cultural and historical elements, on the development of new ways of solidarity, and on respect for nature not by the mercantilization of biodiversity, but by the fact that creating or maintaining a more harmonious relationship between society and nature should be one the fundamentals of sustainable societies (DIEGUES, 2003, p.1, free translation).

The search for a more harmonious relationship between societies and nature should not be based on “consensus” in the environmental guideline, as if it were detached from the social one; socioenvironmental justice should be one of the constituent principles of the desirable horizons of Sustainable Societies (CARVALHO, 2008). Such being the case, the perspective is reinforced that sustainability in the construction of Sustainable Societies extrapolates certain balance in the use versus availability of natural riches scale, “their roots lie in an internal relationship with society, of an economic nature and politically balanced and egalitarian” (RATTNER, 1999, p. 237, free translation).

The building of Sustainable Societies encompasses material and immaterial aspects (COSTA-PINTO, 2019), and “they would be fluid societies, capable of adapting, and guided by their cultural specificities and their way to interpret the world and nature” (IRVING, 2014, p. 35). Consequently, the transition to Sustainable Societies demands a new life organization ethics (ACOSTA, 2012), which requires overcoming or abandoning the current model, even overcoming the instrumental rationalist model that is at the core of the hegemonic development model (GONÇALVES, 2006).

The Sustainable Societies proposal is incorporated into the guidelines in the field of EE, especially after the publication of the Treaty on Environmental Education for Sustainable Societies and Global Responsibility (LIMA, 2009; LOUREIRO, 2014; MEIRA; SATO, 2005). Even with heterogeneous understandings, the Treaty influences important EE public policies in Brazil, such as the national EE Policy and Program (TRAJBER; SORRENTINO, 2006). The various stances towards these issues are influenced by world views and by conceptions such as environment and sustainability (LOUREIRO, 2014; SAUVÉ, 2005). These ideas are inter related, assisting in the characterization of the different actions within the EE field and of how to cope with the socioenvironmental crisis (MEIRA; SATO, 2005; SILVA; CAMPINA, 2011).

Although relevant, the idea of transition to Sustainable Societies is normally silenced by the hegemonic view of Sustainable Development and lacks instruments that may indicate an approximation to people’s conceptions about the counter-hegemonic perspective. Given this gap, the questionnaire prepared articulates these three concepts on this perspective: Environmental Education (EE), Environment (ENV), and Sustainability (SUS). It is thus proposed to establish the Composite Index of Socioenvironmental Conceptions (ICCSA, corresponding acronym in Portuguese), presented below, which indicates people’s stances in relation to Sustainable Societies with correlated concepts as a starting point.

ICCSA construction strategy

Some of the conceptual markers that supported conception of the ICCSA axes are also discussed, intending to present the methodological path followed. In this sense, the initial proposal added another section to work on all three interconnected concepts (EE, ENV and SUS). Grounded on the ideas from the human praxis dimensions (CARVALHO, 2006), it is proposed to segment the axis related to EE into three dimensions: i) EE - Knowledge; ii) EE - Ethics; iii) EE - Participation and Citizenship; a fourth dimension for the Environment, (iv ENV); one for Sustainability, (v SUS); and a last one integrating the concepts, vi) EE, ENV and SUS; each dimension involving a block of items. As no instruments for similar assessments were found in the specialized literature, it is noted that this is an exploratory proposal to validate the use of this type of questionnaire when dealing with topics related to the transition to Sustainable Societies.

The questionnaire here proposed is based on the COCTS model, an acronym in Spanish for Questionnaire on Science-Technology-Society Opinions (VÁZQUEZ-ALONSO; MANASSERO-MAS; ACEVEDO-DÍAZ, 2006). In Brazil, another questionnaire with a similar strategy is the “Students’ Views on the Nature of Science through Contextualization in Ecology” (“Visões de Estudantes sobre a Natureza da Ciência por meio da Contextualização em Ecologia”, VENCCE) (AZEVEDO; SCARPA, 2017a, 2017b). This questionnaire model, based on the degree of agreement with the multiple assertions presented to answer the same question, generates indexes that vary on a scale from (-1) to (+1), where higher values represent a greater approximation to the theoretical framework used. Each item is introduced with a brief contextualization and multiple answers or assertions, presented to complete the statement of the question.

Each of these assertions included in the instrument proposed presents answer options arranged in three categories: in the direction of the transition to sustainable societies (D); partially in the direction of sustainable societies (P, contradictory assertions or partially aligned with the theoretical framework) and in other directions, different from sustainable societies (O). Depending on the assertion category, the value chosen by the participant reflects the index, either positively or negatively (Table 1). Thus, the participants who present a ICCSA value close to (+1) show stances in line with the markers presented about the transition to Sustainable Societies (Figure 1).

The questionnaire was presented in the “Google Forms” platform, a free tool that enables remote and intuitive participation. For each assertion, the participants indicate their agreement in a scale from 1 to 9, with the possibility of abstaining from answering to indicate “I don’t know” or “I didn’t understand”. This pair of answers in considered as reasserting the premises about non-determinism in the use of quantitative instruments, as the research participants do not need to include themselves in the scale proposed. In these cases, the total value of the index cannot be compared to other participants, as the final value is based on the number of assertions per category.

Table 1
Reference for the ICCSA scores by answer categories

The assertions in the dimensions related to EE were prepared based on an academic debate on a plurality of the development of the topic in Brazil (LAYRARGUES, 2004; SILVA; CAMPINA, 2011; SORRENTINO, 2018). Specifically on the i) EE - Knowledge dimension, items on the “disciplinarization” or not of Environmental Education in formal education were included, as well as on the hierarchization of academic knowledge about other forms of knowledge, and on which topics would be specific to EE (CARVALHO, 2006; DE OLIVEIRA, 2007). In the ii) EE - Ethics questions, priority was given to representing different views of the role of EE in the transformation of society’s behaviors (CARVALHO; FARIAS; PEREIRA, 2011; GRÜN, 2010). In question iii) EE - Participation and Citizenship, different views about participation and involvement of EE in the public policy and management processes are included (JACOBI, 2003; RAYMUNDO; BRIANEZI; SORRENTINO, 2015; SORRENTINO et al., 2005).

In the discussion about the Environment (iv), different coexisting conceptions are included, which point to certain plurality of attitudes towards the theme (SAUVÉ, 2005; GONÇALVES, 2006). The concept of Sustainability (v) is included within the possibilities that are presented around the Sustainable Development, or Sustainable Societies, debate, as well as the different dimensions that comprise such debate. Finally, in the last dimension (vi), a proposal to integrate the conceptual markers presented is conceptualized, having already indicated the complex interactions in the production of these meanings.

Submission for the experts’ contribution

The process to define the questionnaire went through 5 main stages organized into two phases: elaboration of the assertions and performance of the internal reliability and validity analyses. The first phase was initiated with the elaboration of the first ICCSA proposal and its submission to a group of experts (Group A). With these answers, internal reliability and content validity were analyzed, thus enabling elaboration of the second ICCSA proposal. The second phase consisted in another group of experts (Group B) filling in the second proposal and in new internal reliability, content validity and discriminant validity analyses. The feedback and results found during the experts’ participation were considered in all these analyses (Figure 2A).

Figure 1
Example of the process to apply the questionnaire until obtaining the participant’s CISEC score. For more information about categories and scores, please consult the text included in Table 1

In the first application, one of the experts reported Internet connection problems while filling in the questionnaire in its first proposal, with the need to start answering it again. The combination of reporting loss of answers, the inconsistent answer pattern and the high frequency of “outlier” answers raised suspicions about the participant’s care when filling out the questionnaire (MEADE; CRAIG, 2012). It is possible that the respondent had used the “satisficing” answer strategy (KROSNICK, 1991), which involves using mental shortcuts to circumvent the cognitive effort required to answer a questionnaire. Consequently, this participant’s answers were discarded from the analyses that justified the adaptations between the first and second versions of the instrument. An update in the Google Forms tool implemented in July 2021 incorporated the answer autosave functionality during filling-in, which avoids data loss, as was the case with this participant.

Figure 2
Schematic representation corresponding to the development of ICCSA with distribution of the assertions in the second proposal, considering the following stages: A. Elaboration of CISEC developed in two phases and five stages. B. Distribution chart corresponding to the assertions in the second proposal of the instrument, considering the entire questionnaire (General), all six dimensions assessed (from i to vi), and all three possible assertion categories

Given the above, the participation of the Group A experts enabled coding the scores assigned to the assertions, defining the category to which each assertion belongs. Categorization of the assertions seeks to position the conceptions within an axis, explaining divergences and convergences in the debate (VÁZQUEZ-ALONSO; MANASSERO-MAS; ACEVEDO-DÍAZ, 2006). Therefore, each assertion received an agreement value, where the mean of the values indicated defines the category of the assertions in the following relation: less than or equal to 3 = “O”; between 3 and 7 = “P”; and greater than or equal to 7 = “D”.

Once the assertion categories had been established, the instrument proceeded to the second assessment phase, with other experts (Group B) and obtaining 32 answers from 80 invitations, during July 2021. Group B is comprised as follows: members of the Center for Research and Extension in Environmental Education at UFSB (Núcleo de Pesquisa e Extensão em Educação Ambiental da UFSB, NUPEEA/UFSB); experts who were unable to respond to the first validation stage; authors of “Part I - Analysis and reflections from the social indicators and control of public policies on Environmental Education” from the same publication used in the first stage; and students and former students of the Specialization in Environmental Education and Sustainability at OCA-USP. This Group is considered as less homogeneous, with some of its invitees in earlier stages of training. This choice is justified for the need of an expanded perspective, taking into account applications of the questionnaire in different populations and contexts, especially for assessing discriminant validity of ICCSA.

In the initial proposal, the mean time for answering the questionnaire was 45 minutes - which characterizes it as tiring, as pointed out by 8 out of the 10 respondents. This description was expected, as an extensive pool of items was presented, surveying possibilities to refine the final instrument. Thus, answering the second proposal of the instrument required nearly 15 minutes.

Decisions involved and validation criteria for the instrument

Given the epistemological context to develop this material, it is sought to explain the decisions and criteria involved in the creation of a quantitative instrument that is intended to be not deterministic - with subsequent more tension in the discussions in the face of the results found in the validation stages. Dialoguing with the qualitative perspectives pointed out, a broad validation is considered necessary, aiming at accuracy and reliability of the quantitative results.

Cronbach’s Alpha (Cronbach’s 𝞪) indicators were adopted to assess reliability, with 0.7 as the reference value for validity (CRONBACH; SHAVELSON, 2004; FIELD, 2009; KLINE, 1999), and item-total correlation with 0.3 as minimum value (FIELD, 2009, p. 678) for the subscales (dimensions). The exclusion of items aiming to reduce the instrument’s length and increase its reliability was accompanied by the questionnaire’s validity analysis. Both analyses were complementary, ensuring that the remaining items would comprise reliable and valid scales (Stage three, Figure 2A).

Statistical analyses should accompany a qualitative analysis in the validation and interpretation of the results (HAIR-JR; PAGE; BRUNSVELD, 2020; VÁZQUEZ-ALONSO; MANASSERO-MAS; ACEVEDO-DÍAZ, 2006). Reflections on internal consistency of the instrument are complementary to its content validity assessment, aiming to ensure “that the content of the test was pertinent and relevant to its purpose” (KLINE, 1999, p. 23). In other words, content validation involves reflections on the pertinence of item selection, within a pool, for the adequate representation of the constructs under analysis (FIELD, 2009; HAIR-JR; PAGE; BRUNSVELD, 2020). Thus, the experts’ comments were analyzed in a qualitative way following Ludke and André (1986), and supported reformulation of the instrument, in dialogue with the reliability indicators - ensuring its content validity during the process to refine the subscales.

In addition, far from exhausting the reflections on validity of the instrument, a broad assessment of its discriminant validity was proposed, which can be understood as “(...) the extent to which the construct is not related to measures that do not differ from it” (HAIR-JR; PAGE; BRUNSVELD, 2020, p. 265). Such criterion is considered as a stage to assess construct validity, which in this study reflects a theoretical differentiation between the perceptions recorded between groups A and B, as the conception under evaluation are historical and individual constructions and, thus, there should be differences in ICCSA between both groups of experts. It is acknowledged that, although the experts act in the same thematic field, they carry their own processes with them according to the underlying guidelines and conceptions of the criteria to take part in this study.

Given the above, it was sought to assess internal-quantitative reliability: Cronbach’s 𝞪 and item-total correlation (CRONBACH; SHAVELSON, 2004; FIELD, 2009); and content and discriminant validity - qualitative (KLINE, 1999; LUDKE; ANDRÈ, 1986) and quantitative (Mann-Whitney test, FIELD, 2009) approaches, respectively - interdependently and reasserting the premises on non-deterministic processes. The results of these analyses are described below, where the calculations presented were performed with the IBM SPSS v 23.0.0. software.

Results to establish and validate ICCSA

Internal reliability

Given the first phase, for the first proposal of the full questionnaire, a satisfactory reliability value was verified (𝞪=0.757), whereas the subscales for some isolated dimensions (𝞪Dim) and questions presented insufficient values, especially the EE-Knowledge dimension (𝞪Dim=-0.027) (Table 2). In this application, the other dimensions presented values that can be considered satisfactory in exploratory studies, with values close to 0.6 (HAIR-JR; PAGE; BRUNSVELD, 2020). It is warned that, for the subscales of the questions, adherence to such premise was not verified for all the questions worked on. Such an initial result was expected, as the Cronbach indicator is an index for the internal consistency of the construct and because the extensive pool of initial items, which would represent categories D, P or O, did not reflect inter-correlated assertions.

Table 2
Cronbach’s Alpha (𝞪) values for the full questionnaire (Total), dimensions (𝞪Dim) and questions (𝞪Q), according to the different phases and to the number of items in each subscale (Ni)

Seeking a significant reduction in the number of items of the instrument, an individual assessment of each item was performed. For the decision to keep the items in the following phase, the following criteria were considered: i) representation of at least two items classified as D, P or O in each dimension; ii) item-total correlation corrected for the item > 0.3 (FIELD, 2009, p. 677); iii) increase in the Cronbach’s 𝞪 value for the dimension if the item is excluded; and iv) adherence to the theoretical premises about the construct unit. It is important to note the articulation between these criteria for the final decision, as the intention is to avoid a mechanical procedure in these decisions.

Reflections on these indicators, together with the participants’ qualitative assessment, led to the elaboration of a new set of assertions, reducing the questionnaire from 103 to 43 assertions (Figure 2B). In the first validation phase, the Cronbach’s 𝞪 values were recalculated based on Group A and only considering the 43 assertions that were kept and, in the second phase, the new version of the instrument was applied to Group B, also calculating the internal consistency values (Table 2). Although the Cronbach’s 𝞪 value of Group A for the full questionnaire was lower in the second proposal, the 𝞪Dim values were more satisfactory, with no negative value and only one below 0.6 (i. EE-Knowledge). Although the number of items in the scale is directly proportional to Cronbach’s 𝞪 value, even with the exclusion of 60 items, the values for the full questionnaire increased from 0.757 in the first phase for Group A to 0.928 in the application with Group B.

The 𝞪 values obtained in the second proposal raised reflections on the sensitivity of the indicator to the population sampled, as the same instrument generates different values with Group A and with Group B, as well as possible redundancies of the instrument due to the value >0.900 (FIELD, 2009), obtained with Group B. Regarding this point, it is noted that the high 𝞪 value was restricted to the full questionnaire, not repeating itself in the dimensions; on the other hand, (HAIR-JR; PAGE; BRUNSVELD, 2020) interpret 𝞪 values >0.950 as redundancy.

In relation to the Group B subscales and based on questions 1 and 2 for each dimension of the first proposal, ideal 𝞪Dim (0.722< 𝞪Dim<0,831) values were verified, except for dimension iv ENV (𝞪Dim=0.493). Such a result is justified in the occurrence of certain diversity regarding ENV conceptions, as expressed by one of the respondents:

This block is probably the one that had the greatest difficulty due to the widespread use of the term “Environment”, precisely because it is more of a social representation than a hegemonic concept; it is polysemic. (...) For each of the answers, it is necessary to take into account what I understand by Environment (Expert 8).

Based on these results, the reliability assessment of the instrument was considered as sufficient for broad application. As Cronbach’s indicator has also already been used as a measure of the construct unidimensionality (Field, 2009), a low internal consistency value can be accepted given the multiple interactions of the concepts addressed, as in the case of iv ENV. In other words, it is understood that such a result is a product of the underlying complexity to the processes to construct its conceptions. Thus, accepting a value below the one considered ideal takes us back to the premises about complexity and non-determinism and about the polysemy of the concepts, therefore constituting an expected element in the construction of representations of the antagonism in the constructs evaluated, especially in relation to the interactions with the conceptions about ENV (SAUVÉ, 2005).

Content validity

First application (Grupo A)

In the first phase, in addition to the general comments about the extended time to answer the entire questionnaire, a common note among the comments was about the importance of reflecting on the concepts. As already pointed out, it is part of the multiple-choice model questionnaire technique to present contradictory or partially correct assertions according to the theoretical framework, which caused strangeness in some participants: “The exercise about reflecting on these questions was very interesting. Some assertions are not very clear, as they involve conflicting concepts (...)” (Expert 4). Other experts point out how some assertions can be confusing and depend on the respondent’s perception of the concepts worked on:

Many of the questions above generate double interpretations depending on the analysis criterion used to answer them. This is because teaching ecology and recycling techniques are not enough and are far from being the solution, as they reinforce a naturalistic, conservative and normative-prescriptive perspective of Environmental Education (EE). However, they are important and necessary at the same time, not being exclusive of critical, emancipating, transforming and libertarian or even revolutionary EE. (Expert 8)

One of the concerns with validation was how the experts would behave when transforming these concepts into numbers on a scale. It was expected that, when being presented the contradictory assertions, the respondents would categorize them as “Partially”. As indicated, they understood this idea:

For example, the last question (...) As I see it, it has some balance. I don’t agree with the first phrase, but I do agree with the second. This creates some difficulty answering. That’s why I give it a 5. But if they were separated, the first phrase would be 1 and the second, 9. (...) (Expert 8, emphasis by the respondent).

In this case, with the assertion categorized as P, the value of ‘5’ generates the highest score for the index, which follows the objective in the use of the multiple-choice model questionnaire (Table 1). Of all 43 assertions selected, only two were left unanswered, by two different participants, and both from the P category.

Second application (Grupo B)

In the second phase, only three respondents pointed out considering that the questionnaire was too long. The few comments about length of the questionnaire proved to be successful in the objective of reducing the number of items in the instrument, preserving its validity and consistency. It is considered that such goal was achieved since, even after answering a shorter questionnaire, the participants showed the need for a reflective perspective to interpret the assertions.

As expected, some experts pointed out that the assertions can generate different interpretations: “Many questions are dubious, they allow for different interpretations depending on how the concepts are understood” (Expert 32), as in the first phase. Also in relation to the doubts while filling in the questionnaire, as was the case in the first phase, the participants showed doubts about how to answer the assertions from category P: “In some questions (I) agreed with half of them and, generally, the contradiction started after the comma. In this situation, it was difficult for me to answer that I hardly agreed” (Expert 25).

Only one respondent behaved differently when finding contradictory assertions, leaving 9 assertions unanswered (4 D, 4 P, 1 O):

The fact that there is more than one argument, many times with different meanings, in several of the assertions presented as “questions”, led me to partially or hardly agree with much of what was expressed in the form.(...) I left several items unanswered, either because I understood that the period consisted of phrases that contradicted each other or because I did not understand the meaning of the assertion as a whole (Expert 37).

Also in relation to the unanswered items, only two experts did not answer some assertions (one, 2 P and 2 D; and the other, 1 P and 2 O), with only one case in which the same assertion was left unanswered by two experts.

ICCSA values and discriminant validity

The multiple-choice model with different dimensions allows calculating the values of these dimensions in different clusters. For the general index of the questionnaire (ICCSAg), Group A obtained a significantly higher value than Group B (U=12, p<0.01), thus evidencing Discriminant Validity (Table 3). The scores obtained in the categories (D, P and O) also suggest a significant difference between both applications, where the second group was able to get closer to the stances towards sustainable categories in category D (U=50.5, p=0.04), whereas in categories P and O the first group obtained a significantly higher score (U=21.5, p<0.01 and U=53.5, p=0.04, respectively).

Table 3
ICCSA scores as a standard deviation measure, as well as the value of the Mann-Whitney Test statistics, in addition to its p _value . The scores are also presented with a standard deviation measure for the isolated dimensions. The highest values for each answer category between the different groups are highlighted (in bold)

	ICCSAg	D	P	O
Group A	0.47 +- 0.05	0.69 +-0.12	0.15 +- 0.24	0.57+-0.11
Group B	0.24+-17	0.79 +-0.14	-0.28+-0.28	0.22+-0.44
	U= 12 p= 0.00*	U =50.5 p= 0.04*	U=21.5 p = 0.001*	U=53.5 p= 0.04*
ICCSA calculated values by dimension
Dimensions		Group A	Group B
i. EE Knowledge		0.45+-0.23	0.32+-0.34	U=104 p= 0.43
ii. EE Participation		0.34+-0.14	0.11+-023	U= 53 p = 0.01*
iii. EE Ethics		0.54+-0.12	0.34+-0.24	U= 62,5 p =0.01*
iv. ENV		0.53+-0.22	0.38+-0.26	U=89.5 p=0.11
v. SUS		0.43+-0.20	0.11+-0.21	U=32 p =0.00*
vi. EE, ENV e SUS		0.52+-0.08	0.17+-0.28	U=37,5 p= 0.00*

• Fonte: The authors
• Notes: *Statistically significant difference by the Mann-Whitney test
• D = In the direction of the transition to sustainable societies; P = Partially in the direction of sustainable societies; O = In other directions, different from sustainable societies. EE = Environmental Education; ENV = Environment; SUS = Sustainability.

These divergences between the groups can signal a greater tendency to agree with D in Group B, as these participants categorized more assertions as “9” or close to “9”, thus increasing the D score and reducing the expression of P and O. For both groups, both in ICCSAg and in each of the dimensions, the P assertions were those that presented the lowest values, being the only category where negative values were observed. This category results in lower values the more extreme the stances in relation to agreement or non-agreement with the P assertions. The low values in this category can reflect the participants’ difficulty when finding contradictory assertions, as reported in their testimonies. In addition to that, in case some participant presents a “satisficing” behavior, especially the answers from category P will show low values (KROSNICK, 1991; MEADE; CRAIG, 2012).

The higher scores obtained by Group A in relation to Group B were maintained in the different dimensions, only i) EE-Knowledge (U=104, p= 0.43) and iv) Environment (U=89.5, p=0.11) did not present significant differences between both groups. These positive results suggest that, even with different compositions, both groups can converge into the political-pedagogical macro tendency of critical EE (LAYRARGUES; LIMA, 2011). In addition, the stance in one these large strands also dialogues with the Environment conceptions (SILVA; CAMPINA, 2011).

Although the same was expected for other dimensions, it is considered that dimensions ii. EE - Ethics and iii. EE - Participation and Citizenship involved values not necessarily explicit or homogeneous in the field of critic EE. The same happens with the results of the Sustainability (v) and Integration (vi) dimensions, which encompass concepts such as views of socioeconomic models and environmental justice (CARVALHO, 2008). Even if the ideas about Sustainability are diverse and polysemic, the highest difference between the groups in these dimensions suggests a structuring distinction between the transition to Sustainable Societies and the hegemonic view of Sustainable Development.

Final considerations about the instrument

The instrument developed in this paper is the first quantitative tool that seeks to identify stances in relation to the transition to Sustainable Societies. In addition to that, the behaviors when answering the questionnaire can be analyzed both in a broad way and in each of the dimensions presented and their inter-relations. The results should be interpreted with caution since, in addition to the errors inherent to filling out any measuring instrument, there is a limitation in the possible answers, which are previously established by those who elaborate them. The thoughtful process during the elaboration and validation of this instrument minimizes these problems, enhancing the advantages of its use, such as its large-scale application.

The analysis of the experts’ comments favors content validity of the questionnaire, and the Cronbach’s Alpha and item-total correlation measures contribute to the reliability of the instrument. Additionally, the participants’s performance during the second phase revealed a distinction in the indices obtained between the different responding groups, which denotes the discriminant validity of the instrument.

It is considered that the questionnaire model chosen can address the topics proposed. By involving these dimensions, the possibility of an analysis by categories or for each assertion presented is created, so that the questionnaire can be adapted or reassessed depending on the question to be answered. The assertions adopted in this paper had a bibliographic survey about the topics as their starting point. Although new questions may be included and removed based on systematic reviews or on primary data collected, a validation stage is recommended for the new future versions.

The validation process based on the comments and the indicators can be improved by monitoring the experts while they answer the instrument, creating the opportunity to question the values assigned to each assertion, in order to elucidate how the respondents understood the scale. Due to the results found, new applications are recommended with validation stages, especially with non-expert populations. The need for special caution in relation to the P assertions is noted, as they generated doubts when answering them, which can even reflect in the index values and generate noise in data interpretation. Finally, the succinct instructions provided about how to answer the scale also represented a limitation of the process described: when presenting the questionnaire, it is suggested to encourage the participants to explore the spectrum presented more in depth, reducing the number of automated answers and doubts about how to fill it. Assisted application of the questionnaire can minimize these problems, allowing the respondents to solve their doubts.

Using ICCSA to assess the conceptions of a given population is contextualized in the field of development of diagnostic instruments to support educational actions, explaining particularities in complex conceptions, both between and within groups. The questionnaire can also be applied before and after an educational intervention, and may indicate possible changes in the participants’ stances due to the training process stimulus. As is the case with the surveys conducted in the STS (Science-Technology-Society) and Nature of Sciences areas, it is inferred that the indicator can be used as a diagnosis for critical gaps or points, both for educational processes and in the implementation of public policies. In synthesis, ICCSA can guide strategic, contextualized and dialogical actions that contribute to the transition to Sustainable Societies.

ICCSA is available to any researcher who might wish to use or modify the instrument. Distributed in their categories and dimensions, all 43 assertions that comprise this proposal are listed in Santos (2022). In order to receive other contents pertinent to the application of the instrument, please send an email message to the lead author.

Acknowledgements

The authors are grateful for the support of Public Notice No. 19/2020/PRPGI/IFBA: “Support and Strengthening of IFBA Stricto Sensu Postgraduate Programs” for enabling the submission and translation of this article.

Referências

• ACOSTA, Alberto. O Buen Vivir: uma oportunidade de imaginar outro mundo. In: BARTELT, Dawid Danilo (org.). Um campeão visto de perto - Uma análise do Modelo de Desenvolvimento Brasileiro. Brasília/DF: Heinrich-Böll-Stiftung. Série Democracia,2012, p. 198-216.
• AZEVEDO, Nathália Helena; SCARPA, Daniela Lopes. Um levantamento em larga escala das concepções de Natureza da Ciência de graduandos de biologia brasileiros e os possíveis elementos formativos associados. Ensaio Pesquisa em Educação em Ciências, Belo Horizonte, v. 19, p. 1-28, 2017a.
• AZEVEDO, Nathália Helena; SCARPA, Daniela Lopes. Decisões envolvidas na elaboração e validação de um questionário contextualizado sobre Concepções de Natureza da Ciência. Investigações em Ensino de Ciências, Porto Alegre, v. 22, n. 2, p. 57, 2017b
• CARVALHO, Luiz Marcelo de. A temática ambiental e o processo educativo: dimensões e abordagens. In: CINQUETTI, H. C.; LOGAREZZI, A. (org.). Consumo e Resíduo: Fundamentos para o trabalho Educativo. São Carlos: EdUFScar, 2006. p. 1-19.
• CARVALHO, Isabel Cristina de Moura. Educação para Sociedades Sustentáveis e ambientalmente justas. Revista Eletrônica do Mestrado em Educação Ambiental, Rio Grande, v. especial, p. 46-55, 2008.
• CARVALHO, Isabel Cristina de Moura; FARIAS, Carmen Roselaine; PEREIRA, Marcos Villela. A Missão “ecocivilizatória” e as novas moralidades ecológicas: A Educação ambiental entre a norma e a antinormatividade. Ambiente & Sociedade, São Paulo, v. 14, n. 2, p. 35-49, 2011.
• CRONBACH, Lee J.; SHAVELSON, Richard J. My Current Thoughts on Coefficient Alpha ans Successor Procedures. Los Angeles: Center for the Study of Evaluation (CSE) National Center for Research on Evaluation, Standards, and Student Testing (CRESST) Graduate School of Education & Information Studies University of California, 2004.
• DIEGUES, Antonio Carlos. Sociedades e comunidades sustentáveis. 2003. Disponível em: <http://nupaub.fflch.usp.br/sites/nupaub.fflch.usp.br/files/color/comsust.pdf>. Acesso em: 19 jun. 2022.
  » http://nupaub.fflch.usp.br/sites/nupaub.fflch.usp.br/files/color/comsust.pdf
• DE OLIVEIRA, Haydée Torres. Educação ambiental - ser ou não ser uma disciplina: essa é a principal questão?! In: MELLO, Soraia Silva De; TRAJBER, Rachel (org.). Vamos cuidar do Brasil: conceitos e práticas em educação ambiental na escola. Brasília: Ministério da Educação, Coordenação Geral de Educação Ambiental: Ministério do Meio Ambiente, Departamento de Educação Ambiental : UNESCO, 2007. p. 103-114.
• FIELD, Andy. Descobrindo a estatística usando o SPSS-5. Porto Alegre: Penso, 2009.
• GARRIDO-PEÑA, Francisco. Sobre la epistemología ecológica. In: GARRIDO, Francisco et al. (org.). El paradigma ecológico en las ciencias sociales. Barcelona: Icaria:Antrazyt, 2007. p. 31-53.
• GONÇALVES, Carlos Walter Porto. Os (des) caminhos do meio ambiente. 14 ed. São Paulo: Contexto, 2006.
• GRÜN, Mauro. Ética e educação ambiental: a conexão necessária. 13. ed. Campinas: Papirus, 2010.
• GUIMARÃES, Roberto Pereira; FEICHAS, Susana Arcangela Quacchia. Desafios na construção de indicadores de sustentabilidade. Ambiente & Sociedade, São Paulo, v. 12, n. 2, p. 307-323, 2009.
• HAIR-JR, Joe F.; PAGE, Michael.; BRUNSVELD, N. Essentials of business research methods. 4. ed. Abringdom: Routledge, 2020.
• IRVING, Marta de Azevedo. Sustentabilidade e O futuro que não queremos: polissemias, controvérsias e a construção de sociedades sustentáveis. Sinais Sociais, Rio de Janeiro, v. 6, n. 26, p. 13-38, 2014.
• JACOBI, Pedro. Educação ambiental, cidadania e sustentabilidade. Cadernos de Pesquisa, São Paulo, n. 118, p. 189-206, 2003.
• KLINE, Paul. The Handbook of psychological testing. 2. ed. Abingdon: Routledge, 1999. DOI: 10.1016/s0191-8869(96)90047-1.
  » https://doi.org/10.1016/s0191-8869(96)90047-1.
• KROSNICK, Jon A. Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, Nova Jersey, v. 5, n. 3, p. 213-236, 1991.
• LANKSHEAR, Colin; KNOBEL, Michele. Pesquisa pedagógica: do projeto à implementação. Porto Alegre: Artmed, 2008.
• LAYRARGUES, Philippe Pomier (ORG.). Identidades Educação Ambiental Brasileira. Brasília: Ministério do Meio Ambiente - Diretoria de Educação Ambiental, 2004.
• LAYRARGUES, Philippe Pomier; LIMA, Gustavo Ferreira da Costa. Mapeando as macro-tendências político-pedagógicas da educação ambiental contemporânea no brasil. VI Encontro “Pesquisa em Educação Ambiental” A Pesquisa em Educação Ambiental e a Pós-Graduação no Brasil., Ribeirão Preto, p. 1-15, 2011.
• LEFF, Enrique. Racionalidade ambiental: a reapropriação social da natureza. Rio de Janeiro: Civilização Brasileira, 2006.
• LIMA, Gustavo Ferreira da Costa. Educação ambiental crítica: Do socioambientalismo às sociedades sustentáveis. Educação & Pesquisa, São Paulo, v. 35, n. 1, p. 145-163, 2009.
• LOUREIRO, Carlos Frederico Bernardo. Sustentabilidade e educação ambiental: controvérsias e caminhos do caso brasileiro. Sinais Sociais, Rio de Janeiro, v. 1, n. 1, p. 39-72, 2014.
• LUDKE, Menga; ANDRÉ, Marli E. D. A. Pesquisa em Educação: Abordagens Qualitativas. São Paulo: EPU, 1986.
• MEADE, Adam W.; CRAIG, S. Bartholomew. Identifying careless responses in survey data. Psychological Methods, Washington, v. 17, n. 3, p. 437-455, 2012.
• MEIRA, Pablo; SATO, Michèle. Só os peixes mortos não conseguem nadar contra a correnteza. Revista de Educação Pública, Cuiabá, v. 14, n. 25, p. 1-19, 2005.
• MORIN, Edgar. Ciência com Conscienciência. 8. ed. Rio de Janeiro: Bertrand Brasil, 2005.
• MORIN, Edgar. La Epistemología de la complejidad. In: GARRIDO, Francisco et al. (org.). El paradigma ecológico en las ciencias sociales. Barcelona: Icaria:Antrazyt, 2007.
• MORIN, Edgar; CIURANA, Emilio-Roger; MOTTA, Raúl Domingo. Educar na Era Planetária: O pensamento complexo como “Método” de aprendizagem no erro e na incerteza humana. São Paulo: Cortez, 2003.
• PAULISTA, Geralda; VARVAKIS, Gregório; MONTIBELLER-FILHO, Gilberto. Espaço emocional e indicadores de sustentabilidade. Ambiente & Sociedade, São Paulo, v. 11, n. 1, p. 185-200, 2008.
• RATTNER, Henrique. Sustentabilidade - uma visão humanista. Ambiente & Sociedade, São Paulo, n. 5, p. 233-240, 1999.
• RAYMUNDO, Maria Henriqueta Andrade; BRANCO, Evandro Albiach; SORRENTINO, Marcos. Avaliação e monitoramento de políticas públicas de educação ambiental no Brasil: transição para sociedades sustentáveis. Piracicaba: MH-Ambiente Natural, 2019.
• RAYMUNDO, Maria Henriqueta Andrade; BRIANEZI, Thaís; SORRENTINO, Marcos (ORG.). Como Construir Políticas Públicas De Educação Ambiental Para Sociedades Sustentáveis? São Carlos: Diagrama Editorial, 2015.
• SANTOS, Boaventura de Sousa. Um discurso sobre as ciências. 5. ed. São Paulo: Cortez, 2008.
• SANTOS, Leandro Fernandes Antonio. Unidades de Conservação e Sociedades Sustentáveis: A Educação Ambiental desenvolvida nas UC da Costa do Descobrimento - BA. Dissertação (Mestrado em Ciências e Tecnologias Ambientais) - Universidade Federal do Sul da Bahia; Instituto Federal de Educação, Ciência e Tecnologia da Bahia, Porto Seguro, 2022. Disponível em: < https://sig.ufsb.edu.br/sigaa/verArquivo?idArquivo=727550&key=b7e7b9437998669c30d5e6d4f30fb0e4>. Acesso em: 19 jun. 2022.
  » https://sig.ufsb.edu.br/sigaa/verArquivo?idArquivo=727550&key=b7e7b9437998669c30d5e6d4f30fb0e4
• SARTORI, Simone; DA SILVA, Fernanda Latrônico; DE SOUZA CAMPOS, Lucila Maria. Sustainability and sustainable development: A taxonomy in the field of literature. Ambiente e Sociedade, São Paulo, v. 17, n. 1, p. 1-22, 2014.
• SAUVÉ, Lucie. Educação ambiental: possibilidades e limitações. Educação e Pesquisa, São Paulo, v. 31, n. 2, p. 317-322, 2005.
• SILVA, Rosana Louro Ferreira; CAMPINA, Nilva Nunes. Concepções de educação ambiental na mídia e em práticas escolares : contribuições de uma tipologia. Pesquisa em Educação Ambiental, Rio Claro, v. 6, n. 1, p. 29-46, 2011.
• SORRENTINO, Marcos. Educação ambiental e políticas públicas : conceitos, fundamentos e vivências. 2. ed. Curitiba: Appris, 2018.
• SORRENTINO, Marcos; TRAJBER, Rachel; MENDONÇA, Patrícia; FERRARO JUNIOR, Luiz Antonio. Educação ambiental como política pública. Educação e Pesquisa, São Paulo, v. 31, n. 2, p. 285-299, 2005.
• TRAJBER, Rachel; SORRENTINO, Marcos. As políticas de educação ambiental do Estado brasileiro: transformações socioambientais para a sociedade sustentável. In: TASSARA, Eda (org.). Mudanças Climáticas e Mudanças Socioambientais Globais: reflexões sobre alternativas de futuro. Brasília: UNESCO, 2006. p. 155-168.
• VÁZQUEZ-ALONSO, Ángel; MANASSERO-MAS, María Antonia; ACEVEDO-DÍAZ, José-Antonio. An analysis of complex multiple-choice science-technology-society items: Methodological development and preliminary results. Science Education, Washington, v. 90, n. 4, p. 681-706, 2006.

• Note

  Acknowledging the political importance of language, we shall adopt a neutral or female gender language as generic whenever possible.

Publication Dates

• Publication in this collection
  17 July 2023
• Date of issue
  2023

History

• Received
  01 Feb 2022
• Accepted
  21 Nov 2022