Analyzing a test as genre: Critical Genre Analysis of the English language ENEM¹

Analisando um teste como gênero: Análise Crítica de Gênero do ENEM de língua inglesa

ABSTRACT

This paper proposes a Critical Genre Analysis (CGA) (Motta-Roth, 2006; 2008a; 2008b) of the English Language test on the Brazilian National High School Exam (EL-ENEM), to, ultimately, discuss the language conception that underlies this test. The corpus of the analysis was composed of eight samples of the EL-ENEM genre. The results point out that EL-ENEM items provide the test takers with a contextualization about their reference texts. This contextualization, in most items, provide information regarding the genres of the reference texts. The items, then, demand the test takers to identify different types of information in the reference texts. The types of information that are demanded the most refer to the text’s content, objective, and topic. This way, the test takers must activate knowledge related to Semantics and Pragmatics, Register (context of situation) and Genre (context of culture), indicating a direction to the view of language as genre, adopted by the official documents that guide ENEM’s design.

Keywords:
Critical Genre Analysis; English Language Reading Test; ENEM

RESUMO

Este trabalho propõe uma Análise Crítica de Gênero (ACG) (Motta-Roth, 2006; 2008a; 2008b) do teste de Língua Inglesa no Exame Nacional do Ensino Médio (ENEM-LI), para, em última instância, discutir a concepção de linguagem que subjaz esse teste. O corpus de análise foi composto de oito exemplares do gênero ENEM-LI. Os resultados apontaram que os itens do ENEM-LI fornecem ao examinando uma contextualização sobre seus textos-base. Essa contextualização, na maioria dos itens, fornece informações a respeito dos gêneros dos textos-base. Os itens, então, demandam que os examinandos identifiquem diferentes tipos de informação nos textos-base. Os tipos de informação que são mais demandados referem-se ao conteúdo, objetivo e tópico do texto. Dessa forma, os examinandos precisam ativar conhecimentos relacionados à Semântica e Pragmática, ao Registro (contexto de situação), e ao Gênero (contexto de cultura), indicando um direcionamento para a visão de linguagem como gênero, adotada pelos documentos oficiais que orientam a formulação do ENEM.

Palavras-chave:
Análise Crítica de Gênero; Teste de Leitura em Língua Inglesa; ENEM

1. Introduction

In Brazil, the National High School Exam (Exame Nacional do Ensino Médio, ENEM) assesses students’ overall performance during high school. In addition, the student’s scores on this test can be adopted to apply to public and private universities and to social programs that facilitate the access to these universities, such as University for All (Prouni) and Student Financing Program (FIES), among others. The rich variety of opportunities that may emerge through the results of the exam demonstrates its national significance, which, consequently, influences the development of research on the topic. ENEM is organized in the following four fields of knowledge: Human Sciences, Languages, Mathematics and Nature Sciences.

The focus of the present study is the Languages field on ENEM, more specifically, the English Language test, hereafter, EL-ENEM. Concerning this test, previous studies have analyzed the language and reading conceptions that underlie its questions (Rauber, 2012); its backwash effect (Blanco, 2013; Avelar, 2015; Oliveira, 2019); the inference processes required to answer its questions (Azevedo et al., 2018); among others. These previous studies provide an overview of EL-ENEM, mainly in relation to its backwash effects, such as its influences in school contexts. However, when it comes to the features of the test itself, these studies analyze a small sample of tests (no more than three tests), and do not provide detailed explanation about its macro and microstructure and contextual features, that is, do not analyze the test as a genre. Thus, this present study concentrates on the internal features of the test, that is, its macro and microstructure, analyzing a larger sample of eight tests.

Therefore, the objective of this study is to analyze EL-ENEM as a genre, using Critical Genre Analysis (CGA) (Motta-Roth, 2006; 2008a; 2008b) theoretical and methodological approach, in order to, ultimately, delineate the language conception that underlies it. CGA’s theoretical and methodological rationale is presented in the following section.

2. Critical Genre Analysis

Critical Genre Analysis (CGA) can be described as a theoretical and methodological approach to genre analysis, which “sees genre as language use constitutive of and constituted by social activities, as a relatively stable communicative event, associated with specific sociocultural contexts” (Motta-Roth; Heberle, 2015, p. 2). This approach was developed by different authors, including Meurer (2002, 2005), Motta-Roth (2006, 2008a, 2008b), Bonini (2010) and Bhatia (2015). Since these authors may approach genre analysis in rather different ways, we consider here Motta-Roth’s perspectives (2006, 2008a, 2008b), and the understanding that this approach necessarily includes textual and contextual analysis; and combines Sociorhetoric, Systemic Functional Linguistics (SFL) and Critical Discourse Analysis (CDA) theories.

CGA draws on several concepts and analytical tools from those three theories. In this section, we focus on the ways those theories contribute to the analysis of the genre EL-ENEM. From Sociorhetoric, which includes Genre Analysis - Motta-Roth (2008a, p. 350) uses this term to refer to the instrumental perspective of genre analysis, such as in Swales (1990) - and English for Specific Purposes (ESP) studies, we consider the analytical design, reviewed by Bawarshi and Reiff (2013, p. 80), of investigating the genre through its communicative purpose, rhetorical moves and steps and linguistic and textual features that realize moves and steps. That is, this approach to genre analysis considers a “given schematic structure found in the text which has specific rules for form and context of use in relation to the function that it performs in the genre” (Motta-Roth, 1995, p. 16). This schematic structure is organized in rhetorical moves: “discoursal or rhetorical unit that performs a coherent communicative function in a written or spoken discourse”, by means of rhetorical steps (Swales, 2004, p. 228-229). Thus, to carry out the analysis of the schematic structure, it is necessary to investigate the content and the form of each linguistic element that performs a specific function on the text.

SFL is a linguistic theory that also stems from the text’s function to analyze its linguistic elements. To accomplish this purpose, it provides several analytical tools, such as the Systemic Functional Grammar (Halliday; Matthiessen, 2014), which aims to explain how language organizes itself in a system, from which users can choose linguistic elements to represent their experiences, to establish social relations and to linguistically organize the text. It is based upon the idea that the choice made by the user is one within several possibilities that are available in the system. In SFL, the language system is represented as a stratification system (Martin, 1992; Halliday; Matthiessen, 2014, p. 24), which was adapted by CGA (for instance, Hendges, 2005; Motta-Roth, 2008a; 2008b). This concept represents genre as social event, in which people participate discursively through linguistic elements in different levels - phonology/graphology, lexicogrammar, semantics/pragmatics, text, discourse - (Motta-Roth, 2008b, p. 247) which work together to accomplish communicative purposes. Figure 1, adapted from Motta-Roth (2008a, 2008b), represents language as genre in six interrelated strata.

Figure 1
The stratification of language

As well as CGA, which is both a theory and a method for genre analysis, the concept of the stratification of language is used both to represent language and to support genre analysis in terms of the different contextual and textual elements that are involved in a communicative activity. In the case of the present study, we do not use the stratification of language to analyze the genre itself (EL-ENEM), but to discuss the linguistic elements, related to each stratum, that must be activated by the users of the genre, that is, the EL-ENEM test takers.

To carry out this investigation, we analyzed EL-ENEM items with the objective of identifying what specifically the test taker must understand in the reference text to answer the question (Do they need to understand the objective of the text? Its topic? Its organization in terms of structure?). Each linguistic element was then related to a language stratum, using Chart 1 as a reference. The chart was designed based on Motta-Roth (2008b), Gouveia (2009) and Halliday e Matthiessen (2014).

Chart 1
Linguistic elements related to the language strata

Language Stratum	What does the test taker must understand/identify on the reference text?
Graphology*	The language’s writing systems.
Lexicogrammar	The actions represented in the text, as well as the participants and the circumstances involved in these actions (through verbal, noun and adverbial groups); and/or the mood used in the text (indicative - declarative or interrogative, and imperative); and/or the textual structure (the position of each information in the text).
Semantics and Pragmatics	The linguistic representations, the text’s subject and/or meaning; and/or the exchanges between author and reader established by the text; and/or the text’s message.
Register	Language uses and linguistic resources typically related to specific contexts.
Genre	The relation among the linguistic system, the context, and the text’s social and communicative objectives.
Discourse	The power and/or hierarchical relations established by the text; the discourses (political, religious, etc.) permeated by the text; the ideologies presented in the text.

• * Since the present study investigates a reading test, the analysis did not include Phonology aspects.
• Source: Authors

According to Halliday and Matthiessen (2014, p. 25), the most concrete stratum (Graphology) realizes the next stratum (Lexicogrammar), which is less concrete. For instance, in order to represent actions through verbal groups, it is necessary to use the language’s writing or sound system. Thus, if the test taker must identify Discourse aspects in the reference text, they must activate aspects of all of the other strata, because the Discourse stratum realizes itself in the Genre stratum, which realizes itself in the Register stratum, and so forth.

Concluding the considerations concerning Sociorhetoric and SFL, which, in the present study, support the genre’s textual analysis, we now present the contributions of CDA, which supports the contextual analysis.

According to Fairclough (1992, p. 4; 63), discourse, that is, “language use as a form of social practice”, and discourse analysis are three-dimensional, because every text is also a discursive practice and a social practice. Consequently, the three dimensions of discourse are ‘text’, ‘discursive practice’ and ‘social practice’. In this conception of discourse and discourse analysis,

the ‘text’ dimension attends to language analysis of texts. The ‘discursive practice’ (...) specifies the nature of the processes of text production and interpretation. The ‘social practice’ dimension attends to (…) the institutional and organizational circumstances of the discursive event (Fairclough, 1992, p. 4).

In this perspective it is possible to understand the importance of the contextual analysis in a genre analysis, since the textual dimension and analysis is one of the three dimensions that must be considered. The contextual analysis in CDA’s perspective should, ideally, include an ethnographic account (for instance, Fairclough, 1985, p. 759; 1992, p. 226-227). However, a detailed analysis considering the documental dimension may also be regarded as the contextual analysis (exemplified in Fairclough, 2001, p. 127-128). It is in this latter perspective that the present study is inspired in CDA. Therefore, the next section presents a summary of the documental analysis of previous studies concerning EL-ENEM that are relevant for the present investigation.

3. EL-ENEM

Among the previous studies that are part of the contextual analysis, the investigation presented in Rauber’s master’s thesis (2012) was particularly useful for the present analysis. In this study, the author discussed the language and reading conceptions that underlies EL-ENEM, besides describing items from two editions of the EL-ENEM. In this analysis, Rauber (2012, p. 62 -67) indicated problems in the items’ design, including unclear question’s instructions with unnecessary information; not-plausible response alternatives; and no correct response alternative (in one item).

In relation to the reading skills tested in EL-ENEM, Rauber (2012, p. 69) indicated that most items evaluate the global and local comprehensions, that is, the abilities of identifying the main idea and specific information in the reference texts. In addition, the items do not explore aspects regarding the reference texts’ genres, which only represent pretexts for testing grammatical and lexical knowledges (Rauber, 2012, p. 80). This way, Rauber (2012, p. 81-82) points out that, in EL-ENEM, reading is considered a passive process of extracting meanings and information from the texts, as the bottom-up reading approach suggests. This reading conception is associated with a fragmented understanding of language: the opposite of what the official guidelines for ENEM’s design suggest, in which the texts’ social uses and functions are prioritized, conducting to the understanding of reading as literacy (Rauber, 2012, p. 82,85).

The second study that supported the present analysis was Blanco’s master’s thesis (2013), which investigated the social impact of the inclusion, since 2010, of additional languages on the test. The participants of this study were two EL teachers and their students from a public high school and a free preparation course for college entrance exam. In both contexts, Blanco (2013, p. 74, 77) identified positive influences of the exam on teaching practices, in terms of inclusion of additional teaching materials related to ENEM, rereading of official documents that guide teaching, and change of classes’ focus: before, the focus was on grammar; then, classes started to focus on processes of reading and interpreting texts and genres. In the same way, positive influences could be identified on students’ practices: they demonstrated more interest and engagement on classes, since they had an immediate context to use the language (Blanco, 2013, p. 75).

In relation to the EL on the exam, students and teachers considered the presence of different genres a positive aspect, which indicates a contextualized view of reading practices (Blanco, 2013, p. 88). However, negative points were also highlighted: one of them refers to the number of questions, considered limited by the participants of the study (five EL questions from 180 in the Languages test) (Blanco, 2013, p. 89). In this context, Blanco (2013, p. 92, 97) agrees that the test could explore more questions per reference text to become more reliable (since each reference text is followed by one and only question).

Finally, the third study that has contributed with the present analysis was Avelar’s PhD dissertation (2015). Like the previous studies presented in this section, Avelar (2015) explored the language and reading conceptions that underlie EL-ENEM in three editions of the exam, and its backwash effect. As well as Rauber (2012), Avelar (2015, p. 58) found out that EL-ENEM items do not match the language and reading conceptions prioritized in the official guidelines, because they evaluate, mostly, reading sub-skills, such as the identification, recognition and inference of information in the reference texts. This way, according to Avelar’s results (2015, p. 58), in EL-ENEM reading is understood in line with the top-down approach, in which the reader, who has the central role in the process, formulates hypothesis about the topic of the text and verifies them during the reading.

Avelar (2015) is also in line with the discussion presented by Rauber (2012) in relation to the role of genres in the test. The author (2015, p. 61-62) indicates that the questions’ instructions only mention the genres, but do not explore aspects concerning their contexts of use or production. At this point, it is important to mention that the present study draws a different conclusion about the role of genres and their contextual features in the test, which are explored by the items’ contextualization as a way of facilitating the test taker’s process of reading. These differences can be explained, as it was indicated in the Introduction section, because of the greater number of tests analyzed by the present study (see the analysis presented in the Results and Discussion section).

To conclude, with respect to the investigation of the backwash effect of the exam, Avelar’s study (2015) concentrated on two contexts of investigation: a public and a private high school, including one EL teacher from each school. Differently from Blanco’s results (2013), Avelar (2015, p. 148-149) found out that, in both contexts of investigation, ENEM does not directly influence teaching practices. In the context of the private high school, the teacher explains that students aim to be prepared for other admission tests for private universities, while the teacher of the public high school explains that it is not the objective of the school to prepare students neither for ENEM nor other admission tests, even though the teacher reports exploring reading activities related to the exam (Avelar, 2015, p. 148-149, 181-183). It is in the reading skill, privileged in both teaching contexts, that the backwash effect of the exam could be identified, although it is not assigned to ENEM, but to the teaching materials (Avelar, 2015, p. 209). Those three previous studies provided an important background for the present study, which analyzed the same study object with different analytical categories, presented in the following section.

4. Methodological Procedures

In this section, we firstly present the procedures to collect and define the corpus of the study, and, then, we describe the analytical procedures and categories to carry out the investigation. All ENEM editions are available for download on the National Institute for Educational Studies and Research “Anísio Teixeira” (Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira, INEP) website (Brasil, 2015a). The scope of our study concerns ENEM editions applied between 2010 (when additional languages were included in the test) and 2017 (the most recent edition on the moment of the corpus collection, on March 2018). In this period, at least two different versions of ENEM were applied each year, and the four fields of knowledge were divided into two different testing-books, applied on different days.

In this context, we downloaded all the testing-books that included the Languages test and were applied between 2010 and 2017, numbering 17 tests. In order to define the corpus, we selected only the first editions of each year. This way, we analyzed eight samples, each of which present five items on the EL test. Consequently, the corpus of the analysis comprises 40 EL items.

The analytical procedures encompassed the contextual and textual analysis, based on CGA approach. The contextual analysis was inspired on CDA perspective, and concentrated on the documental dimension, that is, we read, studied, and used other texts to interpret the results of the textual analysis of the study object. These other texts include previous studies that have investigated EL-ENEM (Rauber, 2012; Blanco, 2013; Avelar, 2015); official documents that regulate ENEM (National Curricular Orientations for High School - Orientações Curriculares Nacionais para o Ensino Médio, OCEM, Brasil, 2006; Guideline for items elaboration - Guia de elaboração e revisão de itens, Brasil, 2010); and official information disclosed on government’s websites (INEP portal, Brasil, 2015a; Ministry of Education - Ministério da Educação, MEC portal, Brasil, 2015b).

The textual analysis concentrated on the textual and discursive elements of the genre, using Sociorhetoric and SFL analytical tools. The first analytical procedure was to investigate the typical rhetorical organization of EL-ENEM genre. This investigation was carried out through the analysis of the form and the content (semantic analysis) of the parts (reference text, question’s instruction, response alternatives) that visibly organize the genre. The detailed analysis of the question’s instruction led to the second analytical procedure, which concentrated on the investigation of the language strata explored by the items. This investigation was conducted through the analysis of the types of information, regarding the item’s reference text, explored by the question’s instruction. Finally, this second analytical procedure made it possible to discuss the language conception that underlies the test, considering that each language stratum, taken separately, implies on different aspects and views of language. The results that emerged through these analytical procedures are presented in the following section.

5. Results and Discussion

This section presents and discusses the results of the analysis of the genre EL-ENEM. We have mentioned in section 2 that CGA presupposes contextual and textual analysis. The contextual analysis was carried out considering the documental dimension, that is, official documents that regulate ENEM’s design and application, used to understand the genre’s social practice. This way, the results of the contextual analysis are used to discuss the results of the textual analysis, the main focus of this section. Firstly, we present the rhetorical organization of the genre and detail its rhetorical moves. We then describe the language strata explored by EL-ENEM’s items, and, based on these results, we discuss the language conception that underlies the test.

Starting with the general organization of the genre, inside each ENEM test, the EL takes up no more than two pages, out about 30 pages of the Languages test. EL-ENEM demands the reading of five texts written in EL, each one followed by a question. Therefore, these five items to be answered by the test takers are considered as the five sections that represent EL-ENEM’s typical rhetorical organization. Each of these sections, or items, is organized on three rhetorical moves: the reference text (Move 1), the question’s instruction (Move 2), and the response alternatives (Move 3). The typical rhetorical organization of EL-ENEM’s items is represented in Figure 2.

Figure 2
Rhetorical organization of EL-ENEM’s items

The first move presents the reference text, which motivates the problem situation proposed by the item (Brasil, 2010, p. 10), and it is organized on three rhetorical steps (in Figure 2, the underlined steps are the ones that occur in the sample question). The first steps are optional and present the pre-textual elements: step 1A presents the text’s title, step 1B presents the subtitle and step 1C presents additional information (recovering the text’s author, publication date and place of publication). In the sample question, the reference text presents step 1A only: “The weatherman”. Steps 2 and 3 are obligatory and they present, respectively, the body of the verbal and/or nonverbal text and the reference for the text’s publication place and/or author. In the sample question, step 2 presents the body of a verbal text and step 3 presents the reference for the text’s publication place. The analysis of the corpus showed that the reference texts present around 103 words: the item’s rhetorical move that features more textual content, even though the texts, in general, do not take up more than half a column in one page.

The second move presents the question’s instruction, which indicates the information the test taker needs to identify on the reference text. This instruction can be structured as a complete sentence, ending with a colon mark, or as a sentence to be completed by the correct alternative (Brasil, 2010, p. 11). The sample question in Figure 2 exemplifies the latter. The analysis showed that move 2 is organized on two obligatory rhetorical steps. Step 1 presents a contextualization, that is, gives information, while step 2 presents a demand for information⁴. In the sample question, step 1 is realized in the sentence “ao conversar sobre a previsão do tempo”, in which the instruction gives the information that the general topic of the reference text is weather forecasting, while step 2 is realized in the sentence “o texto mostra”, which demands specific information about the text’s content. The question’s instructions present around 25 words, with around 13 words on the contextualization and 12 words on the demand for information, indicating a balance between the two parts.

Finally, the third move presents five response alternatives, which are the possibilities of answers for the item’s problem situation (Brasil, 2010, p. 11). The response alternatives can be structured as complete sentences (if the question’s instruction ends with a colon mark), or as complements for the sentence that started on the question’s instruction. The sample question exemplifies the latter. Move 3 is composed of one and only correct answer - step 1, and four incorrect answers, called distractors - step 2. The response alternatives present around 10 words each, being homogeneous in extension, as the sample question shows.

In addition to these results of the analysis of the rhetorical organization of the genre, the analysis with focus on the form of the genre also confirmed that EL-ENEM tests exceptionally the EL reading skill, accomplishing its objective: it presents the reference texts in EL, and the instructions and response alternatives in Portuguese Language, as it is exemplified in the sample question. This way, the test taker is not expected to activate other skills in EL, such as to understand the question’s instruction; they only need to focus on the EL reading skill of a short text.

Since the analysis of the rhetorical organization of a genre presupposes the analysis of its form and content (Motta-Roth, 1995, p. 16), the analysis presented so far focused on the description of the investigated genre’s form, while the next part of the analysis focuses on the content of its rhetorical moves.

As for move 1, the analysis showed that most reference texts are authentic ones, i.e., texts that were actually written to be read in a specific language use context, in contrast to texts that are produced only with the purpose to be used in a language teaching context. This can be considered a positive aspect of the test because authentic texts, according to Nuttal (1996, p. 177), are motivating and carry features of “true discourse: having something to say, being coherent and clearly organized”. In addition, the items explore different genres; some of them are presented as text fragments, while others are full texts. The genres explored by the items are presented, followed by their occurrence in the corpus: news (32,5%), ads (15%), lyrics (10%), comic strips (7,5%), poems (7,5%), didactic texts (5%), cartoons (5%). All the following had only one occurrence in the corpus (2,5%): notice of baggage inspection, biography, letter to the editor, essay, catchphrase, book review, and donation website. This way, most texts belong to the journalistic sphere (news, letter to the editor, book review and donation website). The texts also belong to the artistic-literary (biography, cartoon, essay, catchphrase, lyrics, poems, comic strips), pedagogic (didactic text), advertising (ads), and touristic (notice of baggage inspection) spheres.

This result indicates that test takers will probably be familiar with EL-ENEM’s reference texts, because they will find, in the test, texts that they usually read to be informed about current events, at least in their mother language, such as news. This possible familiarity of the test takers with the reference texts is another positive aspect of the test: when the reader knows the genre of the text, even when reading in their mother language, they are more familiar with its linguistic and content features, which eases the process of reading. In that regard, Coscarelli (2002, p. 23) indicates that the familiarity with the genre enables the reader to develop reading strategies, since they know what to expect of each part of the text. Some texts, on the other hand, to a lesser degree, may be less familiar to the test takers, such as the notice of baggage inspection, which would only be familiar to someone who have already travelled by plane and had their baggage inspected.

Another aspect that may boost the process of reading in an additional language and excelled in the corpus is the presence of texts involving both verbal elements and images. In all editions of the corpus, at least one item proposes the reading of this type of text: they represent 32,5% of the reference texts. Kress & Van Leeuwen (2006, p. 16-17) point out that “outside the school [or, in our case, outside the test] […] images play an ever-increasing role […] [and] most texts now involve a complex interplay of written text, images and other graphic or sound elements”, and it is extremely important that schools provide a space to learn how to read and produce this kind of texts. In this context, two positive aspects can be highlighted. The first is, again, the test takers’ familiarity with the genres. The second is the importance devoted to texts involving both verbal elements and images in a nationally regarded exam, which influences teaching contexts.

The analysis of the item’s reference texts could point out several positive aspects of EL-ENEM. However, some negative points could also be highlighted. Many items changed the original texts’ layout, and some presented incomplete references. These aspects make the process of reading more difficult, because they hide important contextual information regarding the author of the text, its publication place, target audience and objective, which inhibits the understanding of the text by its context. The second negative point is the choice for didactic texts as reference texts (although they represent a limited part of the corpus - 5%). The sample question presented in Figure 2 is an example of a didactic text as a reference text. Nuttal (1996, p. 177) explains that texts that are exclusively produced for language learners not always present the characteristics that are expected of authentic texts, and the simplicity of the language may lead to superficial texts, which only serve to present the language, not being able to communicate a true message. Ending the considerations concerning the reference texts, we will now present the detailed analysis of the questions’ instructions and response alternatives.

All the EL-ENEM items are alternative item type, in which the correct alternative completes the sentence presented on the question’s instruction. As it was previously described, the question’s instruction presents two different types of information: firstly, a contextualization, and lastly, a demand for information. With respect to the contextualization of the question’s instruction, the analysis showed that this part of the item can offer at least seven different types of information, presented on Chart 2, together with their occurrence in the corpus and an example (the part that refers to the contextualization in the question’s instruction is underlined).

Chart 2
Types of information in the question’s instruction contextualization

Type of information in the contextualization	Occurrence (%)	Example in the corpus
1) Generalization regarding the genre	25%**	“As instituições públicas fazem uso de avisos como instrumento de comunicação com o cidadão. (...)”
2) Authorship	15%	“Estes são os versos finais do famoso poema The Road Not Taken, do poeta americano Robert Frost. (...)”
3) General subject/topic	12,5%	“A reportagem apresenta consequências do uso de novas tecnologias para a mente humana. (...)”
4) Specific part/content	12,5%	“Em relação às pesquisas, a utilização da expressão university graduates evidencia a intenção de informar que”
5) Indication to read the reference text	10%	“A partir da leitura dessa tirinha, infere-se que o discurso de Calvin teve um efeito diferente do pretendido, uma vez que ele”
6) Generalization regarding the context or the target audience	7,5%	“A internet tem servido a diferentes interesses, ampliando, muitas vezes, o contato entre pessoas e instituições. (...)”
7) Publication place	7,5%	“Ao ler a matéria publicada na National Geographic, para realização de um trabalho escolar, um estudante descobriu que”

• ** We included here the types of information that had the greatest occurrence in the contextualization of the question’s instructions. Three items of the corpus presented a different type of information each, and one item directly presented the demand for information, without presenting a contextualization.
• Source: Authors

This way, it is possible to point out that in this part of the question’s instruction most items offer information about the context of publication of the reference text, which includes the objective of the genre and information about the text’s author and target audience. After offering this information in the contextualization, the question’s instruction presents the demand for information, which represents, in Halliday & Matthiessen’s terms (2014, p. 698), an interpersonal metaphor, because it is structured in a grammatical mood that do not typically realize this function: the declarative mood. The typical grammatical mood to realize a demand for information, i.e., a question, is the interrogative mood.

As well as the item’s contextualization, the demand for information could be classified on different types. In this context, Chart 3 presents the types of information that the items demand the test takers identify on the reference texts, their occurrence in the corpus and an explanation concerning the expected task.

Chart 3
Demand for information in the items

Type of information demanded in the item	Occurrence (%)	Expected task
1) Specific information regarding the content	42,5%	To identify the demanded information
2) Objective	30%	To identify the objective of the text
3) Topic	15%	To identify the topic of the text
4) Intertextual relations	5%	To identify the meaning relations between the reference text and another cited text, or between the reference text and the text presented on the response alternatives
5) Function of expressions	5%	To identify the meaning and/or the function of an expression used in the reference text
6) Translation	2,5%	To identify the best translation of text fragments in EL to Portuguese Language

• Source: Authors

In relation to the items that demand specific information regarding the reference text’s content, this information can refer, among others, to the opinion the author of the text is expressing; to the persona poetry’s feelings; to the reason why the author of a book review is impressed with a character; and to the objective of a character in using certain arguments in a cartoon. When it comes to comic strips, the items demand information about the reason of a misunderstanding between two characters; the humor effect; and the inference that can be made through the action of a character. In relation to translation, one item of the corpus demands that the test taker translates two fragments of the reference text. This text presents the eight millennium development goals, and each alternative presents two fragments in Portuguese Language - the test taker must identify what alternative presents the two fragments that refer to two millennium development goals presented in EL in the reference text.

In summary, the EL-ENEM test taker must, mainly, identify specific information about the content and the objective of the reference texts. All editions of the corpus present at least one item that demands these two types of information. The topic of the reference texts is also highly demanded by the items. With respect to the reading of comic strips, the items demand other types of information, specific to the dialogue or to the characters that each text presents.

Through a more detailed analysis concerning the item’s demand for information, it was possible to identify the most privileged language strata on EL-ENEM’s items, that is, the ones directly explored by the demand for information: Semantics and Pragmatics (55%) and Register (45%). In all editions of the corpus, at least one item explores the Register stratum, and the others, the Semantics and Pragmatics stratum.

The items that explore the Semantics and Pragmatics stratum are the ones that refer to the meanings of the texts, therefore, they are the ones that demand specific information regarding the content of the reference text, the identification of intertextual relations, the function of expressions, and translation. The items that explore the Register stratum, in turn, refer to the uses of language typically related to its context of use; therefore, they are the ones that demand the topic and the objective of the reference texts.

The emphasis on the Semantics and Pragmatics stratum is another confirmation that EL-ENEM actually tests the reading skill and activates knowledge typically related to this skill, such as inferring and identifying processes of reference on the text. Moreover, the emphasis on the Register stratum demonstrates a concern in directing the test taker to the contextual aspects of the reference texts.

Besides the language strata directly explored on the demand for information, the analysis also showed that the Genre stratum is highly explored on the contextualization, in which there is an offer of information recovering the genre’s objective, context and target audience. Therefore, even if the item does not directly demand that the test taker identify aspects of the Genre on the reference text, information concerning this language stratum is offered on the contextualization, which demonstrates that they are considered relevant information to understand the text. In addition, considering that each language stratum is realized in another stratum (Gouveia, 2009, p. 23), that is, the Discourse stratum is realized in the Genre stratum, which is realized in the Register stratum, and so forth; then we can conclude that the Lexicogrammar and the Graphology strata are also indirectly explored by the items. This way, we propose Figure 3 to present the types of information (presented either in the contextualization - which are positioned on the left side of the Figure, or in the demand for information - positioned on the right side) that explore aspects of each language stratum in EL-ENEM’s items.

Figure 3
The stratification of language and the EL-ENEM

This way, we can conclude that EL-ENEM explores five from the six language strata. The Genre stratum is explored on the contextualization of the question’s instruction, giving information about the genre’s objective and its textual and contextual features. This stratum is realized on the Register stratum, which is realized on the Semantics and Pragmatics stratum; both strata explored as much on the contextualization as on the demand for information. Lexicogrammar and Graphology strata are indirectly explored, because they express the aspects of the others (Kummer, 2015, p. 108), through the choices in terms of vocabulary and grammar (Catto, 2014, p. 5).

This analysis leads to the discussion concerning the language conception that underlies EL-ENEM. The language conception that guides CGA, and, therefore, the present study, is the critical conception of language as genre, that is, according to Motta-Roth (2008a, p. 352), language that works to accomplish objectives and perform activities in order to create and recontextualize social interactions. This language conception was also verified on official documents that guide ENEM, such as the OCEM (Brasil, 2006). In EL-ENEM, we could verify a direction towards the conception of language as genre, since it explores several important aspects concerning the context of the reference texts, as Figure 3 demonstrates. However, we could not identify the critical conception of language as genre, since the test does not explore critical aspects from the reference texts, related to the Discourse stratum.

A critical conception of language as genre implies the understanding of language as the materialization of world representations that suits the interests of specific dominating groups, which ends up dictating and constructing social relations. A critical teaching program helps learners to understand that these world representations are not the only ones, and, consequently, it instigates social change, encouraging a critical reflection on other possible world representations and other ways of constructing social relations. We mention teaching programs here considering that ENEM may influence teaching contexts (as it was shown in Blanco’s (2013) study, for instance). Therefore, we may consider that the language conception that underlies ENEM also underlies teaching practices.

One of the difficulties faced by EL-ENEM on exploring a critical conception of language as genre may be the proposal of only one question per reference text. This is an obstacle because the critical aspect of language is an abstraction, therefore, it is necessary, firstly, to deal with its concrete aspects, such as its objectives and its lexicogrammatical features, and then, it would be possible to concentrate on the abstract aspects. Regardless, the fact that the test explores the contextual features of the reference texts is a positive aspect highlighted by the analysis, which indicates that this test departs from the traditional view of language in which grammar performs the leading role, and approaches a more contextualized view of language.

6. Final Remarks

This study concentrated on a detailed analysis of EL-ENEM, supported by CGA assumptions. It is a pioneer study in considering this test as genre, which can bring contributions not only to the Applied Linguistics field, by exposing features of an important genre that circulates in Brazilian schools’ contexts, but also to the Genre Analysis field, by analyzing a genre that is not-typically regarded in the field. The study also differentiates from the others in relation to the sample size: by analyzing a larger sample, it was possible to identify an important role given to the reference texts’ contexts, indicating that its questions test more than lexical or grammatical knowledge, as it is popularly spread.

The limitations of the study include a more detailed contextual analysis, as one of the important procedures in CGA, which allows a richer interpretation of the genre’s textual features, through interviews and questionnaires with people in charge of designing the questions, for example. As a way of contributing to the school context, which is also something expected by CGA, the authors of this study designed a material to be used by the test takers with instructions on how to solve EL-ENEM questions (Radünz; Marcuzzo, 2020). This pedagogical implication is also applicable for EL teachers, who can use the results of this study to design EL reading classes.

References

• Avelar, Flávia Juliana de Sousa. (2015). O inglês no ENEM e na escola: práticas de dois professores do ensino médio Doctoral dissertation, Universidade Estadual de Campinas, Campinas, SP.
• Azevedo, Bruno de, Pires, Daniel Reschke, & Tomitch, Lêda Maria Braga. (2018). Reading comprehension in English at the Brazilian national high school exam (ENEM): Unveiling inferential processes. Calidoscópio, 16(1), pp. 122-132.
• Bawarshi, Anis S., & Reiff, Mary Jo. (2013). Gênero - história, teoria, pesquisa Translated by Benedito Gomes Bezerra. São Paulo: Parábola.
• Bhatia, Vijay Kumar. (2015). Critical genre analysis: theoretical preliminaries. Hermes - Journal of Language and Communication in Business, 54, pp. 9-20.
• Blanco, Juliana. (2013). A avaliação de língua inglesa no ENEM: efeitos de seu impacto social no contexto escolar Master’s thesis, Universidade Federal de São Carlos, São Carlos, SP.
• Brasil, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. (2019). Portal INEP http://portal.inep.gov.br/web/guest/enem-ppl
  » http://portal.inep.gov.br/web/guest/enem-ppl
• Brasil, Governo do Brasil, Portal do Governo do Brasil. (2016). http://www.brasil.gov.br/noticias/educacao-e-ciencia/2016/12/segunda-aplicacao-do-enem-ocorre-neste-fim-de-semana
  » http://www.brasil.gov.br/noticias/educacao-e-ciencia/2016/12/segunda-aplicacao-do-enem-ocorre-neste-fim-de-semana
• Brasil, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. (2015a). Portal INEP http://portal.inep.gov.br/web/guest/enem
  » http://portal.inep.gov.br/web/guest/enem
• Brasil, Ministério da Educação. (2015b). Portal MEC http://portal.mec.gov.br/ultimas-noticias/212-educacao-superior-1690610854/30781-exame-evolui-desde-a-criacao-ha-17-anos-e-amplia-oportunidades-na-educacao-superior
  » http://portal.mec.gov.br/ultimas-noticias/212-educacao-superior-1690610854/30781-exame-evolui-desde-a-criacao-ha-17-anos-e-amplia-oportunidades-na-educacao-superior
• Brasil, Instituto Nacional de Estudos e Pesquisas Educacionais Anísio Teixeira. (2010). Guia de elaboração e revisão de itens http://darnassus.if.ufrj.br/~marta/enem/docs_enem/guia_elaboracao_revisao_itens_2012.pdf
  » http://darnassus.if.ufrj.br/~marta/enem/docs_enem/guia_elaboracao_revisao_itens_2012.pdf
• Brasil, Ministério da Educação, Secretaria de Educação Básica. (2006). Orientações curriculares para o Ensino Médio http://portal.mec.gov.br/seb/arquivos/pdf/book_volume_01_internet.pdf
  » http://portal.mec.gov.br/seb/arquivos/pdf/book_volume_01_internet.pdf
• Bonini, Adair. (2010). Critical genre analysis and professional practice: the case of public contests to select professors for Brazilian public universities. Linguagem em (dis)curso, 10, pp. 485-510.
• Catto, Natália Rodrigues. (2014). Letramento multimodal: participação em gêneros discursivos multimodais em livros didáticos de língua inglesa. Pesquisas em Discurso Pedagógico, 1.
• Coscarelli, Carla Viana. (2002). Entendendo a leitura. Revista Estudos da Linguagem, 10 (1), pp. 7-27.
• Fairclough, Norman. (1985). Critical and descriptive goals in discourse analysis. Journal of Pragmatics, (9), pp. 739-763.
• Fairclough, Norman. (1992). Discourse and social change Cambridge: Polity Press.
• Fairclough, Norman. (2001). Critical discourse analysis as method in scientific research. In Wodak, R., & Meyer, M. (Eds.). Methods of critical discourse analysis London: Sage, pp. 121-138.
• Gouveia, Carlos Alberto Marques. (2009). Texto e gramática: uma introdução à Linguística Sistêmico Funcional. Matraga, 16(24), pp. 13-47.
• Halliday, Michael Alexander Kirkwood, & Matthiessen, Christian Matthias Ingemar Martin. (2014). Halliday’s Introduction to Functional Grammar New York/London: Routledge.
• Hendges, Graciela Rabuske. (2005). A genre and register analysis of electronic research articles from a systemic functional perspective: new medium, new meanings. PHD project. Florianópolis: PPGI/UFSC.
• Kress, Gunther, & Van Leeuwen, Theo. (2006). Reading images 2. ed. London: Routledge.
• Kummer, Daiane Aline. (2015). Letramento Multimodal Crítico: sob a perspectiva de livros didáticos e de professores de inglês e de biologia Master’s thesis, Universidade Federal de Santa Maria, Santa Maria, RS.
• Martin, James Robert. (1992). English text: system and structure Philadelphia/Amsterdam: John Benjamins.
• Meurer, José Luiz. (2002). Uma dimensão crítica do estudo de gêneros textuais. In Meurer, J. L., & Motta-Roth, D. (Orgs.). Gêneros textuais e práticas discursivas: subsídios para o ensino da linguagem Bauru: Edusc, pp. 17-29.
• Meurer, José Luiz. (2005). Gêneros textuais na análise crítica de Fairclough. In Meurer, J. L., Bonini, A., & Motta-Roth, D. (Orgs.). Gêneros: teorias, métodos, debates São Paulo: Parábola, pp. 81-106.
• Motta-Roth, Désirée. (1995). Rhetorical features and disciplinary cultures: a genre-based study of academic book reviews in Linguistics, Chemistry, and Economics PHD dissertation. Tese de doutorado, Universidade Federal de Santa Catarina, Florianópolis, SC.
• Motta-Roth, Désirée. (2006). Questões de metodologia em análise de gêneros. In Karwoski, A. M., Gaydeczka, B., & Brito, K. S. (Orgs.). Gêneros textuais: reflexões e ensino 2. ed. Rio de Janeiro: Editora Lucerna, pp. 145-163.
• Motta-Roth, Désirée. (2008a). Análise Crítica de Gêneros: contribuições para o ensino de linguagem. D. E. L. T. A., 24(2), pp. 341-383.
• Motta-Roth, Désirée. (2008b). Para ligar a teoria à prática: roteiro de perguntas para orientar a leitura/análise crítica de gêneros. In Motta-Roth, D., Cabanãs, T., & Hendges, G. (Orgs.). Análises de textos e de discursos: relações entre teorias e práticas 2. ed. Santa Maria: PPGL Editores, pp. 243-272.
• Motta-Roth, Désirée, & Heberle, Viviane Maria. (2015). A short cartography of genre studies in Brazil. Journal of English for Academic Purposes, pp. 1-10.
• Nuttall, Christine. (1996). Teaching reading skills in a foreign language Oxford: Heinemann.
• Oliveira, Andrea Barros Carvalho de. (2019). ENEM and the Language Policy for English in the Brazilian Context. Revista Brasileira de Linguística Aplicada, 19(2), pp. 361-383.
• Radünz, Amanda Petry. (2020). Análise Crítica de Gênero do ENEM de Língua Inglesa Master’s thesis. Universidade Federal de Santa Maria, Santa Maria, RS.
• Radünz, Amanda Petry, & Marcuzzo, Patrícia. (2020). Guia prático para mandar bem no ENEM de língua inglesa Recurso eletrônico (e-book): UFSM, DLEM. https://www.ufsm.br/app/uploads/sites/601/2020/06/Guia-Enem-%C3%BAltima-vers%C3%A3o-1.pdf
  » https://www.ufsm.br/app/uploads/sites/601/2020/06/Guia-Enem-%C3%BAltima-vers%C3%A3o-1.pdf
• Rauber, Bárbara Battistelli. (2012). Avaliação em língua estrangeira (inglês) no acesso ao ensino superior: O ENEM em discussão Master’s thesis, Universidade Federal de São Carlos, São Carlos, SP.
• Swales, John M. (1990). Genre analysis: English in academic and research settings Cambridge: Cambridge University Press.
• Swales, John M. (2004). Research genre: exploration and applications. Cambridge: Cambridge University Press.

• 1
  The present article presents an overview of the research conducted during a master’s degree in Applied Linguistics (Radünz, 2020), sponsored by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - Brazil (CAPES) - Finance Code 001.
• 4
  The terms “giving and demanding information” are used, as it was proposed by Halliday & Matthiessen (2014, p. 97), to refer to sentences that, through the declarative, interrogative or imperative mode, offer or demand information.

Publication Dates

• Publication in this collection
  11 June 2021
• Date of issue
  2021

History

• Received
  22 May 2020
• Accepted
  26 Feb 2021