Reading comprehension assessment: effect of order of task application

Accepted: June 18, 2018 Study conducted at Departamento de Fonoaudiologia, Escola Paulista de Medicina – EPM, Universidade Federal de São Paulo – UNIFESP São Paulo (SP), Brasil. 1 Departamento de Fonoaudiologia, Universidade Federal de São Paulo – UNIFESP São Paulo (SP), Brasil. 2 Núcleo de Ensino, Assistência e Pesquisa em Escrita e Leitura – NEAPEL, Universidade Federal de São Paulo – UNIFESP São Paulo (SP), Brasil. Financial support: Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP), processo número 2011/21453-8. Conflict of interests: nothing to the declare. ABSTRACT


INTRODUCTION
One of the greatest difficulties encountered by clinicians and researchers of reading and related disorders is how to select the best instrument to assess reading comprehension performance, considering the large number of skills and competencies involved, as well as the complex mental construction process regarding the read text (1) . Therefore, many opt to use multiple tasks in order to more widely observe an individual's comprehension abilities.
Reading performance depends not only on decoding conditions, but also on the development of other skills: cognitive mechanisms, linked to memory, perception, recognition, organization, integration and association of sensory information, such as acquisition and development of vocabulary and language rules, and the use of metacognitive and metalinguistic strategies, as well as the self-monitoring of language elaboration, or mental representation of the information received (2)(3)(4) . Thus, when considering the completeness and extension of the comprehension process and all the components it involves, the assessment of reading comprehension must take this complexity into account.
Among the different ways of measuring reading comprehension, retelling has been considered a valuable evaluation procedure, since the task demands the production or elaboration of a meaningful representation of textual information in the reader's mind (5)(6)(7)(8) . The mental representation accessed through retelling allows the evaluator to identify the different levels of understanding revealed by elements of microstructure, macrostructure and superstructure (5)(6)(7) . The number of recounted ideas has a direct relationship with the microstructure variables, and reflects how the reader processes and memorizes the information read (7,8) . On the other hand, the analysis of the links established by the reader between the ideas understood, and the connections made, allows the evaluator to understand to what extent the causal chain of the text was understood and organized, resulting in the general structuring of the ideas in a global text scheme, given, in turn, by the general understanding shown by the retelling (5)(6)(7) .
Another instrument used to assess reading comprehension is the use of open-ended questions about the read text. Among the advantages of using open-ended questions is the possibility of using the nature of the question to control the cognitive process used by the reader to handle the information required, or to control the type of processing that inflicts greater difficulty when reading (1,9) . A classification model of questions enables the possibility of identifying the cognitive processes involved. The comprehension of information of a literal nature or by inferences, through text-connecting or Gap-Filling (10,11) , can also be gaged.
The assessment of both the mental representation constructed by the reader, through retelling of the read text, and the processes involved in the construction of that understanding, through open-ended questions about the text, are instruments which complement one another, as they provide different types of information about the reading comprehension, thus favoring the identification of reading problems. In addition, the two forms used to evaluate reading comprehension are shown to be correlated with each other, showing that although they have different cognitive requirements, both assess a common skill that is textual comprehension (8)(9)(10)(11) .
However, when assessing reading comprehension, one must take into consideration the possible influences of the organizing action required when retelling a text, as well as the use of narrative elements (when answering questions), on the results. One must be attentive to the possibility that the questioning that occurs before asking for the text to be retold, may interfere with this process, promoting the use of problem-solving strategies on the part of the learner, that would lead to the correction of possible gaps in the natural process of understanding of what was read (12,13) . In most cases, in reading comprehension assessment tasks, retelling or rewriting a passage read should be a single task that the learner must fulfill independently.
Little or nothing has been investigated about the influence that important questions about the text, in the form of tutoring by the evaluator, could exert on the reading comprehension and, above all, on the retelling of the read text. Considering these aspects, the present study aimed to investigate the influence of order of reading tasks application (open-ended questions and retelling) on the reading comprehension performance by students with and without reading difficulties.
The research hypothesis considered that good readers could benefit from a task requiring them to answer open-ended questions before retelling the text, since these students would use self-monitoring resources triggered by the information declared in the questions, favoring the repair or reinforcement of the mental representations achieved after reading. The opposite, however, was expected for readers with difficulties, who would focus more on the task of decoding the text while reading and lose efficiency in the construction of meaning. With little information about the text, this population would not respond to new information presented in the questions, making it impossible to use metacognitive repair strategies.

Ethics approval and consent to participate
This study was approved by the Research Ethics Committee of the Federal University of São Paulo (CEP / UNIFESP) -Process nº 1328/11. The assessments began after: 1) authorization was given by the school through the signing of the Term of Institutional Consent; 2) clarification of doubts from parents and responsible parties regarding the procedures and voluntary acquiescence by signing the Term of Free and Informed Consent.

Participants
We selected 40 children (23 girls), studying in a public school of the State Education Network, located in the southern zone of the city of São Paulo.
In order to meet the objectives of the study two steps were followed.
First, teachers were asked to recommend students with reading comprehension difficulties and students without reading and writing complaints, difficulties or disorders, or any difficulty in school achievement. The instruction given to teachers requested the indication of children without complaints of reading (decoding and/or understanding) and with reading comprehension difficulties. In order to aid in the judgment of reader performance, the following criteria were given (14) : to consider the ability to decode (read with rhythm / read without rhythm, neither too slowly nor so quickly), to consider the ability to understand what it reads, to respond adequately when questioned about the text read and to orally summarize the content he has read. The following general criteria were also observed by the teachers when recommending the children: absence of auditory or visual (uncorrected) complaints or indicators of cognitive and sensorial deficits, neuromotor and behavioral disorders; regular enrollment in the 4 th or 5 th grade.
Finally, all students who were recommended and signed an Informed Consent Form were evaluated by an oral reading text task (15) . Texts selected for each school series ("A coruja e a águia" -4th grade, "O veterinário maluco" -5th grade), taken from appropriate didactic materials, were presented printed in Arial font 12, with spacing 1.5. The readings were performed individually and recorded for further analysis. No time-limit was set and digital support during reading was not restricted.
By listening to the recordings, the total reading time was computed and words omitted, re-read or inserted during reading were quantified. In case of insertion or re-reading of some words by the student, the inserted or re-read items were incorporated into the total number of words in the text for the reading rate calculation. On the other hand, the number of items omitted during reading was subtracted from the total number of words in the text. To calculate the reading rate (words per minute), the following formula was used: total of words reads X 60 / reading time (in seconds).
For the reading accuracy calculation, the total of correctly reads words was calculated. Accuracy was considered when the words were read correctly and fluently. Hesitations, re-read or read with orthographic decoding, were considered an error. The following formula was applied: total of words reads correctly X 60 / reading time (in seconds).
The test aimed to ensure that they presented reading accuracy values greater than or equal to 40 words read correctly per minute, thus ensuring minimal decoding conditions for access to comprehension (16) . Students with accuracy below 40 words per minute were excluded from the sample (n = 12).
The 40 selected students were grouped as follows: • Research Group (with reading difficulties) -Retelling (RG-r): 10 students with reading complaints, enrolled in the 4th or 5 th grade (n = 5 for each grade), who first retold the read text and then answered open-ended questions about it; • Control Group -Retelling (CG-r): 10 students considered typical readers because they did not present difficulties or indicators of reading disorders, enrolled in the 4 th or 5 th grade (n = 5 for each grade), matched to RG-r students by gender, age and schooling. This group of students initially retold and subsequently answered the open-ended questions about the read text; • Research Group (with reading difficulties) -Questions (RG-q): 10 students with reading complaints, enrolled in the 4 th or 5 th grade (n = 5 for each grade), who first answered the open-ended questions and then retold the read text; • Control Group -Questions (CG-q): 10 students considered typical readers because they did not present difficulties or indicators of reading disorders, enrolled in the 4 th or 5 th grade (n = 5 for each grade), matched to RG-q students by gender, age and schooling, who initially answered the open-ended questions and then retold the read text.
Chart 1 shows the distribution of the groups according to school grade, age and order of application of the research procedures (response to questions-retelling or retelling-response to questions). The following table presents the characterization of groups according to schooling only for didactic reasons.

Reading comprehension task
For the reading comprehension assessment, the narrative text "A aposta" (15) , printed in Arial font 11 with double spacing, was used. The students were instructed to read the text in the way they usually read for comprehension (aloud or silently) and  The participants were individually assessed in a single session, for fifteen minutes approximately, in a room provided by the school administration. The room was well lit and the students sat at a table while reading. The furniture was adequate for the size of the children. The text was removed before the assessment. The retelling and the answers to the questions were recorded for later transcription and analysis.
The analysis of reading comprehension through retelling of the read text was conducted in order to identify microstructure and macrostructure variables (5,17) . For this, we used the analysis of the retelling of the text "A aposta" (15) .
For the analysis of the microstructure, the identification of the central ideas (CI), details (D) and inferences (I) in the retelling were assigned one point for every idea expressed in its entirety. Next, the total number of ideas, central ideas, details and inferences were computed.
The analysis of the macrostructure involved the identification of first-order causal links, essential for connecting central ideas of a causal chain; and second-order links, where a detail can replace a central idea in the causal chain. One point was assigned to each identified causal link. Next, the total number of links, first-order links and second-order links were computed.
A Retelling Reference Standard was also classified from the retelling assessment (5,17,18) . The following points were assigned to this standard: Reference Standard 3 -three points were attributed to the retelling when it contemplated all the first-order causal links (central ideas) or contemplated all moments of the text -the Setting, initial fact, internal response, execution, consequence and reaction -even if some first-order (central) ideas were replaced by second-order (detail) ideas; Reference Standard 2 -two points were attributed to the retelling when it contemplated only causal links considered minimal; relating setting, initial fact, execution and reaction; Reference Standard 1 -one point was attributed to the retelling when it failed to contemplate one or more causal links considered minimal, although it showed first-order and second-order ideas, and was thus a descriptive retelling; Reference Standard 0 -No points were attributed to a retelling that omitted central ideas of the text and showed an absence of causal links, being considered a retelling without structure and/or with the intrusion of elements not present in the text.

Responses to open-ended questions
After reading, the students answered eight open-ended questions designed to evaluate cognitive processes involved in reading comprehension. The questions orally presented were classified according to the presence and type of inference they determined (9,10) : two questions of literal comprehension, that is, that involved an understanding of information clearly expressed in the text; three questions of text-connecting inference, that required the integration of implicit information to establish cohesion between different sentences of the text; three questions of inference by the gap-filling process, whereby the reader uses his/her previous knowledge and the content of the text to fill the gaps and thus integrate the intentions of the text.
The performance of the students was analyzed from a pool of answers, defined by a panel of three speech therapists, and for which one point was assigned to each correct question. Next, the number of correct answers by type of question and the total number of questions answered were computed. The data were tabulated and analyzed statistically.

Statistical method
Statistical analysis was performed using the Statistical Package for Social Sciences (SPSS), version 10.0. Statistical treatment used Shapiro-Wilk and Mann-Whitney Test. The significance level was set at 5% (0.05).
In order to confirm the accuracy of the teacher recommendations, a paired data analysis was carried out. As expected, the research groups (RG-r and RG-q) showed lower decoding performance than the control groups (CG-r and CG-q), as a function of grade, analyzed using an independent-samples t-test. For this analysis, it is important to note that the decoding parameters were converted into z-scores, since different texts were used for assessing 4 th grade and 5 th grade students. This conversion allowed the data to be analyzed together, regardless of the use of different instruments on decoding reading assessment.
On the other hand, considering the proposed objective, it would be desirable for the research groups (RG-r and RG-q), as well as the control groups (CG-r and CG-q), to have the same decoding ability. This condition would be essential to confirm that any differences observed in the comparisons between the groups were due exclusively to the interference of the order of application of the reading comprehension tasks and not to the decoding characteristics of the groups.
The descriptive data for decoding by the different groups are given in Tables 1 and 2.

Effect of order of task application study
The summary and descriptive analysis of the data for the reading comprehension variables are shown in Table 3.    The comparative analysis of CG-r and CG-q groups showed differences in the performance of the groups for second-order links, total number of links and total reference standard, with a better performance for CG-q students ( Table 4). The study of effect size corroborates findings since Cohen's P-values show in the middle range, attesting the minimum quality level of the sample for the intended analyzes (2 nd order links: r =0.56; Total links: r= 0.53; RS: r=0.51).
On the other hand, the results of the comparative analysis of RG-r and RG-q students did not reveal differences in the performance of the groups when considering all the reading comprehension variables investigated (Table 5).

DISCUSSION
The influence of the task of answering open-ended questions on performance in the retelling task was based on the theoretical assumption that understanding involves the use of metacognitive strategies that allow the reader to monitor, during the course of reading, or even after reading, the understanding reached (13,19,20) . Monitoring, it was possible to understand as the ability to review all the comprehended content and identify possible failures in this process, and how itcan activate meta-strategies such as repair (21) , thus favoring comprehension. Repair, as a result of the conscious action of the reader who seeks to solve a reading comprehension problem, can arise from the reflection on the content, promoting the change in the association between the ideas of a text or connecting them to previous knowledge not activated in a first reading, or even after re-reading the text (12,13,21) .
In this study, the open-ended questions were understood as possible triggers for the use of metacognitive strategies and capable, therefore, of enhancing understanding. The retelling after reading, in turn, was understood as the possibility to access, without interference from the evaluator or from the explanation of a reading assessment task, the significant representation of the textual information constructed by the reader through metacognitive strategies. In addition, retelling after reading a text was adopted because it is commonly used to measure reading comprehension (5,7,22) and to have its evaluative properties studied, showing that it is a valid means of accessing the comprehension construct (23) .
The idea of manipulating the moment when the assessment tasks are presented was to verify if the exposure to questions prior to the retelling task could promote a better understanding, evidenced by a greater number of microstructure (central ideas, detail and inferences) or macrostructure (links, retelling reference standard) elements.
The results showed that students considered typical readers without difficulties who answered questions before retelling (CG-q) performed better than those who retold the read text before answering the questions (CG-r). Their performances were better when considering the macrostructure variables (higher number of second-order links, total links and retelling reference standard). These findings suggest that the questions seem to have played a role in activating the metacognitive strategies of monitoring, helping to apply significance to the connections between the ideas read, and acting directly in the global comprehension of the text.
The similar performance of these groups on the number of central ideas and details retold is probably due to the fact that they correctly answered a larger number of questions related to textual cohesion and gap-filling inferences when compared to how well they answered the literal questions. This result indicates that the answers to these inferential questions do not produce an improvement in the recall of ideas in the text, but favor the understanding of the links between ideas already memorized (10,18) .
The literature shows that open-ended questions can trigger, in the competent reader, global metacognitive reading strategies, used for general analysis of the text (24) . The use of these strategies,  therefore, enables these readers to rethink, reorganize and even reconstruct the links between the main ideas of the text and thus modify what they had understood initially (25) . This would be possible by triggering mechanisms related to metacognition, the uses and regulation of which can be improved through a great variety of strategies developed with reading experience, such as external directives capable of overcoming impasses (26) , and teaching strategies (12,13) . Metacognitive regulation improves students' performance in several aspects, favoring more efficient use of attention resources, the use of strategies already acquired, and greater awareness of possible failures in the process of constructing understanding (24) , enabling the competent reader to make repairs. However, the same process of activation of metacognitive resources was not observed in the performance of students with reading difficulties. The data showed that the students who performed the retelling before the questions (RG-r) achieved similar results to those who answered the questions and then retold what they understood from the read text (RG-q) for all the microstructure and macrostructure variables investigated. These findings indicate, therefore, that open-ended questions did not influence the retelling performance in readers with reading and comprehension difficulties. Readers with reading difficulties would be less able to devise strategies for monitoring their mental representations, using these resources less frequently (12,13,26) . The results, therefore, seem to explain the absence of the influence of one assessment task on the other regarding the reading comprehension of the students with reading difficulties, independent of the order in which they were applied.
The progressive increase in the decoding speed developed in the typical learning of reading should facilitate the retention of information in a more precise way, leading to good reading comprehension, especially for students of the investigated grades (11,27,28) . A study carried out with Brazilian schoolchildren shows that children with longer schooling are more able to use metacognitive strategies because they have greater reading proficiency (11) .
However, although a minimum number of 40 words read correctly per minute was fixed to guarantee the possibility of reading comprehension, this development pattern was not observed in the students of the Research Groups, since they demonstrated inferior performance when compared to their peers in the Control Groups in both decoding and word recognition. The poor decoding ability did not guarantee the automaticity required to construct a more comprehensive understanding and may have influenced both cognitive microstructure processing and the triggering of skills, such as self-monitoring. For this reason, these regulatory strategies, which expend a lot of effort as they involve a conscious act on the part of the reader, would be less used by students with difficulties, or even abandoned, due to their need to concentrate their attention on simpler decoding and word recognition processes (29) .
The data presented here are preliminary and should be expanded. Studies with larger numbers of participants and the expansion of research for clinical groups may bring relevant information, especially on the ability to repair errors during reading comprehension process.

CONCLUSION
The analysis of the data obtained indicated that responding first to questions related to the text read influences the reading comprehension and enriches the mental representation of the text, thus favoring the performance of the students without reading difficulties. Improved performance occurs especially in identifying the links between ideas and in the retelling reference standard and favors macrostructural processing of the text. The same, however, does not occur in the presence of reading alterations that require a greater effort in decoding, leading to less effective self-monitoring and direction management strategies. Without these strategies, there are no repairs in the constructed mental representation and, therefore, there is no improvement in performance, either at the micro or macrostructural level.