Development of an instrument for collective assessment of fluency and comprehension of reading in elementary school students II

ABSTRACT

Purpose

The study presents the process of developing of an instrument for collective evaluation of reading fluency and comprehension of secondary elementary school students in grades 6-9 and verifies the effect of schooling on performance in the instrument.

Methods

100 students regularly enrolled in grades 6-9 in secondary elementary public schools participated in the study. The construction of the instrument involved seven steps, with the participation of two judges. The instrument was composed of narrative text appropriate for secondary elementary school students and for 10 multiple choice questions, which five were literal questions and five were inferential questions.

Results

The results showed a better performance for the participants with higher schooling in fluency and in reading comprehension. The reading fluency presented positive and moderate correlations with the reading comprehension.

Conclusion

The instrument is easy to apply and analyze, and can be used in clinical, educational and research context to measure the performance of students in grades 6-9 in fluency and reading comprehension.

Keywords
Reading; Comprehension; Adolescent; Education; Educational Measurement; Speech-Language Therapy

RESUMO

Objetivo

O estudo apresenta o processo de desenvolvimento de um instrumento para avaliação coletiva da fluência e da compreensão de leitura de escolares do ensino fundamental II e verifica o efeito da escolaridade no desempenho no instrumento.

Método

Participaram do estudo 100 escolares regularmente matriculados do 6º ao 9º ano no ensino fundamental de escolas públicas. A construção do instrumento envolveu sete etapas, com participação de duas juízas. O instrumento é composto por um texto narrativo apropriado para escolares do ensino fundamental II e por 10 questões de múltipla escolha, sendo cinco questões literais e cinco inferenciais.

Resultados

Os resultados evidenciaram melhor desempenho, tanto em fluência quanto em compreensão de leitura, para os participantes com maior escolaridade. A fluência de leitura apresentou correlações positivas e moderadas com a compreensão leitora.

Conclusão

O instrumento é de fácil aplicação e análise, podendo ser utilizado em contexto clínico, educacional e de pesquisa para mensurar o desempenho de escolares do 6º ao 9º ano em fluência e compreensão de leitura.

Descritores
Leitura; Compreensão; Adolescente; Educação; Avaliação Educacional; Fonoaudiologia

INTRODUCTION

Reading is an important skill involved in processes of knowledge acquisition, self-criticism and understanding of reality and the world. However, the act of reading is not simple and involves a complex of cognitive skills.

Reading comprehension is generally defined as the aptitude for extracting meaning from the text⁽¹⁾, a process that requires integration of a variety of skills and abilities. These processes include cognitive skills such as decoding (transformation of written code into information), motivation, comprehension of spoken language, vocabulary, language skills, and also full information processing capabilities, such as working memory, reading accuracy (fluency) and rapid automatized naming (information processing speed)⁽²⁾. In addition to these variables, it is possible to include higher mental level component processes, that is, ability to extract information that is implicit in sentences and integrate new information with prior knowledge⁽³⁾, as well as monitoring of the text general meaning. Therefore, developing a reading and comprehension assessment instrument is not an easy task, since comprehension is not measured directly due to its complexity⁽⁴⁾.

Reading fluency is a multidimensional skill. It presents three main dimensions that favor connection with the comprehension skill: accuracy (or precision in decoding), automatic processing (automatic word recognition), and prosody (variations of fundamental frequency, duration and intensity that guarantee expressiveness and impression of attitudes towards reading). It develops gradually over the school years, allowing the reader to develop control over the processing of text surface structures so that he/she can focus on understanding deeper structures of access to meaning⁽⁵⁾. Reading fluency can be assessed by oral reading or silent reading. Oral reading (aloud) enables observation of the reader’s reading routes, accuracy and prosody. Silent reading (visual, without using the voice) favors comprehension, which is enabled by closer dialogue between text and reader. Silent reading is faster than oral reading, because there is a process of mental recovery of the word sound and there is no need to use the vocal tract to access meaning. Nevertheless, both have the same purpose of extracting meaning from the text, involving multiple cognitive processes and at different levels of interaction (linguistic, textual and world knowledge)⁽⁶⁾.

National and international studies indicate that cases of school failure have increased over the years^(7,8). The results of these studies show that students who complete the early years of elementary school have poorer reading and writing performance compared with results expected for the school range and reading difficulty may have a strong relation with these findings.

Being able to measure and identify which aspects are involved in the reading skill where students present weakness is the main way to improve the intervention principles for both clinical and educational use. Such evaluations should include texts that require the general knowledge of the reader, as well as corrective criteria and clear answers, with no room for error induction⁽⁹⁾.

Reading comprehension can be assessed by means of online measures, obtained while the subject is reading, or offline measures, obtained after reading is completed⁽¹⁰⁾. Among online measures, reading time, lexical decision tasks, naming and recognition during reading stand out. Regarding offline measures, the literature highlights retelling, answer to open-ended and closed-ended questions, and problem solving. Online and offline measures have advantages and disadvantages⁽¹⁰⁾. The literature shows that offline measures do not hinder the reading process and are more indicative of the longer lasting representational outcome of these processes, whereas many online measures are disruptive and may lead the reader to use specific strategies for the test which are not normally used⁽¹⁰⁾. However, offline measures tend to be less informative about how certain reading processes operate and are also subject to forgetfulness or reconstructive processes at the time of testing. The literature suggests that textual comprehension can be better understood with convergent evidence obtained with different multiple measures⁽¹⁰⁾.

The retelling task after reading has been used for a long time to assess reading comprehension^(4,11). Nevertheless, as it is of individual application, its use in educational context is difficult. One of the main objectives of this type of evaluation may be to measure how much the reader remembers what he/she read, and what was understood from the text⁽⁴⁾. In the Text Processing Model, when reading a text, the reader should combine the meanings of words to form propositions, which, interrelated, form the microstructure of the text⁽¹²⁾. The microstructure is organized into a global structure called macrostructure. For the formation of the macrostructure, there is recognition of topics and their interrelationships. Together, the macrostructure and the microstructure are called textual basis, which represents the text explicit meaning⁽¹²⁾. In order for deeper text understanding to occur, comprehension should not be restricted to what is explicit in the text⁽¹⁰⁾. In this case, it is necessary to build a situational model (mental model of the situation described in the text). The reader’s mental model can be considered an extended set of propositions, including inferences and propositions extracted from the actual text⁽¹³⁾. The instrument proposed here allows extraction of online and offline measures at the same time.

Multiple choice questions can be considered one of the most practical and objective evaluation techniques, since there is no interference from the reader’s subjectivity. Besides, they are efficient regarding evaluation speed, since enable evaluation of a large number of individuals in a single situation, contributing to time optimization⁽⁹⁾. In addition, they favor control of difficulties of written production.

After defining which method will be used for comprehension evaluation (use of multiple choice questions), it is necessary to define the textual type. According to the literature, text type (narrative, descriptive, etc.), textual genre, its linguistic styles and the various text configurations can bring new issues that are only solved with reading experience⁽¹³⁾. On this subject, simple narratives have been more used for evaluation. The narrative structure presents a logical relation between events and actions of the characters and a macro-textual organization of each of these events⁽⁴⁾. Moreover, they present structural characteristics that demand the proficiency of distinct skills, since they use a variety of time markers, vocabulary and succession of events involving cause and consequence⁽¹⁴⁾.

For evaluation of fluency-related skills, in turn, reading speed is usually measured, where the amount of words read in one minute is calculated for automatic processing, and the amount of words read correctly per minute is used for accuracy. There are other ways of assessing these skills (e.g., measurement of articulation rate, utterance rate), as well as prosodic mastery, but the abovementioned ways are the most practical to obtain information objectively and accurately regarding this aspect of reading⁽⁵⁾.

In clinical and educational practice, there is significant number of students from 11 to 17 years old with complaints of oral language alterations, written language difficulties and disorders, but few instruments are available in the national literature to assess this population’s reading comprehension⁽¹⁵⁾. In general, the instruments are designed for students up to 12 years old or for adults and the elderly. Moreover, no national collective assessment instrument has been found so far enabling assessment of comprehension and textual reading fluency at the same time.

This scarcity hampers identification of adolescents with reading issues in the school environment, as well as the clinical diagnosis and assessment of effectiveness of the interventions proposed. Thus, this study is aimed at presenting the process of developing an instrument for the collective assessment of fluency and comprehension of textual reading for secondary elementary school students, besides verifying the effect of schooling on their performance.

METHODS

This is a cross-sectional analytical observational study, with convenience sampling, approved by the Institution’s Research Ethics Committee under protocol No. 1,722,230.

Participants

The research included 100 Brazilian students of both sexes, aged 11 to 15 years, regularly enrolled at two public secondary elementary schools in the municipality of Belo Horizonte, state of Minas Gerais, divided into four classes: 6^th grade (n = 32), 7^th grade (n = 24), 8^th grade (n = 26), and 9^th grade (n = 18).

The study included students who were considered by the teachers as not presenting learning difficulties in relation to academic performance. Such students signed the Informed Assent Form, and their parents/legal guardians signed the Informed Consent Form. Exclusion criteria were students with current or previous history of neurological or psychiatric disorders and oral and written language changes, reported by their guardians, as well as individuals with uncorrected visual or hearing disorders.

The process of developing the instrument also involved the participation of two judges. The judges were invited for presenting extensive experience in written language and in the development of neuropsychological tests.

Instrument and procedures

Several steps were performed for elaboration of the collective reading comprehension assessment instrument for students in 6th to 9th grade of elementary school. More specifically, it involved seven steps that will be presented below:

Step 1: Initial text selection

Two researchers, Language Speech Therapists, selected seven narrative texts from Brazilian Portuguese books, used in the municipal secondary elementary school of the municipality where the research was conducted.

Step 2: Text selection by a judge

After the initial selection, the texts were submitted to a judge with extensive experience in the written language area, who chose two texts.

Step 3: Computational analysis of texts selected

Both texts were analyzed regarding complexity using the Coh-Metrix-Port 2.0 computational tool⁽¹⁶⁾. The tool analyzed text ambiguity, phrasal structure, semantics, syntax and length, as well as the Flesch Reading Ease Score (FRES), which measures the text complexity, its accessibility in terms of linguistic material and also the reader’s facility in interacting with information made available by that text⁽¹⁶⁾. The Coh-Metrix-Port formula for the Flesch Reading Ease Score is as described in the formula below, where ASL is the average sentence length (the number of words divided by the number of sentences), and ASW is the average number of syllables per word (the number of syllables divided by the number of words): FRES = 206.835 – (1.015 x ASL) – (84.6 x ASW). The results allow identifying four ranges of reading difficulties for the Brazilian Portuguese: texts classified as very easy (index between 75-100), which would be suitable for readers at the beginning of primary elementary school students; easy texts (index between 50-75), which would be suitable for secondary elementary school students; difficult texts (index between 25-50), which would be suitable for students attending high school or university; very difficult texts (index between 0-25), which would generally be suitable for specific academic areas.

Step 4: Narrative text final selection

Posteriorly, the text was analyzed by another judge with experience in elaborating neuropsychological tests. The judge was instructed as to the difficulty classification of the texts analyzed and the language used, selecting what she deemed most appropriate for the school age.

From all analyzes, the text “Por que o morcego só voa à noite” was chosen⁽¹⁷⁾, which is a fable (narrative form) and belongs to the African folktale, from the juvenile literature.

Step 5: Elaboration of questions

For the elaboration of questions, significant propositions that are explicit or implicit in the text were analyzed. The literal or inferential information was verified, as well as the causal relations between these ideas.

The elaboration of the questions was performed by two researchers, using as reference the reading comprehension model⁽¹⁸⁾, and the questions were classified as literal and inferential. Five literal and five inferential questions were elaborated. To elaborate the multiple choice items, a script was used with criteria for elaboration of objective multiple choice questions⁽¹¹⁾. Care has been taken to avoid questions that led to obvious answers or answers easily deducible by the reader’s world knowledge, that is, without having to access knowledge acquired from text reading.

Step 6: Analysis of questions by expert judge

After elaboration of questions and the multiple choice items, the instrument was sent to a judge specialized in the area of reading and writing language, who judged the adequacy of the questions regarding quality, content proposed, and classification of the questions as literal and inferential. Adaptations were made according to the judge’s considerations regarding rewriting of questions with two possible answers, and clarity related to utterance writing and multiple choice items, which could give rise to doubts.

Step 7: Procedure of instrument application on the sample selected.

The application was collective, per school year, and participants were initially informed that the assessment would be performed in two steps, where the first would be text silent reading, and then they should answer the questionnaire related to the text. They were instructed by an examiner with the following orientation: “Read the text very carefully. Start text silent reading and, when I ask you, stop reading and mark the word you are reading with an X. When I signal again, go back to reading from the marked word.” The examiners timed 60 seconds on a digital timer and instructed the adolescents to mark the word they were reading as soon as the evaluator authorized it. After the examiners confirmed the word marking with the students, they restarted reading the text within 15 seconds after the pause. At the end of the reading task, the text was collected and the sheet with questions of text interpretation was handed. The students were instructed as follows: “Read the questions related to the text and mark the answer that you think is correct. Only one alternative will answer the question.”

The classes consisted of approximately 20 students and were monitored by two examiners. The average time to apply the test was 30 minutes, but participants were asked to raise their hands for an examiner to collect the evaluation sheet, and they should leave the instrument application site as soon as they completed the test.

Data analysis

The analysis was performed considering the total of correct answers and the total of correct answers in literal and inferential questions. Reading rate was also calculated, considering the number of words read per minute. Statistical analyzes of textual reading fluency (measured by the number of words read per minute) and reading comprehension (measured by questionnaire performance) performances were conducted using IBM SPSS Statistics 20.0, with a significance level of 5% (0.05). For sampling characterization, mean and standard deviation descriptive analyzes were carried out for the four groups per schooling years.

One-Way Analysis of Variance (One-Way ANOVA) was performed using Bonferroni post hoc test for multiple comparisons to verify differences in reading performance between the groups per grade level (6t^h, 7^th, 8^th and 9^th grades). In a second moment, two groups were formed: Group 1 (6t^h and 7^th grades) and Group 2 (8^th and 9^th grades), since there was no difference between 6t^h and 7^th grades and between 8^th and 9^th grades. To compare the means of the two groups in each reading component, t-test analysis for independent samples was used.

Test power analysis and effect size were analyzed by Cohen's d calculation. To evaluate the association between performance in the tasks of reading fluency, reading comprehension and age, Pearson’s correlation coefficient test was applied.

RESULTS

In step 1, analysis of Brazilian Portuguese didactic books used in a secondary elementary level of a public school in the municipality where the study was conducted resulted in the selection of seven narrative texts.

The seven texts selected were sent to a judge, who elected two of them after careful analysis.

Then, in step 3, the two texts selected were analyzed using the Coh-Metrix-Port 2.0 computational tool⁽¹⁶⁾. The tool investigated text ambiguity, phrasal structure, semantics, syntax and length, and the Flesch Reading Ease Score (FRES). FRES of the texts was rated 50-75 and considered suitable for secondary elementary school students.

In step 4, the texts were analyzed by another judge. From all analyzes, the text “Por que o morcego só voa à noite” was selected⁽¹⁷⁾, a 448-word narrative text that has FRES of 64.4, equivalent to the level “easy” to read and within the educational level of the instrument.

Box 1 presents the narrative text “Por que o morcego só voa à noite”⁽¹⁷⁾.

Box 1
Narrative text “Por que o morcego só voa à noite”

Por que o morcego só voa à noite Há muito e muito tempo, houve uma tremenda guerra entre as aves e o restante dos animais que povoam as florestas, savanas e montanhas africanas. Naquela época, o morcego, esse estranho bicho, de corpo semelhante ao do rato, mas provido de poderosas asas, levava uma vida mansa voando de dia entre as enormes e frondosas árvores à cata de insetos e frutas. Uma tarde, pendurado de cabeça para baixo num galho, ele tirava a soneca costumeira, quando foi despertado bruscamente pelos trinados aflitos de um passarinho: – Atenção, todas as aves! Foi declarada guerra aos quadrúpedes. Todos aqueles que têm asas e sabem voar devem se unir na luta contra os bichos que andam pelo chão. O morcego ainda estava se refazendo do susto, quando uma hiena passou correndo e uivando aos quatro ventos: – Atenção, atenção! Foi declarada guerra às aves! Todos os bichos de quatro patas devem se apresentar ao exército dos animais terrestres. – E agora? Perguntou a si mesmo o aparvalhado morcego. – Eu não sou uma coisa nem outra. Indeciso, não sabendo quem apoiar, resolveu aguardar o resultado da luta: – Eu é que não sou bobo. Vou me apresentar ao lado que estiver vencendo – decidiu. Dias depois, escondido entre as folhagens, viu um bando de animais fugindo em carreira desabalada, perseguidos por uma multidão de aves que distribuía bicadas a torto e a direito. Os donos de asas estavam vencendo a batalha e, por isso, ele voou para se juntar às tropas aladas.

– Uma águia, gigantesca, ao ver aquele rato com asas, perguntou: – O que você está fazendo aqui? – Não está vendo que sou um dos seus? Veja! – disse o morcego abrindo as asas. – Vim o mais rápido que pude para me alistar – mentiu. Oh, queira me desculpar – falou a desconfiada águia. – Seja bem-vindo à nossa vitoriosa esquadrilha. Na manhã seguinte, os animais terrestres, reforçados por uma manada de elefantes, reiniciaram a luta e derrotaram as aves, espalhando penas para tudo quanto era lado. O morcego, na mesma hora, fechou as asas e foi correndo se reunir ao exército vencedor. – Quem é você? – rosnou um leão. – Um bicho de quatro patas como Vossa Majestade – respondeu o farsante, exibindo os dentinhos afiados. – E essas asas? – interrogou um dos elefantes. – Deve ser um espião. Fora daqui! – berrou o paquiderme erguendo a poderosa tromba num gesto ameaçador. O morcego, rejeitado pelos dois lados, não teve outra solução: passou a viver isolado de todo mundo, escondido durante o dia em cavernas e lugares escuros. É por isso que até hoje ele só voa de noite. BARBOSA, Rogério Andrade. Histórias africanas para contar e recontar. São Paulo: Ed. Do Brasil, 2001. p 9-12

In step 5, analysis of propositions made it possible to identify literal and inferential information and causal relations between the ideas in the text. From these analyzes, five literal and five inferential questions were elaborated.

In step 6, the analysis of a judge was performed, as well as necessary adaptations to the multiple choice questions.

Box 2 presents the multiple choice questions elaborated and applied.

Box 2
Multiple choice questions

1 – Como eram os morcegos? (a) Corpo de ave e orelha de rato. (b) Corpo de rato e orelhas de coelho. (c) Corpo de rato e asas de ave. (d) Bico de aves e corpo de hiena.	6 - O que fez o morcego desistir de apoiar as aves? a) As aves não aceitaram o morcego como parte do exército. b) As aves ficaram cansadas durante a guerra e perderam as forças. c) As aves ficaram em desvantagem na guerra após a entrada dos elefantes no grupo dos animais terrestres. d) O morcego não queria mais enganar as aves
2 - Por que o morcego se assustou? (a) Uma hiena passou correndo e uivando anunciando uma guerra entre os quadrúpedes. (b) Um passarinho voou gritando desesperado anunciando uma guerra entre as aves e os quadrúpedes. (c) Uma hiena e um pássaro perseguiram o morcego. (d) Uma hiena e um pássaro estavam lutando em uma guerra.	7 - Qual era o plano inicial do morcego? a) Aguardar o início da guerra e de se juntar ao lado que estivesse vencendo. b) Se esconder e só aparecer quando a guerra acabasse. c) Escolher o lado das aves, já que elas se movimentavam mais rapidamente que os quadrúpedes. d) Escolher o lado dos quadrúpedes, já que eles eram mais fortes do que as aves.
3- O que fez o morcego ficar indeciso ao escolher em qual lado da guerra ficar? (a) O morcego era amigo dos dois grupos de animais e por isso não queria brigar com ninguém. (b) Por ter um corpo com características semelhantes aos dos dois grupos de animais e não saber em qual lado se encaixar. (c) O morcego achava que era um pássaro, mas tinha corpo de rato, então não sabia em qual lado da guerra ficar. (d) : O morcego achava que era um rato e não entendia porque tinha asas, então não sabia em qual lado da guerra ficar.	8 – O que fez o elefante rejeitar o morcego? a) O fato de o morcego ter asas. b) O fato de o morcego ser inimigo dos quadrúpedes. c) O fato de o morcego ser parecido com um rato e elefantes não gostam de ratos. d) O fato de o morcego ter feito amizade com as aves e o elefante havia descoberto.
4- Por que o morcego se refere ao Leão como Vossa Majestade? (a) Porque o leão gostava de ser chamado dessa forma. (b) Porque o morcego e o leão eram amigos, então era uma forma amigável do morcego chamá-lo. (c) Porque o leão é considerado rei da floresta e “Vossa Majestade” é a forma como se refere a um rei. (d) Porque o morcego não sabia qual era o nome do leão, então escolheu uma forma de chamá-lo.	9- Quais eram os hábitos do morcego naquela época? (a) Alimentava-se de sangue e dormia de cabeça para baixo. (b) Voava durante horas e dormia durante toda a tarde. (c) Dormia pendurado em árvores apenas durante o dia. (d) Alimentava-se de insetos e frutas e dormia de cabeça para baixo.
5- Qual o problema do morcego e como ele fez para resolvê-lo? (a) Ele tinha corpo de rato e asas de ave, por isso foi rejeitado pelas aves e pelos quadrúpedes, então decidiu viver recluso em cavernas e só sair à noite. (b) Ele não gostava de brigas e não queria entrar na guerra, então resolveu esconder-se em cavernas e não ser visto por nenhum dos animais. (c) Ele tinha corpo de rato e não gostava de ter asas de ave, então achou melhor viver escondido e só voar à noite para que ninguém visse suas asas (d) Ele não gostava de ter corpo de rato e queria ser uma ave, então decidiu se esconder dos quadrúpedes durante o dia e só voar à noite para que ninguém o visse.	10 – Segundo o texto, o morcego só voa à noite por que: (a) Não sabe voar durante o dia. (b) Ele só enxerga à noite. (c) Ele se alimenta de animais noturnos (d) Tem medo de se encontrar com as aves e com os quadrúpedes.

In step 7, the descriptive analyzes of the performances in textual reading fluency (words read per minute) and reading comprehension (total of correct answers in the questionnaire, total of correct answers in literal and inferential questions) are presented in Chart 1.

Chart 1
Characterization of the participants’ performance in terms of mean, standard deviation and minimum - maximum (N = 100), per group and education level

Grade		N	Mean	Standard Deviation	Minimum	Maximum
WPM	6th grade	32	107.53	41.58	49	265
	7th grade	24	109.71	38.23	49	181
	8th grade	26	124.38	50.34	55	284
	9th grade	18	117.94	31.08	76	182
	Total	100	114.31	41.66	49	284
TCA	6th grade	32	6.44	3.03	1	10
	7th grade	24	5.38	2.49	2	10
	8th grade	26	7.88	1.88	4	10
	9th grade	18	6.94	2.20	3	10
	Total	100	6.65	2.62	1	10
TLQ	6th grade	32	3.09	1.63	1	5
	7th grade	24	2.54	1.74	0	5
	8th grade	26	4.00	1.02	2	5
	9th grade	18	3.44	1.46	1	5
	Total	100	3.26	1.56	0	5
TIQ	6th grade	32	3.34	1.59	0	5
	7th grade	24	2.83	1.27	1	5
	8th grade	26	3.88	1.10	2	5
	9th grade	18	3.50	1.15	1	5
	Total	100	3.39	1.36	0	5

• Caption:. WPM: number of words read per minute; TCA: total of correct answers in the questionnaire; TLQ: total of correct answers in literal questions; TIQ: total of correct answers in inferential questions

The data in Table 1 show increase in the number of words read per minute according as schooling progresses. The comparative analysis between school years showed that there was a statistically significant difference only between 7^th and 8^th grades in the total of correct answers in the questionnaire (p = 0.005, d = 0.60), in literal questions (p = 0.004, d = 0.59), and inferential questions (p = 0.038, d = 0.57), with better performance for 8^th grade students.

No differences were detected between 6^th and 7^th grades or between 8^th and 9^th grades in any component of the reading assessment. From these results, it was decided to form two groups to analyze the data: Group 1 (6t^h and 7^th grades), n = 56, and Group 2 (8^th and 9^th grades), n = 44. Chart 2 presents groups’ performance, the comparison of means and the effect size for reading fluency and comprehension.

Chart 2
Performance (mean, standard deviation) of Group 1 (6^th and 7^th grades), and Group 2 (8^th and 9^th grades), comparison of means (p-value) and effect size (Cohen's d) for reading fluency and reading comprehension

Component	Groups	N	Mean	Standard Deviation	p-value	Cohen’s d
WPM	Group1	56	108.46	39.84	0.89	2.52
	Group 2	44	121.75	43.19
TCA	Group 1	56	5.98	2.84	0.001*	0.60
	Group 2	44	7.50	2.05
TLQ	Group 1	56	2.86	1.68	0.002*	0.60
	Group 2	44	3.77	1.23
TIQ	Group 1	56	3.13	1.47	0.04*	0.45
	Group 2	44	3.73	1.12

• ^*
  p<0.05
• Caption: WPM: number of words read per minute; TCA: total of correct answers in the questionnaire; TLQ: total of correct answers in literal questions; TIQ: total of correct answers in inferential questions.

The comparison between Groups 1 and 2 (Table 2) showed that there was a significant difference in the total number of correct answers in the questionnaire, in the total number of correct answers in literal questions and in the total number of correct answers in inferential questions, with better performance for the higher educated students (Group 2). The effect size was considered strong (d = 2.52). The number of words read per minute did not differ in both groups.

Table 3 shows that there was association between textual reading fluency (words read per minute) and reading comprehension (grand totals, literal and inferential questions). A positive association of moderate intensity (r = 0.302; p≤0.01) was observed between the number of words read per minute and the total of correct answers in the questionnaire.

Table 3
Correlation between reading fluency performance, reading comprehension performance and age

	WPM	TCA	TLQ	TIQ
Idade	−0.009	0.056	0.047	0.055
WPM		0.302**	0.277**	0.264**
TCA			0.911**	0.880**
TLQ				0.605**

• ^**
  p≤0.01
• Caption: WPM: number of words read per minute; TCA: total of correct answers in the questionnaire; TLQ: total of correct answers in literal questions; TIQ: total of correct answers in inferential questions

DISCUSSION

In this study, the development of an instrument for collective assessment of reading fluency and reading comprehension of secondary elementary school students was described, as well as the results of the application of such instrument. The effect of schooling on performance in this instrument was also verified.

To elaborate the instrument proposed by this study, the same text and multiple choice questions were used for all school levels evaluated. The elaboration of the instrument involved seven steps. The text was selected in steps 1 to 4. A narrative text was selected due to the students’ greater familiarity with this textual typology. The literature shows that students may find it easier to interpret a narrative text⁽¹⁹⁾, since narratives allow the reader to recreate his/her knowledge. In addition, they present chronological development, events occur in a certain order, which facilitates reading organization and comprehension⁽¹⁹⁾.

In steps 5 and 6, the questions were elaborated and the significant propositions that were explicit or implicit in the text were analyzed. Literal and inferential information and the causal relations between the ideas were verified. The analysis was performed due to the need for the instrument to contain literal and inferential questions, since the literature proposes that text comprehension occurs at three levels⁽¹⁹⁾. The first would be superficial understanding, called base text, which allows the reader to remember the text, summarize key ideas, and answer questions about the content. The second level is the integration of the information contained in the text with the reader’s prior knowledge. The third level would be self-regulation, which allows the reader to identify problems occurred in the text, seeking solutions to solve them⁽¹⁹⁾.

Multiple choice questions were chosen in order to eliminate interference from the linguistic demands required in open-ended and retelling questions, for example. The literature shows that multiple choice questions allow evaluating the skills involved in comprehension, providing research on the contextual meaning of words, author’s intention and access to literal and inferential information contained in the text⁽²⁰⁾.

According to the results of the performance analysis (step 7), there was only difference between 7^th and 8^th grades regarding answers to the multiple choice questions (reading comprehension), with better performance for the 8^th grade students. No effect of schooling on textual reading fluency was observed. In an attempt to increase the sample size in each group and better analyze performance differences between schooling levels, the students were grouped into 6^th and 7^th grades (Group 1) and 8^th and 9^th grades (Group 2). The effect of schooling was maintained for answer to multiple choice questions, with medium effect⁽²¹⁾. The results indicate that the power of the assessment proposed is moderate for the groups studied⁽²¹⁾. The number of words read per minute did not differ in both groups. However, the effect size was strong, which may indicate that difference may appear between the groups if the number of participants is greater.

Multiple choice tests have many advantages, such as their objectivity, easy administration and punctuation, and the possibility of group application, which is very useful for educational practices in the school environment. However, it is important to point out that such instruments, although providing clearer and more direct analysis of the answers, are subject to the practice of frequently choosing alternatives of random answers. Another problem pointed out by the literature is the possibility of answering some questions without actually reading the passage, since there is probability of using the reader’s world knowledge⁽²²⁾. Bearing these issues in mind, the instrument was built to minimize such problems.

With regard to comparison between the groups, the analysis revealed increased reading fluency (assessed by the number of words per minute) with advancing education, although no statistically significant difference was found. The fact that there were no differences between the groups may point to the stabilization of textual reading fluency in adolescence. The literature shows increased reading fluency during schooling and advancing age⁽²³⁾. Nevertheless, the average reading fluency of adolescents in this study is below the average observed in other studies with readers of American English, European Portuguese, and Brazilian Portuguese^(5,24,25). All the studies cited also used texts to evaluate reading fluency; however, they were different texts regarding type, ambiguity, phrasal structure, semantics, syntax, and length. Another different methodological factor was the way the assessment was carried out. In this study, adolescents were evaluated collectively through silent reading, in which the main objective was comprehension. In the referenced studies, reading was performed orally in order to evaluate reading fluency and decoding. Cultural, educational and socioeconomic factors may also have interfered with these results, given that the Human Development Index (HDI) that evaluates human development in three dimensions – income, education and health – of these three countries are quite distinct, that is, the United States ranks 10^th, with the HDI of 0.920, Portugal ranks 41^st, with the HDI of 0.843, and Brazil ranks 79^th, with the HDI of 0.744⁽²⁶⁾.

The study proposed by the Program for International Student Assessment (PISA, 2015) assesses the knowledge and skills of students aged 15 to 16 years in reading, math and science, contrasting with results of students’ performance from other countries. PISA test assesses students’ mastery of three aspects of reading: locate and retrieve information; integrate and interpret, and reflect and analyze. Among the 70 nations participating in the assessment, Brazilian students ranked 59^th in reading. The performance average showed the second fall since 2009⁽²⁷⁾. Due to this reality, development of validated and standardized instruments that allow teachers to monitor students’ evolution throughout the year becomes one of the ways of continuous control of reading development, besides being and early indication of difficulties for referrals and guidance of individuals and groups at the risk of learning issues or educational strategies.

When comparing Groups 1 and 2, it is possible to observe better performance of older students when answering literal questions and questions that required the construction of inferences, which indicates that, with increased education, there is improvement in deductive reasoning ability, since students tend to develop and improve reading processes according as they develop the techniques and practice this skill. Other studies have also shown improved reading comprehension with increased education^(5,25,28).

The results show positive correlations between reading fluency and reading comprehension with a moderate correlation magnitude. This may indicate that there is a relation between textual reading fluency and reading comprehension, as evidenced in the literature^(5,29).

Several studies show that as the reader reaches a functional level of decoding, the importance of reading fluency for reading comprehension decreases^(23,30). Another study, conducted with students from 1^st to 6^th grade (current 2^nd and 7^th grades according to the Ministry of Education, 2005), revealed that reading fluency is not the most significant factor for reading comprehension in 6^th grade, and it is more important in early grades⁽³⁰⁾. Thus, it is relevant to consider age and educational level of the population studied when coming to a conclusion on the processes involving reading and their associations.

The assessment of reading comprehension is paramount, since it detects possible difficulties presented by adolescents, which may generate several negative consequences during school years. This instrument can assist in the assessment of this target audience by various professionals involved in reading learning (teachers, speech therapists, psychopedagogy professionals, psychologists, neuropsychologists, etc.). It can help educators regarding teaching proposals, monitoring the evolution of reading and comprehension skills and identifying possible reading difficulties, associated with assessments of other skills, and then verify the need for making specific referrals; and it can help clinicians by evaluating and detecting difficulties in order to monitor the evolution of the case, allowing the professional to elaborate and direct the interventions to be performed.

The great advantage is the possibility of collective administration of the instrument, which can be conducted in the classroom or in assistance groups, facilitating application and interpretation of results, useful in clinical and educational actions. Moreover, the instrument proposed here allows online and offline measure extraction at the same time, since reading time is monitored and multiple choice questions are presented after reading.

Due to the limitations of the study, such as a small sample and coming from only one type of school, the need for the instrument to be applied to greater number of adolescents is emphasized. Thus, it can present recognized reliable and valid measures, in order to provide the researcher, the teacher and the clinician with the possibility of detecting adolescents with comprehension issues, aiming to adequately direct the interventions that will be used and thus gather evidence that will support their scientific, clinical and scholarly reasoning. It is also suggested that the studies should be conducted with adolescents from public and private schools, with and without learning disabilities in order to validate, standardize and increase the reliability of the instrument experimentally applied in this study.

CONCLUSION

This article aimed to present the development process and the pilot study of an instrument to evaluate textual reading fluency and reading comprehension in adolescents. The analyzes carried out allowed investigating the results of the development of an instrument for collective assessment of reading comprehension of students in 6t^h to 9^th grade, verifying the evolution of the performance of adolescents over the school years.

The results evidenced increased reading comprehension performance as schooling progressed. The instrument presented statistically significant results with medium effect and reading fluency showed positive and moderate correlations with reading comprehension.

The emphasis is on the need for further research that should be performed using this instrument so that to increase the number of students evaluated, make comparisons between public and private institutions, and between typical adolescents and adolescents with oral and written language disorders.

• Study conducted at Departamento de Fonoaudiologia, Universidade Federal de Minas Gerais – UFMG - Belo Horizonte (MG), Brasil.
• Financial support: nothing to declare.

REFERÊNCIAS

• ¹
  Reynolds MR, Turek JJ. A dynamic developmental link between verbal comprehension-knowledge (Gc) and reading comprehension: verbal comprehension-knowledge drives positive change in reading comprehension. J Sch Psychol. 2012;50(6):841-63. http://dx.doi.org/10.1016/j.jsp.2012.07.002 PMid:23245503.
  » http://dx.doi.org/10.1016/j.jsp.2012.07.002
• ²
  Solari EJ, Grimm R, McIntyre NS, Lerro LS, Zajic M, Mundy PC. The relation between text reading fluency and reading comprehension for students with autism spectrum disorders. Res Autism Spectr Disord. 2017;41/42:8-19. http://dx.doi.org/10.1016/j.rasd.2017.07.002
  » http://dx.doi.org/10.1016/j.rasd.2017.07.002
• ³
  Tiffin-Richards SP, Schroeder S. The component processes of reading comprehension in adolescents. Learn Individ Differ. 2015;42:1-9. http://dx.doi.org/10.1016/j.lindif.2015.07.016
  » http://dx.doi.org/10.1016/j.lindif.2015.07.016
• ⁴
  Corso HV, Sperb TM, Salles JF. Desenvolvimento de instrumento de compreensão leitora a partir de reconto e questionário. Rev SLAN. 2012;4(2):22-32.
• ⁵
  Chang EM, Avila CRB. Compreensão leitora nos últimos anos dos ciclos I e II do Ensino Fundamental. CoDAS. 2014;26(4):276-85. http://dx.doi.org/10.1590/2317-1782/201420130069 PMid:25211686.
  » http://dx.doi.org/10.1590/2317-1782/201420130069
• ⁶
  Kim YS, Wagner R, Foster E. Relations among oral Reading fluency, silent reading fluency, and reading comprehension: A latent variable study of first-grade readers. Sci Stud Read. 2011;15(4):338-62. http://dx.doi.org/10.1080/10888438.2010.493964 PMid:21747658.
  » http://dx.doi.org/10.1080/10888438.2010.493964
• ⁷
  Santos AAA. O Cloze como técnica de diagnóstico e remediação da compreensão em leitura. Interação Psicol. 2004;8(2):217-26. http://dx.doi.org/10.5380/psi.v8i2.3257
  » http://dx.doi.org/10.5380/psi.v8i2.3257
• ⁸
  Cartwright KB, Coppage EA, Lane AB, Singleton T, Marshall TR, Bentivegna C. Cognitive flexibility deficits in children with specific reading comprehension difficulties. Contemp Educ Psychol. 2017;50:33-44. http://dx.doi.org/10.1016/j.cedpsych.2016.01.003
  » http://dx.doi.org/10.1016/j.cedpsych.2016.01.003
• ⁹
  Gorjian B. The effect of passage content on multiple-choice reading comprehension test. Procedia Soc Behav Sci. 2013;84:160-4. http://dx.doi.org/10.1016/j.sbspro.2013.06.528
  » http://dx.doi.org/10.1016/j.sbspro.2013.06.528
• ¹⁰
  Kintsch W, Rawson KA. Comprehension. In: Snowling M, Hulme C. (Eds.). The science of reading: a handbook. Oxford, UK: Blackwell; 2005.p.209-226. http://dx.doi.org/10.1002/9780470757642.ch12
  » http://dx.doi.org/10.1002/9780470757642.ch12
• ¹¹
  Cunha VLO, Capellini SA. Construção e validação de instrumento de avaliação da compreensão de leitura para escolares do terceiro ao quinto ano do Ensino Fundamental. CoDAS. 2014;26(1):28-37. http://dx.doi.org/10.1590/s2317-17822014000100005 PMid:24714856.
  » http://dx.doi.org/10.1590/s2317-17822014000100005
• ¹²
  Kintsch W, Van Dijk TA. Toward a model of text comprehension and production. Psychol Rev. 1978;85(5):63-94. http://dx.doi.org/10.1037/0033-295X.85.5.363
  » http://dx.doi.org/10.1037/0033-295X.85.5.363
• ¹³
  Perfetti CA, Landi N, Oakhill J. The acquisition of reading comprehension skill. In: Snowling M, Hulme C (Eds.). The science of reading: a handbook. Oxford, UK: Blackwell; 2005.p. 227-247. http://dx.doi.org/10.1002/9780470757642.ch13
  » http://dx.doi.org/10.1002/9780470757642.ch13
• ¹⁴
  Coelho CLG, Correa J. Compreensão de leitura: habilidades cognitivas e tipos de texto. PSICO. 2017;48(1):40-9. http://dx.doi.org/10.15448/1980-8623.2017.1.23417
  » http://dx.doi.org/10.15448/1980-8623.2017.1.23417
• ¹⁵
  Corso HV, Piccolo LR, Miná CS, Salles JF. Coleção Anele - Avaliação da Compreensão de Leitura. Vol. 2. São Paulo: Vetor Editora; 2017.
• ¹⁶
  Scarton CE, Aluísio SM. Análise da Inteligibilidade de textos via ferramentas de Processamento de Língua Natural: adaptando as métricas do Coh-Metrix para o Português. Lingua. 2010;2010(2):45-62.
• ¹⁷
  Barbosa RA. Histórias africanas para contar e recontar. São Paulo: Editora Do Brasil; 2001.p. 9-12.
• ¹⁸
  Kintsch W. Comprehension: A paradigm for cognition. New York: Cambridge University Press; 1998.
• ¹⁹
  Sánchez EM. Compreensão e redação de textos. Dificuldades e ajudas. Porto Alegre: Artmed; 2002.
• ²⁰
  Kida ASB, Chiari BM, Ávila CRB. Escala de leitura: proposta de avaliação das competências leitoras. Rev Soc Bras Fonoaudiol. 2010;15(4):546-53. http://dx.doi.org/10.1590/S1516-80342010000400012
  » http://dx.doi.org/10.1590/S1516-80342010000400012
• ²¹
  Cohen J. Statistical power analysis for the behavioral sciences. 2. ed. Hillsdale, NJ: Lawrence Earlbaum Associates; 1988.
• ²²
  Keenan JM, Betjemann RS, Olson RK. Reading comprehension tests vary in the skills they assess: differential dependence on decoding and oral comprehension. SSSR. 2008;12(3):281-300. http://dx.doi.org/10.1080/10888430802132279
  » http://dx.doi.org/10.1080/10888430802132279
• ²³
  Fernandes S, Querido L, Verhaeghe A, Marques C, Araújo L. Reading development in European Portuguese: relationships between oral reading fluency, vocabulary and reading comprehension. Read Writ. 2017;30(9):1987-2007. http://dx.doi.org/10.1007/s11145-017-9763-z
  » http://dx.doi.org/10.1007/s11145-017-9763-z
• ²⁴
  Fernandes S, Simões C, Querido L, Verhaeghe A. Fluência na leitura oral de texto e de palavras: estudo transversal com adolescentes portugueses. RIDEP. 2015;39(1):113-24.
• ²⁵
  Hasbrouck J, Tindal GA. Oral reading fluency norms: A valuable assessment tool for reading teachers. Read Teach. 2006;59(7):636-44. http://dx.doi.org/10.1598/RT.59.7.3
  » http://dx.doi.org/10.1598/RT.59.7.3
• ²⁶
  Jahan S. Human Development Report 2016: Human Development for Everyone. New York. United Nations Development Programme; 2016.
• ²⁷
  PISA: Programa Internacional de Avaliação de Estudantes, OCDE- Organização para a Cooperação e Desenvolvimento Econômico [Internet]. Brasil no PISA 2015: Análises e reflexões sobre o desempenho dos estudantes brasileiros. São Paulo: Fundação Santillana; 2016 [citado em 2018 Nov 24]. Disponível em: http://download.inep.gov.br/acoes_internacionais/pisa/resultados/2015/pisa2015_completo_final_baixa.pdf
  » http://download.inep.gov.br/acoes_internacionais/pisa/resultados/2015/pisa2015_completo_final_baixa.pdf
• ²⁸
  Joly MCRA, Bonassi J 2nd, Dias AS, Piovezan NM, Silva DV. Avaliação da compreensão de leitura pelo Sistema Orientado de Cloze (SOC). Fractal. Rev Psicol. 2014;26(1):223-42.
• ²⁹
  Bigozzi L, Tarchi C, Vagnoli L, Valente E, Pinto G. Reading fluency as a predictor of school outcomes across grades 4-9. Front Psychol. 2017;8:200. http://dx.doi.org/10.3389/fpsyg.2017.00200 PMid:28261134.
  » http://dx.doi.org/10.3389/fpsyg.2017.00200
• ³⁰
  Ouellette G, Beers A. A not-so-simple view of reading: how oral vocabulary and visual-word recognition complicate the story. Read Writ. 2010;23(2):189-208. http://dx.doi.org/10.1007/s11145-008-9159-1
  » http://dx.doi.org/10.1007/s11145-008-9159-1

Publication Dates

• Publication in this collection
  02 Mar 2020
• Date of issue
  2020

History

• Received
  31 Jan 2019
• Accepted
  30 May 2019