Brain activation for reading and listening comprehension: an fMRI study of modality effects and individual differences in language comprehension

SPECIAL SECTION ON LANGUAGE AND COGNITIVE NEUROSCIENCE

Brain activation for reading and listening comprehension: an fMRI study of modality effects and individual differences in language comprehension

Augusto Buchweitz^I; Robert A. Mason^I; Lêda M. B. Tomitch^II; Marcel Adam Just^I

^ICarnegie Mellon University, USA

^IIUniversidade Federal de Santa Catarina, Brazil. Marcel Adam Just, Carnegie Mellon University, USA

^{Correspondence regarding this article should be directed to}

ABSTRACT

The study compared the brain activation patterns associated with the comprehension of written and spoken Portuguese sentences. An fMRI study measured brain activity while participants read and listened to sentences about general world knowledge. Participants had to decide if the sentences were true or false. To mirror the transient nature of spoken sentences, visual input was presented in rapid serial visual presentation format. The results showed a common core of amodal left inferior frontal and middle temporal gyri activation, as well as modality specific brain activation associated with listening and reading comprehension. Reading comprehension was associated with more left-lateralized activation and with left inferior occipital cortex (including fusiform gyrus) activation. Listening comprehension was associated with extensive bilateral temporal cortex activation and more overall activation of the whole cortex. Results also showed individual differences in brain activation for reading comprehension. Readers with lower working memory capacity showed more activation of right-hemisphere areas (spillover of activation) and more activation in the prefrontal cortex, potentially associated with more demand placed on executive control processes. Readers with higher working memory capacity showed more activation in a frontal-posterior network of areas (left angular and precentral gyri, and right inferior frontal gyrus). The activation of this network may be associated with phonological rehearsal of linguistic information when reading text presented in rapid serial visual format. The study demonstrates the modality fingerprints for language comprehension and indicates how low- and high working memory capacity readers deal with reading text presented in serial format.

Keywords: fMRI, language comprehension, reading span.

Introduction

Linguistic information can be conveyed in the form of speech and written text, but it is the content of the message that is ultimately essential for the higher-level processes in language comprehension, such as making inferences and associations between text information and knowledge about the world. The goal of this study was to investigate the brain activation for listening and reading comprehension processes and the effects of modality of language input on brain activation for language comprehension. The study also aimed to investigate individual differences in brain activation for individuals with high or low language processing capacity, as indexed by the Daneman and Carpenter (1980) reading span test.

Good readers tend to be good listeners, and good listeners tend to be good readers. Behavioral studies have shown that listening and reading comprehension are two closely-related skills. As schooling increases, so does the strength of the correlation between reading and listening comprehension performance (Just & Carpenter, 1987). Skilled readers retrieve phonological information faster and more automatically than less skilled readers (Booth, Perfetti, & MacWhinney, 1999; Booth, Perfetti, MacWhinney, & Hunt, 2000). Successful reading relies on an interaction between decoding linguistic visual input and accessing phonological information.

Models of language comprehension describe higher-level cognitive processes of text comprehension (inference-making and semantic access, for example) as amodal processes. This means that higher-order cognitive processes draw on the manipulation of text information in abstract form (Mason & Just, 2006). Kintsch (1998) proposed that discourse processing is based on the semantic structure of text, which is broken down into amodal units of meaning called propositions. According to the model, comprehension thrives on the integration of this amodal, propositional information. Booth et al.'s (2002a; 2002b) model of language comprehension, which is based on evidence from brain imaging studies, postulates that although auditory and visual word form processing involves distinct cortical areas, semantic processing is independent of input modality. Semantic processes are associated with the same network of brain activation whether the input is visual or auditory.

Brain imaging studies of language comprehension have demonstrated that there is a comparable network of areas of the brain activated in higher-order cognitive processes of reading and listening comprehension. There is a high similarity in cortical areas recruited for listening and reading comprehension processes at the word, sentence, and discourse level (Jobard, Vigneau, Mazoyer, & Tzourio-Mazoyer, 2007). Brain imaging studies convincinly point to the same direction of behavioral studies, indicating that the higher-order processes of listening and reading comprehension are intertwined rather than separate. On the one hand, there are differential cortical areas recruited by modality-specific processes, such as the processing of word form by the visual word form area (Cohen et al., 2002). Yet there are comparable areas recruited by amodal processes such as inference-making and other higher-level cognitive processes (Booth et al., 2002a; Carpentier et al., 2001; Constable et al., 2004; Jobard, Crivello, & Tzourio-Mazoyer, 2003; Michael, Keller, Carpenter, & Just, 2001). Brain imaging studies of complexity effects on brain activation have also shown comparable modulation of brain activity by task complexity in both listening and reading comprehension tasks (Carpentier et al., 2001; Constable et al., 2004; Just, Carpenter, Keller, Eddy, & Thulborn, 1996; Keller, Carpenter, & Just, 2001; Michael et al., 2001).

The amodal network of cortical areas involved in language processing, called the language network, centralizes higher-order language comprehension processes. The modality-specific, or dedicated, primary areas are involved in lower-level processing (Jobard et al., 2007; Mesulam, 1998). One of the amodal centers of language processing in the brain, or amodal computation node, is the left inferior frontal gyrus (LIFG). In turn, the primary areas, or modality-specific areas, include the primary auditory and visual cortices. Booth and colleagues (2002a) reported modality-independent activation in semantic processing over the LIFG and left middle temporal gyrus, but modality-specific activation of the fusiform gyrus (written words) and of the superior temporal gyrus (spoken words). Brain imaging studies have mapped specific linguistic integrative and input-tomeaning functions onto these known cortical structures: (1) the LIFG, for amodal processes, integration of textual information, and response to language task complexity; (2) the auditory cortex, for the primary processing of speech and the decoding of acoustic features in speech (prosody, for example); and (3) the visual cortex and the fusiform gyrus, for the processing of visual input to meaning.

The processing of abstract, amodal information in language

The well-known left-hemisphere language network implicated in the processing of language and discourse includes the LIFG, the superior and middle temporal gyri, the inferior temporal gyrus, and the angular gyrus (Bookheimer, 2002). The LIFG is implicated in a variety of language tasks, which include production and comprehension (Schlosser, Aoyagi, Fullbright, Gore, & McCarthy, 1998) and retrieval of meaning (Wise & Price, 2006). Lesions to the portion of the LIFG known as Broca's Area (the LIFG pars opercularis, or BA 44, according to Broca's original definition, but currently including a larger portion of the IFG (Bookheimer, 2002)) are associated with language production disorders, such as articulatory and speech aphasias and naming deficits (Bookheimer, 2002; Obler & Gjerlow, 1999).

The LIFG is implicated in several specialized language tasks that require some level of manipulation and integration of amodal information. LIFG activation has been associated with syntactic and semantic processing (Keller et al., 2001), increasing comprehension workload (Constable et al., 2004; Just et al., 1996; Michael et al., 2001), increasing complexity of language units (word, sentence, text) (Jobard et al., 2007), and reading tongue-twister sentences (Keller, Carpenter, & Just, 2003). The LIFG has been described as part of a network of cortical areas activated in the integration of information into the reader's understanding of text. This is similar to what models of comprehension call the mental (situation) model (Mason & Just, 2006).

The decoding of speech

The posterior portion of the superior temporal gyrus, or STG (BA 42), and the superior temporal sulcus (BAs 22 and 42) are also associated with general processing of auditory input (Cabeza & Nyberg, 2000; Jobard et al., 2003; Schlosser et al., 1998). A number of functional imaging studies of comprehension have reported differential activation of STG in listening comprehension relative to reading comprehension (Carpentier et al., 2001; Constable et al., 2004; Jobard et al., 2007; Michael et al., 2001).

The decoding of written text

The primary visual cortex decodes visual information from printed words, sentences, and text. Neuroimaging studies of lexical tasks have also revealed the importance of the fusiform gyrus for lexical processing in writing systems of several languages, including alphabets and logographs (Bolger, Perfetti, & Schneider, 2005; Cohen et al., 2000; 2002; Jobard et al., 2007; Tan, Laird, Li, & Fox, 2005). In terms of subspecialization of cortical function, there is a portion of the fusiform gyrus referred to as the visual word form area (VWFA) that is paramount for the lexical processes that bridge the gap between linguistic visual input and speech representations.

The VWFA is largely implicated in mapping visual information to meaning or retrieval of meaning (Cohen et al., 2000; 2002; Cohen & Dehaene, 2004), especially in reading ideograms (Bolger et al., 2005). It processes information of fine-grained visual form such as (but not exclusively) information required for discriminating between words and for combining visual and verbal linguistic information (Devlin, Jamison, Gonnerman, & Mathews, 2006; Vigneau, Jobard, Mazoer, & Tzourio-Mazoer, 2005). Lesions to the VWFA are associated with impairments in oral reading and oral naming tasks (Hills et al., 2005).

However, based on a metanalysis of 35 neuroimaging studies, Jobard and colleagues (2003) argued that the concept of a VWFA invites more refined investigation and that neuroimaging studies have yet to consistently corroborate the concept of a written word lexicon in the brain. It is possible that VWFA activation is more specific to reading words in deep orthographies.

The different levels of complexity of the mapping between the printed word and the sounding out of that word may be associated with different cognitive processes, and may be underpinned by different networks of cortical activation. For example, differences in brain activation have been reported for Italian (transparent orthography) and English (nontransparent, deep orthography) reading comprehension. For Italian reading comprehension there was a language-specific left superior temporal activation, whereas for English reading there was a language-specific left inferior posterior temporal activation (Paulesu et al., 2000). Portuguese, in a putative continuum of orthographic transparency, is less transparent than Italian and Spanish, but it is more transparent than French and, especially, English (Seymour, Aro, & Erskine, 2003).

For shallow orthographies, the development of reading skill may be more strongly associated with phonological processing abilities. For deep orthographies, the development of reading skill may be more strongly associated with the ability to combine visual and phonological processing abilities. Paulesu and colleagues (2001) found that dyslexic readers in Italian (transparent orthography) perform better in reading tasks than dyslexics who read English and French. The study shows an effect of cultural diversity on language processing. Of course, independently of the language, dyslexic readers always performed worse than normal, control readers. A cross-linguistic comparison among different orthographies showed that children learning shallow orthographies become accurate and fluent in reading words sooner than children who must learn to read French, Danish, and particularly English (Seymour et al., 2003). An investigation of children learning to read in Portuguese showed a strong correlation of higher-span reading skills with phonological processing abilities and a weak correlation with visual processing abilities (Capovilla, Capovilla, & Suiter, 2004a; Capovilla & Capovilla, 2004b).

Individual differences in reading comprehension skill, right-hemisphere spillover of activation, and reading text in rapid serial visual presentation format

Reading comprehension is a complex, higher-level cognitive process in which there are systematic individual differences in skill and performance. For example, good readers are faster and more accurate at comprehension of syntactically complex sentences than are poor readers (Just & Carpenter, 1987). One of the questions that brain imaging studies have attempted to answer is which individual differences in brain functioning underpin individual differences in reading skill.

Individual differences in reading comprehension are likely to be associated with a quantifiable measure of consumption of brain resources during task performance. Resource consumption can be measured by the amount of brain activation in different areas of the brain. In a study of sentence comprehension with varying lexical and syntactic difficulty, Prat, Keller and Just (2007) found greater right-hemisphere activation in less-skilled participants than in skilled participants. In other words, to perform the more difficult tasks, less-skilled participants required additional recruitment of brain activation in right-hemisphere homologues of the areas usually activated during language comprehension (see Prat & Just, 2008 for a discussion of the brain bases of individual differences in language comprehension).

In the present study the reading task was a relatively easy one for college-level students. We did not modulate sentence syntactic or lexical difficulty. However, because the reading task was presented in a novel, unconventional form (serial presentation), we expected to find evidence of different brain activation between high and low capacity readers for dealing with the reading task. The visual stimuli were presented in rapid serial visual presentation format (RSVP), where words appear on the screen one at a time. RSVP is an unconventional form of rapid reading developed by Forster (1970). RSVP differs from normal reading because the duration of gaze on each word is not under the control of the reader, words cannot be skipped, and the words that have already been read cannot be read again (no backtracking).

Method

Participants

Twelve right-handed speakers of Portuguese as a first language (eight males), mean age 29.9 years (SD = 5.74; range 20-40 years), were recruited for the study. Participants were highly educated: Eleven were enrolled in graduate school programs and one was in the senior year of college. All participants were financially compensated for the practice session and the fMRI data collection. Each participant gave signed informed consent approved by the University of Pittsburgh and Carnegie Mellon University Institutional Review Boards.

Stimulus

The stimuli consisted of 24 statements (12 in print, 12 spoken, all in Portuguese) about general world knowledge, for example: O Everest é conhecido como a montanha mais alta do mundo e está localizado no Nepal ("The Everest is known as the highest mountain in the world and is located in Nepal"). The sentences were controlled for topic, length, and duration of play. The auditory and visual sentences were constrained to 12 to 16 words in length. Both auditory and visual stimuli were presented for a total of six seconds. Visual stimuli were further constrained to 65 to 85 characters, to maintain consistency in word presentation speed across sentences. Several other precautionary steps were taken: Auditory sentences were digitally recorded and (1) spoken with little or no prosody; (2) digitally edited for comparable volume (Goldwave v5.06, Goldwave inc.); and (3) edited for onset at time zero and completion within milliseconds of time = 6 sec. Visual sentences had no punctuation other than the final period.

Visual sentences were presented one word at a time to control for visual processing rate, which would be somewhat up to the participant if sentences were presented in their entirety on the screen. Both auditory and visual sentences had to be processed sequentially and within the same amount of time (6 sec). Hence, the reading stimuli did not allow for backtracking.

fMRI scanner and acquisition parameters

Imaging was done on a Siemens Allegra 3.0 Tesla scanner used in conjunction with a commercial birdcage, quadrature-drive radio-frequency whole-head coil. Sixteen oblique-axial images were selected to maximize coverage of the entire cortex. The images were collected using an EPI acquisition sequence, with TR = 1000 ms, TE = 30 ms, flip angle = 60º, and a 64 x 64 acquisition matrix with a voxel size of 3.135-mm x 3.125-mm x 5-mm with a 1-mm gap. The volume scan was constructed from 160 3DMPRAGE oblique-axial images that were collected with TR = 2000 ms, TE = 3.34 ms, 7º flip-angle, and a 256 × 256 FOV, resulting in 1-mm x 1-mm x 1-mm voxels.

Procedure

Participants were positioned in the scanner bed as comfortably as possible, with their head strapped to the head coil to help avoid excessive motion. An angled mirror was adjusted in front of the eyes to reflect the visual stimuli projected onto a rear-projection screen in the bore of the magnet. Audio stimuli were transmitted via special headphones designed to fit the ears snugly and reduce scanner noise interference. Visual stimuli were displayed word-by-word on the center of the screen using rapid serial visual presentation (RSVP). RSVP word presentation rate for each word for each sentence was calculated separately for each stimulus sentence. Word presentation rate for each sentence was calculated with a formula that incorporated each word length times 50 milliseconds, then the sum of all these multiplications was subtracted from six seconds, and the result then divided by the number of words in the sentence. The result of this subtraction-and-division is a variable intercept, which was added to the number obtained in the multiplication of word length times 50 milliseconds. This addition gave the total time a word was to be presented on the screen.

Sentences were displayed as a single condition (Portuguese listening alone or Portuguese reading alone) in two blocks of six sentences for each condition. In between each block of six sentences, an "X" was presented on the center of the screen (fixation condition). During fixation, participants were instructed to clear their mind. To ensure that participants were reading and listening to the sentences, there were True or False (T/F) questions after each sentence. Twenty-five percent of sentences were false.

Behavioral data analysis

Participants responded to sentences presented visually or auditorily using mice buttons. Anatomical mapping of response fingers was used such that the index finger on the left hand was used to respond to true statements, and the index finger on the right hand to respond to false statements. The response options were rear projected onto a screen 20 cm from the participant. True or false responses and response times were collected in the scanner as participants pressed the mice buttons. Response times were recorded by the Coglab experimental software.

fMRI data analysis - Distribution of activation

Data were analyzed using SPM2 (Wellcome Department of Cognitive Neurology, University College, London). Images were corrected for slice acquisition timing, motion-corrected, normalized to the Montreal Neurological Institute (MNI) template, resampled to 2-mm x 2-mm x 2-mm voxels, and smoothed with an 8-mm Gaussian kernel to decrease spatial noise. Statistical analysis was performed on individual and group data by using the general linear model and Gaussian random field theory as implemented in SPM2 (Friston et al., 1995). Group analyses were performed using a random-effects model. Statistical maps were superimposed on normalized T1-weighted images. Automated anatomical labeling (AAL) (Tzourio-Mazoyer et al., 2002), as implemented in the SPM2 software, was employed to name activation cluster centroids and adjacent areas of activation.

Correlation with reading span scores

The reading span scores for each participant were correlated with the individual images for each participant's contrast between reading comprehension and fixation, as implemented in SPM2. The reading span scores were also correlated with a measure of recruitment of right-hemisphere areas: the ratio of total voxels recruited in left hemisphere in relation to voxels recruited in the right hemisphere for reading comprehension. The participants' reading span scores ranged from 2 to 5 (M = 2.79; SD = 0.86).

Results

Behavioral results

There were no significant differences between the results for Portuguese listening and reading comprehension accuracy (listening comprehension M = 0.90, SD = 0.10; reading comprehension M = 0.90, SD = 0.12) or response times (listening comprehension M = 907 ms, SD = 329; reading comprehension M = 1030 ms, SD = 323). As expected, input modality did not have an effect on speed of processing or accuracy of sentence comprehension.

fMRI results - Common network of activation for listening and reading comprehension

Listening and reading comprehension activated a common left-lateralized network of areas in left inferior frontal gyrus (LIFG) and left middle temporal gyrus (LMTG). These areas are part of a modality-independent network of activation associated with language comprehension (Booth el al., 2002a; Constable et al., 2004). Right inferior frontal gyrus was also activated for both reading and listening comprehension. The common areas of activation are shown in white in Figure 1. Listening comprehension showed large clusters of activation in bilateral (posterior and anterior) superior temporal and middle temporal gyri. Bilateral posterior superior temporal lobe activation has been consistently associated with auditory comprehension in brain imaging studies of language comprehension (Constable et al., 2004; Jobard et al., 2007; Michael et al., 2001). In turn, reading comprehension showed activation of bilateral fusiform gyri. The left fusiform gyrus is an area associated with the processing of written words.

Modality fingerprints in brain activation: listening > reading and more overall activation of the brain for listening comprehension

The contrast between listening and reading comprehension showed more activation for listening comprehension in bilateral superior temporal gyri. There was also more activation for listening comprehension in bilateral middle temporal gyri, right angular gyrus, and right insula. Listening comprehension was also associated with more overall activation of the whole brain when compared to reading comprehension (listening > fixation: 22,682 total voxels activated; reading > fixation: 3,579 total voxels for the contrasts across all participants, see Tables 1 and 2).

Reading > listening and left-lateralized brain activation for reading comprehension

Reading comprehension showed more activation in the left inferior occipital lobe, including the left fusiform gyrus (Figure 2, Table 3). Activation of the occipital lobe is associated with processing visual stimuli and is consistent with reading-specific activation found in other studies (Constable et al., 2004). In most participants, the brain activation for reading comprehension was leftlateralized. Eight participants had more activated voxels in the left hemisphere than in the right hemisphere for reading comprehension. The average ratio of left-toright hemisphere voxels was 2.5 (SE = 1.2) (average number of voxels activated in left hemisphere = 1864.5; SE = 680.5; average number of voxels activated in right hemisphere = 1718.1, SE = 630.7).

The brain activation for reading comprehension was more left-lateralized than the brain activation for listening comprehension. Left-lateralized brain activity for reading comprehension has been widely reported in studies of reading comprehension (Constable et al., 2004; Jobard et al., 2007; Just et al., 1996; Michael et al., 2001). Reading comprehension had a significant number of active voxels in areas associated with visual processing. There was activation in clusters in the left inferior occipital cortex and in the right inferior occipital lobe (Table 2).

Individual differences in brain activation for reading comprehension

The results show that the ratio of voxels recruited in the left hemisphere versus voxels recruited in the right hemisphere was positively correlated with reading span scores (r = .76; p < .01). This indicates that lower capacity readers had to recruit significantly more voxels in right-hemisphere areas of the brain than higher capacity readers. This result corroborates previous findings of spillover of brain activation in less-skilled readers (Prat et al., 2007).

The correlation between reading span scores and brain activation for reading comprehension showed two characteristics of individual differences in brain activation for reading comprehension in rapid serial format. Lower capacity readers (low spans) showed more activation in the left middle frontal gyrus, an area associated with executive control and strategic processes. Higher capacity readers (high spans) showed more activation in left angular, precentral and postcentral gyri, and right inferior frontal gyrus. The network of activation for better readers may be associated with phonological rehearsal. The differences in correlation of reading ability and brain activation may be evidence of different strategies for reading comprehension in RSVP format between higher and lower capacity readers (Figure 3). There were no significant correlations between reading span scores and the brain activation for listening comprehension.

Discussion

The results demonstrate the modality fingerprints for brain activation for listening and reading comprehension of Portuguese sentences and the brain activation associated with individual differences in reading text presented in serial format. The brain activation for listening and reading comprehension showed that processing of speech and print resulted in differential activation in modality-sensitive areas. In addition, listening comprehension resulted in more overall activation over the whole brain. The extensive red-colored clusters of activation in Figure 1 illustrate the difference in overall brain activation for listening comprehension. The brain activation for reading comprehension was more left-lateralized. These results corroborate previous studies of listening and reading comprehension (Constable et al., 2004; Jobard et al., 2007; Michael et al., 2001).

It is interesting to compare the results from the present study to studies of the influence of input modality in other languages. The present study showed that the activation for reading comprehension was more left-lateralized, and listening comprehension, more bilaterally distributed. The laterality difference between activation for comprehension of print and speech corroborates studies that compared brain activation for listening and reading activation in other languages (Constable et al., 2004; Jobard et al., 2007; Michael et al., 2001). One explanation proposed for the difference in laterality of brain activity for reading and listening comprehension is developmental. Listening comprehension begins at a much earlier stage in life than reading comprehension. Consequently, an established left-laterality of language processing may influence a left-disposition of brain response to visual linguistic stimulus (Michael et al., 2001).

Other studies of modality effects on brain activation have reported differences in the location of LIFG activation for listening and reading comprehension. The difference in LIFG activation was modulated by task difficulty. The studies showed an increase in activation in pars triangularis (BA 45) associated with increasing difficulty of the comprehension tasks (Carpentier et al., 2001; Constable et al., 2004; Michael et al., 2001). However, the studies also showed more modality-specific (and not task-specific) activation in LIFG for reading relative to listening comprehension (Carpentier et al., 2001; Constable et al., 2004). In the current study, LIFG activation for listening and for reading comprehension was identified in two overlapping clusters of activation. Our results did show a larger cluster of activation in LIFG for reading comprehension in the contrast with fixation, but the difference was not statistically significant in the group comparison between reading and listening comprehension. The interpretation of the differences in LIFG activation in other studies is that there may be some degree of modal subspecialization in this area of the brain (Constable et al., 2004). This subspecialization of LIFG activation was not found in the present study; however, task difficulty was not manipulated in the present stimuli.

Modality fingerprints for listening comprehension

The modality-specific bilateral superior temporal cortex activation in listening comprehension, relative to reading comprehension, replicated previous findings of the bilaterality of activation for this process. Both the STG and MTG are well known for their association with early speech processing, and with spoken-word recognition tasks (Cabeza & Nyberg, 2000). Listening comprehension also showed more activation in right angular gyrus. Left angular gyrus activation has been associated with phonology-to-orthography (and vice-versa) conversion in lexical tasks (Booth et. al, 2002b). Listening comprehension also showed more activation than reading comprehension in the superior frontal gyrus (SFG). The activation of superior frontal areas of the brain has been associated with phonological processing without visual input (Katzir, Misra, & Poldrack, 2005).

Modality fingerprints for reading comprehension

The contrast between reading comprehension and its auditory counterpart showed clear modality-specific activation. Reading comprehension activated more left inferior occipital cortex, including the left fusiform gyrus. The fusiform gyrus is activated in word reading tasks in different writing systems (Bolger et al., 2005; Tan et al., 2005).

The activation for reading comprehension contrasted with listening did not show more activation of the visual word form area (VWFA) for reading. The VWFA is a modality-specific area of the brain known for its response to written visual stimuli. The distance between the peak of fusiform activation closest to the VWFA (located at MNI coordinates x = -42, y = -52, z = -22) was approximately 20.0 mm from the standard VWFA coordinates reported in a meta-analysis of 16 brain imaging studies of word reading (MNI x = -44, y = -58, z = -15) (Vigneau et al., 2005). This distance is above the ~5.0 mm standard deviation reported for the coordinates encompassing the visual word form area (Cohen et al., 2000; 2002; Cohen & Dehaene, 2004). The left inferior occipital activation found for reading comprehension extends slightly to posterior left fusiform gyrus, but it was more posterior than the activation usually reported for reading tasks (Bolger et al., 2005; Carpentier et al., 2001; Constable et al., 2004; Tan et al., 2005).

Studies have shown that brain activation is sensitive to the consistency of the mapping between orthography and sound. Paulesu and colleagues (2000) showed that reading Italian (transparent, consistent spelling-tosound rules) resulted in more activation of left superior temporal areas (phoneme processing), while reading English (deep, inconsistent spelling-to-sound rules) resulted in more activation of left posterior inferior temporal gyrus. Buchweitz, Mason, Hasegawa and Just (2009) found that reading different writing systems within the same language (Japanese) resulted in different brain activation for reading a logographic (kanji) and a syllabic (hiragana) writing system. Reading sentences in the logographic system (more visually complex ideograms) activated more occipitotemporal lobe areas, associated with visual processing. Reading sentences in the syllabic system activated more areas associated with phonological processes. It has been shown that the VWFA is more active for real words than for consonant strings, which indicates that it is an area that becomes attuned to the orthographic regularities that constrain letter combination (Cohen et al., 2002).

It is plausible that activation of the VWFA is more prominent in deep orthographies such as English and French. More transparent orthographies, such as Spanish, Italian, and Portuguese, have a more uniform mapping of print to the sounds of the language. The same group of letters within these languages will likely lead to the same pronunciation. In transparent orthographies reading can be achieved by a regular grapheme-phoneme route (graphophonological route). However, in deep orthography languages, like English and French, there are several letter combinations that will read differently (for example, in English, "int" in "pint" or "mint," or "eard" in "heard" or "beard"). To correctly read those words that do not follow the most usual spelling-to-sound rules, it is necessary to engage another route, called the direct or lexicosemantic route (Jobard et al., 2003). In a deep orthography, readers use the lexicosemantic route to build on the associations between arbitrary mappings of print to sound as they meet new words (Jobard et al., 2003). The lexicosemantic route of reading is based on the associations between word form and meaning. Word forms, in deep languages, are like images arbitrarily associated with meaning and sound. Hence, the VWFA may play a much more important role in reading deep orthographies than in reading more shallow orthographies, like Portuguese. Of course, the present study does not rule out the role of the VWFA in shallow orthographies. It simply shows that reading sentences in Portuguese, relative to listening to sentences in Portuguese, did not show more activation that could be related to processing word form. The contrast between reading and fixation in the present study did show activation located at the VWFA coordinates specified above. Further investigation into the modulation of VWFA activation by different orthographies is warranted.

More overall brain activation for listening comprehension and left-laterality of activation for reading comprehension

Listening comprehension showed more overall activation of the brain than reading comprehension. One explanation for the difference in total active voxels is that the greater activation in listening could be a result of the transient nature of auditory input. Auditory information is presented sequentially, while reading allows for backtracking, if necessary (Michael et al., 2001).

In the present study, however, reading stimuli were presented in sequential format, which did not allow for backtracking (RSVP format). In RSVP format, the visual input mirrors the transient characteristic of auditory input. The sequential presentation did not seem to result in an increase in overall reading activation, possibly ruling out the explanation that more overall activation for listening comprehension is due to the transient nature of auditory input. The results showed that despite the RSVP presentation of visual sentences, the difference in overall brain activation for listening, compared to reading, remains. This difference has been reported in other studies that investigated how modality influences sentence comprehension (Constable et al., 2004; Michael et al., 2001). It seems that a different explanation for the greater overall activation in listening is warranted.

The developmental explanation for the bilateral distribution of brain activation in listening versus the leftlateralized activation in reading (Michael et al., 2001) may be a more suitable explanation for the difference in laterality of activation in listening and reading brain activation, and of greater overall activation in listening. This explanation posits that once humans learn to read, the brain response is grafted onto the existing left-lateralized, amodal brain network for language comprehension. Later-learned reading comprehension is postulated as a second-order skill that is grafted onto the already-learned listening comprehension skill (Michael et al., 2001). Thus, as reading comprehension evolves into an automatic skill (like listening comprehension) and as humans learn to successfully operate higher-order cognitive processes in reading (inference-making, text integration, syntactic parsing), the cognitive responses to these processes are mapped onto the same areas of the brain that are associated with higher-level processes of listening comprehension. As a first-order language comprehension skill, listening comprehension retains most of the cognitive response that is intrinsic to its input form, whereas reading comprehension is mapped onto a preexisting brain response to language comprehension.

Individual differences in reading comprehension in RSVP format: Spillover, executive control, and phonological rehearsal

The study showed the brain activation for reading comprehension in rapid serial visual presentation. The transient nature of RSVP places additional load on working memory processes during reading possibly because it removes the possibility of backtracking in reading comprehension. The results indicate that lower capacity readers recruited more cortical resources from right-hemisphere areas of the brain. Spillover of activation to the right-hemisphere is a known mechanism of recruitment of additional cortical resources in lower capacity readers (Prat & Just, 2008). The results also show that lower capacity readers had more activation in the prefrontal cortex, specifically in left middle frontal gyrus. Activation of the prefrontal cortex has been consistently associated with executive control, and the left middle frontal gyrus is part of the executive network of cortical areas. The executive network consists of cortical areas activated in situations associated with controlled, strategic, and goal-oriented cognitive operations. It is a domain-general network of the brain, held to be mediated by a frontal-parietal neural system of processing centers (including dorsal and ventral left middle and inferior frontal cortex, medial frontal gyrus, and parietal cortices), and it is interpreted as a system engaged in dealing with novel cognitive tasks (D'Esposito et al., 1995).

The correlation of reading span scores and brain activity for higher-capacity readers showed significantly more activation in a frontal-posterior network of areas. The areas more activated for higher-span readers were left angular and precentral gyri, and RIFG. The activation of this network may be associated with phonological rehearsal of linguistic information.

The results indicated that higher and lower capacity readers may have resorted to different strategies while reading sentences in RSVP format. Lower capacity readers, in addition to recruiting more right-hemisphere areas of the brain, may have resorted to more executive control processes to help keep track of transient visual input and avoid comprehension processes from breaking down in a relatively novel type of reading task. Activation of the executive network of cortical areas is usually found in tasks that demand increased attention to the stimulus and the ability to maintain previous information active in working memory, such as in dual tasks (D'Esposito et al., 1995; Jaeggi et al., 2003). More activation in left middle frontal gyrus may indicate that reading in RSVP format may not have been a seamless process for lower capacity readers.

Higher-capacity readers, in turn, activated areas of the brain associated with phonological rehearsal of information. Rather than resorting to executive control processes, these readers may have been better able to adapt to the task of reading comprehension in transient, serial form. More activation in skilled readers may therefore reflect more successful subvocalization of the materials. More activation in areas associated with phonological rehearsal (left inferior parietal lobe and LIFG) has been associated with the increased demand of reading tongue-twisters (Keller et al., 2003) and with keeping sound to meaning representations active in a second language (Buchweitz et al., 2009). In the present study, the high-span readers activated more left angular gyrus and RIFG, the right-hemisphere homologue of LIFG. The activation of RIFG may represent spillover of activation from its left-hemisphere homologue.

The activation of this network of areas for high span readers may indicate that they were able to resort to phonological rehearsal of the words presented in serial format. Phonological rehearsal may have helped keep sound-to-meaning representations active after the written stimulus was no longer available on the screen. It is necessary to note that the study has a reduced number of participants and that there was no manipulation of task difficulty. Further studies with a larger population of participants, and that manipulate RSVP sentence difficulty, should be carried out to further confirm the present findings and show evidence of a correlation between reading span scores with brain activity and with better performance.

Conclusion

The study shows the modality fingerprints for the processing of spoken and written sentences. The differences in brain activation for reading and listening comprehension were found mostly in unimodal areas of the brain. The differences in brain activation between reading and listening comprehension corroborate other studies, even though the present study used a technique for serial presentation of written text. The study also shows individual differences in brain activation for reading comprehension in RSVP. The individual differences may be associated with different strategies for reading transient text. The results provide evidence to support one of the premises underlying models of human comprehension: Higher-order language comprehension processes are amodal.

Acknowledgments

This research was supported by the National Institute of Mental Health Grant MH029617. During the time of data collection Augusto Buchweitz was supported by the Brazilian Ministry of Education, CAPES BEX 3356-04-3. Thanks to Jennifer Moore and Chantel Prat for comments on a previous version of this manuscript.

Received 5 September 2009

received in revised form 16 October 2009

accepted 28 October 2009

Available on line 29 December 2009

• Bolger, D.J., Perfetti, C.A., Schneider, W. (2005). A cross-cultural effect on the brain revisited. Human Brain Mapping, 25, 91-104.
• Bookheimer, S. (2002). Functional MRI of language: New approaches to understanding the cortical organization of semantic processing. Annual Review of Neuroscience, 25, 151-188.
• Booth, J. R., Perfetti, C. A., MacWhinney, B. (1999). Quick, automatic, and general activation of orthographic and phonological representations in young readers. Developmental Psychology, 35, 3-19.
• Booth, J. R., Perfetti, C. A., MacWhinney, B., Hunt, S. B. (2000). The association of rapid temporal perception with orthographic and phonological processing in reading impaired children and adults. Scientific Studies of Reading, 4, 101-132.
• Booth, J.R., Burman, D.D., Meyer, J.R., Gitelman, D.R., Parrish, T.B., Mesulam, M.M. (2002a). Modality independence of word comprehension. Human Brain Mapping, 16, 251-61.
• Booth, J.R., Burman, D.D., Meyer, J.R., Gitelman, D.R., Parrish, T.B., Mesulam, M.M. (2002b). Functional anatomy of intra- and cross-modal lexical tasks. NeuroImage, 16, 7-22.
• Buchweitz, A., Mason, R. A., Hasegawa, M., Just, M. A. (2009). Japanese and English sentence reading comprehension and writing systems: An fMRI study of first and second language effects on brain activation. Bilingualism: Language and Cognition, 12, 141-151.
• Cabeza, R. and Nyberg, L. (2000). Imaging cognition II: An empirical review of 275 PET and fMRI studies. Journal of Cognitive Neuroscience, 12, 1-47.
• Capovilla, A.G.S, Capovilla, F.C., Suiter, I. (2004a). Processamento cognitivo em crianças com e sem dificuldades de leitura. Psicologia em Estudo, 9, 449-458.
• Capovilla, F.C. and Capovilla, A.G.S. (2004b). Research on the role of phonology, orthography and cognitive skills in reading, spelling and dyslexia in Brazilian Portuguese. In: I. Smythe, J. Everatt, and R. Salter (Eds.). International Book of Dyslexia: A cross-language comparison and practice guide (pp. 152-172). West Sussex, UK: Wiley.
• Carpentier, A., Pugh, K.R., Westerveld, M., Studholme, C., Skrinjar, O., Thompson, J.L., Spencer, D.D., Constable, R.T. (2001). Functional MRI of language processing: Dependence on input modality and temporal lobe epilepsy. Epilepsia, 42, 1241-1254.
• Cohen, L., Dehaene, S., Naccache, L., Lehéricy, S., Dehaene-Lambertz, G., Hénaff, M., Michel, F. (2000). The visual word form area: Spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain, 123, 291-307.
• Cohen, L., Lehéricy, S., Chochon, F., Lemer, C., Rivard, S., Dehaene, S. (2002). Language-specific tuning of visual cortex? Functional properties of the visual word form area. Brain, 125, 1053-1069.
• Cohen, L. and Dehaene, S. (2004). Specialization within the ventral stream: the case for the visual word form area. NeuroImage, 22, 466-476.
• Constable, R.T., Pugh, K.R., Berroya, E., Mencl, W.E., Westerveld, M., Ni, Weijia, Shankweiler, D. (2004). Sentence complexity and input modality effects in sentence comprehension: an fMRI study. NeuroImage, 22, 11-22.
• D'Esposito, M., Detre, J.A., Alsop, D.C., Shin, R.K., Atlas, S., Grossman, M. (1995): The neural basis of the central executive system of working memory. Nature, 378, 279-281.
• Devlin, J.T., Jamison, H.L., Gonnerman, L.M., Matthews, P.M. (2006). The role of the posterior fusiform gyrus in reading. Journal of Cognitive Neuroscience, 18, 911-922.
• Forster, K. I. (1970). Visual perception of rapidly presented word sequences of varying complexity. Perception & Psychophysics, 8, 215-221.
• Friston, K., Ashburner, J., Frith, C., Poline, J.-B., Heather, J., Frackowiak, R. (1995). Spatial registration and normalization of images. Human Brain Mapping, 2, 165-189.
• Hills, A.E., Newhart, M., Heidler, J., Barker, P., Herskovits, E., Degaonkar, M. (2005). The roles of the 'visual word form area' in reading. NeuroImage, 24, 548-559.
• Jaeggi, S.M., Seewer, R., Nirkko, A.C., Eckstein, D., Schroth, G., Groner, R., Gutbrod, K. (2003): Does excessive memory load attenuate activation in the prefrontal cortex? Load-dependent processing in singe and dual tasks: Functional magnetic resonance imaging study. NeuroImage, 19, 210-225.
• Jobard, G. Crivello, F, Tzourio-Mazoyer, N. (2003). Evaluation of the dual route theory of reading: a metanalysis of 35 neuroimaging studies. NeuroImage, 20, 693-712.
• Jobard, G, Vigneau, M., Mazoyer, B., Tzourio-Mazoyer, N. (2007). Impact of modality and linguistic complexity during reading and listening tasks. NeuroImage, 34, 784-800.
• Just, M. A., and Carpenter, P. A. (1987). The psychology of reading and language comprehension Newton, MA: Allyn and Bacon.
• Just, M. A., Carpenter, P. A., Keller, T. A., Eddy, W. F., Thulborn, K. R. (1996). Brain activation modulated by sentence comprehension. Science, 274, 114-116.
• Katzir, T., Misra, M., Poldrack, R.A. (2005). Imaging phonology without print: assessing the neural correlates of phonemic awareness using fMRI. NeuroImage, 27, 106-115.
• Keller, T.A., Carpenter, P.A., Just, M.A. (2001). The neural bases of sentence comprehension: an fMRI examination of syntactic and lexical processing. Cerebral Cortex, 11, 223-237.
• Keller, T.A., Carpenter, P.A., Just, M.A. (2003). Brain imaging of tongue-twister sentence comprehension: Twisting the tongue and the brain. Brain and Language, 84, 189-203.
• Kintsch, W. (1998). Comprehension: A paradigm for cognition. Cambridge, UK: Cambridge University Press.
• Mason, R. A., and Just, M. A. (2006). Neuroimaging contributions to the understanding of discourse processes. In M. Traxler and M. A. Gernsbacher (Eds.), Handbook of Psycholinguistics (pp. 765-799). Amsterdam: Elsevier.
• Mesulam, M. (1998). From sensation to cognition. Brain, 121, 1013-1052.
• Michael, E.B., Keller, T.A., Carpenter, P.A., Just, M.A. (2001). fMRI investigation of sentence comprehension by eye and by ear: modality fingerprints on cognitive processes. Human Brain Mapping, 13, 239-252.
• Obler, L., and Gjerlow, K. (1999). Language and the Brain. Cambridge, UK: Cambridge University Press.
• Paulesu, E., McCrory, E., Fazio, F., Menoncello, L., Brunswick, N., Cappa, S.F., ... Frith, U. (2000). A cultural effect on brain function. Nature Neuroscience, 3, 91-96.
• Prat, C., Keller, T. A., Just, M. A. (2007). Individual differences in sentence comprehension: A functional Magnetic Resonance Imaging investigation of syntactic and lexical processing demands. Journal of Cognitive Neuroscience, 19, 1950-1963.
• Prat, C. and Just, M. A. (2008). Brain bases of individual differences in cognition. Psychological Science Agenda, Volume 22, Issue 5.
• Schlosser, M.J., Aoyagi, N., Fulbright, R.K., Gore, J.C., McCarthy, G. (1998). Functional MRI studies of auditory comprehension. Human Brain Mapping, 6, 1-13.
• Seymour, P.H.K., Aro, M., Erskine, J.M. (2003). Foundation literacy acquisition in European orthographies. British Journal of Psychology, 94, 143-174.
• Tan, L.H., Laird, A.R., Li, K., Fox, P.T. (2005). Neuroanatomical correlates of phonological processing of Chinese characters and alphabetic words: A meta-analysis. Human Brain Mapping, 25, 83-91.
• Tzourio-Mazoyer, N., Landeau, B., Papathanassiou, D., Crivello, F., Etard, O., Delcroix, N., ... Joliot, M. (2002). Automated anatomical labeling of activations in SPM using a macroscopic anatomical parcellation of the MNI MRI single subject brain. NeuroImage 15, 273-289.
• Vigneau, M., Jobard, G., Mazoyer, B., Tzourio-Mazoyer, N. (2005). Word and nonword reading: what role for the visual word form area? NeuroImage, 27, 694-705.
• Wise, R.J.S. Price, C.J. (2006). Functional neuroimaging of language. In: R. Cabeza and A. Kingstone (Eds.) Handbook of functional neuroimaging of cognition (pp. 191-227). Cambridge, MA: The MIT Press.

Publication Dates

• Publication in this collection
  17 Jan 2011
• Date of issue
  Dec 2009

History

• Accepted
  28 Oct 2009
• Reviewed
  16 Oct 2009
• Received
  05 Sept 2009